Fix building github.com/popcorn-official/popcorn-desktop

If you try to build from development branch, the result is not usable. I’ll explain the hacks you need to make it work.

Problem 1: 403 on nw:build

As reported on github: https://github.com/popcorn-official/popcorn-desktop/issues/202

Background on nw-builder

Basically what nw-builder does is that it fetches the prebuilt binaries of nw.js and copy the source code of your app and bundles it into a package. It is also possible to directly run your app without bundling it using the method described in the nw.js documentation:

cd /path/to/your/app
/path/to/nw .

According to this comment:

Just for you know that we have an strong dependency on NW, which is compiled PCT team.
PCT NW are compiled with anothers video codecs (something like that).
Maybe you should know this.

Note: I copy the entire contents from links in case of DMCA takedowns

Since the PCT binary URL is returns 403 now (not sure if it’s due to DMCA takedown), I thought about using the vanilla binaries instead, but this way we would lose a few codecs. Then I figured I can extract the binaries from the working packaged version 0.3.9 (it’s on the front page).

Download it, unzip it, copy Popcorn-Time.app to popcorn-desktop/cache/0.12.2/osx64/nwjs.app, and rm -r popcorn-desktop/cache/0.12.2/osx64/nwjs.app/Resources/app.nw otherwise nwjs will simply look for source code inside app.nw to run instead of your repo source code at popcorn-desktop.

Now we can start popcorntime from your source code with:

cd popcorntime-desktop
cache/0.12.2/osx64/nwjs.app/Contents/MacOS/nwjs –development .

Note: I read somewhere of the --development flag, but don’t observe any change in behavior, I’ll just leave it there.

Soon you’ll hit error like this (debug log at popcorn-desktop/cache/0.12.2/osx64/nwjs.app/Contents/MacOS/debug.log):

[43854:0710/142626:INFO:CONSOLE(27)] ""added" "TVShowTime" "to provider registry"", source: app://host/src/app/lib/providers/generic.js (27)
[43854:0710/142626:INFO:CONSOLE(35)] ""loaded" "tvshowtime.js"", source: app://host/src/app/bootstrap.js (35)
[43854:0710/142626:INFO:CONSOLE(27)] ""added" "Watchlist" "to provider registry"", source: app://host/src/app/lib/providers/generic.js (27)
[43854:0710/142626:INFO:CONSOLE(35)] ""loaded" "watchlist.js"", source: app://host/src/app/bootstrap.js (35)
[43854:0710/142626:INFO:CONSOLE(27)] ""added" "ysubs" "to provider registry"", source: app://host/src/app/lib/providers/generic.js (27)
[43854:0710/142626:INFO:CONSOLE(35)] ""loaded" "ysubs.js"", source: app://host/src/app/bootstrap.js (35)
[43854:0710/142626:INFO:CONSOLE(21)] ""%c[%cERROR%c] Error starting up" "color: black;" "color: red;" "color: black;"", source: app://host/src/app/app.js (21)
[43854:0710/142626:INFO:CONSOLE(16)] ""[%cWARNING%c] Show Disclaimer" "color: orange;" "color: black;"", source: app://host/src/app/app.js (16)

Problem 2: Error starting up

With my limited Javascript knowledge, I added the lines to print out debug messages at app/database.js#447:

446             .catch(function (err) {
447                 console.log(err.message);
448                 console.log(err.stack);
449                 win.error('Error starting up', err);

Got the error:

Error: Cannot find module 'butter-provider-MovieApi'
at Function.Module._resolveFilename (module.js:327:15)
at Function.Module._load (module.js:264:25)
at Module.require (module.js:356:17)
at require (module.js:375:17)
at window.require (\u003Canonymous\u003E:4:17)
at Object.getProvider [as get] (app://host/src/app/lib/providers/generic.js:54:60)
at app://host/src/app/lib/config.js:227:42
at Function._.map._.collect (app://host/src/app/vendor/underscore/underscore.js:172:24)
at Object.Config.getProviderForType (app://host/src/app/lib/config.js:226:26)
at app://host/src/app/bootstrap.js:85:42
at Array.map (native)
at app://host/src/app/bootstrap.js:83:56
at Promise_then_fulfilled (/Users/pellaeon/project/popcorn-desktop/node_modules/q/q.js:766:44)
at Promise_done_fulfilled (/Users/pellaeon/project/popcorn-desktop/node_modules/q/q.js:835:31)
at Fulfilled_dispatch [as dispatch] (/Users/pellaeon/project/popcorn-desktop/node_modules/q/q.js:1229:9)
at Pending_become_eachMessage_task (/Users/pellaeon/project/popcorn-desktop/node_modules/q/q.js:1369:30)"", source: app://host/src/app/database.js (448)

grepping butter-provider-MovieApi in the source tree returns no result, neither does googling.

Then I tried copying the entire working source tree from the packaged version Popcorn-Time.app/Contents/Resources/app.nw/, this doesn’t make the error go away either, which indicates that the problem lies not in the source tree. But where could the problem be?

The only one I could think of was the 2 files you need to download separately from popcorntime.sh to build it: popcorntime.sh/torrent_collection.js and popcorntime.sh/ysubs.js (instructions in readme). Anyway this doesn’t work either.

I came across issue#102:

The popcorntime.sh/ysubs.js downloaded is invalid, after pct load the file, it is not added to provider registry. I changed ysubs.js minified content by ysubs.js from branch feature/electron (old version of the file) and it works.

Hmmm, very suspicious.

But diffing ysubs.js doesn’t seem the downloaded version have anything different that would make the error go away, nor does actually installing it, nor is torrent_collection.js, which lies inside the view directory, even less likely.

So looks like I need to dig in to the trace to see what I have missed.

Digging into the code

With the trace above, I decided to go with bootstrap.js first, between line 83 and 85, let’s print out the variable first:

83             return _.filter(_.keys(Settings.providers).map(function (type) {
84                 console.log('========1');
85                 console.log(type);
86                 return {
87                     provider: App.Config.getProviderForType(type),
88                     type: type
89                 };

Running again, the log gives:

[46099:0710/164418:INFO:CONSOLE(84)] ""========1"", source: app://host/src/app/bootstrap.js (84)
[46099:0710/164418:INFO:CONSOLE(85)] ""movie"", source: app://host/src/app/bootstrap.js (85)

Hmmm, nothing too special, don’t look deeper just now.

Let’s see what it later calls: getProviderForType(). See the vars again:
config.js

225             } else if (provider instanceof Array || typeof provider === 'object') {
226                 return _.map(provider, function (t) {
227                     console.log('==========2');
228                     console.log(t);
229                     return App.Providers.get(t);
230                 });
231             } else {
232                 return App.Providers.get(provider);
233             }

Log:

[46099:0710/164418:INFO:CONSOLE(227)] ""==========2"", source: app://host/src/app/lib/config.js (227)
[46099:0710/164418:INFO:CONSOLE(228)] ""MovieApi?&apiURL=https://movies-v2.api-fetch.website/,"", source: app://host/src/app/lib/config.js (228)

…wait, isn’t that the non-existent module’s name? “butter-provider-MovieApi“, and after that a URL that doesn’t look valid at all: movies-v2.api-fetch.website, verryyy suspicious (Note: I later found out it is actually valid URL). grep it:

popcorn-desktop$ grep -R 'MovieApi' *
node_modules/butter-settings-popcorntime.io/index.js:             uri: ['MovieApi?'

What ?!! node_modules ! It gets settings from node_modules ?!

grep in packaged version returned no result. So this is what differs between 2 versions!

I copied index.js from packaged version to popcorn-desktop/node_modules..., and this time it started successfully, and is able to load all movies and TV shows!

What really happened

Looking at package.json from the 2 different butter-settings-popcorn.io directory, seems like the two do differ in versions, master uses c745869979405d02d5a4cfea066c227801ee0fc2 (of butter-settings-popcorn.io), 0.3.9 uses 6b8075e0065efbe7276e4a469ca00dbc43c2aaa9. diff:

git diff 6b8075e0065efbe7276e4a469ca00dbc43c2aaa9..HEAD
diff --git a/index.js b/index.js
index dce0e24..ccfcec1 100644
--- a/index.js
+++ b/index.js
@@ -16,10 +16,10 @@ module.exports = {
movie: {
order: 1,
name: 'Movies',
-             uri: ['yts?'
+             uri: ['MovieApi?'
+'&apiURL='
-                     + 'https://movies.api-fetch.website/,'
-//                     + 'cloudflare+https://xor.image.yt,'
+                     + 'https://movies-v2.api-fetch.website/,'
+//                     + 'cloudflare+https://movies-v2.api-fetch.website,'
//                     + 'cloudflare+http://xor.image.yt'
]
},
@@ -28,8 +28,8 @@ module.exports = {
name: 'Series',
uri: ['TVApi?'
+'&apiURL='
-                     + 'https://tv.api-fetch.website/,'
-//                     + 'https://odgoglfi7uddahby.onion.to/,'
+                     + 'https://tv-v2.api-fetch.website/,'
+//                     + 'cloudflare+https://tv-v2.api-fetch.website,'
//                     + 'http://tv.ytspt.re/'
]
},

Testing the URLs in it I found it is actually valid (must be the new gTLDs).

Seems like the name before the question mark is the key, there is butter-provider-yts but no butter-provider-MovieApi in node_modules, so it couldn’t load the latter and errored.

Butter will automatically load any npm package installed (listed in package.json) that matches the =/butter-provider-.*/ regex.

I guess this is in startup stage, and in later stage someone would request the MovieApi or yts provider, but I still haven’t found the code.

Anyway, this fixes the build, so another time!

Neutron VXLAN multicast – disable IGMP snooping

openstack-ansible liberty 6e3815d, l2pop=off

At first I encountered this issue: https://bugs.launchpad.net/openstack-ansible/liberty/+bug/1563448 , but after deleting vxlan interfaces and restarting neutron agents, there are still problems.

  1. Unicast packets in VM tenent networks works, which means from agent namespaces you can ping VM with their IP in that subnet.
  2. Broadcast packets in VM tenant networks doesn’t work, which means VM can’t get DHCP reply, nor ARP reply, and nor can the DHCP agents. but strangely in my case L3 agent can get VM’s ARP reply.
  3. After further investigation, I found that vxlan-X interface on compute and agent containers doesn’t receive broadcast packets that others sent.
  4. Further, on physical interface (in my case `eth1.30`, slave of  `br-vxlan`), I can see packets going out to `vxlan_group` (239.1.1.1 by default), but not packets from 239.1.1.1
  5. Packet capturing on the switch found that the switch is not forwarding the packets.
  6. On #openstack-ansible @jimmdenton pointed me this article: http://movingpackets.net/2013/06/04/multicast-problems-juniper-ex/ , suggest to disable IGMP snooping
  7. It works!

Thanks to Rackspace guys at #openstack-ansible, who spend a lot of time helping me debug the issue.

This article helped me understand how VXLAN and L2 population works: https://kimizhang.wordpress.com/2014/04/01/how-ml2vxlan-works/ . (Don’t use L2 population! Neutron developers suggest against it!)

巴拿馬文件相關資訊 Panama Papers related links

4/5 10:55 蘋果日報抄 UDN 的錯誤報導還沒被撤下 http://www.appledaily.com.tw/realtimenews/article/international/20160404/831292/

4/5 10:50 UDN 網站出現這篇報導「台灣版「巴拿馬文件」尚未公布」 http://udn.com/news/story/1/1608667

4/5 10:43 UDN 聯合新聞網撤下魏應充的報導

4/5 10:10

我錯了,台灣媒體除了天下雜誌其他媒體應該沒有拿到這次的資料,UDN 顯然是把 2013 年的 Offshore Leaks 跟這次的資料搞混了,根本亂報。

Offshore leaks 有開放的可以搜尋的資料庫: https://offshoreleaks.icij.org/

我自己是進不去,搜尋台灣顯示錯誤。朋友表示搜尋結果跟 UDN 報的八成像。

4/5 9:50

Reddit 上面有人在問要怎麼取得原始資料: https://www.reddit.com/r/MachineLearning/comments/4d7t1h/panama_papers_dataset/

http://www.wired.com/2016/04/reporters-pulled-off-panama-papers-biggest-leak-whistleblower-history/

The ICIJ’s developers then built a two-factor-authentication-protected search engine for the leaked documents, the URL for which they shared via encrypted email with scores of news outlets including the BBC, The Guardian, Fusion, and dozens of foreign-language media outlets. The site even featured a real-time chat system, so that reporters could exchange tips and find translation for documents in languages they couldn’t read.

ICIJ 有給各國媒體使用的內部搜尋引擎。

台灣媒體已經拿到了:
http://udn.com/news/story/9688/1608548
http://udn.com/news/story/9688/1608342

ICIJ查詢網站昨天開放後,網站查詢速度極慢,到近深夜時,查詢網站一度無法連線。記者稍早下載與台灣相關的近1.7萬筆資料比對

台灣記者似乎故意讓人誤會資料是開放的。

有沒有人認識台灣的記者啊?


4/4

這次的巴拿馬洩密案,吹哨者釋出了 2.6 TB 的資料給西方媒體,幾家媒體正在整理資料,會陸續報導出來。

南德日報的報導: http://panamapapers.sueddeutsche.de/en/
國際調查記者協會 ICIJ: http://panamapapers.icij.org/

也有人揭露 ICIJ 背後和大企業的關係,因為目前釋出的報導幾乎都集中在西方政府和企業的敵人——普亭、阿塞德、中共,只有一個小小的冰島總理被犧牲,「平衡一下」,西方的大企業和政府官員幾乎都沒有被報導。他也呼籲媒體釋出完整的資料。 https://www.craigmurray.org.uk/archives/2016/04/corporate-media-gatekeepers-protect-western-1-from-panama-leak/

4/5 更新:南德日報總編輯在 twitter 表示:「等著看接下來我們會釋出什麼吧」

維基解密也在呼籲公開完整的資料,目前在媒體手上的 1100 萬份文件,只有 149 份被公開釋出。 https://www.documentcloud.org/public/search/Source:%20%22Internal%20documents%20from%20Mossack%20Fonseca%20%28Panama%20Papers%29%22/p12

這次洩密的主角是 Mossack Fonseca,是間專門幫人設立海外避稅公司的公司。維基解密不知道哪裡弄來的資料,顯示在台灣設立的空殼公司高達 2725 家(從 1970 至今),也有 46 個台灣客戶。

90 行的開源貢獻

https://github.com/pellaeon/fb-post-bulk-privacy-edit

去年我花了幾個晚上用僅有的 Python 技能寫了一隻很小的只有 90 行的 script ,可以幫你大量編輯一段時間內的 Facebook 貼文隱私設定。

今天早上收到信,感謝我寫了這個,然後問了一個問題。

回去看那個 repo ,居然也有另一個人回報了 issue 。
我覺得超驚訝,這樣一個小小的 90 行的程式也能造成這麼大的影響啊。

其實開源貢獻真的沒有很難。

遵守規矩很難嗎?

https://www.facebook.com/hauyu.wang/posts/10201726466241484?pnref=story

遵守規矩不難,但是培養一個獨立思考、具備理性思辨能力的人非常困難。

要培養一個具備理性思辨能力的人其中很重要的一點,就是要有一個鼓勵理性思辨的環境,也就是讓學生覺得,理性思辨能力,真的能夠改變世界或至少改變他週遭的人事物。

用不合理的規定強加在學生身上,只不過是讓他們不相信理性思辨,而去相信權力。

「再合理都沒有用,必須要有權力,才能造成影響」

為了達成他們想要的事情,未來他們會覺得,理性思辨不重要,制定規則的權力才能造成影響。

所以他們學習模仿那些大人是如何去怎麼壓迫沒有權力的人,而不是去討論去思辨去說服。

等到他們長大,也只會複製一樣的模式來壓迫後輩。

時間都在這些不合理的規則浪費掉了,台灣不會變得更好,世界也不會變得更好。

Pull request 接受率的性別差異研究

https://peerj.com/preprints/1733/

此研究未經同儕審查。

分析男女的 PR 在 github 被 merge 的機率的研究,樣本約有 140 萬 github 使用者。

簡單整理一下我看過之後的結果:

  1. 女生發的 PR 被 merge 的可能性比較高
  2. 女生在精進的過程中持續維持高接受率
  3. 女生的 PR 去修正專案當前的需求(像是 bug)的可能性比較低(女生在 PR 裡面比較少提到另一個 issue)
  4. 女生所做的更動平均比男生大(行數、commit 數、檔案數)
  5. 女生在程式語言檔案 (.php, .py, etc) 的 PR 接受度比非程式語言檔案 (.md, .txt, .rst, etc) 還要高
  6. 女生是專案的外部開發者,並且可以被辨識為女生的時候, PR 接受度較低

為什麼數位音訊只要記錄振幅就可以了?

最近想要持續改進 ownCloud Music app ,現在在 Chrome 上面跑的很順,但是我常用的 Firefox 播 FLAC 音樂還是經常會有 glitch (我的 Firefox 本來就已經很頓了,雖然我有 150+ 個分頁,但是我有裝自動 unload 的 plugin 啊,任一時間同時開啟的分頁應該不會超過 10 個,為什麼會這麼慢我也搞不懂),為了要改善效能,我先研究了一下 web worker ,看看有沒有可能把 codec 跑在 web worker 裡面,這樣就不會受到 main thread 的影響了,看看 web worker ,發現有 AudioWorker 這東西,就又再挖進去探究一下(筆記在這)。看 AudioWorker 發現自己實在不懂數位音效處理,就又開始找資源研讀。

關於數位音訊處理,我先前有看過 Xiph.org 講得非常好的技術理論的影片。這次找到 Indiana University 清楚簡明的介紹

我之前一直不太懂數位格式音訊(振幅)是怎麼轉換成實際的聲音的,我不懂為什麼數位音訊裡面不包含頻率資料(譬如說某個頻率的聲音在某個時間小段的振幅),卻還是可以還原本來的聲音?看了這段之後我懂了:

Frequencies will be recreated later by playing back the sequential sample amplitudes at a specified rate. It is important to remember that frequency, phase, waveshape, etc. are not recorded in each discrete sample measurement, but will be reconstructed during the playback of the stored sequential amplitudes.

然後加上這張圖:

d-to-a

Each sample is “clocked" into the DAC’s register. A ‘1’ in a register place will add a voltage to the sum of that sample proportionate to its binary value. In this hypothetical case, we have a sample whose binary value is 5. The gates or switches for the binary places of ‘4’ and ‘1’ are closed, and the value of 5 mvolts is sent out the DAC and held until the next sample is clocked into the register.

所以說,為什麼只要收集一個時間小段裡面的振幅資訊就可以還原本來的音效呢?

對於 DAC (Digital to Analog Converter) 來說,它做的事情就是在每 1/44100 秒內(數字依照取樣率決定), 輸出音訊檔指定的電壓大小,並在那一小段時間維持電壓大小。每個時間小段的振幅資訊在音訊檔裡面叫做一個 frame ,但一個 frame 可能有很多值,一個聲道 (channel) 一個值,譬如說雙聲道有 2 個值,5.1 環繞音效就有 6 個值。

每一個電壓值都是那個時間之內喇叭振動體的位置,電壓值不變,振動體位置就不變,但聲音是空氣的震動,振動體位置不變,空氣就不會被震動,也就沒有聲音。所以舉例來說,如果要產生一個特定頻率、音量的 sine 聲波,做法不是讓 DAC 在時間內輸出某個固定的電壓值,而是讓 DAC 輸出一個不斷變大,到了極大又逐漸變小的電壓值,從極大到極小的速度(譬如說每個極大到下一個極大值的差距是 2205 個 frame),決定了輸出的 sine 波的頻率。

更複雜一點來說,所有的聲音都是不同頻率、音量(最大振幅)、相位的 sine 波疊加出來的,所以,取樣所得的資訊,即是在那個取樣時間間隔內,附近聲音的所有波的振幅疊加。下一個取樣,每個波又會依照自己本來的規律而有不同的振幅。所有波,不同頻率、音量、相位的波,在那小段時間內的振幅資訊,都是疊加在一起被收集起來的,所以不需要「把每個頻率的波的資訊分開來收集」。

註:我記得國中理化好像有提到,振幅本來就有「最大」的意思,就是波的峰和谷的高度差,不過我忘了那個描述某個時間點介質的位置的名詞是什麼了,不過我想你知道我的意思。

不過你或許也注意到了,這邊有 2 個頻率:取樣頻率DAC 變換值的頻率,如果兩個頻率不一樣,還原出來的聲音的頻率就會有變化。

If samples are clocked into the DAC at the rate they were sampled, then the original frequencies will be reproduced up to the Nyquist frequency. If the samples are clocked in at twice the rate, then the frequency will be doubled.

如果取樣頻率和 DAC 變換頻率相同,原始錄音時小於 Nyquist frequency 的頻率就會被重現,如果 DAC 變換頻率是取樣頻率的兩倍,還原出來的頻率也會是原本錄音頻率的兩倍。

懂了這點之後,原本似懂非懂的 Nyquist frequency 也懂啦。

(我不是這方面專業,本文有任何錯誤還請指正)