Android Island work profile apps and AFWall

Island is an app that allows you to isolate, clone, freeze, hide, archive apps.

Unfortunately when you use it with iptables firewall app like AFWall the apps placed in the work profile have no access to the network. (because there aren’t rules allowing them in the firewall)

AFWall doesn’t support multiple profiles – there’s an experimental option called “multi-user support" but only works in mode where selected items are blocked:


AFWall doesn’t show work profile apps. So I need a way to allow work profile app traffic.

With a bit of digging, I found the rule for work profile (UNIX) uid : user_id*100000+app_uid , the primary user has user id of 0, the work profile Island created has user id 10.

So you can go look for app uid, add user_id*100000 , and it becomes the (UNIX) uid of the app in that work profile.

A quick way to know app_uid is go to AFWall preferences > User interface > “Show UID for apps".

Then you need to write your own rules that allows traffic to pass through, the number after --uid-owner is the UID you want to change, in this example I allowed 3 apps:

Place it in /data/local/, chmod +x it, add its path to AFWall custom script, then apply rules in AFWall. You should now get internet in work profile apps.


尤其是要爬像是 Facebook 這種蓄意想要鎖住資訊,把 Internet 變成一個巨大內網的邪惡公司的時候。

之前在 Github 看到 rss-bridge 這個專案,用爬蟲去爬網頁然後產生 RSS/Atom feed ,原本是想要把它用在 Facebook 上面,這樣就可以訂閱粉專的每一篇貼文、照時間排序、不用被 Facebook 的 filter bubble 審查資訊,也不會因為你訂閱什麼就汙染你的 stream 。


結果隔天早上醒來發現所有 feed 都失效,因為 Facebook 要求 Captcha 驗證。原本的 rss-bridge 會把驗證圖片 proxy 顯示給瀏覽 feed 的使用者,使用者可以解完之後送出,rss-bridge 會再送給 Facebook ,但這功能實測的時候發現無論如何送出都會失敗,把這 bug 解掉之後又發現 captcha 只會當次有效,也就是要發另一個 request 的時候又會被跳 captcha 。

自己開瀏覽器測試了一下發現,瀏覽器的 captcha 只要解一次,未來的 request 就都會過,想說大概是因為 rss-bridge 的爬蟲不支援 cookie 的問題,所以就花了好幾天時間把 rss-bridge 的爬蟲換成 PHP curl (原本是用 file_get_contents() ),因為 PHP curl 有 cookie jar 的功能,管理 cookie 比較方便。

結果新的 code 終於會動,但儘管送了 cookie 還是每次被跳 captcha ,挖到底層去測試了很多東西,檢查 request header 和 response header ,確定不是我實作有問題之後,我發現問題不在 cookie。

問題在於 rss-bridge 的爬蟲預設都會多送一個 GET 參數 _fb_noscript=1 ,有這個參數的時候顯示出來的 captcha 會不一樣!

_fb_noscript=1 的時候顯示的 captcha 會直接是一個 <img src="..."> ,而且這個 captcha 也只有單次有效!下次再請求,就算有 cookie, 也會再被要求解另一個 captcha 。

沒有那個參數的時候的 captcha 圖片會使用 AJAX 載入,然後解開之後設定的 cookie 也會多次有效。

我拿舊的實作改成 curl 的時候沒有特別改這個東西,結果到現在才發現這件事。

所以,這次學到的教訓是,寫爬蟲的時候,不要仁慈,直接開 PhantomJS/CasperJS 吧……

P.S. 我改過的 rss-bridge 在這兒:

更新:不想用 Javascript 的話 Python 的 Scrapy + Splash (Splash 是 headless browser)看起來還不錯(沒用過)。還有 django-dynamic-scraper 可以直接把爬到的東西結構化成 Django model object ,每一個 field 直接關聯一個 selector ,scrape 的時候讀進去,太神啦。

2017/2/24 更新2:後來又繼續試著解決這個問題。研究了一下 django-dynamic-scraper , 幾乎找不到如何處理 captcha 的說明和範例,雖然說應該是可以解但是感覺又要研究 scrapy 很久,所以還是決定回去幫 rss-bridge 加上 proxy 的功能。改出來的東西在 。然後還花了一些心力研究要怎麼取得免費的 proxy 清單,最後發現 的不錯,寫了個小小的 Javascript 去撈 proxy 清單

Fix building

If you try to build from development branch, the result is not usable. I’ll explain the hacks you need to make it work.

Problem 1: 403 on nw:build

As reported on github:

Background on nw-builder

Basically what nw-builder does is that it fetches the prebuilt binaries of nw.js and copy the source code of your app and bundles it into a package. It is also possible to directly run your app without bundling it using the method described in the nw.js documentation:

cd /path/to/your/app
/path/to/nw .

According to this comment:

Just for you know that we have an strong dependency on NW, which is compiled PCT team.
PCT NW are compiled with anothers video codecs (something like that).
Maybe you should know this.

Note: I copy the entire contents from links in case of DMCA takedowns

Since the PCT binary URL is returns 403 now (not sure if it’s due to DMCA takedown), I thought about using the vanilla binaries instead, but this way we would lose a few codecs. Then I figured I can extract the binaries from the working packaged version 0.3.9 (it’s on the front page).

Download it, unzip it, copy to popcorn-desktop/cache/0.12.2/osx64/, and rm -r popcorn-desktop/cache/0.12.2/osx64/ otherwise nwjs will simply look for source code inside app.nw to run instead of your repo source code at popcorn-desktop.

Now we can start popcorntime from your source code with:

cd popcorntime-desktop
cache/0.12.2/osx64/ –development .

Note: I read somewhere of the --development flag, but don’t observe any change in behavior, I’ll just leave it there.

Soon you’ll hit error like this (debug log at popcorn-desktop/cache/0.12.2/osx64/

[43854:0710/142626:INFO:CONSOLE(27)] ""added" "TVShowTime" "to provider registry"", source: app://host/src/app/lib/providers/generic.js (27)
[43854:0710/142626:INFO:CONSOLE(35)] ""loaded" "tvshowtime.js"", source: app://host/src/app/bootstrap.js (35)
[43854:0710/142626:INFO:CONSOLE(27)] ""added" "Watchlist" "to provider registry"", source: app://host/src/app/lib/providers/generic.js (27)
[43854:0710/142626:INFO:CONSOLE(35)] ""loaded" "watchlist.js"", source: app://host/src/app/bootstrap.js (35)
[43854:0710/142626:INFO:CONSOLE(27)] ""added" "ysubs" "to provider registry"", source: app://host/src/app/lib/providers/generic.js (27)
[43854:0710/142626:INFO:CONSOLE(35)] ""loaded" "ysubs.js"", source: app://host/src/app/bootstrap.js (35)
[43854:0710/142626:INFO:CONSOLE(21)] ""%c[%cERROR%c] Error starting up" "color: black;" "color: red;" "color: black;"", source: app://host/src/app/app.js (21)
[43854:0710/142626:INFO:CONSOLE(16)] ""[%cWARNING%c] Show Disclaimer" "color: orange;" "color: black;"", source: app://host/src/app/app.js (16)

Problem 2: Error starting up

With my limited Javascript knowledge, I added the lines to print out debug messages at app/database.js#447:

446             .catch(function (err) {
447                 console.log(err.message);
448                 console.log(err.stack);
449                 win.error('Error starting up', err);

Got the error:

Error: Cannot find module 'butter-provider-MovieApi'
at Function.Module._resolveFilename (module.js:327:15)
at Function.Module._load (module.js:264:25)
at Module.require (module.js:356:17)
at require (module.js:375:17)
at window.require (\u003Canonymous\u003E:4:17)
at Object.getProvider [as get] (app://host/src/app/lib/providers/generic.js:54:60)
at app://host/src/app/lib/config.js:227:42
at (app://host/src/app/vendor/underscore/underscore.js:172:24)
at Object.Config.getProviderForType (app://host/src/app/lib/config.js:226:26)
at app://host/src/app/bootstrap.js:85:42
at (native)
at app://host/src/app/bootstrap.js:83:56
at Promise_then_fulfilled (/Users/pellaeon/project/popcorn-desktop/node_modules/q/q.js:766:44)
at Promise_done_fulfilled (/Users/pellaeon/project/popcorn-desktop/node_modules/q/q.js:835:31)
at Fulfilled_dispatch [as dispatch] (/Users/pellaeon/project/popcorn-desktop/node_modules/q/q.js:1229:9)
at Pending_become_eachMessage_task (/Users/pellaeon/project/popcorn-desktop/node_modules/q/q.js:1369:30)"", source: app://host/src/app/database.js (448)

grepping butter-provider-MovieApi in the source tree returns no result, neither does googling.

Then I tried copying the entire working source tree from the packaged version, this doesn’t make the error go away either, which indicates that the problem lies not in the source tree. But where could the problem be?

The only one I could think of was the 2 files you need to download separately from to build it: and (instructions in readme). Anyway this doesn’t work either.

I came across issue#102:

The downloaded is invalid, after pct load the file, it is not added to provider registry. I changed ysubs.js minified content by ysubs.js from branch feature/electron (old version of the file) and it works.

Hmmm, very suspicious.

But diffing ysubs.js doesn’t seem the downloaded version have anything different that would make the error go away, nor does actually installing it, nor is torrent_collection.js, which lies inside the view directory, even less likely.

So looks like I need to dig in to the trace to see what I have missed.

Digging into the code

With the trace above, I decided to go with bootstrap.js first, between line 83 and 85, let’s print out the variable first:

83             return _.filter(_.keys(Settings.providers).map(function (type) {
84                 console.log('========1');
85                 console.log(type);
86                 return {
87                     provider: App.Config.getProviderForType(type),
88                     type: type
89                 };

Running again, the log gives:

[46099:0710/164418:INFO:CONSOLE(84)] ""========1"", source: app://host/src/app/bootstrap.js (84)
[46099:0710/164418:INFO:CONSOLE(85)] ""movie"", source: app://host/src/app/bootstrap.js (85)

Hmmm, nothing too special, don’t look deeper just now.

Let’s see what it later calls: getProviderForType(). See the vars again:

225             } else if (provider instanceof Array || typeof provider === 'object') {
226                 return, function (t) {
227                     console.log('==========2');
228                     console.log(t);
229                     return App.Providers.get(t);
230                 });
231             } else {
232                 return App.Providers.get(provider);
233             }


[46099:0710/164418:INFO:CONSOLE(227)] ""==========2"", source: app://host/src/app/lib/config.js (227)
[46099:0710/164418:INFO:CONSOLE(228)] ""MovieApi?&apiURL=,"", source: app://host/src/app/lib/config.js (228)

…wait, isn’t that the non-existent module’s name? “butter-provider-MovieApi“, and after that a URL that doesn’t look valid at all:, verryyy suspicious (Note: I later found out it is actually valid URL). grep it:

popcorn-desktop$ grep -R 'MovieApi' *
node_modules/             uri: ['MovieApi?'

What ?!! node_modules ! It gets settings from node_modules ?!

grep in packaged version returned no result. So this is what differs between 2 versions!

I copied index.js from packaged version to popcorn-desktop/node_modules..., and this time it started successfully, and is able to load all movies and TV shows!

What really happened

Looking at package.json from the 2 different directory, seems like the two do differ in versions, master uses c745869979405d02d5a4cfea066c227801ee0fc2 (of, 0.3.9 uses 6b8075e0065efbe7276e4a469ca00dbc43c2aaa9. diff:

git diff 6b8075e0065efbe7276e4a469ca00dbc43c2aaa9..HEAD
diff --git a/index.js b/index.js
index dce0e24..ccfcec1 100644
--- a/index.js
+++ b/index.js
@@ -16,10 +16,10 @@ module.exports = {
movie: {
order: 1,
name: 'Movies',
-             uri: ['yts?'
+             uri: ['MovieApi?'
-                     + ','
-//                     + 'cloudflare+,'
+                     + ','
+//                     + 'cloudflare+,'
//                     + 'cloudflare+'
@@ -28,8 +28,8 @@ module.exports = {
name: 'Series',
uri: ['TVApi?'
-                     + ','
-//                     + ','
+                     + ','
+//                     + 'cloudflare+,'
//                     + ''

Testing the URLs in it I found it is actually valid (must be the new gTLDs).

Seems like the name before the question mark is the key, there is butter-provider-yts but no butter-provider-MovieApi in node_modules, so it couldn’t load the latter and errored.

Butter will automatically load any npm package installed (listed in package.json) that matches the =/butter-provider-.*/ regex.

I guess this is in startup stage, and in later stage someone would request the MovieApi or yts provider, but I still haven’t found the code.

Anyway, this fixes the build, so another time!

Neutron VXLAN multicast – disable IGMP snooping

openstack-ansible liberty 6e3815d, l2pop=off

At first I encountered this issue: , but after deleting vxlan interfaces and restarting neutron agents, there are still problems.

  1. Unicast packets in VM tenent networks works, which means from agent namespaces you can ping VM with their IP in that subnet.
  2. Broadcast packets in VM tenant networks doesn’t work, which means VM can’t get DHCP reply, nor ARP reply, and nor can the DHCP agents. but strangely in my case L3 agent can get VM’s ARP reply.
  3. After further investigation, I found that vxlan-X interface on compute and agent containers doesn’t receive broadcast packets that others sent.
  4. Further, on physical interface (in my case `eth1.30`, slave of  `br-vxlan`), I can see packets going out to `vxlan_group` ( by default), but not packets from
  5. Packet capturing on the switch found that the switch is not forwarding the packets.
  6. On #openstack-ansible @jimmdenton pointed me this article: , suggest to disable IGMP snooping
  7. It works!

Thanks to Rackspace guys at #openstack-ansible, who spend a lot of time helping me debug the issue.

This article helped me understand how VXLAN and L2 population works: . (Don’t use L2 population! Neutron developers suggest against it!)


最近想要持續改進 ownCloud Music app ,現在在 Chrome 上面跑的很順,但是我常用的 Firefox 播 FLAC 音樂還是經常會有 glitch (我的 Firefox 本來就已經很頓了,雖然我有 150+ 個分頁,但是我有裝自動 unload 的 plugin 啊,任一時間同時開啟的分頁應該不會超過 10 個,為什麼會這麼慢我也搞不懂),為了要改善效能,我先研究了一下 web worker ,看看有沒有可能把 codec 跑在 web worker 裡面,這樣就不會受到 main thread 的影響了,看看 web worker ,發現有 AudioWorker 這東西,就又再挖進去探究一下(筆記在這)。看 AudioWorker 發現自己實在不懂數位音效處理,就又開始找資源研讀。

關於數位音訊處理,我先前有看過 講得非常好的技術理論的影片。這次找到 Indiana University 清楚簡明的介紹


Frequencies will be recreated later by playing back the sequential sample amplitudes at a specified rate. It is important to remember that frequency, phase, waveshape, etc. are not recorded in each discrete sample measurement, but will be reconstructed during the playback of the stored sequential amplitudes.



Each sample is “clocked" into the DAC’s register. A ‘1’ in a register place will add a voltage to the sum of that sample proportionate to its binary value. In this hypothetical case, we have a sample whose binary value is 5. The gates or switches for the binary places of ‘4’ and ‘1’ are closed, and the value of 5 mvolts is sent out the DAC and held until the next sample is clocked into the register.


對於 DAC (Digital to Analog Converter) 來說,它做的事情就是在每 1/44100 秒內(數字依照取樣率決定), 輸出音訊檔指定的電壓大小,並在那一小段時間維持電壓大小。每個時間小段的振幅資訊在音訊檔裡面叫做一個 frame ,但一個 frame 可能有很多值,一個聲道 (channel) 一個值,譬如說雙聲道有 2 個值,5.1 環繞音效就有 6 個值。

每一個電壓值都是那個時間之內喇叭振動體的位置,電壓值不變,振動體位置就不變,但聲音是空氣的震動,振動體位置不變,空氣就不會被震動,也就沒有聲音。所以舉例來說,如果要產生一個特定頻率、音量的 sine 聲波,做法不是讓 DAC 在時間內輸出某個固定的電壓值,而是讓 DAC 輸出一個不斷變大,到了極大又逐漸變小的電壓值,從極大到極小的速度(譬如說每個極大到下一個極大值的差距是 2205 個 frame),決定了輸出的 sine 波的頻率。

更複雜一點來說,所有的聲音都是不同頻率、音量(最大振幅)、相位的 sine 波疊加出來的,所以,取樣所得的資訊,即是在那個取樣時間間隔內,附近聲音的所有波的振幅疊加。下一個取樣,每個波又會依照自己本來的規律而有不同的振幅。所有波,不同頻率、音量、相位的波,在那小段時間內的振幅資訊,都是疊加在一起被收集起來的,所以不需要「把每個頻率的波的資訊分開來收集」。


不過你或許也注意到了,這邊有 2 個頻率:取樣頻率DAC 變換值的頻率,如果兩個頻率不一樣,還原出來的聲音的頻率就會有變化。

If samples are clocked into the DAC at the rate they were sampled, then the original frequencies will be reproduced up to the Nyquist frequency. If the samples are clocked in at twice the rate, then the frequency will be doubled.

如果取樣頻率和 DAC 變換頻率相同,原始錄音時小於 Nyquist frequency 的頻率就會被重現,如果 DAC 變換頻率是取樣頻率的兩倍,還原出來的頻率也會是原本錄音頻率的兩倍。

懂了這點之後,原本似懂非懂的 Nyquist frequency 也懂啦。


Web Audio API

Web Audio 是在瀏覽器裡面合成、filter、輸出音效的 API 。

音訊處理可以視為一系列的函數,有輸入和輸出,這個概念在 Web Audio 裡面稱作 Node ,有:

  • 音訊來源的 node ,像是直接從程式產生 sine wave ,或是從 buffer 裡面讀取 PCM 資料的 AudioBufferSourceNode
  • 中間處理、形變的 node ,像是改變音量的 GainNode
  • 輸出的 node, 像是輸出至喇叭的 AudioDestinationNode

以上這些瀏覽器內建的常用音效功能實際上都是在獨立的 thread 執行的,但是若用自訂的 Javascript 操作音訊資料,就要使用 ScriptProcessorNode ,但是糟糕的是,它和 DOM 使用同一個 thread ,所以會影響畫面的 rendering ,反之亦然。

因此,若要用自訂的 Javascript 產生音效,像是 Aurora.js 一系列的各種音訊格式的解碼器,內建的簡單 sine wave generator 應該不夠,就必須使用效能差的 ScriptProcessorNode (但是我沒有找到 source code 裡面有那個?可能在解碼器裡面,但他們絕對不會是使用 web worker ,那樣效能會更差,因為 web worker 在不同 thread 之間傳遞資料的效率很差)。

AudioWorker comes to the rescue!

AudioWorker 是基於 web worker ,加強 web worker 資料傳遞功能,成為專門用來在瀏覽器裡面處理音訊的 worker 。

現在 (2015/1/9) AudioWorker 的 spec 還沒完全確定,但似乎已經在最後修改文字的階段了。

使用 Django REST Framework 實作 model 的 CRUD


class MonitorList(generics.ListCreateAPIView, mixins.RetrieveModelMixin, mixins.UpdateModelMixin):
queryset = Monitor.objects.all()
serializer_class = MonitorSerializer

urlpatterns = [
url(r'^monitors/$', views.MonitorList.as_view()),

urlpatterns = format_suffix_patterns(urlpatterns)

後來發現不行,Retrieve 和 Update 必須使用自己獨立的 route 和 view, 必須和 ListView 分開,所以必須要這樣寫:

class MonitorList(generics.ListCreateAPIView):
queryset = Monitor.objects.all()
serializer_class = MonitorSerializer

class MonitorDetail(generics.RetrieveUpdateDestroyAPIView):
queryset = Monitor.objects.all()
serializer_class = MonitorSerializer

urlpatterns = [
url(r'^monitors/$', views.MonitorList.as_view()),
url(r'^monitors/(?P.+)/$', views.MonitorDetail.as_view()),

urlpatterns = format_suffix_patterns(urlpatterns)