What my broken Synology NAS taught me about mdadm , LVM and testdisk

The fail

I was trying to set up SSL client certificate authentication on nginx. In the config nginx needs to read the client certificate CA from custom file location. Then I found out nginx on Synology DSM wouldn’t start because of AppArmor. I tried to modify the apparmor profile of usr.bin.nginx but can only find cache of the profile under /etc/apparmor.d/cache. So I had to completely disable apparmor for nginx, by removing the cache file.

Then it’s some weeks of trial and error to get it working on nginx. In the end I couldn’t get it to work. (I forgot why.)

The problem came when I tried to enable apparmor for nginx by moving back the profile cache and reloading apparmor:

sudo synoservice --restart apparmor

I lost communication with it after this command. It does not respond to HTTP or SSH or any requests anymore. I unplugged it from power and plugged back, it couldn’t boot. The blinking blue light of death.

Disk structure



Below are some operations that I did, not in order, I only want to document how to achieve each individual task.

Mounting the data partition and system partition: https://xpenology.com/forum/topic/7004-tutorial-how-to-access-dsms-data-system-partitions/

On first mount of the system partition, it is readonly, I used the following command to make it readwrite:

sudo mdadm --readwrite /dev/md2

Since the data partition lives on a LVM LV inside a MD array, here is the correct procedure to remove the disk:

  1. umount the filesystem
  2. Deactivate the volume group: sudo vgchange -an vg1000 , if you have more than one VG named vg1000, vgchange will ask you to specify uuid, like this: vgchange -ay vg1000 --select vg_uuid=22aBQe-iiUm-XJEd-cjvF-S9qV-U26U-WpsG3p
  3. Use sudo dmsetup table to check the VG is actually disabled. (If disabled it won’t show up.)
  4. Stop DSM system and data MD arrays: sudo mdadm --stop /dev/md126 , sudo mdadm --stop /dev/md127
  5. Tell the kernel to remove the device: echo 1 > /sys/block/sde/device/delete (issue this as root) (reference)
  6. You should hear the disk power down, wait for it to power down completely, then you can safely unplug the disk.

View MD status:

cat /proc/mdstat

Repairing DSM (failed)

First after mounting the system partition, I took a backup using tar .

Then I slide in another disk to the NAS, ask it to install a new operating system on the disk. Then copied the fresh DSM system partition content into the old system partition. This didn’t work, it still wouldn’t boot.

I also tried the official HDD migration method. Didn’t work.

Then I tried powering on the NAS without a disk, and insert the original disk. Hoping it would detect it as an “migrated" data disk. It asked me if I want to install DSM on the disk, I thought, well, let’s see if it can install the system in existing system partition, and, usually, before any formatting is done, there is a warning. – This was a mistake, I clicked install and it immediately started to format the disk without warning. I immediately unplugged the power, hoping to save the data from being overwritten.


Now I got a disk with broken partition table. I had to recover files from it.

Directly asking testdisk to work on the whole disk device: testdisk /dev/sdb . It did find the MD and LV header, but did not find the ext4 partition inside the LV. I tried waiting on testdisk longer to do “deep search" to see if it finds the ext4 superblock. It didn’t. (I imagined the ext4 superblock should be near the LV header, but seems that it is not.)

Fortunately, I soon find that, the LV is still detected by my system. Not sure if it’s the original LV or the new one created. But I asked testdisk to work on the LV: testdisk /dev/vg1000/lv . And this time it quickly found the ext4 filesystem and the directory structure. The directory structure and content are still preserved.

With this I can start dumping the files using testdisk.


Don’t buy a product that you don’t know how it works. (And have bad documentation on how it works.)

If you ever buy such product, don’t use it in “unsupported" use cases, or try to customize it too much. (but sometimes how much is too much can be hard to tell)

If I need a NAS, buy Node 304 chassis, put ASRock J5005-ITX motherboard in it, and install FreeNAS.

LXC Simple Bridge Networking with existing iptables rules

Environment: RPi 3B+ , Raspbian 9

I set up LXC with simple bridge networking according to the instructions here: https://ubuntu.com/blog/converting-eth0-to-br0-and-getting-all-your-lxc-or-lxd-onto-your-lan .

Same instruction but other ways are documented as well: https://wiki.debian.org/LXC/SimpleBridge#Host_device_as_bridge

The problem I had was, after setting up, the container still couldn’t communicate with outside. I found this Q&A that taught me a nice way to test if it is firewall rules that is interfering:

echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

This will make bridge networking bypass iptables.

After testing, simply change it back to 1.

Then on the second answer it describes my exact same problem.

So I installed iptables-persistent. Initially I couldn’t figure out the format of /etc/iptables/rules.v4. So I issued:

netfilter-persistent save

This will save the current rule into that file.

The next answer is exactly my issue. So I integrated their rule into mine and it worked:

-A FORWARD -o lxcbr0 -m comment --comment "allow packets to pass from lxd lan bridge" -j ACCEPT
-A FORWARD -i lxcbr0 -m comment --comment "allow input packets to pass to lxd lan bridge" -j ACCEPT

Different methods to intercept Android app SSL traffic (public version)

This is a record of my process trying to intercept SSL traffic on Android apps.

Obtaining APK files

Most tutorials would instruct you to download the APK file from third-party sites like APKpure and APKmirror, or install the APK first on your device then pulling it to the computer using adb pull. Here I use another way: manually downloading from Aurora Store. Aurora Store is an unofficial, FOSS client to Google’s Play Store. This gives some advantages:

  • Third-party APK sites might not have latest version of the APK.
  • Third-party APK sites might not have the APK variant (CPU architecture, screen resolution, locale, etc) that fits your device.
  • Aurora Store allows you to easily spoof device models and regions.


  1. If you want to download apps only available in a specific region, connect to a VPN of that region.
  2. Select “Anonymous" when using Aurora Store
  3. Search for the app. The search result will contain apps available in that region.
  4. Go to the app info page.
  5. In the upper right 3-dot menu, select “Manual Download"
  6. The text field will be filled with the latest version code. If you want to download an older version you can change the code here. Note: a) old versions are not always available, b) every app has their own scheme naming the code, c) the code corresponds to android:versionCode property in AndroidManifest.
  7. Click “Download". This will download all split APKs (or Dynamic Delivery) that fits your device profile into Internal Storage/Aurora/.
  8. You don’t have to install the app, because we’re going to modify it.

Using rootless Xposed

Unmodified version of the app failed to run (black screen) on VirtualXposed.

TaiChi even failed to install the app. Taichi will ask to uninstall the app outside Taichi’s container, then after uninstallation nothing follows.

Modifying the APK

Modifying APKs usually involves these steps:

  • Decompile: apktool does a good job
  • Modify:
    • Source code: edit smali, I don’t need it here
    • Resources: AndroidManifest.xml and other XML files, turns out there are quite some difficulty with this part.
  • Rebuild package: apktool usually does a good job, but there seems to be some methods that an app developer can take to make apktool fail rebuilding.
  • Signing: no problem


Since Android 7, user supplied CAs are not trusted in apps anymore. Most guides (like this one) will suggest you to modify networkSecurityConfig in AndroidManifest to make the app trust user supplied CAs.

Unfortunately when I do this for some apps, apktool will fail to rebuild the app. There are many strange errors, some can be solved by instructing apktool to use aapt2, but I encountered many cases where even aapt2 wouldn’t even help. So I had to seek other ways.

AndroidManifest have a complex binary format. It is usually quite easy to decode binary AndroidManifest into readable XML text, in the process the decoder (apktool) will flatten special “binary pointers to external resource files" into text paths and handle other special structures to text. But when encoding text AndroidManifest into binary, there are ambiguities to external references, causing the encoding to fail. (I don’t fully understand this part.) Fortunately, for most apps, there is another way: mark the app debuggable.


For many apps, the networkSecurityConfig is defined like this:

<?xml version="1.0" encoding="utf-8"?>
    <base-config cleartextTrafficPermitted="true">
            <certificates overridePins="true" src="system" />
            <certificates src="user" />

This means that if the app is debuggable, it will accept user supplied CAs.

Modifying debuggable property in AndroidManifest.xml is a much smaller change than modifying networkSecurityConfig path. In the worse case I can just use a hex editor to flip a few bits. And since the change is minimal, apktool should be able to rebuild the app.

In order to modify debuggable property, I tried many tools:

But in the end I had success with Ele7enxxh‘s AmBinaryEditor. (Documentation: http://ele7enxxh.com/AndroidManifest-Binary-Editor.html )

I also wrote some shell scripts to help handle decoding, building and signing split APKs: https://github.com/pellaeon/AddSecurityExceptionAndroid

Full process

Assumption: you already have all split APKs stored in a directory apk1/

1. Decode all APKs:

~/projects/AddSecurityExceptionAndroid/splitApktool.sh decode apk1/
# Decoded apks are put under apk1_tmp/

2 . Binary edit AndroidManifest.xml using AmBinaryEditor:

cd apk1_tmp/xxxx.apk_unpack/
~/projects/AmBinaryEditor/bin/Release/ameditor attr --modify application -d 1 -n debuggable -t 18 -v true -i AndroidManifest.xml -o AndroidManifest.xml1
mv AndroidManifest.xml1 AndroidManifest.xml

3 . Build the APKs

~/projects/AddSecurityExceptionAndroid/splitApktool.sh build apk1/ # It will build from apk1_tmp/

4 . Install the split APKs onto the device

~/projects/AddSecurityExceptionAndroid/adbinstallsplitapk.sh apk1_new/

5 . Check if the package is installed as debuggable:

$ adb shell
j3y17lte:/ $ for p in $(pm list packages | cut -d : -f 2); do (run-as $p id >/dev/null 2>&1 && echo $p); done
com.xxx # If it shows the package id, you have success


# Uninstall APK using pm. Sometimes a package will not remove completely when you use the GUI, causing the installation to fail.

adb shell pm uninstall <com.xxx.packageid>

Forking AmBinaryEditor

During testing I fixed a few quirks of AmBinaryEditor, they are documented in the readme. https://github.com/pellaeon/AmBinaryEditor

For apps that doesn’t have debuggable property already defined

In the aforementioned scenario, debuggable property already exists in AndroidManifest.xml. But if it does not already exist, we need to add an attribute using AmBinaryEditor.

# WON'T WORK: Use this command to add an debuggable attribute to the application tag
~/projects/AmBinaryEditor/bin/Release/ameditor attr --add application -d 1 -n debuggable -r 16842767 -t 18 -v true -i AndroidManifest.xml -o AndroidManifest.xml1

Note: when adding attributes, we need to specify the resource id using -r and a decimal number. Refer to the android source code for system global resource ids. Resource id for debuggable is 0x0101000f , so in decimal it is 16842767.

Unfortunately the debuggable attribute would not be accepted when modified in this way. (It would still install fine but not debuggable.)

To solve this problem, I inspect the APK using aapt:

$ aapt list -v -a apk2.apk
    E: application (line=106)
      A: android:theme(0x01010000)=@0x7f12001e
      A: android:label(0x01010001)=@0x7f110bbe
      A: android:icon(0x01010002)=@0x7f0e0000
      A: android:name(0x01010003)="[REDACTED]"
      A: android:persistent(0x0101000d)=(type 0x12)0x0
      A: android:launchMode(0x0101001d)=(type 0x10)0x3
      A: android:alwaysRetainTaskState(0x01010203)=(type 0x12)0xffffffff
      A: android:allowBackup(0x01010280)=(type 0x12)0x0
      A: android:largeHeap(0x0101035a)=(type 0x12)0xffffffff
      A: android:supportsRtl(0x010103af)=(type 0x12)0xffffffff
      A: android:resizeableActivity(0x010104f6)=(type 0x12)0x0
      A: android:networkSecurityConfig(0x01010527)=@0x7f150003
      A: android:roundIcon(0x0101052c)=@0x7f0e0000
      A: android:appComponentFactory(0x0101057a)="android.support.v4.app.CoreComponentFactory" (Raw: "android.support.v4.app.CoreComponentFactory")
      A: android:isSplitRequired(0x01010591)=(type 0x12)0xffffffff
      A: android:debuggable(0x0101000f)=(type 0x12)0x1

Compared with aapt output from using ameditor attr --modify on another APK with existing debuggable attribute:

$ aapt list -v -a apk1.apk | less
    E: application (line=111)
      A: android:theme(0x01010000)=@0x7f12002e
      A: android:label(0x01010001)=@0x7f110c19
      A: android:icon(0x01010002)=@0x7f0e0000
      A: android:name(0x01010003)="[REDACTED]"
      A: android:persistent(0x0101000d)=(type 0x12)0x0
      A: android:debuggable(0x0101000f)=(type 0x12)0x1
      A: android:launchMode(0x0101001d)=(type 0x10)0x3
      A: android:alwaysRetainTaskState(0x01010203)=(type 0x12)0xffffffff
      A: android:allowBackup(0x01010280)=(type 0x12)0x0
      A: android:largeHeap(0x0101035a)=(type 0x12)0xffffffff
      A: android:supportsRtl(0x010103af)=(type 0x12)0xffffffff
      A: android:resizeableActivity(0x010104f6)=(type 0x12)0x0
      A: android:networkSecurityConfig(0x01010527)=@0x7f150003
      A: android:roundIcon(0x0101052c)=@0x7f0e0000
      A: android:appComponentFactory(0x0101057a)="android.support.v4.app.CoreComponentFactory" (Raw: "android.support.v4.app.CoreComponentFactory")
      A: android:isSplitRequired(0x01010591)=(type 0x12)0xffffffff

One difference I spotted is that, in the previous one, the debuggable attribute is positioned last, and in the latter one, it is positioned in the middle. And in the latter one, attributes are sorted in their resource ID (for debuggable the resource id is 0x0101000f, see the android source code for all resource ids).

Next, looking at AmBinaryEditor’s source code, in function AddAttribute:

            if (list->next == NULL)
            list = list->next;
        list->next = attr;
        attr->prev = list;

It appears that attributes are stored in linked lists, and when adding a new attribute, it is added to the end of the list. This fits our observation from aapt output.

So, in order to make it work, I need to insert the debuggable attribute in correct position. I quickly modified the AmBinaryEditor source code with a hard-coded position index 3:

        for ( int i=0; i<=3; i++ )
            if (list->next == NULL)
            list = list->next;
        ATTRIBUTE *attr_orignext = list->next;
        list->next = attr;
        attr->prev = list;
        attr->next = attr_orignext;

Then try to insert the attribute and build the APK again:

$ ~/projects/AmBinaryEditor/bin/Release/ameditor attr --add application -d 1 -n 'debuggable' -r 16842767 -t 18 -v true -i AndroidManifest.xml -o AndroidManifest.xml1
$ mv AndroidManifest.xml1 AndroidManifest.xml
$ cd -
$ ~/projects/AddSecurityExceptionAndroid/splitApktool.sh build apk2/

This time the attribute is correctly inserted in the middle, and successfully parsed upon installation! Success!



With root, everything is possible. But I didn’t need to go down this path. Well, I had to admit that intercepting traffic in unrooted environment is time spent digging unimportant hole. Eventually I still rooted the phone to intercept the traffic, that is, after all, my real goal.

Magisk only

Xposed doesn’t work on Android 8.1+ yet, so if you need to get this to work, use Magisk, it works on most versions of Android.

This Magisk module will copy the user CA store into system CA store: https://github.com/NVISO-BE/MagiskTrustUserCerts

As of Magisk 20.3 . the repo above doesn’t take any effect, seemingly because it’s using an older Magisk module template that is no longer supported. A pull request is opened to fix it but has not been accepted by the original author yet. In the mean time download the module from here https://github.com/giacomoferretti/MagiskTrustUserCerts to get the working version. (Follow the Installation section to generate installable Magisk zip file.)

Note: How did I discover that the module was not working?
Go to /data/adb/modules/trustusercerts , and I found that post-fs-data.sh , which does exist in the repo, missing. If it installed properly, it should have existed in that directory.

In the end, after I finally got it to work, I found that the particular app that I was looking at seemingly employs some custom SSL pinning, so the Magisk module only allowed me to intercept some of the HTTPS messages. So I had to move on to the next approach.

Objection and Frida

  • Running android sslpinning disable from Objection shell – doesn’t work
  • Tried some popular Frida scripts, doesn’t work either.


In the end, I got everything working with EdXposed and this module: https://github.com/Fuzion24/JustTrustMe

I was able to intercept all traffic from the app.


General walk-through articles

Other tools


KDE Plasma VNC remote session

  • Remote VNC get new session separate from local console
  • Session will remain after disconnect by closing the window
  • Server will be terminated upon logging out from start menu
  • Need SSH forwarding for security
  • Screen resolution cannot be changed
  • Remote user logs in directly with VNC password, no user password needed
  • Need to generate VNC password using `vncpasswd` for the first time

Reference: https://wiki.archlinux.org/index.php/TigerVNC#System_mode


Upgrading my LineageOS on SM-G930F

  1. Download and install latest version: https://forum.xda-developers.com/galaxy-s7/development/lineageos-15-android-8-0-0-herolte-t3710107
  2. Check if TWRP is updated: https://dl.twrp.me/herolte/
  3. Uninstall Magisk (seems that sometimes you need to do it twice)
  4. Install latest Magisk: https://forum.xda-developers.com/apps/magisk/official-magisk-v7-universal-systemless-t3473445
  5. Update OpenGapps – arm64 nano
  6. Install Xposed-magisk from Magisk Manager
  7. Install modified Xposed installer https://forum.xda-developers.com/xposed/unofficial-systemless-xposed-t3388268/post67074423#post67074423


After installing new version of LineageOS. Apps that rely on Google Service Framework always crashes. Usually disabling Magisk and re-enabling works, but this time it doesn’t. In the end upgrading the XprivacyLua module solved the issue.





寫完上面那些之後,我去搜尋了一下有沒有唐鳳跟 FB、LINE 就報導主題對談的紀錄,因為唐鳳大部分的對談都是有公開逐字稿的。結果我沒找到,只有 2017 年 FB 去拜訪他的紀錄。








看到 CPJ 這篇很紅,再看了 HN 底下很多留言,記一下想法。


What is described is better than China, but it’s not ideal. However I do think they should be commended from explicitly avoiding censorship to the degree that they do.

However, communications from the government to counter other communications identified as disinformation are government propaganda. If they are not identified as such, then they have the same problem as the original disinformation.

Secondly, making it easy to flag propaganda in apps and then blocking it as she describes is in fact building in a type of censorship. It’s just a brilliant type of crowd-sourced censorship. I know the theory is that that would only be applied to false information. However, censorship doesn’t work like that. The end result is going to be a worldview that is reduced and shaped by a dominant group rather than the free flow of information. Especially if it relates to all of the information that people in a country can see. This can be very dangerous.

So I will just say what I think would be ideal even if it may not be very realistic. The history of government propaganda and censorship should be part of the public context. I think it is good to be able to flag articles as being propaganda and from what source. However I think that removing things from being visible entirely is very dangerous for freedom. Taiwanese/US/Chinese/etc. government propaganda or counter-propaganda should be identified as such if possible.


This gives me some weird feelings. I mean, if you ask a website to turn some content “unpreferred" without user consensus, isn’t that also some type of censorship?


> How is that reconciled? How does one know when the government is being truthful versus being propagandists?

If there are multiple conflicting narratives, at least some people will be motivated to do more research to resolve the contradiction. If the government is being more truthful, its statements can point to resources that provide independent verification, accelerating the process. At a minimum that will blunt the effect of the lies.


> Once they do that, Facebook promises, by June, that this will inform the Facebook’s algorithm so that it will stop being preferred to show on people’s newsfeed, but it’s not censorship. If you look specifically for that friend, that post is still there, but they have a warning that says it’s already fact-checked as false.

That’d be very interesting if the Taiwanese government was in direct communication with Facebook over something like this. CPJ wasn’t able to get a hold of Facebook, I wonder if anyone here has visibility into something like that? Given that the US government took a more “combative" approach (dragging executives in front of Congress) I’d be curious how more tame approaches like this were being received.

我覺得台灣政府和 FB 在這件事情的合作上面透明度不夠。

This is much better than removing the post, but it still makes me nervous. Just imagine that we’re in the early 2000s, and you can envision a banner saying “The claims in this article are FALSE. As confirmed by the intelligence community and the New York Times, Iraq is actively working on its nuclear weapons program." Perhaps they should have a policy limiting the use of this tool when it comes to issues that may lead to war.


>notably, for all its merits, the taiwanese system still positions the government in an information vetting capacity, which may lead some to think that it is merely another system of propaganda or quasi-censorship

This is kind of how I see it too. I think if your goal is to combat disinformation or propaganda you should just come out straight up and say it because it’s a perfectly defensible position to hold, but creating a government counter-narrative isn’t any more harmless than just censoring something, it’s just sounds nicer than having to use the word “censor" which tends to make people in some parts of the word jump up in panic.


  1. 權力傾向被濫用

政府有了這樣跟 FB 和 LINE 合作的權力,現在打擊不實訊息效果可能很好,現在的政府可能也意圖良善,但是哪天換了一個政府呢?就想,如果國民黨上台,他們會怎麼用這個機制?這些合作的公司又有多少獨立的力量可以抵抗來自政府的要求?


  1. 知名度集中在唐鳳身上,知名度會給一個人更大的權力


  1. 台灣政府和 FB 和 LINE 的合作並不透明


  • 合作的範圍和具體定義
    • 原文 “Facebook promises, by June, that this will inform the Facebook’s algorithm so that it will stop being preferred to show on people’s newsfeed” 這裡的 preferred 是什麼意思?減低多少程度?
      • 不過從另一方面想起來,依照 FB 過去的糟糕紀錄,台灣政府恐怕也不知道這個詞是什麼意思,因為 FB 公共關係部門不會跟他們講清楚的。而且 FB 的公共關係部門大概也不知道這詞到底代表什麼具體的程式行為。
      • FB 演算法相關資訊的透明度大概比北韓政府還低,台灣政府選擇跟這個不透明的公司建立合作而且也沒有立法要求透明,很失敗。(放任壟斷市場的企業繼續不透明,還跟他們合作)
    • LINE 也是
  • 政府和平臺中間具體的溝通機制
    • 溝通管道
    • 主管
    • 實行細則、SOP
    • 決策流程
  • 民間團體的角色
    • 如何監督政府和 FB?
    • 是否具有制衡力?