Building an existing Ubuntu package on Open Build Service

I’ve complained a lot on how the Snap version of Firefox sucks. In the end I found that it’s not possible to run Firefox snap in my own custom network namespace, so I decided to switch to the mozillateam PPA (had to configure APT pinning). But, just in case someday they stop updating the PPA too, I decided to learn to package my own Firefox. Turned out it was not that difficult!

Open Build Service

Open Build Service is provided by SUSE. It’s essentially a CI server for Linux packages. For me the advantages of using OBS are:

  1. OBS has all the package building environment set up, so I don’t have to set up my own.
  2. I don’t have to learn how to *build* the package. I can just download a source package from a PPA and upload it to OBS and it builds it for me.
  3. OBS provides package hosting as well. There’s no extra effort needed from me if I want to let somebody else use my package.

Technically I can also use PPA to achieve those as well, but OBS provides a possibility to also target other distributions. (It looks like a lot of work to configure that though.)

The source of my Firefox package can come from mozillateam PPA, Linux Mint, or PopOS.

OBS Concepts

OBS provides a User Guide, but I found that the openSUSE wiki gives a much better explanation around basic concepts in OBS.

Each project contains the resources needed to build one or more packages (i.e. RPMs/DEBs/etc.). These resources include source archives, patch files, spec files, etc. The output of a project is one or more repositories. A repository is an old familiar concept: simply a bunch of RPMs organized in a directory hierarchy along with some index/meta-data files that make it easy for tools like zypper to search and resolve dependencies.

  • Project: it’s also a namespace for configurations such as build constraints and build depedency preferences. Projects can have Subprojects, which is just an entirely separated project, only with similar names.
  • Repository: repositories can also be used as sources of other projects’ build dependency. The resulting package of a project’s build is also put into the project’s repository.

Each project is also essentially a version-controlled folder (in the folder are those resources mentioned above), managed by the osc commandline tool.

Importing an existing Debian package

I learned about the osc dput command from a talk at DebConf. But when running osc dput it complained “There is no sha256 sum for file _meta.". I worked around it by just downloading the source package files and running osc add on them.

Source package files contains:

  • <package_name_and_version>.dsc : the Debian source control file, which describes a package.
  • <package_name_and_version>.orig.tar.xz : archive file containing the original tarball (source code).
  • <package_name_and_version>.debian.tar.xz : archive file containing Debian build configurations, patches, changelogs, and so on.

After running osc add, run osc ci to commit and upload the changes.

Providing build dependency source repositories

Once uploaded, OBS will immediately start building the package. However it soon returned error saying that it couldn’t find some build dependencies. After rewatching the DebConf talk I realized I have to import the Ubuntu repositories.

After I manually added the update repo

At first, I simply clicked “Add from a Distribution". But OBS still complained that it couldn’t get new enough version of some build dependency. Then I realized that when adding Ubuntu:22.04, only universe is added but not update. A list of all repositories for Ubuntu 22.04 can be found here. I don’t know what universe-update is but I added it anyways.

OBS Project Configuration for Firefox

Two more things I had to change are the build constraints and prefer depedency settings.

Repotype: debian
Constraint: hardware:disk:size unit=G 30
Constraint: hardware:memory:size unit=G 8
Prefer: libncurses-dev
  • Constraint: constraints to the build worker machine. I took the information from here.
  • Prefer: when resolving build dependencies, when multiple packages fits the criteria, OBS doesn’t just randomly choose one out of them. You’re required to explicitly tell it which one to use. If I don’t specify, the build will show a warning message.
    • I checked apt policy libtinfo-dev on my machine and it shows that it’s only a transitional package. Therefore I selected the other option libncurses-dev.

All available project config options are here:

The build should be successful since we did not change anything from the original package. The build artifact (resulting package) will be available in the repository like at

Modifying the package

I will leave this for Part 2.

Firefox snap 依舊雷翻天,我不該浪費時間研究 snap 的,我錯了

把主力機升到 Ubuntu 22.04 之後,我這兩天繼續浪費時間搞 Firefox snap,目前 Firefox snap 主要的問題是:

  1. native messaging 不會動,導致 Mailvelope 這類的插件沒辦法和 snap 環境外的 gpg 程式溝通,不過官方好像已經有解法在 beta 了
  2. 我平常使用 Firefox 會分好幾個 profile ,不同 profile 過不同的 network namespace ,然後在 network namespace 裡面設定 VPN,整體使用 vopono 設定起來非常方便愉快,但是!snap 自己也有用自己的 network namespace 的樣子,所以只要在自己的 network namespace 裡面就無法使用 snap,我已經解掉很多問題成功把 firefox snap 跑起來了,但是最後還是卡在 snap 無法自訂 /etc/resolv.conf ,導致 firefox 無法使用 VPN 的 DNS server ,功敗垂成
  3. 即使界面語言選擇正體中文,仍然顯示英文給你看,這我就懶得修了,我就看英文吧,但這種低級的 bug 居然可以過 Mozilla 和 Canonical 的品管??


所以是時候拋棄 snap ,改用回傳統套件包的 firefox 了。

關於我 debug 在 netns 裡面跑 snap 的過程,我也有在論壇上記錄


我先試了 Linux Mint 的來源,好不容易把 APT pinning 搞定之後,要裝的時候發現 Linux Mint 的版本依賴 Mint 的 ubuntu-system-adjustments 套件,會做一些 Mint 自己的系統變更(比如說設定拒絕安裝 snapd,還有 grub 訊息改成 Linux Mint 之類的)。

所以,還是得要回去用 mozillateam 的 PPA

套件包 GPG 簽章設定

為了增加 Mint 的 APT 來源到我的系統,也要增加 Mint 的 GPG 公鑰,為此順便也學習了 apt-key 被棄用的原因。

簡單來說,原本的 apt-key 的設計是,匯入了一個簽章之後,這個簽章就是全域受信任的,不管套件的來源是哪一個 APT source ,只要有任何一個受信任公鑰的簽名,該套件就會被視為受信任的。這有一個嚴重的問題是,通常不同 APT 來源的發行者是不同人,自然也會擁有不同的簽章,但上面那個設計會導致任何一個受信任簽章持有人都可以發行一個和系統內某個套件同名的套件來蓋掉它

因此現在才改成將 GPG 公鑰寫入 /usr/share/keyrings ,然後額外在每個 APT source 裡面標注該來源使用的簽章:

deb [arch=amd64 signed-by=/usr/share/keyrings/signal-desktop-keyring.gpg] xenial main


垃圾爛系統,毫無客製選項和自由度,唯一會正常工作的情境是開發者想像中的一般使用者的環境,其餘情境下慘不忍睹。我不敢說預設設定的環境不常見,但會用 Linux 的人就是為了可以大幅度客製化自己的環境啊,所以有客製化環境的人所佔的比例肯定比 windows 和 macos 高很多。

Snap 這設計邏輯,堪稱微軟。看看微軟要不要趕快挖角一下 Snap 這垃圾,把它領走好不好,整個子系統都給微軟維護最適合不過了。

自己編譯 Firefox ?

過陣子再來自己研究看看…在 Reddit 上面看到有日本人之前自己維護的優化版 Firefox,似乎可以來自己研究一下怎麼修改編譯選項,改成一些適合我處理器的選項。gcc znver3 選項平均可以提升程式 11% 的效能呢


Ubuntu Snap customization and patching

I hate Snap for a few reasons:

  1. Regression when compared to non-snap versions
    1. In the past, for some apps Chinese IME does not work, such as in Telegram. This seems to have been fixed now.
    2. For some apps user font configurations are not respected, such as Telegram and Spotify.
    3. Bugs bugs bugs, laughable bugs.
  2. The attitude of forcing people to use immature software that loses previously available features. Immature software is fine with me, I agree that to achieve isolation some customizability and features can be sacrificed in the beginning, or even forever, if you just clearly state that those customizability and features are unsupported and allow people to use alternative ways safely. What snap is currently doing is refusing to support widespread legitimate use cases, this is fine by itself, but what snap/Mozilla also does is remove previously-supported ways to use software without snap. These two combined simply goes against the free software philosophy.
  3. Bad documentation: there’s the incomplete official documentation. The details have to be filled in by forum posts by developers under the tag “doc".
  4. Supposedly slower
    1. It’s known that snaps are slower on first launch but “it shouldn’t compromise runtime performance".
    2. But I see numbers that say otherwise, according to the post it’s not snap’s fault but rather mozilla somehow packaged a worse performing binary with snap.
  5. Unclear “security benefits"
    1. Snap executables are “sandboxed" with AppArmor, etc, but who gets to decide the AppArmor rules? The publisher themselves?

Well but since Snap is already here I thought I’d make use of it.

Patching snaps

The spotify snap has a terrible bug. Opening a menu on AMD graphics would freeze the entire app. This also happens on older version of Chromium. Some suggest this is fixed by changing the snap base to core20.

To download the snap and repack it with a new base:

  1. snap download spotify
  2. unsquashfs spotify_*.snap
  3. cd squashfs-root/
  4. Edit meta/snap.yaml to change base: core18 to base: core20
  5. cd ..
  6. snap pack ./squashfs-root
  7. snap install --dangerous spotify_1.1.84.716.gc5f8b819_amd64.snap (no need to remove existing spotify snap, the new one will install alright and inherits user data from the existing snap.)
  8. Note that since it’s a manual install the snap won’t be updated automatically.

Disconnecting interfaces

By default the spotify snap connects the :home interface, which gives access to the home directory. It isn’t really needed for people who don’t use the “local file" feature. We can easily disconnect it with:

snap disconnect spotify:home :home

Also, from what I’m reading, there’s no way of inserting a custom directory in place of /home. i.e. I cannot create a fake home directory at /tmp/home and pass that into the snap as /home. I don’t see why it was designed so badly like this. Passing in arbitrary outside directory is an important use case for sandboxing.


Using ThinkPad media keys F9~F11 on Linux

On the 2021 generation of Thinkpads, the special function keys such as volume, mute, mute microphone, flight mode, etc works out of the box, but the function keys at position F9~F11 does not. (Probably because there isn’t a clear action that matches these icons – what application should the OS launch when users press the “message" key? users have different preferred applications.)

When I tried to map KDE actions to those special keys in System Settings, KDE never detects those keys when I press them. (As if no keys were pressed at all.) So I used xev to test, and verified that those keys were not even handled by Xorg.

To use those keys, one have to modify the udev hwdb to add the mapping of keycodes and action codes. Here’s my settings:

$ cat /etc/udev/hwdb.d/x13-keyboard.hwdb 
evdev:name:ThinkPad Extra Buttons:dmi:bvn*:bvr*:bd*:svnLENOVO*:pn*:*

4B, 4C, 4D are F9~F11 respectively. The keycodes could be obtained by running sudo evtest and looking at the “value" field. The action codes playpause, audionext, phone seem to be called “keysym" technically. However, Xorg and KDE seem to support different sets of keysym, so I actually don’t know where I can get a complete list of all keysyms. I can only test each ones from different lists. The ones listed here can all be detected by KDE.

After we added the new hwdb file, use the following commands to make it into effect.

systemd-hwdb update
udevadm control --reload-rules
udevadm trigger
udevadm info /sys/class/input/eventXYZ

Output for the last command should include the settings added in our new hwdb file, otherwise it means that the settings had not been applied.


Multihome for Docker containers

Environment on docker host:

  1. eth0:, default gateway, which NAT to WAN 1
  2. eth1: directly connected to WAN 2 modem’s bridge port, can dial PPPoE

Goal for my docker container network setup:

  1. Containers can receive connections from eth0 (WAN 1)
  2. Connections initiated by the containers should go via WAN 2

This setup is useful for setting up proxy or VPN servers.

Docker network

Key point here is to create two docker networks, the first one for outgoing traffic, the second one for incoming. If there were only one interface in the container, routing wouldn’t be able to tell the difference between:

  1. Response packets due to an incoming connection, which should go through WAN 1
  2. Packets of connections initiated by the container itself, which should go through WAN 2
docker network create -d bridge --subnet --opt --opt wan2

docker network create -d bridge --subnet --opt incoming

Note that the wan2 bridge was created with custom MTU to fit PPPoE.

Source-based (policy-based) routing

To route traffic to different WAN, we need to set up source-based routing. Essentially, we create a new routing table on the host, and add an ip rule that packets coming from should look up our new table instead of the default one.

First, add a line to /etc/iproute2/rt_tables to create a new routing table:

101     wan2

Next, specify ip rule:

sudo ip rule add from lookup wan2

Then add those routes to the wan2 routing table:

ip route add default via $IPREMOTE dev $IFNAME table wan2                                                                                                                                                       
ip route add $IPREMOTE dev $IFNAME proto kernel src $IPLOCAL table wan2                                                                                                                                         
ip route add dev $docker_bridge_ifname proto kernel scope link src table wan2                                             
  • $IPREMOTE is the remote PPP endpoint
  • $IFNAME is the PPP interface, usually ppp0
  • $IPLOCAL is the local PPP IP address
  • $docker_bridge_ifname is the Linux bridge name created for the docker network, find out by checking ip addr


Accept traffic from host going to the container, without SNAT:

iptables -t nat -A POSTROUTING -s -j ACCEPT

Use iptables to SNAT the outgoing traffic from the container to WAN 2:

iptables -t nat -A POSTROUTING -s -j SNAT --to-source $IPLOCAL

The first rule is meant to prevent SNATing traffic from (the host IP on the docker network wan2) to containters. (So it should be run first.)

Starting a docker container

When starting a docker container, only one network could be attached with docker run:

docker run -d --name container1 --network incoming --dns -p 3128:3128 --cap-add=NET_ADMIN ubuntu/squid

So we must attach the second network manually:

docker network connect wan2 container1

Docker also does not support setting a default route, so we have to docker exec into the container and changing the default route to wan2:

ip route del default
ip route add default via dev eth1

Or we have to change the container route from the outside manually with ip netns.

Strangely, docker always set the container default route to use the incoming network, regardless if I specify incoming or wan2 during docker run.

pppd automation

I’ve added scripts so that pppd automatically sets up and tears down those special route and iptables rules.



export PATH

docker_bridge_ip=$(ip -o -4 addr list $docker_bridge_ifname | awk '{print $4}' | cut -d/ -f1)

iptables -t nat -A POSTROUTING -s $docker_bridge_ip -j ACCEPT
iptables -t nat -A POSTROUTING -s $docker_bridge_subnet -j SNAT --to-source $IPLOCAL

ip route add default via $IPREMOTE dev $IFNAME table $routing_table_for_docker_bridge
ip route add $IPREMOTE dev $IFNAME proto kernel src $IPLOCAL table $routing_table_for_docker_bridge
ip route add $docker_bridge_subnet dev $docker_bridge_ifname proto kernel scope link src $docker_bridge_ip table $routing_table_for_docker_bridge



export PATH

docker_bridge_ip=$(ip -o -4 addr list $docker_bridge_ifname | awk '{print $4}' | cut -d/ -f1)

iptables -t nat -D POSTROUTING -s $docker_bridge_ip -j ACCEPT
iptables -t nat -D POSTROUTING -s $docker_bridge_subnet -j SNAT --to-source $IPLOCAL



Dead ends:

Hacking Dasan Optical Network Terminal H640GO to change low-level configuration

H640GO is a Passive Optical Network Terminal made by Dasan. It has one fiber upstream and 4 Ethernet ports.

With my ISP’s configuration, port 0 is configured to be in bridge mode, port 1-3 is configured to NAT mode. The terminal runs a PPPoE client to obtain dynamic IP address for NAT. My port 0 is connected to my Wifi router configured to dial PPPoE to obtain static IP address.

I want to use the dynamic IP address for the guest Wifi network, periodically redialing to change the address to reduce tracking. Based on my test, the terminal does not run PPPoE relay on port 1-3, so the only way to force redial is to probably reboot the terminal. Not ideal.

Unfortunately the web interface (on does not seem to allow me to change a port into bridge mode. I also looked at its command line interface (similar to other network equipment CLIs, such as configure terminal to enter configuration mode), but there doesn’t seem to be such a command either.

Eventually I figured it out by disassembling the configuration shell binary. I’ll show the research result first and document the research process in later sections.

Switching a port to bridge mode

On a host that is connected to port 1-3:


Username: admin , password: vertex25ektks123

Dasan’s configuration shell dsh will be the default. Use:


to start a UNIX shell.

In this example, I want to set port 1 as bridge mode (from NAT mode). In the shell:

brctl delif br0 eth1

This will remove eth1 from the NAT LAN (br0).

Now a bit of magic is needed:

echo "hybrid port 1 bridge" > /proc/ds_route

Port 1 should be in bridge mode now!

Switching a port to NAT mode

Similarly, start the UNIX shell. Then add the interface back to br0:

brctl addif br0 eth1

Then echo the keyword “route":

echo "hybrid port 1 route" > /proc/ds_route

These settings likely won’t persist after reboot but that’s fine because there’s no need to frequently reboot them either.


Connecting to the terminal

Initially I was only limited to the web interface with user / user. admin / admin didn’t work. Port scan revealed that telnet is running on the terminal. I tried searching in exploit-db but didn’t find anything useful (the exploits were old so probably already patched). Though in one note it seemed that Dasan doesn’t respond to security issues at all. And looking at their other exploits it seemed that their software is really crappy and probably contain a lot of vulnerabilities.

Further googling (keyword: “Dasan Networks GPON ONT admin password") led me to Minh’s post, which listed the telnet password: admin / vertex25ektks123

Locating the shell binary

From reading Dasan’s documentation, it seems that most functionalities are controlled with the specialized configuration shell (`dsh`). But underneath the configuration shell is the Linux operating system. So, the functionalities provided by dsh must be implemented by changing the underlying Linux system. If I want to change the system behavior in a way not allowed by dsh, I need to reconfigure the Linux system directly. And to do that, I need to learn how dsh does it by reverse engineering it.

I can type ? in the shell to see what commands are available. There is a command show process that prints the process list:

H640GO# show process 
    1 admin     3340 S    init  
    2 admin        0 SW   [kthreadd]
    3 admin        0 SW   [ksoftirqd/0]
    4 admin        0 SW   [watchdog/0]
    5 admin        0 SW   [events/0]
    6 admin        0 SW   [khelper]
    9 admin        0 SW   [async/mgr]
   84 admin        0 SW   [sync_supers]
   86 admin        0 SW   [bdi-default]
   88 admin        0 SW   [kblockd/0]
  105 admin        0 SW   [rpciod/0]
  114 admin        0 SW   [kswapd0]
  115 admin        0 SW   [aio/0]
  116 admin        0 SW   [nfsiod]
  117 admin        0 SW   [crypto/0]
  149 admin        0 SW   [mtdblockd]
  170 admin        0 SW   [lilac_spi.0]
  198 1         2140 S    /sbin/portmap 
  203 admin     3456 S    /usr/sbin/inetd 
  218 admin        0 SW   [ubi_bgt1d]
  249 admin        0 SW   [ubi_bgt5d]
  272 admin        0 SW   [ubifs_bgt5_2]
  284 admin        0 SW   [ubi_bgt3d]
  293 admin        0 SW   [ubi_bgt4d]
  311 admin     3340 S    /sbin/syslogd -O /etc/.config/syslog.0 -s 1024 -l 6 
  314 admin     3340 S    /sbin/klogd -c 6 
  356 admin        0 SW   [Pon]
  357 admin        0 SW   [TSSI checker]
  368 admin        0 SW   [MacAgingThread]
  369 admin        0 SW   [pstat_collector]
  380 admin        0 DW   [HwMonitorTask]
  693 admin     3488 S    lighttpd -f /etc/lighttpd.conf 
  718 admin    18548 S    /bl/bin/onuapp -s 0000000000000001 -w 00000000000000000000 
  725 admin     7400 S    /sbin/autoprovision 
  732 admin     5864 S    /sbin/autoupgrade 
  900 admin     3456 S    /sbin/pppd call dsl-provider +ipv6 
  908 admin     5344 S    /bin/gather_optic_power 
  909 admin     5344 S    /bin/factory_reset_monitor 
  910 admin     5396 S    /sbin/snmp_bcm 
  911 admin     3404 S    /sbin/getty -L -f/etc/issue 115200 ttyS0 vt100 
 1647 admin     3340 S    /usr/sbin/udhcpd /etc/udhcpd.conf 
 1952 admin        0 SW   [flush-ubifs_5_2]
 1953 admin     3524 S    telnetd -i > /dev/null 2>&1 
 1954 admin     2888 S    -sh 
 1960 admin     6584 S    /bin/dsh 
 1961 admin     3404 R    ps

So obviously the configuration shell just calls ps (BusyBox). And I guessed /bin/dsh to be the shell binary, which turned out to be right.

Copying files

The terminal has wget and tftp, but nc would be way easier.

Start listening for the file on the receiver:

nc -lp 1999 > dsh

Send from the terminal:

nc -w 2 1999 < [FileName]

Disassembling the binaries

I put the dsh binary into Ghidra. It was able to decompile it into pretty readable C.

I additionally looked at, which seems to be Dasan’s support library for issuing system commands. It has functions that takes care of saving and applying system configs. (It tries to implement a system-wide configuration file, which can then be parsed and implemented by changing configurations to the underlying Linux system and daemons.)

There’s also sysinfo.cgi, which renders the system info web page. By reading it I was hoping to learn where the web interface got its source of truth.

However, I wasn’t able to find anything that controls the NAT / bridge mode of ports.


Poking around the system, I found that a lot of the device’s low-level controls are exposed under /proc, many prefixed with ds_* (probably short for “Dasan"). One thing I found related to my goal was /proc/ds_route:

H640GO@:~ # cat /proc/ds_route 
route mode:HYBRID
 PORT1:NAT (Enabled)
 PORT2:BRIDGE (Enabled)
 PORT3:BRIDGE (Enabled)
 PORT4:BRIDGE (Enabled)

Searching for “/proc/ds_route" led me to this function in dsh:

Note that some of the function names had been changed by me.

The function allowed me to reconstruct the syntax of writing into /proc/ds_route, and that I also need to brctl to change port status.

Other interesting bits

The system might also actually support VLANs because there’s /proc/ds_macctrl:

H640GO@:/proc # cat /proc/ds_macctrl 
vlan-filter on port=0, bridge=0: table={0,000000000000000000000000000000000000000000000000}, untagged=y
vlan-filter on port=1, bridge=0: table={0,000000000000000000000000000000000000000000000000}, untagged=y
vlan-filter on port=2, bridge=0: table={0,000000000000000000000000000000000000000000000000}, untagged=y
vlan-filter on port=3, bridge=0: table={0,000000000000000000000000000000000000000000000000}, untagged=y
vlan-filter on port=5, bridge=0: table={0,000000000000000000000000000000000000000000000000}, untagged=y

There are lots of other interesting things about the system. I will dig into those next time.

It’s also very interesting to learn about how ISPs deploy their network with PON!


Linux 下使用 Thinkpad 特殊按鍵 F9~F11

最新一代的 Thinkpad 在 F9~F11 的位置有幾個特殊的功能鍵在 Linux 下預設無法使用,但其他功能鍵都正常,例如我按 F8 按鍵的時候電腦就切換為飛行模式。

我要在 KDE 的鍵盤捷徑裡面設定使用這些功能鍵的時候,按下按鍵它始終偵測不到,所以我猜大概就是底層的 Xorg 根本沒有處理這個按鍵,使用 xev 測試驗證了我的猜測。

要使用這些特殊按鍵,就要去修改 udev 的 hwdb,加上這些按鍵對應的功能碼。我設定成這樣:

$ cat /etc/udev/hwdb.d/x13-keyboard.hwdb 
evdev:name:ThinkPad Extra Buttons:dmi:bvn*:bvr*:bd*:svnLENOVO*:pn*:*

其中 4B, 4C, 4D 分別是 F9~F11 ,這個編碼可以利用 sudo evtest 獲得(在 value 欄位),然後 = 後面的 playpause, audionext, phone 這些字串似乎叫做 “keysym",猜測是 key symbol 的意思。問題是 Xorg, KDE 似乎分別有不同支援keysym ,所以我其實不知道要從哪裡得知完整可用的 keysym 列表,只能從不同來源的列表中一個一個測試。上面列出的都是我測試過 KDE 可以正常偵測的。


systemd-hwdb update
udevadm control --reload-rules
udevadm trigger
udevadm info /sys/class/input/eventXYZ



處理 ZFS 永久資料錯誤,還有 ZFS 沒考慮到的事

現在安裝 Linux 我都會使用 ZFS 為檔案系統,但最近遇到一個問題讓我開始懷疑是否該繼續用 ZFS。

通常使用 ZFS 都建議要有冗餘,也就是有一個以上的資料副本,通常是分散在不同硬碟。不過如果是筆電就很難有一顆以上的硬碟。


永久資料錯誤 (permanent data error) 通常是在 scrub 的時候會發現,可以透過 zpool status -v 檢視錯誤的物件。發現永久資料錯誤的時候 pool 的狀態會變成 DEGRADED。

zpool status -v 會顯示類似這樣的訊息:

    errors: Permanent errors have been detected in the following files:


冒號前的字串是 dataset 編號,冒號後的字串是物件編號,若編號是可識別的就會顯示它的名字(檔名或是 dataset 名稱)。有以下幾種可能

  • <metadata>:<0x...> 全 pool 的 metadata 區域的物件
  • <0x...>:<0x...> 無法識別的 dataset,因為 dataset 無法識別所以當然也無從得知物件的名稱
  • /some/path/name 現在已掛載的 dataset 中此路徑的檔案或目錄物件損毀
  • dsname:/some/path 在可識別但目前未掛載的 dataset 當中的某一路徑的檔案或目錄損毀,dsname 也有可能是 snapshot
  • dsname:<0x...> 在 dataset “dsname" 裡面的某個無法識別的物件損毀,也有可能是因為該檔案已經被刪除


  • <metadata>:<0x...> 這我沒遇過所以不確定處理方式。可能要試試看回滾所有 dataset (rollback) 到完整的 snapshot ,再不然就只能 zpool destroy 整個 pool 然後重建。但 metadata 區塊很小,所以真的遇到永久資料錯誤的機率應該不高。而且據說 ZFS 針對 metadata 有其他特別的機制可以保護,避免損壞。
  • <0x...>:<0x...> 無法識別的物件損毀,所以很可能已經刪除了,再 scrub 一次或兩次就會清除錯誤,因為 zpool status -v 會顯示過去兩次 scrub 的錯誤。
  • /some/path/name 若該檔案有備份,則直接將備份複製然後覆蓋到這個損毀的檔案上,再執行 scrub 就會清除錯誤。如果沒有備份,可以刪除那個檔案然後 scrub 兩次清除錯誤。
  • dsname:/some/path 只能刪除該 dataset ,但有可能會造成其他問題(下面詳述),或許可以掛載該 dataset 然後用備份覆蓋裡面的檔案或是刪除(不確定可不可以)。若直接刪除該 dataset ,如果有其他 dataset (例如比較新的 snapshot)去 reference 同一個損毀的區塊,再 scrub 一次的話就會變成另一個 dataset 顯示錯誤。
  • dsname:<0x...> 應該也是 scrub 一次或兩次就會清除錯誤。

以上處理方式,其實官方文件寫得沒有很清楚,因為似乎官方文件認為大家都有 redundant vdev 所以這些錯誤不太可能會發生。我還要額外收集好多其他資料才彙整出這些處理方式。

ZFS 沒有考慮到的事情

ZFS 的設計上極力避免永久資料錯誤的發生,說明裡面也經常警告使用者一定要有 redundant vdev,但,ZFS 對於永久資料錯誤發生之後如何修復或是認可錯誤的支援,似乎非常不周到。

從前文可見,ZFS 永久資料錯誤發生之後,幾乎就只有兩個選擇,但使用上仍然非常不直覺:

  • 如果檔案有其他備份,可以直接覆蓋損壞的檔案,但這招並不適用於 snapshot 當中損毀的檔案,有人建議了一個用 zfs send 和 rollback 結合的迂迴處理方式,我還沒試過但顯然非常複雜。
  • 如果檔案沒有其他備份,就只能直接刪除那個檔案,但如果檔案存在於 snapshot 當中,只能刪除整個 snapshot。另一個或許可行的迂迴處理方式就是把那個 snapshot 轉換成 clone (可寫入的 snapshot),刪除 clone 當中損毀的檔案,然後刪除原本損毀的 snapshot ,再把 clone 轉換回同名的 snapshot,但這樣大概也會影響到 reference 同一個損毀檔案的其他 snapshot,等於是這些 snapshot 都要做同樣一次上述操作。

除此之外,ZFS 的另一個問題是,一旦某個檔案發生永久資料錯誤,應用程式在讀取那個檔案的時候一旦遇到錯誤的 block ,ZFS 就會回傳一個終止讀取錯誤,讓應用程式不能讀到錯誤 block 後面的所有資料(即使後面有些未損壞的 block)。舉例來說,一個 10MB 的檔案,其中第 2~3 MB 的區塊損毀了,一般的應用程式就會正常讀取 0~2 MB,然後遇到錯誤就終止,即使 3~10 MB 範圍的資料仍然是未損壞的。雖然可以理解這樣的設計,就是要避免應用程式讀到損壞的資料,但為了避免這件事情就讓應用程式幾乎整個檔案不能讀取,有時候也不太適合,畢竟並不是所有應用程式都不能忍受一點點資料損毀。這個問題之前也有人遇過,他用 dd conv=noerror 忽略讀取錯誤,將 ZFS 拒絕讀取的區域留空,複製出來一個大致完整的檔案。

ZFS 也沒有任何「認可錯誤」的功能,也就是我希望有辦法可以跟 ZFS 說:「這個區塊壞掉了沒關係,你從現在開始就把它當成正確的。」對於很多已經有內建 redundancy (某些資料庫檔案?)或是可以容許誤差的檔案類型(例如影片、圖片、音訊,大部分狀況下 100 秒的影片其中遺失了一秒並不會大幅影響使用)這非常有用。

上述的問題有幾個相關的回報和 PR,但都還沒有被接受:

Ubuntu 和 ZSys 也有點雷

  • Ubuntu ZFS 安裝的 bpool (拿來裝 /boot 的 pool)只有 2GB 左右,很容易就塞滿了。這老問題了即使不用 ZFS 也會遇到,只是有了 ZFS + Zsys 自動 snapshot 這空間會吃得更快。我參考了這篇來解決。
  • 但一直解決不了,因為 grub-probe 會一直叫 failed to get canonical path of rpool。查了很久才知道原來是因為我的 rpool 處於 DEGRADED 狀態…這訊息提示也太糟了吧。
  • 然後我還踩到這個還沒解決的 bug:zsysctl state remove 找不到明明就存在的 state ,後來之後手動 zfs destroy 那個 state 的 snapshot…


Linux 的 overlayfs 不支援使用 ZFS 作為 upperdir 哦。

有股想回去用 FreeBSD 的衝動,但筆電的硬體支援可能還是不行啊,唉。


Guessing, reconstructing and recovering data from a partially wiped disk (which turned out to be encrypted HFS+)

A friend asked me to help recover data from a encrypted external drive. He didn’t know how it was encrypted, but he has the password.

First examination of the drive showed that it has only one 200MB FAT partition in the front sectors labeled “EFI System Partition", the rest are unallocated. Since it’s a 2TB drive, the unallocated space probably contained partitions that had been deleted.

My first idea was to create an image of the disk to prevent further overwriting the data using dd. dd ran for about six hours and exited, reading only about 900GB of data. dmesg showed a lot of read errors, so the drive was probably also corrupted.

GNU ddrescue

ddrescue is a tool that is not derived from dd, designed to image a failing disk, salvaging as much uncorrupted data as possible. It has a smart algorithm to first dump the healthy sectors and gradually moving on to more difficult parts. It also saves a “mapfile" which is like a map of the disk, marking tried and non-tried areas, etc. Mapfile allows for continuation of interrupted dumps.

I first ran:

ddrescue -n /dev/sdc sdc.img mapfile


ddrescue -r 1 /dev/sdc sdc.img mapfile

Side note with ZFS: I forgot that the disk I’m saving sdc.img to only has 1TB, but ddrescue completed successfully, ls reported that the image file is indeed 2TB. So I’m probably saved by ZFS compression!

Is it BitLocker?

The crucial next step to recover data would be to rebuild the partition table to align with the actual partitions living on the disk. Once partitions have been identified, I can then mount the filesystem within it (or repair the filesystem). To do so, I must determine what exactly is the filesystem of the lost partition.

I decided to look at the files on the only healthy partition: the EFI System Partition. Mounting this partition was easy, it was just FAT. In the partition I found a few files generated by Windows, such as WPSettings.dat , so I assumed the disk to have been encrypted with BitLocker.

The dislocker-find utility from the dislocker project is capable of searching BitLocker partitions. However, it can only check a partition already in existence in the partition table, but not search for one in a disk.

Reading the dislocker-find source code shows that the program simply checks for a signature on partitions to determine if it contains BitLocker. Searching “signatures" in the repo shows two relevant files, one is dislocker.c which defines the Ruby signature variables (dislocker-find is written in Ruby); the other one is common.h which defines the signature constants:


#define NTFS_SIGNATURE           "NTFS    "


I should be able to find the Bitlocker partition boundaries by searching for these strings in the entire disk. Bitlocker To Go for this case is more likely since it’s an external drive. I did so first with simple grep:

grep -a -b "MSWIN4.1" sdc.img #FAILED

This failed because grep would ingest a whole line into memory before processing. This is a binary file without newline, so during ingestion grep would eat all the memory and crash. Instead, one should use dd and pipe to fold then grep:

dd if=sdc.img | fold | grep -b -a "MSWIN4.1"

Before searching with the Bitlocker signature I also confirmed it working with different strings known to exist on the disk. However it couldn’t find any occurrence of the Bitlocker signature. (I only let it searched through a few gigabytes on the disk, because the first partition only occupied the first 200MB space, it wouldn’t make sense to create a second one with a ton of space in between.) I also tried “-FVE-FS-" with no result.

The findings so far indicates that either my method was incorrect, or that the disk really didn’t contain a Bitlocker partition. To prove my methods, I spawned up a Windows VM, attached a virtual USB disk (disk image) to it, enabled Bitlocker on the USB disk, and examines the disk image.

Note: To enable Bitlocker, the USB disk has to be first formatted in NTFS, otherwise the bitlocker options won’t show up.

There were two options to enable Bitlocker given by Windows: encrypt the entire disk, or encrypt only the files (I forgot the exact wording).

Before actually testing the Windows created Bitlocker disk image with grep, I noticed neither of the two methods created a “EFI System Partition", which made me suspicious whether it was actually encrypted with Bitlocker.

Note: some useful information I found on recovering Bitlocker.

Recover partitions with testdisk, is it HFS+?

I grew suspicious of the disk being in Bitlocker, so I decided to take testdisk for another spin. testdisk found an additional HFS+ partition that comes after the EFI System Partition, I added it back to the partition table, and tried to mount it but failed.

Reading the fsck.hfsplus manpage, seems like it’s able to repair the B-tree in the filesystem. I don’t actually know the internals of HFS+ but B-tree sounds like a internal structure to supports storing files.

Before running fsck I need to set up a loop device that contains only the HFS+ partition. Then I can use -fryd to repair the B-tree:

fsck.hfsplus -fryd /dev/loop22

Unfortunately it failed to repair. I also tried using fsck_hfs on macOS which reported the same error.

Note: to mount a full disk image on macOS to a virtual device, use:

hdiutil attach -imagekey diskimage-class=CRawDiskImage -nomount filename

This command would create a new device node /dev/diskX where X is a number depending on your device. Then simply run fsck_hfs -fryd /dev/diskXsY

Is it encrypted HFS+?

I vaguely remembered the days when Apple had to maintain and build new features upon the antique HFS+ because APFS was still in development. HFS+ does not support encryption, and Apple’s way of adding disk encryption support was to develop a “volume manager" called Core Storage that can do encryption on the block level and create the actual file system (HFS+) in a encrypted volume provided by Core Storage. Core Storage serves a similar purpose to LVM on Linux. This is probably why I couldn’t mount the HFS+ partition previously.

To serve as a comparison, I decided to create a “encrypted HFS+ external disk" with macOS. But Apple being Apple, removed the feature from its Disk Utility in recent macOS versions. Fortunately, I can easily create macOS VirtualBox VMs with this script. I used the High Sierra version. Note: though there are other projects that provision macOS KVM VMs, I decided to use VirtualBox because its GUI to passthrough USB devices (or to create virtual USB disks) is much easier to use than editing qemu commandline arguments.

Here’s the disk layout of encrypted HFS+ created by macOS:

sudo fdisk -lu /dev/sdd
Disk /dev/sdd: 7.45 GiB, 8000110592 bytes, 15625216 sectors
Disk model: X
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: X

Device        Start      End  Sectors  Size Type
/dev/sdd1        40   409639   409600  200M EFI System
/dev/sdd2    409640 15363031 14953392  7.1G Apple Core storage
/dev/sdd3  15363032 15625175   262144  128M Apple boot

That 200MiB EFI System partition is exactly the same as the one on my corrupted disk!

Verifying Core Storage signatures on disk

To verify that it is actually Apple Core storage and Apple boot partitions sitting on the disk, I have to find the partition signatures on it. The locations of signatures are also important because I need to recreate the partition table based on the actual partition location.

libfvde has documented the Core Storage signature and its location: at 88 bytes from the start of the partition, there is a signature “CS". “CS" translates to byte sequence “0x4353″ in ASCII. (Another reference here)

I then used SearchBin to search for the byte sequence in the disk image:

python3 SearchBin/ -p 004353 sdc.img

“00″ was added to rule out false positives. The field technically contains other header data and might not always be “00″, but on my sample healthy Core Storage partition it is “00″ so I used it anyways.

SearchBin quickly gave me a match at offset 209735767, which translates to almost 200MiB. This could very well be the beginning of the missing Core Storage partition, since the preceding partition is exactly 200MiB in size.

On my healthy sample, the Apple Core storage partition starts at sector 409640, 409640 times sector size of 512 bytes equals 209735680 bytes, close to 200MiB. 209735767, the signature location, minus 87 (offset), equals 209735680 ! This means that the corrupted Apple Core storage partition starts at the same sector (409640) as the healthy one! This makes sense if both of them are created with default settings.

I also used hexdump to verify that the data on both samples are aligned and similar:

hexdump -C -n 10K -s 209735680 sdc.img

Finding the Apple boot partition

Next, I have to find the boundaries of the missing Apple boot partition. Looking at the healthy sample, the Apple boot partition occupies near the end of the disk, but leaves some trailing sectors.

The healthy disk:

  • has 15625216 sectors in total
  • 15363032 to 15625175 is the apple boot partition

The boundaries of the apple boot partition are probably calculated backwards from the end of the disk, since this would leave most of the space in the middle for actual Core Storage data.

Assuming these won’t change across different disks:

  • distance from end of apple boot partition to end of disk
  • distance from start to end of apple boot partition

I can work out the start and end of the missing apple boot partition from the size of the disk (3907029168 sectors):

  • start: 3906766984
  • end: 3907029127

And, according to the healthy disk, there are also no space between the Core Storage partition and the Apple boot partition, so I can work out the end of the Core Storage partition:

  • start: 409640 (already known)
  • end: 3906766983 (one sector less then the start of apple boot partition)

Similarly, I also used hexdump to verify the data on both disks are aligned (looking at the location of zeros vs non-zeros).

Recreating the lost partitions in the partition table

Now I have all information I need to recreate the lost partitions with fdisk sdc.img:

Note that, when specifying the type of partition, one cannot simply type the names of the types like “Apple boot" or “Apple Core storage". One has to use the GUIDs. All GUIDs supported by fdisk are also not listed if you press ‘L’ to list. This is weird but I don’t know why it is the case.

$ fdisk sdc.img                                                                       
Welcome to fdisk (util-linux 2.36.1).                                                                                 
Changes will remain in memory only, until you decide to write them.                                                   
Be careful before using the write command.                                                                            
命令 (m 以獲得說明): p                                                                                               
Disk sdc.img: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors                      
Units: sectors of 1 * 512 = 512 bytes                                                                                 
Sector size (logical/physical): 512 bytes / 512 bytes                                                                 
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt                                        
Disk identifier: X

所用裝置        Start       結束       磁區  Size 類型
sdc.img1         40     409639     409600  200M EFI System
sdc.img2     409640 3906766983 3906357344  1.8T Apple boot
sdc.img3 3906766984 3907029127     262144  128M Apple HFS/HFS+

命令 (m 以獲得說明): t                                    
分割區編號 (1-3, default 3): 2
Partition type or alias (type L to list all): 53746F72-6167-11AA-AA11-00306543ECAC

Changed type of partition 'Apple boot' to 'Apple Core storage'.

命令 (m 以獲得說明): t                                    
分割區編號 (1-3, default 3): 3
Partition type or alias (type L to list all): 426F6F74-0000-11AA-AA11-00306543ECAC

Changed type of partition 'Apple HFS/HFS+' to 'Apple boot'. 

命令 (m 以獲得說明): p                                    
Disk sdc.img: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt                                        
Disk identifier: X

所用裝置        Start       結束       磁區  Size 類型
sdc.img1         40     409639     409600  200M EFI System
sdc.img2     409640 3906766983 3906357344  1.8T Apple Core storage
sdc.img3 3906766984 3907029127     262144  128M Apple boot

命令 (m 以獲得說明): w                                    
The partition table has been altered.
Syncing disks.                                             

After using fdisk to recreate the partition tables on the disk image, I used the same hdiutil attach command above to attach the disk image. macOS immediately detected the disk, and prompted for password. Once I entered the password, I can see everything inside. Success!

Revisiting testdisk

I was curious why testdisk did not identify the Apple Core storage partition, so I checked its source code. Quickly searching the signatures of Core Storage yields nothing. It seems that testdisk is not able to identify Core Storage partitions. Not sure why not, because it seems easy, maybe simply relying on such as short single signature would give too many false positives?

Lessons learned

  1. Most filesystems are robust in a way that corrupted disk sectors would only cause damage to whatever that is sitting on it, but little else. i.e. Entire file system would not be brought down because of a few corrupted sectors. Even if that’s a encrypted filesystem.
  2. Most filesystems mark their beginnings on disk with signatures or magic bytes. Deleting a partition from the partition table doesn’t change the partition data itself. Partition recovery could generally be achieved by finding those signatures, using the location of signatures to determine partition boundaries, and recreating the lost partitions in the partition table.
  3. Well-known easy-to-use tools can only automate so much, and they’re still dumb. Data recovery, forensics, reverse engineering requires a high level of human ingenuity, that calls on out-of-band information (such as I know the person is more likely to be using macOS encryption) and recognize patterns and weigh the nuances.

2021: 武漢肺炎封鎖第二年的雜想

從 2020 年封鎖以來,或許描述我心理狀態最簡單概要的一句話就是「想要見面跟人講話但又沒有」。如今封鎖已經快要兩年,又值 2021 年末,想要藉著本文來回顧一下自己今年做了什麼事,想了什麼事。


不確定是因還是果,2020、2021 這兩年另兩個主要的感覺是「無聊」、「重複」。儘管嘗試做了很多以前沒做過的事情(釣魚、爬小山),但總得不到什麼樂趣,仔細想想我甚至還是覺得最近在研究 fontconfig 和字型渲染比較有趣。2020 我最愛的 Men I Trust 樂團有句歌詞一語中的:

Days will be the same,
in a different way.

Men I Trust 是某一個晚上酒後 Youtube 推薦給我的,一開始是 Lauren:

Lauren 一定要配 MV 聽,沉浸感極強。

聽到後來,Spotify 都說我是 Men I Trust 前 0.01% 的粉絲。


不過 Men I Trust 是 2020 了(而且我覺得他們最好聽的都是 2020 以前的歌),2021 我一路聽 Men I Trust 聽到九月吧,直到我和朋友一起聽了 L’Impératrice 的現場演出:

這沉浸感更不得了。它其實也不是真的現場,是虛擬巡迴。然後 2021 我接下來都在聽 L’Impératrice 了,所以說 2021 我的代表曲目應該還是它。但 Men I Trust 大概還是第二名。











  • 寫待辦事項
  • 每天回顧自己今天完成的自己,肯定自己今天所完成的事情
  • 番茄鐘


  • 訂定一個每天的工作時段,時數是自己每天最多可以維持高度專注的時數,一般人不超過五小時,我那時訂了四小時
  • 每天工作時段的開始,先打電話和另一個人講今天四小時要完成的事情,對方聽不懂沒關係,重點是有人聽
  • 當天工作時段內只做這些事情,做完就提早下班,不要多做,做不完也照樣下班,不要繼續做
  • 每週回顧工作狀態




這兩年分心的標的經常是 Youtube 上面的冷知識短片,Vox, Wendover Productions 之類的。雖然當中的確有些好的內容、可以學到一點東西、又滿足好奇心,但 Youtube 平臺的設計上很容易讓人一部接著一部看下去,經常不小心一看兩小時,只吸收到一些碎片知識、一點好奇心滿足的快樂。

相比之下,兩小時如果去看電影,可以獲得精心設計、專業製作的娛樂;兩小時去看 MIT 開放課程,可以獲得一些有結構的知識,兩個都比兩小時的碎片好。甚至如果真的想要一點滿足好奇心的冷知識的話,去看紀錄片所獲得的東西還比較深入。









我到了陌生城市的朋友家度假,決定嘗試三天兩夜的登山,朋友很罩,借我一堆東西還幫我背。第一天走了七八公里,海拔從 300 升到 1300 左右(沒記錯的話),整個白天都在山裡走路,沒有網路。我很累而且腳痛,但紮營完畢、吃完晚餐,那一晚的夢讓我嚇到了,夢很長,以前的朋友、情人都出現了,還有好多以前的想法、感覺,比平常的夢真實十倍,而且有一部分是清醒的(可以用意識控制夢裡場景和事件的清醒夢),睡了十二個小時醒來,我還花了一小段時間回想為什麼我會在這裡。





這句話是幾年前從朋友的發文看到的。學生時代,我所吸收的資訊有一大部分來自於同儕。開始工作之後,少了學生時代的同儕,即使是同事,所提供的資訊量已經比學生同儕少了。遠端工作之後,所吸收的資訊幾乎都是來自網路上不認識的人了。記得以前高中的時候朋友花了一個下午的時間試圖教我 Big O amortized analysis(但我聽完還是似懂非懂),大學的時候偶爾還有社團朋友會聊到然後就開始解釋 Java garbage collection 還有為什麼那個很爛(記得我們在討論為什麼 minecraft server 會這麼吃記憶體,然後我們後來還發現 C++ 實作的 minecraft server, 結果用起來 bug 很多只好繼續用 Java 版本…)。現在類似的機會很少了。


大部分社交網站給的動態消息都是內容推薦演算法為了最大化「engagement」組合出來的,個人難以控制。所以我開始儘量訂閱有興趣的網站的 RSS ,RSSHub 也很有幫助。但 Facebook 粉絲專頁目前還沒辦法產出 RSS,因為 Facebook 有很討厭的機器人偵測。我最近的一個 side project 正設法解決。

另外一部分,當然還是儘量多看點書,少看點 Youtube。



但總覺得這樣不只是少了許多樂趣,也很容易產生盲點無法察覺。所以 2021 開始,我自己立了一個目標就是要增加產出,無論是筆記、部落格文章、程式,或是演講;同時,減少不斷地吸收資訊而沒有消化。雖然吸收資訊滿足好奇心很快樂,為吸收的資訊做筆記相對來說就沒那麼令人愉悅,但若不作筆記的話所吸收的那些資訊很難長久留存在記憶,或是未來融會貫通,只會是過眼雲煙而已。


大概從 2019 年開始,媒體開始廣為報導各大社交網站公司令人作嘔的行為。雖然我之前就對這些壟斷網際網路的商人們沒什麼好感,但他們令人作嘔的程度甚至還超過了我的想像。所以決定儘量減少在 Facebook 發文,改用 Mastodon, 多寫部落格。


寫這篇的時候正值每年年底的 Chaos Communication Congress,已經連續兩年被迫變成純線上的活動了。線上活動其實跟沒有活動差不多,好一點的狀況下活動前幾天會意識到哦原來有個線上活動,報名的時候發現報名居然已經關了,幸好線上直播是開放的。看了幾場今年 CCC 的直播,實在沒什麼令人印象深刻的演講。但因為時差關係我也沒看很多議程,等錄影出來我再好好挑有興趣的來看看。



第一家是 Tendon Tenya, 日本開來台灣的平價連鎖天婦羅丼飯店,他當然不是那種高級的天婦羅(高級的天婦羅我也沒吃過),但三百上下的價錢可以吃到的品質在台灣已經堪稱一絕。武漢肺炎來襲前就已經歇業了。聽說其實有賺錢但因為不明原因日本總部決定結束台灣市場。但或許在武漢肺炎來襲前就歇業也算是好事吧。

台灣 Tendon Tenya 和我在日本吃到的品質相當接近,反觀開在 Tendon Tenya 隔壁的吉豚屋,這幾年店越開越多,品質也穩定下降(Hamasushi 亦同),或許 Tendon Tenya 早早結束台灣市場也是讓我免於他品質下降的遺憾。

第二家是心目中台北(可能台灣)最好吃的越南牛肉河粉店:開在敦化南路一段190巷內的美越牛肉河粉。已經關了不知道幾年了,但從那至今在台灣沒吃過比它好吃的牛肉河粉了。所幸後來在多倫多發現了 Phở Hưng 和 Pho Linh,但 Phở Hưng 在武漢肺炎期間直接暫停營業連外帶生意都不做,害我都吃不到。