Output audio to Apple HomePod on Linux

Conclusions first: it’s doable but very unstable at the moment.

My OS version: Ubuntu 23.04.

Background

Just searching for “homepod linux" would yield results mostly related to connecting HomePod to Homebridge or Home Assistant, which isn’t what I’m aiming for.

The closest project I could find was BabelPod, but it hadn’t been updated for years.

The openairplay project has an old list of Airplay compatible libraries and programs. I’ve found a few more in addition to that:

I wasn’t aware of the distinction between the different protocols talked by Homepod and Apple TVs. But at some point I came across the Pulseaudio-roap2 project, then I know I’m looking for a ROAP client.

I also found that Pipewire seems to support ROAP. From this issue it seems it might work.

Testing Pipewire 0.3.65, the default version on Ubuntu 23.04

I loaded the roap-discover module, printing debug messages:

PIPEWIRE_DEBUG=3 pw-cli -m load-module libpipewire-module-raop-discover

In my audio panel I immediately see the device detected. (Don’t know why it’s duplicated though.)

I selected the device as output, and started playing some sound. The Homepod seems to detect the stream because it stopped the music that I was originally playing via another Apple device. However, there was no sound from the Homepod!

Tracing the debug message and Wireshark packet capture showed that the server (Homepod) returned a message with timing_port=0, which makes Pipewire thinks that the response is invalid.

Looking through newer commits to the file, I found one commit that should fix the issue. However the commit is only available after version 0.3.68.

Building the latest Pipewire and copying over the module file

So now the goal becomes to upgrade Pipewire on my system to at least 0.3.68. My first idea is to build the entire project manually and just copy over the module file libpipewire-module-raop-sink.so to /usr/lib/x86_64-linux-gnu/pipewire-0.3/.

Result? the older version of Pipewire wouldn’t load the plugin file from a newer version.

So now I have to really upgrade the whole Pipewire.

Upgrading to Pipewire 0.3.71 from Debian experimental

Add the package archive. /etc/apt/sources.list.d/debian-experimental.list :

deb [signed-by=/usr/share/keyrings/debian-archive-buster-automatic.gpg] http://httpredir.debian.org/debian/ experimental main contrib non-free

Install the keyring:

sudo apt install debian-archive-keyring

Next is to identify the packages that needs to be pinned. I also found this excellent tutorial on APT pinning.

The easiest way is to add every binary package built from the pipewire source package to the pin list.

In case there is any unmet dependency requirements when trying to upgrade pipewire, just add those dependencies to the pin list.

Here’s my final pin list, /etc/apt/preferences.d/pin-experimental :

Explanation: use only specific packaeges from debian experimental
Package: *
Pin: release o=Debian,a=experimental,n=rc-buggy,l=Debian
Pin-Priority: -10

Package: gstreamer1.0-pipewire pipewire pipewire-alsa pipewire-audio pipewire-audio-client-libraries pipewire-jack pipewire-libcamera pipewire-tests pipewire-v4l2 pipewire-bin libpipewire-0.3-modules libpipewire-0.3-0 libspa-0.2-bluetooth ilibspa-0.2-dev libspa-0.2-jack libspa-0.2-modules libpipewire-0.3-common pipewire-pulse
Pin: release o=Debian,a=experimental,n=rc-buggy,l=Debian
Pin-Priority: 500

Then simply run apt upgrade and restart pipewire.

Use PIPEWIRE_DEBUG=3 pw-cli -m load-module libpipewire-module-raop-discover to load the module again, and select Homepod as the main audio output.

Note: before I realized that I can just get the Debian keys in Ubuntu package debian-archive-keyring

I’m lazy to learn the apt-key transition again.

  • The archive signing keys page does not show which key signs the experimental channel package, perhaps there is something that I’ve mistaken.
  • Before installing the correct keys, apt update would complain key 648ACFD622F3D138 is not found. Looking it up, it is the debian 10 key.
  • To trust the debian 10 key, one needs to download the key, create a new keyring and add it to the keyring, and finally configure apt to use that keyring with that package source.

Testing Pipewire 0.3.71: it works! (sort of)

Now I’m able to play sound to the Homepod!…

But it will only keep playing for about 40 seconds and then there is no sound. This error was shown:

[E][02299.572523] default      | [   rtsp-client.c:  462 on_source_io()] 0x55d067414b50: got connection error -32 (管線損壞)
[E][02299.572567] mod.raop-sink | [module-raop-sink: 1481 rtsp_error()] error -32
[I][02299.572628] mod.raop-sink | [module-raop-sink: 1474 rtsp_disconnected()] disconnected
[I][02303.821931] pw.stream    | [        stream.c:  606 impl_send_command()] 0x55d0673eecc0: command Spa:Pod:Object:Command:Node:Suspend
[I][02303.822024] pw.node      | [     impl-node.c:  412 node_update_state()] (raop_sink.woshi.local.192.168.212.40.7000-0) running -> suspended
[I][02303.822222] pw.node      | [     impl-node.c: 1985 pw_impl_node_destroy()] (raop_sink.woshi.local.192.168.212.40.7000-0) destroy
[I][02303.822444] pw.node      | [     impl-node.c: 1985 pw_impl_node_destroy()] (raop_sink.woshi.local.192.168.212.40.7000-0) destroy
[I][02303.822586] default      | [   rtsp-client.c:  101 pw_rtsp_client_destroy()] destroy client 0x55d06744f140
[I][02303.822614] default      | [   rtsp-client.c:  101 pw_rtsp_client_destroy()] destroy client 0x55d067414b50

Strangely I don’t see any connection error while recording the packets with Wireshark.

I’ll debug this problem some other day…

USB-C, DisplayPort, HDMI, color format

I’m using 2 external monitors with my Thinkpad X13 Gen 2 AMD. Since the laptop only has one HDMI output, the second monitor had to be connected with a USB-C to HDMI adapter.

However I found that when using the adapter, the colors on the monitor will look washed out. Which prompted me to look into this topic.

First, I’ve seen people suggesting setting “Broadcast RGB" to “Full":

xrandr --output HDMI-1 --set "Broadcast RGB" "Full"

However I later found out that “Broadcast RGB" setting only worked on Intel graphic cards.

Another answer to the same question suggested setting “output_csc" for AMD. However again I later found that “output_csc" is only available on radeon driver, which is the old open-source Linux driver for AMD graphic cards. Modern AMD graphic cards generally use amdgpu driver.

Unfortunately you can’t currently set color format setting in the amdgpu driver. There are discussion to expose that setting to the userspace.

Another answer suggested to set amdgpu.dc=0 kernel parameter. It turns off the Display Core module, which is the basis for modern Linux graphic driver stack. Without it, there would be no 3D acceleration.

From my xrandr output I learned that the laptop is sending DisplayPort signal to the adapter. So that means the adapter turns DisplayPort signal into HDMI. I suspect there could be something wrong with the process so I tried to figure out if it’s possible to use HDMI Alt mode instead of DP Alt mode. (Using the Type-C cable to transport HDMI signal instead of DP signal.) It appears that HDMI alt mode is now rarely used and is about to be deprecated. I’m also not sure if my graphics card will be able to output two HDMI signals at the same time.

Then I looked into the monitor OSD and found the color format setting (called “RGB computer range" here):

If I change it to “RGB(0~255)" the colors look normal! It was set to Automatic previously. Maybe it wasn’t able to automatically select the correct setting because of of the adapter converting DisplayPort to HDMI.

Links:

Bliss OS on Virtualbox Installation Notes

Bliss OS version: Bliss-v15.8.5-x86_64-OFFICIAL-gapps-20230308

VM specifications:

  • OS: Ubuntu 64bit
  • Processor: 4
  • System > Enable EFI
  • Display
    • Memory: 128MB
    • Controller: VBoxVGA

Installation:

  1. Use GPT? No
  2. Create the first primary partition, 500MB, set bootable
  3. Create the second primary partition, the rest of disk
  4. Write to disk
  5. Select the second partition as installation target
  6. Use ext4
  7. Install EFI GRUB? Yes

Boot:

  1. Press ‘e’ to edit the first entry
  2. Add nomodeset VIRT_WIFI=1 between ram0 and $src, resulting line: linux $kd/kernel stack_depot_disable=on cgroup_disable=pressure root=/dev/ram0 nomodeset VIRT_WIFI=1 $src $@
  3. Press Ctrl-x to boot

I have to do this every time because I haven’t found a way to make it permanent.

References:

Downgrading Ubuntu packages that are no longer available on package mirrors

My computer has been experiencing random freezes after upgrading to Ubuntu 22.10. I wan’t able to find the cause, but today I came across this bug report alleging that Mesa 22.2 was the culprit. So I decided to downgrade Mesa to see if it resolves the issue.

If the version is still available in the archive

https://www.linuxuprising.com/2019/02/how-to-downgrade-packages-to-specific.html

But I need to downgrade to Mesa 22.1, which is not available in the archive.

Finding the old version deb file

https://askubuntu.com/questions/1209643/where-do-i-find-old-version-of-deb-files

1. First go to the source package page: https://launchpad.net/ubuntu/+source/mesa/

2. Click on your release, my is Kinetic https://launchpad.net/ubuntu/kinetic/+source/mesa

3. Click on the version you want under Releases in Ubuntu https://launchpad.net/ubuntu/+source/mesa/22.1.7-0ubuntu1

4. Click on amd64 under Build

Now download the debs.

Checking which debs are needed

  1. Use apt list -a --installed <package_name>*to check if the package has an already installed version
  2. Delete the deb if not.

Installing the deb

Have the debs ready in a folder, run sudo apt install ./* . It should automatically resolve the dependency. apt should say DOWNGRADING some packages, while not uninstalling anything else.

Use sudo apt --dry-run install ./* to verify before actually installing.

Hold the old versions

sudo apt-mark hold libegl-mesa0 libgbm1 libgl1-mesa-dri libglapi-mesa libglx-mesa0 libxatracker2 mesa-va-drivers mesa-vdpau-drivers mesa-vulkan-drivers

To avoid the old versions from being replaces when you run apt upgrade next time.

References

https://www.reddit.com/r/linux_gaming/comments/ynue9u/how_can_i_check_what_mesa_driver_is_in_use/

Ubuntu 上的 pi-hole 和 systemd-resolved 衝突

最近要把家中的 pi-hole 從舊的 Raspbian + Rpi3b+ 移到新的 Ubuntu + Rpi4,沒想到遇到麻煩的 systemd-resolved 衝突問題,這篇做一些筆記。

現在比較新的桌面版 Linux 都有內建 systemd-resolved ,systemd-resolved 實作了一個 stub resolver,會聆聽在 127.0.0.53:53 回應 DNS 查詢。另外 systemd-resolved 預設也會去編輯 /etc/resolv.conf,當中把網域名稱伺服器設為 127.0.0.53,這樣一來不支援 glibc 或是 DBUS DNS 查詢的程式就會利用 stub resolver 解析域名。

不過這樣一來本機的 port 53 就被佔用,pi-hole 就沒辦法聆聽。

下了一堆錯誤關鍵字繞了一大圈才發現 pi-hole 文件就有特別說明 Installing on Ubuntu or Fedora

基本原理是這樣:

  1. 把 stub resolver 關閉(不能把 resolved 整個關閉,因為這樣 DBUS 和其他查詢 API 會失效)
  2. 確保 /etc/resolv.conf 指向 resolved 控制的 /run/systemd/resolve/resolv.conf
  3. 修改 netplan 設定確保該網卡界面使用 127.0.0.1 作為域名伺服器

至於為什麼要有 systemd-resolved 這麼複雜又麻煩的設計?主要原因是出於如果讓各應用程式和網卡界面直接去修改 resolv.conf 的話會讓新啟動的界面去覆蓋原有界面的 resolv.conf 設定。例如有多重 VPN 連線提供不同路由目標 (route destination) 連線的時候,新啟動的 VPN 如果覆蓋了先前啟動的 VPN 設定的域名伺服器,就會讓所有 DNS 查詢送往新 VPN 指定的域名伺服器,破壞了先前啟動的 VPN 的設定,造成 DNS query leak 。

關於為什麼要有 systemd-resolved,這篇文章解釋得非常詳盡清楚

參考資料:

Linux 上顯示中文字型的研究、Kubuntu 的 bug (56-kubuntu-noto.conf)

我對字型繪製有興趣很久了,高中時有稍微研究過,但那時候對 Linux 的桌面軟體元件的組成不太瞭解,研究不出個所以然。最近把日常主要電腦從 macOS 換成 Linux ,才又重新開始研究。本文記錄了我這次研究結果的梗概。

本文分成三部分,一是關於字型的一些基本知識的筆記,二是關於 Linux 桌面環境中調整字型設定和除錯的方式,三是總結我認為可以提升 Linux 字型繪製效果的方法。

一、基本知識

字型檔案當中僅包含字型形狀的向量,實際上要把字型顯示在螢幕上的時候需要一套軟體把向量轉換成一個像素的矩陣,當中包含每個像素的顏色。這個軟體稱為 rasterizer(中文我稱它為繪製器),Linux 上面是 freetype。

在繪製的過程當中有各種轉換可以套用,讓最後的效果「更好」,就是下面講到的 hinting, antialiasing, kerning。

Hinting 字型微調

因為目前大部分螢幕的像素密度不夠高,如果不是 HiDPI 螢幕的話,大部分的 DPI 都在 150 以下,這個像素密度下並沒有足夠的像素數來展現不同字的形狀向量。Tonsky 這篇 “Not enough pixels" 段落解釋得很好。

Hinting 有很多種策略和技術可以達成,以 Linux 上面設定字型繪製的 fontconfig 來說,hinting 選項可以設定啟用或是停用。啟用 hinting 後,接下來要選擇「如何 hint」,主要有兩種方法:autohint 和字型內嵌的 hinting instruction(微調指令)。

autohint 是 freetype 的一個元件,裡面是一套演算法可以把任何字型向量繪製到像素上面,autohint 還可以透過 fontconfig 裡面的 hintstyle 設定值進一步設定。autohint 和忽略任何字型裡面內嵌的微調指令,因此繪製出來的結果會較不忠於字型原本的設計。

第二種方法則是在字型檔內嵌入微調的指令,告訴繪製器該如何繪製,這些指令都是字型設計師撰寫的,因此每個字型通常會不一樣,根據微調指令繪製出來的字型也會比較忠於本來的設計。

Antialiasing and Kerning

  • Kerning:個別字元間距的調整,比如 AW 兩字因為形狀的關係如果用尋常的字距,視覺效果會看起來空白異常大(但其實是大腦解讀的問題),所以為了讓整體顯示看來平衡,kerning 會縮短 AW 兩字的字距。這對中文方塊字幾乎沒有影響。
  • Anti-aliasing 反鋸齒,在低解像度的螢幕上顯示比解像度密度更高的線條,會因為無法顯示而造成鋸齒,需要反鋸齒的模糊化效果讓顯示看起來正常平滑。

Windows 字型繪製策略

根據我從各種地方看到的資料,Windows 的字型繪製策略大致上是以「清楚」為優先,捨棄字型本身的設計。變清楚的方法主要是透過微軟的 ClearType 技術(hinting 的一種),裡面包含了各種技巧,主要是 hinting 吧,還有像素對齊之類的。當然各個版本的 windows 略有不同,10 之後可能是鑑於一般電腦的 DPI 平均來講都提高了,所以不像以前採用那麼強烈的演算法來讓字體變清楚。

macOS 字型繪製策略

macOS 的字型繪製策略比較偏向以「維持字型設計」為優先,捨棄字體的清晰度。長久以來完全不使用 hinting,僅使用反鋸齒。

關於 macOS 和 Windows 兩種繪製策略的比較,可參考這篇文章

Linux 字型繪製策略

Linux 的字型繪製策略在預設狀況下通常是介於 Windows 和 macOS 之間,我想主要是因為兩個原因:

  1. 早年進階的 hinting 演算法受到專利限制沒有在 freetype 裡面實作,因此 freetype 只能做最基本,效果比較差的 hinting
  2. macOS 很可能內部有些特殊的演算法讓字型繪製結果比較漂亮又忠於原設計,Linux 裡面自然也沒有這些演算法,所以沒辦法做到像 macOS 一般漂亮的字型繪製

不過 Linux 下的字型繪製系統具有非常大的彈性,且文件完整,可以自己設定。再加上近年來螢幕像素密度提升,降低了 hinting 的需要,且比較高的像素密度也代表即使沒有厲害的字型繪製演算法,單純把向量繪製成像素矩陣,效果也不會太差。所以我現在已經覺得 Linux 下的字型繪製效果堪比 macOS。

個人喜好大不同

有位發文抱怨 Linux 字型渲染(渲染=繪製)的作者喜歡 Windows 的策略,我可能是因為最近幾年主要使用 macOS ,所以比較喜歡 macOS 的策略。

二、Linux 字型設定檔和調校程式

會影響一個系統字型顯示效果的因素大概有兩個:

  1. 如何選擇正確的字型或是 glyph
  2. 選定 glyph 之後要如何把 glyph 內指定的曲線繪製成像素陣列

在 Linux 系統內,1 是由 fontconfig 決定,2 是由 freetype2 決定。

但是 freetype 的 rasterization algorithm 就我所知是沒有使用者可以調整的選項的,所以 Linux 使用者要調整字型一般只要專注調整 frontconfig 設定就好,fontconfig 裡面設定的選項會在繪製時傳遞給 freetype,包含是否要啟用 hinting, anti-aliasing 等等。

另外有一些除錯工具像是 `fc-match` 可以測試 fontconfig 規則。

pango-view 也很有用,pango-view 是一個簡單的小工具,可以從命令列生成一個陽春的視窗,內嵌指定的文字,繪製視窗的文字時便會使用到 fontconfig 。可以配合 FC_DEBUG 環境變數來查看 fontconfig 選擇字型的過程。舉例來說,我會這樣用:

FC_DEBUG=4 PANGO_LANGUAGE=zh-tw pango-view --font="Noto Sans CJK TC"  -t "一旦商品出貨"

如此一來就會生成一個視窗,同時在命令列輸出 fontconfig 的除錯訊息:

FC_DEBUG 這個變數控制要輸出的除錯訊息種類,文件中 《Debug Applications》段落有說明。

三、Linux 字型美化策略

以相似字型取代常見版權字型

Linux 下另一個讓字型沒那麼好看的常見問題是應用程式(主要是網頁)設計者會指定 Windows 或 macOS 上面常見的專有字型,這些字型在 Linux 上面沒有內建,然後 Linux 又沒有設定好適當的規則來用已有的字型來替換這些專有字型。

如果替換規則沒寫好,使用不恰當的字型去替代本來的專有字型,就會有外觀風格不符合的問題。

從使用者端改善這個問題的方式有兩個方向:

  1. 直接在 Linux 上面安裝 Windows 和 macOS 的字型,但這個很可能是字型授權條款所不允許的(當然也沒有一定要遵循授權條款啦)。參考這個專案
  2. 在 Linux 上面使用相似專有字型的開源字型來替代,就是 fedora-better-fonts 這個專案在做的事情。

順帶一提,或許是因為目前常用的中日韓字型數量相較於西文少很多,開發者僅需要專注設定少數中日韓字型,所以在 Linux 上安裝的開源中日韓字型大致上都有設定好它們會取代什麼專有字型,所以上述問題比較不嚴重。如果有網頁使用特殊的中日韓字型大多也會以 web fonts 的方式使用(也就是由網頁提供字型,不需要使用者系統有安裝)。

Kubuntu bug: 56-kubuntu-noto.conf

這邊要講一個長久以來困擾我的問題,也是為什麼我會再次開啟對字型的研究:我主要使用 Kubuntu 發行版,而我發現 Kubuntu 上面中文字型總顯得纖瘦,好像不是正方形而是高的長方形。仔細測試發現,在 KDE 環境下開啟的所有 Gnome 的應用程式都沒有這個問題,令我更加困惑。

與此同時,我注意到 Gmail 界面使用的 Roboto 字型(Google 似乎現在已經換成 Google Sans)會在 l 和 e 中間有不成比例的空白,其餘的字距也顯得不一致:

這才讓我搜尋到這個問題: https://askubuntu.com/questions/1289600/firefox-hint-issue-when-editing-text-in-roboto-font-how-to-fix-it

底下的回答提到 56-kubuntu-noto.conf 這個設定檔不只會造成 Roboto 的問題,很多其他字型也會受到影響。

於是我就試了把該檔案移除,沒想到真的解決了我的問題!

移除設定檔前
移除設定檔後

至此,終於解決了我長久以來的困擾!

後來我再稍微調查了一下為何會有這個設定檔,我拼湊起來的資訊看來是這樣:

  • 這個設定檔的來源是 2006~2009 年間的 Chrome OS,ChromeOS 大多跑在低像素密度的裝置上面,因此中文字型顯示需要開比較強的 hinting 才能顯得清晰。後來 ChromeOS 2012 左右就把這個設定檔移除不再使用了。
  • KDE Neon 因為某種原因引入了這個設定檔到系統中,下游的 Kubuntu 也把它收入
  • 這個設定檔造成了很多奇怪的字型顯示問題(不只影響中日韓文字),在其中一個 bug report 裡面有人留言說他去問了當時引入這個設定檔的開發者,他看起來沒興趣把這個設定檔移除

四、相關資訊

DivestOS

Hacker News 看到的,現在才知道,這 OS 竟然比 LineageOS 還早。

它是一個 LineageOS 的 fork,設計宗旨是儘可能提供安全性給原廠已經不支援的手機。

為此,DivestOS 自己開發了一個自動 patch kernel CVE 的工具,且:

  • 支援 bootloader relocking ,可以確保開機安全
  • 提供每月安全更新

這些特性讓它已經比 LineageOS 安全了。

開發者還有追蹤 Android, Linux kernel,和他支援的各裝置的漏洞修補狀況

如果有舊手機的人,這應該是比 LineageOS 還要安全的選擇。

不過相比之下,DivestOS 設計上並不考慮 Google Play Services ,雖然可能可以自己裝,如果要用 Play Services 的人可能還是用 LineageOS 比較好。

同場加映

Crawling Instagram/Facebook for RSS feed, while benefiting others

Bibliogram has a feature to generate RSS / Atom feeds and I’m using it a lot. However, there are a few limitations, which also applies to RSSHub.

First, Bibliogram on your IP address could be blocked by Instagram. This happens so often that a big part of discussion around Bibliogram is centered around this. I’ve tried a few solutions to this:

  • Flowing the traffic through VPNs: doesn’t work, Instagram/FB has required login on all server IP sections by default.
  • Flowing the traffic through “residential proxies": most residential proxies are not really ethical because they rely on embedding exit nodes to third-party applications. Application developers may integrate with their SDKs and the users of the application will be used as exit nodes, most often without clearly communicating the risks and benefits to them.
  • Flowing the traffic through Cloudflare workers: same as VPNs, server IP ranges are mostly blocked. There’s also an added issue that each request from Cloudflare workers will use randomly different outbound IPs, and it’s not possible to bind to a single one.
  • Flowing the traffic through mobile internet IPs: most mobile internet providers utilize carrier-grade NATs (CGNAT) so that tons of devices will use the same public IP. To avoid collateral damage of banning a ton of devices, IG/FB is very tolerant with the mobile IPs. Barely any rate limiting at all. However mobile IPs are expensive (compared to other solutions), to save data quota, I had to set up Squid proxy to route only the most critical requests (the main HTML file or some API calls) through the mobile internet while keeping the others (CDN requests) using an existing link.
  • Flowing the traffic through the dynamic IP address provided by my home ISP: this is the solution that I settled on. I can configure the modem to re-dial PPPoE periodically to obtain different IPs. See Multihome for Docker containers for my PPPoE and docker setup.

However, despite all of this, request quotas towards FB/IG is still a scarce resource. So that’s why I thought: once I crawled a page and generated a RSS feed, how can I benefit others safely?

Second, when only using a simple reverse proxy, every request is handed directly over to Bibliogram, without any caching. So that means every update pull from the RSS reader will trigger a new crawl, consuming precious quota. So I realized it was necessary to set up caching on the reverse proxy.

Third, the images in the RSS feed entries are originally served by FB/IG’s CDNs. The URL looks like:

https://scontent-tpe1-1.cdninstagram.com/v/t51.2885-15/311989025_797227311577987_2533783926321053094_n.jpg?stp=dst-jpg_e35_s1080x1080&_nc_ht=scontent-tpe1-1.cdninstagram.com&_nc_cat=107&_nc_ohc=q7iUiYhKInsAX-YuXi3&edm=AOQ1c0wBAAAA&ccb=7-5&oh=00_AT-vwyBEJjJ70fuXIuI06Vmjdmay2jfzx6o6SXvJFQzAdg&oe=63597DE5&_nc_sid=8fd12b

In the GET parameters contains a access key which is only valid for around a week (according to my testing). After which the CDN will no longer serve the file. The file is still there but you have to crawl the original post source again to get a new access key. However, once the RSS feed entry is published, the reader usually doesn’t update the entry. Even if it does, crawling the same post still eats up precious request quota.

The best solution I can currently think of is to cache the image with Nginx as well. It will take up disk spaces but the images are already heavily compressed, so usually just a few hundred kilobytes a piece. Disk spaces also isn’t expensive nowadays. For RSSHub which just point the image src to CDNs, I’ll have to modify the code and make it point to Nginx. For Bibliogram, it already rewrites the image src to /imageproxy/Cj_XXXX.jpg, which can be easily cached by Nginx.

Nginx proxy_store

Enter Nginx proxy_store. Nginx proxy_store basically just saves a copy of the response on disk, with path equivalent to the request URI. It’s usually used to set up mirrors, of which objects usually don’t expire.

However it does not offer expire functionality, so objects will have to be deleted from the cache manually.

Nginx geo: differentiating client IPs

The geo directive sets a nginx config variable to different values depending on the client IP address. This is useful for us to:

  • Give trusted IPs access to the crawler (Bibliogram or RSSHub) and the cache.
  • Give untrusted IPs access to only the cache.

This way, untrusted IPs can’t make our crawlers crawl, but they can still use the contents the crawler has already generated.

Putting it all together

I marked the config sections “Section N" to explain the config flow.

geo $whitelist {
	default					static;#only allowed to cache
	1.2.3.0/24				app;#allowed to invoke app
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name bibliogram.example.com;

	access_log /var/log/nginx/bibliogram.access.log;
	error_log /var/log/nginx/bibliogram.error.log;

    root /var/www/html;

	# https://docs.nginx.com/nginx/admin-guide/web-server/serving-static-content/
	sendfile	on;
	tcp_nopush	on;
	tcp_nodelay	on;

	# https://serverfault.com/a/1033578/54651
    # Section 1
	location / {
		root               /var/www/bibliogram;
		try_files $uri @${whitelist};
		add_header	X-Tier	"tryfiles";
	}

	# Don't cache the user html page, otherwise the filename will conflict
	# with the directory name /u/<username>/rss.xml
    # Section 2
	location ~ /u/[\w\._-]*$ {
		root               /var/www/bibliogram;
		try_files $uri @${whitelist}_nocache;
		add_header	X-Tier	"tryfiles_nocache";
	}
	# Section 3
	location @static {
		root               /var/www/bibliogram;
		try_files $uri =404;
	}
    # Section 4
	location @static_nocache {
		root               /var/www/bibliogram;
		try_files $uri =404;
	}
    # Section 5
	location @app {
        proxy_pass http://localhost:10407;
        proxy_redirect     off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header   Host $host;
		proxy_store        on;
		proxy_store_access user:rw group:rw all:r;
		proxy_temp_path    /tmp/;
		add_header	X-Tier	"app_cache";

		root               /var/www/bibliogram;
	}
    # Section 6
	location @app_nocache {
        proxy_pass http://localhost:10407;
        proxy_redirect     off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header   Host $host;
		add_header	X-Tier	"app_nocache";

		root               /var/www/bibliogram;
	}

    ssl_certificate /etc/letsencrypt/live/bibliogram2/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/bibliogram2/privkey.pem; # managed by Certbot
}

This settings enable to following behavior:

  1. For trusted client IPs, the $whitelist variable will be set to app
    • Section 1: if the URI is already cached, serve from the cache, if not go to the named location @$whitelist, which expands to @app (Section 5).
    • Section 5: pass the request to Bibliogram, save the response under /var/www/bibliogram.
  2. For untrusted client IPs, the $whitelist variable will be set to static
    • Section 1: if the URI is already cached, serve from the cache, if not go to the named location @$whitelist, which expands to @static (Section 3).
    • Section 3: the same configuration as Section 1, which serves from the cache. If the URI is not present in cache, 404 is returned.
  3. This special case takes precedence over case 1 and 2. If request URI ends in /u/<username>, which is the URI for HTML rendered pages of a user. The response should never be cached, because this page is only visited from browsers, and I want Bibliogram to crawl for the latest information for the user. This configuration also forbids untrusted IPs from accessing user HTML pages, since The the responses are never cached and the untrusted IPs can only access the cache. (Maybe I’ll change this in the future.)

Result

It turns out that, strangely, my RSS reader (Inoreader) will request a RSS feed file very frequently. Before setting up caching, this would make Bibliogram start to crawl Instagram every time, wasting the request quota. I guess this is because Inoreader uses the RSS lastBuildDate attribute to determine next crawl time. And since Bibliogram generates a new feed file every time Inoreader requests it, Inoreader thinks that the feed is updated very often, so tries to request the feed even more often. However in all of these times the pubDate of the newest post is actually what matters.

Next steps

With the cache in place, I can see from the access logs that Inoreader is starting to get a lot of 304 Not Modified responses from Nginx. This would prevent Bibliogram from having to crawl too often.

With the cache in place, I can also decide on my own when to update the feed files, by deleting the one I want to expire in the cache. Currently this process is all manual, but I should define a cron job to delete the feed files more than 2 days old, for example.

An even better solution would be to decide when to expire the feed by looking at the feed’s post interval.

Future enhancement: rewriting to dedicated image hosts

Cloudflare workers can be used to turn cloud drives into static file hosts using ondrive-vercel-index. In the future if the cache directory became too large to serve locally, maybe we can redirect the cache request to the domain served by onedrive-vercel-index. Then we can move some older cache objects to the larger cloud drive.

References

References:

Dead-ends:

Building an existing Ubuntu package on Open Build Service

I’ve complained a lot on how the Snap version of Firefox sucks. In the end I found that it’s not possible to run Firefox snap in my own custom network namespace, so I decided to switch to the mozillateam PPA (had to configure APT pinning). But, just in case someday they stop updating the PPA too, I decided to learn to package my own Firefox. Turned out it was not that difficult!

Open Build Service

Open Build Service is provided by SUSE. It’s essentially a CI server for Linux packages. For me the advantages of using OBS are:

  1. OBS has all the package building environment set up, so I don’t have to set up my own.
  2. I don’t have to learn how to *build* the package. I can just download a source package from a PPA and upload it to OBS and it builds it for me.
  3. OBS provides package hosting as well. There’s no extra effort needed from me if I want to let somebody else use my package.

Technically I can also use PPA to achieve those as well, but OBS provides a possibility to also target other distributions. (It looks like a lot of work to configure that though.)

The source of my Firefox package can come from mozillateam PPA, Linux Mint, or PopOS.

OBS Concepts

OBS provides a User Guide, but I found that the openSUSE wiki gives a much better explanation around basic concepts in OBS.

Each project contains the resources needed to build one or more packages (i.e. RPMs/DEBs/etc.). These resources include source archives, patch files, spec files, etc. The output of a project is one or more repositories. A repository is an old familiar concept: simply a bunch of RPMs organized in a directory hierarchy along with some index/meta-data files that make it easy for tools like zypper to search and resolve dependencies.

  • Project: it’s also a namespace for configurations such as build constraints and build depedency preferences. Projects can have Subprojects, which is just an entirely separated project, only with similar names.
  • Repository: repositories can also be used as sources of other projects’ build dependency. The resulting package of a project’s build is also put into the project’s repository.

Each project is also essentially a version-controlled folder (in the folder are those resources mentioned above), managed by the osc commandline tool.

Importing an existing Debian package

I learned about the osc dput command from a talk at DebConf. But when running osc dput it complained “There is no sha256 sum for file _meta.". I worked around it by just downloading the source package files and running osc add on them.

Source package files contains:

  • <package_name_and_version>.dsc : the Debian source control file, which describes a package.
  • <package_name_and_version>.orig.tar.xz : archive file containing the original tarball (source code).
  • <package_name_and_version>.debian.tar.xz : archive file containing Debian build configurations, patches, changelogs, and so on.

After running osc add, run osc ci to commit and upload the changes.

Providing build dependency source repositories

Once uploaded, OBS will immediately start building the package. However it soon returned error saying that it couldn’t find some build dependencies. After rewatching the DebConf talk I realized I have to import the Ubuntu repositories.

After I manually added the update repo

At first, I simply clicked “Add from a Distribution". But OBS still complained that it couldn’t get new enough version of some build dependency. Then I realized that when adding Ubuntu:22.04, only universe is added but not update. A list of all repositories for Ubuntu 22.04 can be found here. I don’t know what universe-update is but I added it anyways.

OBS Project Configuration for Firefox

Two more things I had to change are the build constraints and prefer depedency settings.

Repotype: debian
Constraint: hardware:disk:size unit=G 30
Constraint: hardware:memory:size unit=G 8
Prefer: libncurses-dev
  • Constraint: constraints to the build worker machine. I took the information from here.
  • Prefer: when resolving build dependencies, when multiple packages fits the criteria, OBS doesn’t just randomly choose one out of them. You’re required to explicitly tell it which one to use. If I don’t specify, the build will show a warning message.
    • I checked apt policy libtinfo-dev on my machine and it shows that it’s only a transitional package. Therefore I selected the other option libncurses-dev.

All available project config options are here: https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.prjconfig.html

The build should be successful since we did not change anything from the original package. The build artifact (resulting package) will be available in the repository like at https://build.opensuse.org/repositories/home:pellaeon

Modifying the package

I will leave this for Part 2.

Firefox snap 依舊雷翻天,我不該浪費時間研究 snap 的,我錯了

把主力機升到 Ubuntu 22.04 之後,我這兩天繼續浪費時間搞 Firefox snap,目前 Firefox snap 主要的問題是:

  1. native messaging 不會動,導致 Mailvelope 這類的插件沒辦法和 snap 環境外的 gpg 程式溝通,不過官方好像已經有解法在 beta 了
  2. 我平常使用 Firefox 會分好幾個 profile ,不同 profile 過不同的 network namespace ,然後在 network namespace 裡面設定 VPN,整體使用 vopono https://github.com/jamesmcm/vopono 設定起來非常方便愉快,但是!snap 自己也有用自己的 network namespace 的樣子,所以只要在自己的 network namespace 裡面就無法使用 snap,我已經解掉很多問題成功把 firefox snap 跑起來了,但是最後還是卡在 snap 無法自訂 /etc/resolv.conf ,導致 firefox 無法使用 VPN 的 DNS server ,功敗垂成
  3. 即使界面語言選擇正體中文,仍然顯示英文給你看,這我就懶得修了,我就看英文吧,但這種低級的 bug 居然可以過 Mozilla 和 Canonical 的品管??

我跑去問了問題但還是沒有人回。

所以是時候拋棄 snap ,改用回傳統套件包的 firefox 了。

關於我 debug 在 netns 裡面跑 snap 的過程,我也有在論壇上記錄

套件包

我先試了 Linux Mint 的來源,好不容易把 APT pinning 搞定之後,要裝的時候發現 Linux Mint 的版本依賴 Mint 的 ubuntu-system-adjustments 套件,會做一些 Mint 自己的系統變更(比如說設定拒絕安裝 snapd,還有 grub 訊息改成 Linux Mint 之類的)。

所以,還是得要回去用 mozillateam 的 PPA

套件包 GPG 簽章設定

為了增加 Mint 的 APT 來源到我的系統,也要增加 Mint 的 GPG 公鑰,為此順便也學習了 apt-key 被棄用的原因。

簡單來說,原本的 apt-key 的設計是,匯入了一個簽章之後,這個簽章就是全域受信任的,不管套件的來源是哪一個 APT source ,只要有任何一個受信任公鑰的簽名,該套件就會被視為受信任的。這有一個嚴重的問題是,通常不同 APT 來源的發行者是不同人,自然也會擁有不同的簽章,但上面那個設計會導致任何一個受信任簽章持有人都可以發行一個和系統內某個套件同名的套件來蓋掉它

因此現在才改成將 GPG 公鑰寫入 /usr/share/keyrings ,然後額外在每個 APT source 裡面標注該來源使用的簽章:

deb [arch=amd64 signed-by=/usr/share/keyrings/signal-desktop-keyring.gpg] https://updates.signal.org/desktop/apt xenial main

Snap

垃圾爛系統,毫無客製選項和自由度,唯一會正常工作的情境是開發者想像中的一般使用者的環境,其餘情境下慘不忍睹。我不敢說預設設定的環境不常見,但會用 Linux 的人就是為了可以大幅度客製化自己的環境啊,所以有客製化環境的人所佔的比例肯定比 windows 和 macos 高很多。

Snap 這設計邏輯,堪稱微軟。看看微軟要不要趕快挖角一下 Snap 這垃圾,把它領走好不好,整個子系統都給微軟維護最適合不過了。

自己編譯 Firefox ?

過陣子再來自己研究看看…在 Reddit 上面看到有日本人之前自己維護的優化版 Firefox,似乎可以來自己研究一下怎麼修改編譯選項,改成一些適合我處理器的選項。gcc znver3 選項平均可以提升程式 11% 的效能呢

參考