Bibliogram has a feature to generate RSS / Atom feeds and I’m using it a lot. However, there are a few limitations, which also applies to RSSHub.
First, Bibliogram on your IP address could be blocked by Instagram. This happens so often that a big part of discussion around Bibliogram is centered around this. I’ve tried a few solutions to this:
- Flowing the traffic through VPNs: doesn’t work, Instagram/FB has required login on all server IP sections by default.
- Flowing the traffic through “residential proxies": most residential proxies are not really ethical because they rely on embedding exit nodes to third-party applications. Application developers may integrate with their SDKs and the users of the application will be used as exit nodes, most often without clearly communicating the risks and benefits to them.
- Flowing the traffic through Cloudflare workers: same as VPNs, server IP ranges are mostly blocked. There’s also an added issue that each request from Cloudflare workers will use randomly different outbound IPs, and it’s not possible to bind to a single one.
- Flowing the traffic through mobile internet IPs: most mobile internet providers utilize carrier-grade NATs (CGNAT) so that tons of devices will use the same public IP. To avoid collateral damage of banning a ton of devices, IG/FB is very tolerant with the mobile IPs. Barely any rate limiting at all. However mobile IPs are expensive (compared to other solutions), to save data quota, I had to set up Squid proxy to route only the most critical requests (the main HTML file or some API calls) through the mobile internet while keeping the others (CDN requests) using an existing link.
- Flowing the traffic through the dynamic IP address provided by my home ISP: this is the solution that I settled on. I can configure the modem to re-dial PPPoE periodically to obtain different IPs. See Multihome for Docker containers for my PPPoE and docker setup.
However, despite all of this, request quotas towards FB/IG is still a scarce resource. So that’s why I thought: once I crawled a page and generated a RSS feed, how can I benefit others safely?
Second, when only using a simple reverse proxy, every request is handed directly over to Bibliogram, without any caching. So that means every update pull from the RSS reader will trigger a new crawl, consuming precious quota. So I realized it was necessary to set up caching on the reverse proxy.
Third, the images in the RSS feed entries are originally served by FB/IG’s CDNs. The URL looks like:
https://scontent-tpe1-1.cdninstagram.com/v/t51.2885-15/311989025_797227311577987_2533783926321053094_n.jpg?stp=dst-jpg_e35_s1080x1080&_nc_ht=scontent-tpe1-1.cdninstagram.com&_nc_cat=107&_nc_ohc=q7iUiYhKInsAX-YuXi3&edm=AOQ1c0wBAAAA&ccb=7-5&oh=00_AT-vwyBEJjJ70fuXIuI06Vmjdmay2jfzx6o6SXvJFQzAdg&oe=63597DE5&_nc_sid=8fd12b
In the GET parameters contains a access key which is only valid for around a week (according to my testing). After which the CDN will no longer serve the file. The file is still there but you have to crawl the original post source again to get a new access key. However, once the RSS feed entry is published, the reader usually doesn’t update the entry. Even if it does, crawling the same post still eats up precious request quota.
The best solution I can currently think of is to cache the image with Nginx as well. It will take up disk spaces but the images are already heavily compressed, so usually just a few hundred kilobytes a piece. Disk spaces also isn’t expensive nowadays. For RSSHub which just point the image src to CDNs, I’ll have to modify the code and make it point to Nginx. For Bibliogram, it already rewrites the image src to /imageproxy/Cj_XXXX.jpg, which can be easily cached by Nginx.
Enter Nginx proxy_store. Nginx proxy_store basically just saves a copy of the response on disk, with path equivalent to the request URI. It’s usually used to set up mirrors, of which objects usually don’t expire.
However it does not offer expire functionality, so objects will have to be deleted from the cache manually.
Nginx geo: differentiating client IPs
The geo directive sets a nginx config variable to different values depending on the client IP address. This is useful for us to:
- Give trusted IPs access to the crawler (Bibliogram or RSSHub) and the cache.
- Give untrusted IPs access to only the cache.
This way, untrusted IPs can’t make our crawlers crawl, but they can still use the contents the crawler has already generated.
Putting it all together
I marked the config sections “Section N" to explain the config flow.
geo $whitelist {
default static;#only allowed to cache
1.2.3.0/24 app;#allowed to invoke app
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name bibliogram.example.com;
access_log /var/log/nginx/bibliogram.access.log;
error_log /var/log/nginx/bibliogram.error.log;
root /var/www/html;
# https://docs.nginx.com/nginx/admin-guide/web-server/serving-static-content/
sendfile on;
tcp_nopush on;
tcp_nodelay on;
# https://serverfault.com/a/1033578/54651
# Section 1
location / {
root /var/www/bibliogram;
try_files $uri @${whitelist};
add_header X-Tier "tryfiles";
}
# Don't cache the user html page, otherwise the filename will conflict
# with the directory name /u/<username>/rss.xml
# Section 2
location ~ /u/[\w\._-]*$ {
root /var/www/bibliogram;
try_files $uri @${whitelist}_nocache;
add_header X-Tier "tryfiles_nocache";
}
# Section 3
location @static {
root /var/www/bibliogram;
try_files $uri =404;
}
# Section 4
location @static_nocache {
root /var/www/bibliogram;
try_files $uri =404;
}
# Section 5
location @app {
proxy_pass http://localhost:10407;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_store on;
proxy_store_access user:rw group:rw all:r;
proxy_temp_path /tmp/;
add_header X-Tier "app_cache";
root /var/www/bibliogram;
}
# Section 6
location @app_nocache {
proxy_pass http://localhost:10407;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
add_header X-Tier "app_nocache";
root /var/www/bibliogram;
}
ssl_certificate /etc/letsencrypt/live/bibliogram2/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/bibliogram2/privkey.pem; # managed by Certbot
}
This settings enable to following behavior:
- For trusted client IPs, the
$whitelist variable will be set to app
- Section 1: if the URI is already cached, serve from the cache, if not go to the named location
@$whitelist, which expands to @app (Section 5).
- Section 5: pass the request to Bibliogram, save the response under
/var/www/bibliogram.
- For untrusted client IPs, the
$whitelist variable will be set to static
- Section 1: if the URI is already cached, serve from the cache, if not go to the named location
@$whitelist, which expands to @static (Section 3).
- Section 3: the same configuration as Section 1, which serves from the cache. If the URI is not present in cache, 404 is returned.
- This special case takes precedence over case 1 and 2. If request URI ends in
/u/<username>, which is the URI for HTML rendered pages of a user. The response should never be cached, because this page is only visited from browsers, and I want Bibliogram to crawl for the latest information for the user. This configuration also forbids untrusted IPs from accessing user HTML pages, since The the responses are never cached and the untrusted IPs can only access the cache. (Maybe I’ll change this in the future.)
Result
It turns out that, strangely, my RSS reader (Inoreader) will request a RSS feed file very frequently. Before setting up caching, this would make Bibliogram start to crawl Instagram every time, wasting the request quota. I guess this is because Inoreader uses the RSS lastBuildDate attribute to determine next crawl time. And since Bibliogram generates a new feed file every time Inoreader requests it, Inoreader thinks that the feed is updated very often, so tries to request the feed even more often. However in all of these times the pubDate of the newest post is actually what matters.
Next steps
With the cache in place, I can see from the access logs that Inoreader is starting to get a lot of 304 Not Modified responses from Nginx. This would prevent Bibliogram from having to crawl too often.
With the cache in place, I can also decide on my own when to update the feed files, by deleting the one I want to expire in the cache. Currently this process is all manual, but I should define a cron job to delete the feed files more than 2 days old, for example.
An even better solution would be to decide when to expire the feed by looking at the feed’s post interval.
Future enhancement: rewriting to dedicated image hosts
Cloudflare workers can be used to turn cloud drives into static file hosts using ondrive-vercel-index. In the future if the cache directory became too large to serve locally, maybe we can redirect the cache request to the domain served by onedrive-vercel-index. Then we can move some older cache objects to the larger cloud drive.
References
References:
Dead-ends:
噴火獸號：裴列恩之艦 