As you may know, the is a Betterbird subreddit at https://www.reddit.com/r/Betterbird/. Reddit offers to deliver content via RSS feeds by simply appending .rss to the end of a URL. So we're subscribed to two feeds:
https://www.reddit.com/r/Betterbird/.rss
https://www.reddit.com/r/Betterbird/comments/.rss
not because we love it so much, but to monitor support queries coming in via the channel.
A while ago we noticed that those Reddit feeds were in error, see picture above. So we went to debug this. Reddit was in fact replying to the request to fetch the feed with a 403 error "Forbidden". First we compared the behaviour with Thunderbird on a different profile, which actually worked. Goodness gracious, Betterbird wasn't better than Thunderbird? We couldn't rest until the issue was solved.
Carefully analysing the code changes in the Feeds code, we didn't see anything that would explain the different behaviour. So we went in with Mozilla's very helpful error console and looked at the requests. Here they are:
GET /r/Betterbird/comments/.rss HTTP/2
Host: www.reddit.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:140.0) Gecko/20100101 Thunderbird/140.6.0
Accept: application/atom+xml,application/rss+xml;q=0.9,application/feed+json;q=0.9,application/rdf+xml;q=0.8,application/xml;q=0.7,text/xml;q=0.7,application/json;q=0.7,*/*;q=0.1
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
If-Modified-Since: Sat, 01 Jan 2000 00:00:00 GMT
DNT: 1
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
On our test with Thunderbird, we noticed that it sent:
GET /r/Betterbird/comments/.rss HTTP/2
Host: www.reddit.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:140.0) Gecko/20100101 Thunderbird/140.5.0
Accept: application/atom+xml,application/rss+xml;q=0.9,application/rdf+xml;q=0.8,application/xml;q=0.7,text/xml;q=0.7,*/*;q=0.1
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
If-Modified-Since: Sat, 01 Jan 2000 00:00:00 GMT
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Spot the difference? Yes, Betterbird also accepts application/feed+json and application/json since it supports JSON feeds, but more importantly the DNT: 1 header. That's the "do not track" header which gets sent when preference privacy.donottrackheader.enabled is enabled. And lo and behold, that preference was set in the profile that showed the error, but was not set in the profile used for testing with Thunderbird.
More information on Mozilla's "Do Not Track" an be found here.
So what's the take-away? Well, If something doesn't work, try it on a new profile first. You many have set a preference years ago and forgotten about it, and suddenly this preference becomes the determining factor of a new malfunction. But why suddenly a new malfunction? This is what AI had to say:
Reddit applies heuristic request filtering to unauthenticated RSS feed access. Depending on request headers, such as the presence of the DNT (Do Not Track) header, otherwise valid feed requests may be classified as automated traffic and rejected with HTTP 403. This appears to be part of Reddit’s general effort to limit uncontrolled scraping by non-browser clients, while still allowing access from regular browsers and well-known search engine crawlers.