CheckRSS

CheckRSS

The CheckRSS fetcher

If your server logs show a request from CheckRSS, it came from one of three things: someone entered an address on your site into our feed checker, a page displayed a status badge for your feed, or our feed directory rechecked a feed it lists.

How to recognise it

User-Agent: CheckRSS/1.0 (+https://checkrss.com/bot/)

Requests come from Cloudflare's network, where the service runs. The directory run is the only part that happens without a person waiting, and it runs from our own machines.

What it does

  • A feed check is one request for the feed. If the address is a web page, the page is read to find its feed, and when none is announced a handful of common feed addresses (such as /feed and /rss.xml) are tried.
  • A podcast check also requests the cover image and the first two bytes of the newest audio file, to see how the server answers. The audio itself is not downloaded.
  • A status badge requests the feed at most about once an hour and reads only its beginning.
  • A directory recheck reads robots.txt, then your home page, then the feeds it finds: at most eight requests for your site in a run, one at a time, with a pause between them. It does not crawl beyond that. The list of sites it works through is edited by hand, not discovered by following links.

What we keep

A feed check keeps nothing. A directory recheck records what the feed said about itself so the page can show it: the feed address, its format, how many items it held, the date of the newest one, and the titles and links of the newest handful of items. The titles link straight back to you and carry rel="nofollow ugc". We do not copy article text, we do not host or proxy the feed, and your readers fetch it from you.

If you would rather not be listed, write to [email protected] and the site comes off the list.

Limits it keeps

  • Only GET requests, only on ports 80 and 443, never with credentials or cookies.
  • A timeout of 12 seconds on a check, 15 on a directory run, and a size cap on every download.
  • At most six redirects are followed.
  • A 403, a 429 or a bot wall ends the directory run for that site. Nothing here retries behind a block or pretends to be a browser.

How to block it

The directory run reads robots.txt and obeys it, including Crawl-delay. A rule for CheckRSS or for * is enough to keep it away:

User-agent: CheckRSS
Disallow: /

A check someone asks for by hand does not read robots.txt, because it is one request for one address on behalf of a person, in the way a browser or a feed reader would do it. To refuse that as well, block the user agent CheckRSS in your server or firewall. The checker will then report your feed as blocked to whoever asks.

If the fetcher misbehaves, write to [email protected] with a line from your log and we will look into it.