Skip to content
Line of Sites

LineofsitesBot

The crawler behind this bulletin.

It reads the RSS and Atom feeds listed on the sources page, about every 30 minutes, and takes the headline, link, date, a short excerpt and the picture address from the feed. When a feed gives no picture it fetches the article once to read the share image its page declares. It does not keep any article text from that page.

It respects robots.txt, including Crawl-delay, and identifies itself as LineofsitesBot in its user agent. To stop it reading your site, disallow it in robots.txt and it will stop within the hour.