FavoritePosts.com
(last updated February 10, 2008)

I am Favibot, the super-friendly bot!

Favibot is our web crawling robot and RSS aggregator. Here at FavoritePosts, we are working on cutting-edge technologies that will empower our upcoming spam-free content consumption and discovery service. That is why we are keeping track of sites like yours. The reason Favibot comes back to your site and/or fetches your RSS feed regularly is because we believe know you produce quality content. We applaud your efforts, and will do our very best to send new readers your way (plus a bunch of other ultra-secret stuff we can't reveal yet!). Our innovative service has been in the works for a few months now, and we can't wait to show off what we have accomplished!

What is the crawler's HTTP user-agent string?

Favibot/1.0 (+http://www.favoriteposts.com/crawler.html; crawler@favoriteposts.com; X subscribers)   ← now reporting subscription counts!

How can I prevent Favibot from crawling my site?

The The Robot Exclusion Standard is a method that allows Web site administrators to indicate to visiting robots which parts of their site should not be visited by the robot. We will always honor your /robots.txt file. If you do not wish us to visit your site, so be it :)

About those RSS fetches... Hmm. Can I just ping you directly?

We will soon accept XML-RPC pings. Until then, we hope you don't mind that we access your RSS feed regularly (the frequency depends on the popularity of your blog and careful analysis of your posting history). In essense, having us fetch your RSS feed is no different than a single reader who leaves his/her RSS reader open all day :) Please don't get upset. We know this is not the ideal way to do it, and we will ask bloggers to ping us directly in the near future.

Do you implement HTTP's Conditional GET mechanism? Compression?

Yes. Our crawler takes advantage of the ETag and Last-Modified headers to reduce bandwidth consumption by only downloading feeds that have changed. We fully support gzip content encoding.

Are you hiring?

Maybe. Are you a Python hacker? Do you do semantic analysis in your spare time? Ontologies/taxonomies excite you? Expert in data extraction and harvesting? Topic detection / segmentation skills? We're always looking to expand our team. Email us at jobs at favoriteposts.com. Thanks!

Feel free to contact us with any questions or concerns you may have: crawler@favoriteposts.com