Brian Dennis recently looked at web crawlers.
Some great links to papers and ideas, especially the idea of a personal crawler, something which would live on your desktop and crawl things you were interested in. Tie something like that into filtering and recommendations and you could have something very cool.
Posted by Josh at December 12, 2006 05:55 AM