In late 2000 at the tail end of the bubble bursting, there was a search engine company trying to build on a platform of push notifications instead of crawling.
I know that at my current company the plurality if traffic comes from crawlers. We don’t want to throttle them because that’s biting the hand that feeds but it sure sucks.
I wonder often, how many crawlers you need before it’s cheaper for a website to volunteer up when pages change or new ones arrive.
There are N existing standards for websites to notify crawlers about new content. Few websites use them, and the ones that do are often buggy. A few are even malicious.
I know that at my current company the plurality if traffic comes from crawlers. We don’t want to throttle them because that’s biting the hand that feeds but it sure sucks.
I wonder often, how many crawlers you need before it’s cheaper for a website to volunteer up when pages change or new ones arrive.