Hacker News new | past | comments | ask | show | jobs | submit login

I built a search engine that was able to crawl Amazon and Yelp. The toughest sites were reddit and facebook.



at scale? millions of pages a week? And now? I wrote a crawler that could crawl Amazon as early as a year ago too, but now it doesn't work.


And google sucks at those too.




Consider applying for YC's Summer 2025 batch! Applications are open till May 13

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: