Hacker News new | past | comments | ask | show | jobs | submit login

That's basically asking to close the market in favor of the current actors.

New actors have the right to emerge.




They have the right to try to convince me to let them scrape me. Most of the time they're thinly veiled data traders. I haven't seen any new company try to scrape my stuff since maybe Kagi.

Kagi is welcome to scrape from their IP addresses. Other bots that behave are fine too (Huawei and various other Chinese bots don't and I've had to put an IP block on those).


No they don't.

There's no rule that you have to let anyone in who claims to be a web crawler.


So who decides that you can be one? Right now it's Cloudflare, a litteral monopoly...

The truth is that I sympathize with the people trying to use mobile connections to bypass such a cartel.

What Cloudflare is doing now is worse than the web crawlers themselves and the legality of blocking crawlers with a monopoly is dubious at best.


which is why they will stop claiming to be one.


so what happened to competition fostering a better outcome for all then?




Consider applying for YC's Summer 2025 batch! Applications are open till May 13

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: