Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
ohgodplsno
on Aug 7, 2023
|
parent
|
context
|
favorite
| on:
GPTBot – OpenAI’s Web Crawler
Like every time you put content on the internet: you depend on their good will to respect these tags, or robots.txt. OpenAI can decide to ignore it. It's wishful thinking.
copperx
on Aug 7, 2023
[–]
The next version of GPT might have better citations, and they could just refuse to cite things they were not allowed to crawl.
However, it's trivial to know whether the bot crawled your site or stopped at robots.txt.
Join us for
AI Startup School
this June 16-17 in San Francisco!
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
