Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
benyami
on May 18, 2015
|
parent
|
context
|
favorite
| on:
What one may find in robots.txt
Check out the Internet Archive FAQ on how to remove a document from their archives.
https://archive.org/about/exclude.php
It looks like they used robots.txt to do that.
neil_s
on May 18, 2015
[–]
Huh, so the wild-card user-agent will block not just searchbots, but also archivebots. Wonder how OP managed to get screenshots of archive.org having archives available for those documents.
Join us for
AI Startup School
this June 16-17 in San Francisco!
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
It looks like they used robots.txt to do that.