That's a functionally meaningless distinction. If you setup a web server that responds to requests, then you're choosing to make content available because your server can choose to not respond to requests. The entire protocol includes mechanisms to negotiate access.
And yet it is legal to produce and redistribute summaries as sufficiently transformative derivative works, and this has been court tested[1]. Of course in Australia we passed rather specific laws to the contrary, because lo and behold Rupert Murdoch wanted money and gosh darn it our government was going to give it to him[2].
This is a meaningless simplification. In this framework "robots.txt" has no role, because your server "can choose" not to respond. Heck, even DDOS is fine, because "protocol"
