|
Poster:
|
amnion |
Date:
|
July 29, 2012 12:25:37pm |
|
Forum:
|
web
|
Subject:
|
Archive appears sometimes, other days it won't |
I've gone to an old website of mine several times and sometimes it works just fine, but other times it won't pull up anything saying there's a robot.txt file. I know for a fact that I never had one on any of my sites. Why does the archive work sometimes but not others?
How would there be a robot.txt file on the archived site now but there wasn't before?
|
Poster:
|
amnion |
Date:
|
August 09, 2012 03:54:12pm |
|
Forum:
|
web
|
Subject:
|
Re: Archive appears sometimes, other days it won't |
Wouldn't that only apply to the current versions of that site? What about the versions of the site where there were no robots.txt in place? The website has already been crawled, when it was okay to crawl it and archive it. The robots.txt just says "do not crawl this website". That would mean it can't be archived right now, at this moment. It doesn't seem fair that a new domain owner can put out a decree stating that this website, and all versions of this website from the past (which I did not own) are mine to control and must not be archived.
I am only trying to find the versions that have already been archived, which I know exist, because I've seen them.
TVNews
OpenLibrary
Jeff Kaplan