Robots.txt and the Wayback Machine...

[Duke87 1,070]

Now, I know that nothing from the past couple of years is archived on the Wayback Machine because the site started blocking bots from accessing it....

...but now it would seem as though the older pages which were archived have retroactively been blocked out:

"robots.txt", eh? Well, such a file does indeed exist here. It says the following as of this posting:

User-agent: Googlebot

Disallow:

User-agent: googlebot-image

Disallow:

User-agent: googlebot-mobile

Disallow:

User-agent: *

Disallow: / quote>

That last item with the asterisk (which, if I understand correctly, serves to disallow anything and everything) seems to be the sticky point here. Wayback probably modified their policy such that that not only disallows their bots from making new archives but now also denies access to the old ones.

Was anyone aware of this? Is there anything that can be done about it?

[Korot 6]

Kindly ask Dirk to add a line called:

User-Agent: Wayback Machine

Allow:

Regards,

Korot

[Duke87 1,070]

Originally posted by: Korot

Kindly ask Dirk to add a line called:

User-Agent: Wayback Machine

Allow:quote>

According to them, the agent is called "ia_archiver", not "Wayback Machnie".

[un1 40]

Does that block the Google bots that take caches of the pages? (in case of a hack.... again)

So, I checked it out, seems like it isn't blocked and the caches were fairly new (as in taken 2 minutes ago).

[simmaster07 1,220]

Originally posted by: Korot

User-Agent: Wayback Machine

Allow:quote>

Should be

User-Agent: ia_archiver

Allow: /

quote>

Can't forget to include the root directory or the robots.txt file is incorrect.