Maybe to keep someone from registering it for malicious purposes? At one point, archive.org would follow robots.txt retroactively. So if you lost control of the domain, someone else could wipe all of its history.