Their explanation for ignoring robots makes sense - they say they ignore it because their crawler only runs when a human enters a URL and archives it, they also link to Google as this is what they do.