is robots.txt dead? lets ask dropbox

First off I’d like to give creds to Francis Brown and Rob Ragan who presented their talk Tenacious Diggity at Defcon20¬†where I found out about the apparent steam-rolling of Dropbox’s robots.txt file. For as far back as I can remember, the robots.txt file has been a ban-list of places that search engine crawlers are supposed to ignore when crawling a site. Recently however there is some talk that the preferred way of disallowing crawlers is to control them using alternative methods such as metatags and javascript.

It may appear however, that Google has already decided that robots.txt is merely a bug heading towards it’s windscreen and is indexing pages that are excluded via the robots.txt as is apparent with Dropbox.