User-agent: *
Disallow: /cgi-bin/ # CGI, no spiders please

User-agent: ia_archiver
Disallow: /

Sitemap: http://directory.nycfilm.com/sitemap.xml.gz