By Their Robots.txt Ye Shall Know Them?

by Emily Gertz · 2009-01-20 17:04:00 UTC
Topics:

Is this a sign (a geeky one) of a more transparent White House?

An interesting catch by Jason Kottke: Most every web site has a file on it called "robots.txt." Search engines are guided by this file on what to include in their indexes when they spider the site, and what to leave out.

Here's the robots.txt file from whitehouse.gov on Jan 19:

User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /omb/search
Disallow: /omb/query.html
Disallow: /expectmore/search
Disallow: /expectmore/query.html
Disallow: /results/search
Disallow: /results/query.html
Disallow: /earmarks/search
Disallow: /earmarks/query.html
Disallow: /help
Disallow: /360pics/text
Disallow: /911/911day/text
Disallow: /911/heroes/text

As Jason Kottke notes, "And it goes on like that for almost 2400 lines!" Click here to see the entire thing.

Here's the entire robots.txt file on whitehouse.gov today:

User-agent: *
Disallow: /includes/

Emily Gertz
PREVIOUS STORY:
Obama's Call to Action on Energy, Global Warming
NEXT STORY:
Stopping the Water Grab in Nevada

COMMENTS (7)

    Comment Policy

    · All fields are required to comment.

    [X]

    Comments on Change.org are meant for further exploration and evaluation of the campaign on Change.org. To that end, we welcome constructive comments. However, we reserve the right to delete comments which, as determined solely in our discretion: (1) are offensive, abusive, or off-topic; (2) include content solely intended to personally attack the campaign creator, (3) are designed to subvert or hijack comment threads rather than contribute to them; and/or (4) violate our terms of service and/or privacy policy. Repeat offenders may be permanently removed from the site at our discretion. Please also be advised that: (A) we do not actively curate and/or monitor in any manner whatsoever the comments made on the Change.org platform, and (B) the creator of each campaign on Change.org may remove any comment at her/his/its discretion.