<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
		>
<channel>
	<title>Comments on: GoDaddy &amp; Phantom Robots.txt Files</title>
	<atom:link href="http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/</link>
	<description>Search engine optimization, search marketing strategies, and web solutions company. Our efforts also include linkbait, niche content, articles, social networking, blogoshpere networking, application development, web development, site audits, and usability testing.</description>
	<lastBuildDate>Thu, 14 Jan 2010 05:01:10 -0700</lastBuildDate>
	<generator>http://wordpress.org/?v=2.8.5</generator>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<xhtml:meta xmlns:xhtml="http://www.w3.org/1999/xhtml" name="robots" content="noindex" />
	<item>
		<title>By: Sarah</title>
		<link>http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/comment-page-1/#comment-2911</link>
		<dc:creator>Sarah</dc:creator>
		<pubDate>Mon, 29 Sep 2008 09:18:32 +0000</pubDate>
		<guid isPermaLink="false">http://www.squareoak.com/blog/?p=137#comment-2911</guid>
		<description>I went through this thread and it was a life saver. I was trying to figure out how someone could see my robots.txt file but I couldn&#039;t find it anywhere in my wordpress installation...I host with godaddy as well. Something fishy was definitely going on....but all I did was upload a new robots.txt file to my root installation for wordpress and made it so it will allow everything. I went back to Google webmaster tools and I could upload my sitemap with no problem. Thanks guys.</description>
		<content:encoded><![CDATA[<p>I went through this thread and it was a life saver. I was trying to figure out how someone could see my robots.txt file but I couldn&#8217;t find it anywhere in my wordpress installation&#8230;I host with godaddy as well. Something fishy was definitely going on&#8230;.but all I did was upload a new robots.txt file to my root installation for wordpress and made it so it will allow everything. I went back to Google webmaster tools and I could upload my sitemap with no problem. Thanks guys.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Ryan</title>
		<link>http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/comment-page-1/#comment-2894</link>
		<dc:creator>Ryan</dc:creator>
		<pubDate>Wed, 13 Aug 2008 21:44:31 +0000</pubDate>
		<guid isPermaLink="false">http://www.squareoak.com/blog/?p=137#comment-2894</guid>
		<description>Just checking to see if you have made any head-way for your robots.txt issue yet. I am going through the same issue and have a godaddy host with a wordpress blog. I have been back and forth to all of them and haven&#039;t been able to fix the problem. I have follow up with every bit of advice/information from everyone on this thread but still have not made any progress.</description>
		<content:encoded><![CDATA[<p>Just checking to see if you have made any head-way for your robots.txt issue yet. I am going through the same issue and have a godaddy host with a wordpress blog. I have been back and forth to all of them and haven&#8217;t been able to fix the problem. I have follow up with every bit of advice/information from everyone on this thread but still have not made any progress.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Brendan Picha</title>
		<link>http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/comment-page-1/#comment-2857</link>
		<dc:creator>Brendan Picha</dc:creator>
		<pubDate>Sat, 21 Jun 2008 02:44:54 +0000</pubDate>
		<guid isPermaLink="false">http://www.squareoak.com/blog/?p=137#comment-2857</guid>
		<description>@Otto Yes I&#039;m well aware that Wordpress does this. I don&#039;t think this is what was happening and I haven&#039;t seen any further changes in my robots.txt file. When you&#039;ve had a site for a given amount of time and all settings/plugins stay the same and then one day something like the robots.txt file mysteriously changes, one has to wonder.</description>
		<content:encoded><![CDATA[<p>@Otto Yes I&#8217;m well aware that Wordpress does this. I don&#8217;t think this is what was happening and I haven&#8217;t seen any further changes in my robots.txt file. When you&#8217;ve had a site for a given amount of time and all settings/plugins stay the same and then one day something like the robots.txt file mysteriously changes, one has to wonder.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Otto</title>
		<link>http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/comment-page-1/#comment-2856</link>
		<dc:creator>Otto</dc:creator>
		<pubDate>Fri, 20 Jun 2008 19:39:01 +0000</pubDate>
		<guid isPermaLink="false">http://www.squareoak.com/blog/?p=137#comment-2856</guid>
		<description>I&#039;m Otto, a moderator on the WordPress.org support forums.

The latest versions of WordPress DO generate robots.txt files via rewrites, if there is no robots.txt file already present in the main WordPress directory.

If you go to the Settings-&gt;Privacy menu option in WordPress, then you&#039;ll see an option there that changes WordPress to block or allow search engines to index the site. If you switch this, then you&#039;ll notice the &quot;pretend&quot; robots.txt file changes.

So yes, WordPress is absolutely generating this fake file. GoDaddy hosting is NOT at fault here.</description>
		<content:encoded><![CDATA[<p>I&#8217;m Otto, a moderator on the WordPress.org support forums.</p>
<p>The latest versions of WordPress DO generate robots.txt files via rewrites, if there is no robots.txt file already present in the main WordPress directory.</p>
<p>If you go to the Settings-&gt;Privacy menu option in WordPress, then you&#8217;ll see an option there that changes WordPress to block or allow search engines to index the site. If you switch this, then you&#8217;ll notice the &#8220;pretend&#8221; robots.txt file changes.</p>
<p>So yes, WordPress is absolutely generating this fake file. GoDaddy hosting is NOT at fault here.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Brendan Picha</title>
		<link>http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/comment-page-1/#comment-2854</link>
		<dc:creator>Brendan Picha</dc:creator>
		<pubDate>Fri, 20 Jun 2008 02:00:24 +0000</pubDate>
		<guid isPermaLink="false">http://www.squareoak.com/blog/?p=137#comment-2854</guid>
		<description>@sapphirecat Yeah, just added the &#039;email updated comments&#039; plugin :). In any event, the content of the phantom robots.txt file was mentioned before but I&#039;ll mention it again:

Godaddy&#039;s version:
User-agent: *
Disallow:

My Original version:
User-agent: *
Disallow: /blog/wp-login</description>
		<content:encoded><![CDATA[<p>@sapphirecat Yeah, just added the &#8216;email updated comments&#8217; plugin <img src='http://www.squareoak.com/blog/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /> . In any event, the content of the phantom robots.txt file was mentioned before but I&#8217;ll mention it again:</p>
<p>Godaddy&#8217;s version:<br />
User-agent: *<br />
Disallow:</p>
<p>My Original version:<br />
User-agent: *<br />
Disallow: /blog/wp-login</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: sapphirecat</title>
		<link>http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/comment-page-1/#comment-2852</link>
		<dc:creator>sapphirecat</dc:creator>
		<pubDate>Thu, 19 Jun 2008 20:45:17 +0000</pubDate>
		<guid isPermaLink="false">http://www.squareoak.com/blog/?p=137#comment-2852</guid>
		<description>Hrm. If the tiny little &quot;Update&quot; paragraph was there when I commented before, I missed it. In any event, since nothing was ever said about what the content of the phantom robots.txt file was, it wasn&#039;t obvious that you were talking about some other content than what I was seeing when I commented. Otherwise it would have been a lot more obvious that something had changed, and I wouldn&#039;t have wasted our time.</description>
		<content:encoded><![CDATA[<p>Hrm. If the tiny little &#8220;Update&#8221; paragraph was there when I commented before, I missed it. In any event, since nothing was ever said about what the content of the phantom robots.txt file was, it wasn&#8217;t obvious that you were talking about some other content than what I was seeing when I commented. Otherwise it would have been a lot more obvious that something had changed, and I wouldn&#8217;t have wasted our time.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Brendan Picha</title>
		<link>http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/comment-page-1/#comment-2846</link>
		<dc:creator>Brendan Picha</dc:creator>
		<pubDate>Fri, 13 Jun 2008 15:26:02 +0000</pubDate>
		<guid isPermaLink="false">http://www.squareoak.com/blog/?p=137#comment-2846</guid>
		<description>@Dak, Good info here. Thank you for taking the time to write this but please see here: http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/#comment-2821</description>
		<content:encoded><![CDATA[<p>@Dak, Good info here. Thank you for taking the time to write this but please see here: <a href="http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/#comment-2821" rel="nofollow">http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/#comment-2821</a></p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Brendan Picha</title>
		<link>http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/comment-page-1/#comment-2845</link>
		<dc:creator>Brendan Picha</dc:creator>
		<pubDate>Fri, 13 Jun 2008 15:22:11 +0000</pubDate>
		<guid isPermaLink="false">http://www.squareoak.com/blog/?p=137#comment-2845</guid>
		<description>@Bill, The robots.txt file that you see now was not the file that GoDaddy had in there before. The file that&#039;s there now is the file that&#039;s SUPPOSED to be there. Previously there was a robots.txt that wasn&#039;t mine, I hadn&#039;t uploaded it and I had no idea where it came from. I was then sent an article by a friend that mentioned GoDaddy uploading robots.txt files to customer accounts. Whether this is intentional or a mistake I just don&#039;t know. The theory you talk about isn&#039;t mine but that of the folks from Geek Daily, the article I reference in this post. I&#039;ve stated the evidence over and over again and find myself replying to comments like yours, where it&#039;s quite obvious that the article and previous comments haven&#039;t been thoroughly read through. Read what I&#039;ve posted above in the comments completely and please do tell us what you think afterwards.</description>
		<content:encoded><![CDATA[<p>@Bill, The robots.txt file that you see now was not the file that GoDaddy had in there before. The file that&#8217;s there now is the file that&#8217;s SUPPOSED to be there. Previously there was a robots.txt that wasn&#8217;t mine, I hadn&#8217;t uploaded it and I had no idea where it came from. I was then sent an article by a friend that mentioned GoDaddy uploading robots.txt files to customer accounts. Whether this is intentional or a mistake I just don&#8217;t know. The theory you talk about isn&#8217;t mine but that of the folks from Geek Daily, the article I reference in this post. I&#8217;ve stated the evidence over and over again and find myself replying to comments like yours, where it&#8217;s quite obvious that the article and previous comments haven&#8217;t been thoroughly read through. Read what I&#8217;ve posted above in the comments completely and please do tell us what you think afterwards.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: dak</title>
		<link>http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/comment-page-1/#comment-2844</link>
		<dc:creator>dak</dc:creator>
		<pubDate>Fri, 13 Jun 2008 09:17:09 +0000</pubDate>
		<guid isPermaLink="false">http://www.squareoak.com/blog/?p=137#comment-2844</guid>
		<description>@Brendan, There appears to be a rewrite rule that is trying to remove trailing slashes in a way that doesn&#039;t work. This is a minor misconfiguration and has very little impact, aside from being confusing. The rule would look something like &quot;RewriteRule (.*)/$ $1 [R]&quot; and should actually be &quot;RewriteRule (.*)/$ $1 [R,L]&quot;. It may be in your .htaccess file, or it could be part of GoDaddy&#039;s configuration.

What&#039;s happening is that the [R] flag is rewriting a URL like &quot;robots.txt/&quot; into an absolute URL &quot;http://www.squareoak.com/robots.txt&quot;. It also changes the requested filename from something like &quot;/path/to/robots.txt&quot; into &quot;http://www.squareoak.com/path/to/robots.txt&quot;. It does this to signal to Apache to send an external redirect, instead of just silently rewriting the URLs. The problem is, without the [L] flag, we continue evaluating other rewrite rules after we&#039;ve done this conversion. If those rules rely on the requested file being right, you&#039;re going to get weird behavior.

One of the rewrite rules we evaluate after doing this trailing slash removal bit is WordPress&#039;s robots.txt rule. It basically says, &quot;If the requested file does not exist and the requested URL is for robots.txt, act like the user requested index.php?robots=1&quot;. Well, since the [R] has changed the requested filename to be invalid, but the URL is still for robots.txt, this rule gets applied.

How does this explain what you&#039;re seeing? I think you may have initially gone to http://www.squareoak.com/robots.txt/ (with the trailing slash). The incorrect rewrite rule basically caused you to see WordPress&#039;s dynamically-generated robots.txt. You noticed this wasn&#039;t your robots.txt file. You went in and deleted your robots.txt file in an attempt to verify that there was a &quot;phantom&quot; robots.txt file being served by GoDaddy. But when you did this, going to http://www.squareoak.com/robots.txt (with or without the trailing slash) caused WordPress to serve its index.php?robots=1. So you think you&#039;ve confirmed that there&#039;s a phantom robots.txt. You come back later, you put your robots.txt file back, you check http://www.squareoak.com/robots.txt (without the trailing slash), see it&#039;s working now, and think, &quot;Ah, GoDaddy fixed the problem.&quot;

The bright side is, GoogleBot (and every other bot) is requesting &quot;robots.txt&quot; without a trailing slash, so the right file is being served.</description>
		<content:encoded><![CDATA[<p>@Brendan, There appears to be a rewrite rule that is trying to remove trailing slashes in a way that doesn&#8217;t work. This is a minor misconfiguration and has very little impact, aside from being confusing. The rule would look something like &#8220;RewriteRule (.*)/$ $1 [R]&#8221; and should actually be &#8220;RewriteRule (.*)/$ $1 [R,L]&#8220;. It may be in your .htaccess file, or it could be part of GoDaddy&#8217;s configuration.</p>
<p>What&#8217;s happening is that the [R] flag is rewriting a URL like &#8220;robots.txt/&#8221; into an absolute URL &#8220;http://www.squareoak.com/robots.txt&#8221;. It also changes the requested filename from something like &#8220;/path/to/robots.txt&#8221; into &#8220;http://www.squareoak.com/path/to/robots.txt&#8221;. It does this to signal to Apache to send an external redirect, instead of just silently rewriting the URLs. The problem is, without the [L] flag, we continue evaluating other rewrite rules after we&#8217;ve done this conversion. If those rules rely on the requested file being right, you&#8217;re going to get weird behavior.</p>
<p>One of the rewrite rules we evaluate after doing this trailing slash removal bit is WordPress&#8217;s robots.txt rule. It basically says, &#8220;If the requested file does not exist and the requested URL is for robots.txt, act like the user requested index.php?robots=1&#8243;. Well, since the [R] has changed the requested filename to be invalid, but the URL is still for robots.txt, this rule gets applied.</p>
<p>How does this explain what you&#8217;re seeing? I think you may have initially gone to <a href="http://www.squareoak.com/robots.txt/" rel="nofollow">http://www.squareoak.com/robots.txt/</a> (with the trailing slash). The incorrect rewrite rule basically caused you to see WordPress&#8217;s dynamically-generated robots.txt. You noticed this wasn&#8217;t your robots.txt file. You went in and deleted your robots.txt file in an attempt to verify that there was a &#8220;phantom&#8221; robots.txt file being served by GoDaddy. But when you did this, going to <a href="http://www.squareoak.com/robots.txt" rel="nofollow">http://www.squareoak.com/robots.txt</a> (with or without the trailing slash) caused WordPress to serve its index.php?robots=1. So you think you&#8217;ve confirmed that there&#8217;s a phantom robots.txt. You come back later, you put your robots.txt file back, you check <a href="http://www.squareoak.com/robots.txt" rel="nofollow">http://www.squareoak.com/robots.txt</a> (without the trailing slash), see it&#8217;s working now, and think, &#8220;Ah, GoDaddy fixed the problem.&#8221;</p>
<p>The bright side is, GoogleBot (and every other bot) is requesting &#8220;robots.txt&#8221; without a trailing slash, so the right file is being served.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Bill</title>
		<link>http://www.squareoak.com/blog/godaddy-phantom-robotstxt-files/comment-page-1/#comment-2843</link>
		<dc:creator>Bill</dc:creator>
		<pubDate>Fri, 13 Jun 2008 04:58:31 +0000</pubDate>
		<guid isPermaLink="false">http://www.squareoak.com/blog/?p=137#comment-2843</guid>
		<description>Let&#039;s pretend like this wasn&#039;t a Wordpress issue, which it seems clearly to be. Why would Go Daddy put up a robots.txt that disallows bot access to wp-login.php? Your stated theory of saving bandwidth just doesn&#039;t ring true in this instance because wp-login.php just isn&#039;t a high-bandwidth page (moreover, if you go over on bandwidth you get a charge so it seems like they&#039;d encourage usage). Presumably not every site hosted there has a Wordpress blog, so it seems far-fetched to think that this was something that they did to every account.

You casually accuse them of fraud, but to what end? That&#039;s a serious question. You seem to be jumping to malfeasance without warrant: there&#039;s plenty of alternate explanations and culprits that fit the scenario.</description>
		<content:encoded><![CDATA[<p>Let&#8217;s pretend like this wasn&#8217;t a Wordpress issue, which it seems clearly to be. Why would Go Daddy put up a robots.txt that disallows bot access to wp-login.php? Your stated theory of saving bandwidth just doesn&#8217;t ring true in this instance because wp-login.php just isn&#8217;t a high-bandwidth page (moreover, if you go over on bandwidth you get a charge so it seems like they&#8217;d encourage usage). Presumably not every site hosted there has a Wordpress blog, so it seems far-fetched to think that this was something that they did to every account.</p>
<p>You casually accuse them of fraud, but to what end? That&#8217;s a serious question. You seem to be jumping to malfeasance without warrant: there&#8217;s plenty of alternate explanations and culprits that fit the scenario.</p>
]]></content:encoded>
	</item>
</channel>
</rss>
