Displaying 1 result from an estimated 1 matches for "msnbot".
Did you mean:
isnot
2010 Jan 16
3
httpd and robots.txt
would anyone out there care to share their robots.txt experience using
centos as a webserver and their robots.txt files?
i realize this is a somewhat simple exercise, yet i am sure there are both
large and small hosters out there and possibly those that have high traffic
modify their robots.txt files differently that others ???
please share if you can or care to please?
for years we have just