I'm not an expert on Robots.txt but from what I can gather it's a raw text file with a few lines which is used to instruct the Google bots when they visit your site.
For a real life example on where you would use Robots.txt on your affiliate site I suggest you read the following thread. The thread is based around the concept of re-directs (do a search on this forum for additional threads on redirects or on the net) which is a very important concept if you are planning to run an affiliate site which links directly to a merchant.
Submitted by Jeremy Palmer on Wed, 02/21/2007 - 21:33.
A robots.txt file isn't necessary for a search engine to crawl your site, but it can help you control the bot crawling your site. Most webmasters use robots.txt to block crawlers from content they don't want indexed - e.g. company intranet, members-only area, etc.
Robots.TXT Question
Hi Matt
I'm not an expert on Robots.txt but from what I can gather it's a raw text file with a few lines which is used to instruct the Google bots when they visit your site.
For a real life example on where you would use Robots.txt on your affiliate site I suggest you read the following thread. The thread is based around the concept of re-directs (do a search on this forum for additional threads on redirects or on the net) which is a very important concept if you are planning to run an affiliate site which links directly to a merchant.
Redirects thread (with Robots.txt example)
http://quityourdayjob.com/members/forums/viewtopic.php?t=807&highlight=lucky
Hope the above helps.
Slainte
Robots.TXT Question
For more information on robots.txt and how they work try this site:
http://www.mcanerin.com/EN/search-engine/robots-txt.asp
Robots.TXT Question
Thanks for all your help.
Matt
Robots.TXT Question
A robots.txt file isn't necessary for a search engine to crawl your site, but it can help you control the bot crawling your site. Most webmasters use robots.txt to block crawlers from content they don't want indexed - e.g. company intranet, members-only area, etc.
For more info:
http://sitemaps.blogspot.com/2006/02/using-robotstxt-file.html