Sales & Support 1-888-220-7361

A robots.txt file is a file you put on your server to give specific instructions to web robots. These are also called crawlers, or spiders. All the major search engines have them. Most spammers and malware hawkers have them too. They’re all over the place, honestly.

But there are a couple of things you should know about the robots.txt file.

  1. It’s a public document, meaning anyone can open the file and see which sections of your website you don’t want the search engines to see or index. In other words, you can’t use it to hide information.
  2. Robots can ignore your robots.txt file, and many do.

So if robots can ignore the file, then what good is it?

That’s a good question. I’ll answer that by saying that good robots typically obey the commands you give in your robots.txt file while bad robots don’t. So if your intention is to create a file that prevents bad robots from accessing your website, then it’s probably pointless.

If your intention, however, is to communicate specific instructions to good robots – for example, search engine robots – then it’s a safe bet that those commands will be obeyed.

The most common use of the robots.txt file is to prevent search engines from indexing a website, or portions of the website. It’s not the only use, however.

The robots.txt file is a useful tool for webmasters, but it has its weaknesses. Do some additional reading to learn more about how you can use the robots.txt file for title=”robots.txt web development”>your website, or whether you need it at all.

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>