Many search engines use programs called robots to locate web pages for indexing. These programs are not limited to a pre-defined list of web pages, instead they follow links on pages they find, which makes them a form of intelligent agent. The process of following links is called spidering, wandering, or gathering. Once they have a page or document, the parsing and indexing of the page begins

Robot spiders cannot index unlinked files, so they will ignore all the miscellaneous files you may have in your web server directory. Web publishers can control which directories the robots should index by editing the robots.txt file, and web page creators can control robot indexing behavior using the Robots META tag (see below).

<meta name="robots" content="index,follow" />