|
|
What is robots.txt?
it's a file that makes google filter the search result or not
did you edit the robots.txt file?
robots.txt - meme gif
robots.txt - video
Robots.txt - what is it?
File that stops search engines from crawling a specified page or directory.
Disallow all search engines to crawl the specified directories - /images and /cgi-bin:
User-agent: *
Disallow: /cgi-bin/
Disallow: /images/