How does robots txt work?

Related questions

1. Do I need robots txt on my site?
2. Where is my robots txt file?
3. What is custom robots txt in Blogger?
4. Does Yoast create a sitemap?
5. What is Sitemap XML SEO?
6. What is http mj12bot com?
7. What is Yandex bot?
8. What is Bingbot htm?
9. Is robots txt necessary for SEO?
10. How long does it take for robots txt to take effect?

Tags

1. #processed
2. #robot
3. #inform
4. #simply
5. #specifies
6. #robots
7. #exclusion
8. #scanned
9. #communicate
10. #websites
11. #other
12. #about
13. #standard
14. #areas
15. #which
16. #website
17. #protocol
18. #known
19. #should
20. #crawlers