What is crawling a website?

Related questions

1. What are spider bots?
2. Do I need robots txt on my site?
3. What is http mj12bot com?
4. Do I need a robots txt?
5. What is Spider Yandex com?
6. What BOT does Google use for Web search?
7. Is robots txt necessary for SEO?
8. Should Sitemap be in robots txt?
9. What is custom robots txt in Blogger?
10. What is Sitemap in robots txt?

Tags

1. #methodical
2. #search
3. #program
4. #script
5. #robot
6. #particular
7. #crawler
8. #spidering
9. #sites
10. #legitimate
11. #known
12. #engines
13. #process
14. #manner
15. #means
16. #automated
17. #which
18. #browses
19. #world
20. #spider
21. #crawling
22. #providing
23. #called