How Web Crawlers Work07/01/2016 Tagged With: code, html, html to xml converter, internet, source code, spider, web, web crawler, xmlA web crawler (also known as a web spider or web robot) is a program or automated script which browses the internet seeking for web pages to process.