On a monthly basis our proprietary spider crawls and indexes more than 270 million websites in 50 countries and structures the data. So governments, statistical agencies, open source intelligence and companies can use it for analytical, business and research purposes.
Our Dataprovider.com spider
"Mozilla/5.0 (compatible; Dataprovider.com)"
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="robots" content="noindex, nofollow"> </head>
User-agent: dataprovider Disallow: /
For more information on how to block spiders check http://www.robotstxt.org/robotstxt.html
Alternatively, you can contact us and opt out by filling in the form.