Our crawler: EverroadBot

Everroad reads franchise dealers' own websites to check the facts it lists: hours, service options and posted fees. Our crawler identifies itself as EverroadBot, with this user agent: Mozilla/5.0 (compatible; EverroadBot/1.0; +https://everroad.org/data/crawler/)

It obeys robots.txt (the EverroadBot group first, then *), makes one request at a time to a site, waits at least 10 seconds between requests to one site, runs no JavaScript, keeps no cookies, and never rotates its user agent or IP address.

A block, a challenge page or repeated errors ends crawling of that site for the cycle; we never try to get around it. Facts for that store then come from other public sources, or stay off the page.

To ask us to stop or slow down, add a robots.txt rule for EverroadBot or write to hello@everroad.org.