Our crawler

Clickwalker/1.0

If you saw Clickwalker/1.0 (+https://clickwalker.com/bot) in your logs, this page explains what it is.

What it does

Clickwalker runs site checks and flow tests that its users set up for their own websites. A check loads one page, its images and scripts, and sends a quick request to the links on that page to see whether they work. A flow test clicks through the steps the site owner wrote.

It doesn't crawl whole sites, index content, or train AI models on what it sees.

How it behaves

  • It identifies itself with Clickwalker/1.0 (+https://clickwalker.com/bot) at the end of a normal Chrome user agent.
  • When checking links, it sends at most three requests at a time to any one server, usually a HEAD request.
  • Runs are started by a person, or by a schedule they chose. Scheduled runs happen at most every six hours.

Allowing it through a firewall

If you use Cloudflare or another bot protection service and want to test your own site, allow requests whose user agent contains Clickwalker.

Stopping it

If Clickwalker is visiting a site you run and you didn't set it up, email hello@clickwalker.com with the domain and we'll stop the checks and remove the site from the account that added it.