Join the waitlist

VozRadarBot

What VozRadarBot does on your site.

VozRadarBot is the web crawler behind VozRadar. It fetches a small number of public pages, identifies itself on every request and follows the rules you set in robots.txt.

What VozRadarBot is

VozRadar measures how AI assistants (answer engines) mention, recommend and cite brands in specific markets. To do that, it needs to read some public web pages: the pages of the brands our customers track and the pages that AI answers cite. VozRadarBot is the crawler that fetches them.

Every request identifies itself with this user agent:

VozRadarBot/0.1 (+https://vozradar.com/bot)

The token to use in robots.txt is VozRadarBot. Matching is case-insensitive.

What it visits

VozRadarBot only fetches pages for a reason tied to a VozRadar customer. There are three cases.

  • The customer's own website. When a customer connects their website to a project, the bot runs a setup crawl of up to about 250 pages. It also runs an "AI access" check that requests the homepage, /robots.txt and /llms.txt.
  • Competitor websites. For the competitors a customer tracks, the bot fetches up to 100 pages per site, starting with the pages that AI answers cite. The most-mentioned competitors are refreshed about once a week; a customer can also ask for a manual refresh, at most once a day.
  • Pages cited in AI answers. To see which brands a cited page mentions, the bot fetches a small number of public third-party pages on demand, up to 20 pages per check.

How it behaves

  • It fetches public pages only. It never logs in, submits forms, bypasses paywalls or tries to reach anything private.
  • Before crawling, it reads robots.txt and follows the group for VozRadarBot, or the * group when there is no VozRadarBot group. Allow and Disallow are supported, including * and $ patterns. The one exception is the AI access check on a customer's own website, which requests the homepage, /robots.txt and /llms.txt to report on them.
  • When it crawls a site it makes one request at a time, at least one second apart, and honours a Crawl-delay in robots.txt up to 10 seconds.
  • It does not run JavaScript. It reads the HTML as served.
  • Every request has a size limit of 2 MB and a timeout.
  • It only uses http and https on the standard ports.

How to block or slow it down

Add rules for VozRadarBot to your robots.txt. Changes take effect the next time the bot reads the file, which it does at the start of each crawl. The AI access check on a customer's own website still requests the homepage, /robots.txt and /llms.txt so that it can report on them.

Block it from the whole site:

User-agent: VozRadarBot
Disallow: /

Keep it out of one folder:

User-agent: VozRadarBot
Disallow: /private/

Slow it down to one request every five seconds:

User-agent: VozRadarBot
Crawl-delay: 5

What we store

For competitor websites, we store page metadata only: the URL, the title, the meta description, the main headings, the language, the dates the page publishes, the HTTP status and the word count. We do not store the page text.

For third-party pages cited in AI answers, we store the URL, the title, the HTTP status and which tracked brands the page names. We do not store the page text.

For a customer's own website, we store the page text, so that the recommendations we give that customer can use it.

Contact

Questions, complaints or a request to stop crawling a site: [email protected]. If you ask us to stop crawling a site, include its domain.