# See how AI crawlers actually request your site.

> Classify AI crawler requests, measure site coverage, inspect response status and validate robots, middleware and routing changes.

Canonical: https://stage.virent.app/bots
Last updated: 2026-07-29

Classify crawler requests at the server, then find the pages machines reach, miss, or revisit.

## Move from server-log noise to a useful map of machine access.

- **Which crawler families request the site, and which routes do they reach?** Answered in Analytics / Crawlers: Requests grouped by crawler family, with the routes each one reaches.
- **Which high-value pages have little or no observed crawl activity?** Answered in Analytics / Crawlers: Routes ranked by crawler requests in the period, so the ones with little or none stand out.
- **Where do crawlers receive redirects, blocks, or server errors?** Answered in Analytics / Crawlers: Response statuses crawlers received, so redirects, blocks and errors stand out by route.
- **What changed after a robots, middleware, or routing update?** Answered in Analytics / Crawlers: Crawler requests to the same routes before and after the date of a change.

## Crawler activity happens without running your browser analytics.

Most crawlers request HTML without executing browser JavaScript, so Virent captures requests in the server, edge, middleware, reverse proxy, or plugin hook instead. Server secrets stay in server-side environment variables.

- [Choose a framework in Documentation](https://stage.virent.app/docs#installation)

## Core measurements

### Crawler classification

Group requests using maintained user-agent and crawler-family rules, with the reported agent preserved for inspection. A classification is not proof of cryptographic identity.

### Request volume and timing

Review counts and trends by family, route, section, method, and time range. A spike is a prompt to inspect, not automatically harmful.

### Coverage

Compare the pages crawlers requested with the inventory that matters. Coverage describes observed requests, not guaranteed indexing or model use.

### Response status

See the status returned to crawlers and find repeated redirects, blocked routes, missing pages, or server failures.

### Rules and change context

Compare behavior around robots, middleware, proxy, or routing changes to validate the observed effect rather than the intent.

## Being accessible, being selected, and being cited are different states.

A request proves that a machine reached an observed layer. It does not prove that a provider indexed the page or cited it in an answer. Virent makes the gap measurable:

1. inspect crawler coverage and response behavior for the pages that matter;
2. compare them with citation URLs from repeatable prompt runs;
3. separate access problems from relevance and source-selection questions;
4. repeat the comparison after a change.

- [Explore LLM Visibility](https://stage.virent.app/visibility)
- [Explore Content Agent](https://stage.virent.app/content-agent)

## Workflow

1. **Install at the request layer**: Use the recommended hook for your framework. Exclude API routes, static assets, favicons, robots files, and sitemaps unless the measurement plan needs them.
2. **Verify with a controlled request**: Send a request with a known crawler user agent and confirm the event appears. This tests the integration, not the external crawler's network identity.
3. **Establish coverage**: Group routes into business-relevant sections and compare observed requests with the expected page inventory.
4. **Investigate changes**: Use status, route, timing, and citation evidence to decide whether the next step is technical, editorial, or simply more observation.

## For teams

### SEO teams

Connect crawl behavior to visibility and citations without reducing it to a single score.

[Explore LLM Visibility](https://stage.virent.app/visibility)

### Developers and platform teams

Inspect concrete routes and responses, keep secrets server-side, and validate changes with controlled requests.

### Content teams

Find pages that receive crawler attention but never appear as sources in monitored answers.

## Frequently asked questions

### Why not use the browser script for crawler analytics?

Many crawlers do not run the browser script. Capture has to happen before browser JavaScript, at the server or another request-interception layer.

### Does Virent verify that every request came from the named company?

Virent classifies observed signals such as user agents using its maintained taxonomy, and surfaces stronger verification separately where it is available. A classification alone is not cryptographic identity proof.

### Does a successful crawl mean the page can appear in an AI answer?

No. Access is one prerequisite. Indexing, retrieval, source selection, and answer generation are controlled by external providers.

### Can Virent change our robots rules?

No. Virent measures and supports decisions. It does not push robots, canonical, middleware, or content changes to your site.