Capture AI crawlers on Fastly.

An HTTPS logging endpoint on your Fastly service streams one JSON line per request to Virent, including cached responses.

You need your site ID and server secret from Settings → Connection (see Hosting integrations). In the dashboard, Connect analytics → Hosting integrations → Fastly shows these steps with your site ID, domain and secret filled in. Below, replace <your-site-id>, <paste-your-server-secret> and example.com with your own values.

Configure Fastly so example.com sends reliable AI crawler request telemetry to Virent.

Site ID: <your-site-id>
Use the full Virent server secret from the Connect dialog. Do not use the masked preview "vha_sk_••••" as the real value.

Use the current provider documentation: https://www.fastly.com/documentation/guides/integrations/logging-endpoints/protocol-based-and-self-hosted/log-streaming-https/

Steps:
1. Create an HTTPS endpoint: On the Fastly CDN service, add an HTTPS logging endpoint. Virent answers Fastly’s ownership validation challenge at this destination.

https://stage.virent.app/api/ingest/fastly

2. Authenticate delivery: Set Content type to application/x-ndjson and configure this custom header. Keep Placement on Format Version Default.

x-api-key: <paste-your-server-secret>

3. Set the request log format: Paste this exact log format. It emits one Virent-compatible JSON object per request and safely escapes request headers.

{"timestamp":"%{begin:%Y-%m-%dT%H:%M:%SZ}t","status_code":%>s,"request_method":"%m","request_path":"%{json.escape(req.url.path)}V","query_string":"%{json.escape(req.url.qs)}V","content_type":"%{json.escape(resp.http.Content-Type)}V","client_ip":"%a","hostname":"%{json.escape(req.http.host)}V","user_agent":"%{json.escape(req.http.User-Agent)}V","referrer":"%{json.escape(req.http.Referer)}V"}

4. Activate and verify: Create the endpoint, activate the service version on Production, send an AI crawler request to example.com, and verify it in Virent.

Keep the secret server-side, preserve 100% request coverage where the provider supports sampling, and verify both normal site delivery and a crawler request after activation.

Steps#

1. 01

   Create an HTTPS endpoint

   On the Fastly CDN service, add an HTTPS logging endpoint. Virent answers Fastly’s ownership validation challenge at this destination.

   Endpoint URL

   https://stage.virent.app/api/ingest/fastly

2. 02

   Authenticate delivery

   Set Content type to application/x-ndjson and configure this custom header. Keep Placement on Format Version Default.

   Custom header

   x-api-key: <paste-your-server-secret>

3. 03

   Set the request log format

   Paste this exact log format. It emits one Virent-compatible JSON object per request and safely escapes request headers.

   Log format

   {"timestamp":"%{begin:%Y-%m-%dT%H:%M:%SZ}t","status_code":%>s,"request_method":"%m","request_path":"%{json.escape(req.url.path)}V","query_string":"%{json.escape(req.url.qs)}V","content_type":"%{json.escape(resp.http.Content-Type)}V","client_ip":"%a","hostname":"%{json.escape(req.http.host)}V","user_agent":"%{json.escape(req.http.User-Agent)}V","referrer":"%{json.escape(req.http.Referer)}V"}

4. 04

   Activate and verify

   Create the endpoint, activate the service version on Production, send an AI crawler request to example.com, and verify it in Virent.

Check Fastly's current documentation if its screens differ from these steps.

Verify#

Send a test crawler request to your site. The setup dialog in Virent shows "Virent is receiving AI crawler traffic" once the first request arrives.