Key Takeaways
- Agentic AI traffic is autonomous systems that browse, click, and complete tasks on a person’s behalf. It is a distinct, fast-growing traffic category beyond training crawlers and retrieval bots.
- A page can rank well and look fine to a human while still failing an AI agent trying to use it; the gap shows up in access control, rendering, semantic HTML, structured data, and session handling.
- Access and crawl control come first: audit robots.txt and CDN or hosting-level blocks separately, since agent traffic can be silently blocked upstream of your own server.
- Content that only appears after JavaScript runs is often invisible to retrieval bots and slows down agentic browsers, so server-side rendering matters more than ever.
- The accessibility tree, not just visual design, is now a primary way agents read and interact with a page, making semantic HTML and ARIA a functional requirement, not just an accessibility nicety.
- Structured data, session persistence, and bot-detection settings all affect whether an agent can complete a task.
- A scannable, priority-and-effort-ranked checklist is included further down to help you triage where to start.
- A few third-party tools, including Google’s Lighthouse and Cloudflare’s isitagentready.com, can give you a quick agent-readiness signal alongside the checks in this article.
Your website already receives non-human traffic from multiple sources. Search engine crawlers, AI training bots, and retrieval bots that power AI search citations have been part of the traffic mix for a while now. But there’s a newer category emerging worth paying attention to: agentic AI traffic.
Unlike crawlers that read and leave, agentic AI traffic behaves more like an autonomous actor. These autonomous systems browse, navigate, click, and try to complete tasks on a person’s behalf, rather than simply collecting information. Most websites were never built to distinguish between the two, and that gap is starting to surface in places a traditional SEO audit rarely checks.
This is where agentic web optimization comes in. The goal is to make a website accessible, understandable, and usable by AI web agents as they navigate the agentic web.
The problem is that an AI agent doesn’t experience a page the way a person does. A page can rank well and look perfectly functional to a human visitor while it fails an agent trying to use it.
The familiar idea of being “crawlable” still applies here: Googlebot needs to be able to reach and index a page, that part hasn’t changed. But a broader term is emerging alongside it: sites are increasingly described as “agent-ready” or “agent-accessible” when an agent can also understand, navigate, and act on it without getting stuck.
Why should you care? By the time agentic traffic appears as a measurable line item in your analytics, hygiene issues may have already cost you a failed form submission, an abandoned checkout, or a task an agent simply gave up on. To help you prioritize, a scannable checklist further down in this article organizes each item by priority and estimated dev effort.
If you’re still weighing whether to allow AI bots on your site at all, that’s a separate, earlier decision to make first. For that, see Should You Block AI Bots from Your Website?
Understanding How Agentic AI Traffic Differs
Before you jump to the checklist to streamline agentic web optimization, it helps understand why these checks matter, not just what to check.
Three Types of AI Traffic
There are roughly three categories of non-human visitors worth distinguishing: training crawlers, retrieval bots, and agentic AI traffic.
Training crawlers harvest content to improve AI models, but they generally don’t drive traffic back to your site:
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Meta-ExternalAgent (Meta)
- Bytespider (ByteDance)
Google-Extended and Applebot-Extended are also worth knowing, but they’re robots.txt-only tokens that allow you to decide whether the content already fetched by Google and Apple can be used for training. These aren’t separate crawlers that fetch pages themselves, so they won’t show in your server logs.
Indexing and retrieval bots are the ones powering AI citations and are the most relevant if your goal is to appear in AI-generated answers.
- OAI-SearchBot (OpenAI)
- PerplexityBot (Perplexity)
- Claude-SearchBot (Anthropic)
- Googlebot (Google, also feeds Google’s own AI features)
Agentic AI traffic is the third and newest category: autonomous systems that browse, navigate, and act, rather than simply read. It’s also the fastest-growing category and the one that most sites are least prepared for. For a closer look at the tools driving this category and what they mean for the broader search landscape, see What Agentic Browsers Mean for SEO.
- ChatGPT-User and Claude-User (triggered when someone asks an assistant to visit a specific page)
- Fully agentic browsers such as ChatGPT’s built-in browser, Perplexity Comet, and Claude’s own browser/computer-use integrations
Unlike crawlers, these agentic browsers don’t always announce themselves in server logs. Some agent-triggered visits carry a distinct signal, ChatGPT-User and Claude-User both identify themselves in the user-agent string, but a fully agentic browser often runs on a standard browser engine and can present an ordinary Chrome user-agent identical to a human visitor. When that happens, logs alone won’t reliably tell the two apart, and distinguishing them usually comes down to behavioral signals instead, such as unnaturally consistent timing or pixel-precise mouse movement.
How AI Agents See a Web Page
How an agent actually interacts with a page is another question worth raising to understand the checklist better. Per Google’s own developer guidance on agent-friendly site design, there are three main channels:
- Screenshots: A vision model reads the rendered page visually, pixel by pixel. This can capture what a human sees, but it’s computationally expensive.
- HTML and the DOM: The agent can inspect the page’s structural hierarchy and the relationships between elements.
The DOM, or Document Object Model, is the browser’s internal, structured representation of a page: every HTML element nested inside the next, forming a tree. This is what the previous point means by the page’s structural hierarchy.
- Accessibility tree: A browser-native representation originally designed for screen readers, exposing the roles, names, and states of elements on the page.
The accessibility tree is a related, simpler tree built from the DOM, originally designed for screen readers, that reduces a page to its actual interactive parts: each element’s role (button, link, dropdown), its name or label, and its current state (expanded, disabled, selected).
Most production agentic systems (agent frameworks and browsers built for real-world, at-scale use rather than research demos), including Microsoft’s Playwright MCP, can combine these modalities. But the accessibility tree is particularly valuable because it gives an agent a structured map of what is on the page and what each element can do, without requiring the cost of interpreting a screenshot.
The practical implication: websites that present clean signals across all three channels (semantic HTML, a stable layout, and machine-readable content) tend to perform better with agents. The reverse holds too. Heavy client-side rendering, vague calls to action, and loose structure create the conditions for agents to fail at tasks that would take a human two seconds.
The Agentic AI Traffic Readiness Checklist at a Glance
Use this as a scannable summary. Each item is grouped by section, with a priority tier and a rough sense of dev effort. Click any item to jump straight to that section.
Access & Crawl Control
This is also where AI crawler optimization begins. Before worrying about how an agent interprets buttons, forms, or page structure, make sure the relevant AI crawlers and agents can actually reach the resources they need.
#2.1 Audit robots.txt and AI Bots for Unintended Blocks
robots.txt is the file that tells bots and crawlers which parts of your site they’re allowed to visit, which is one of the oldest and simplest ways a website communicates with automated traffic. A common mistake is a blanket Disallow: / rule aimed at “AI bots” that accidentally blocks legitimate indexing and retrieval crawlers, along with training crawlers. It’s worth knowing the key user-agent strings: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Google-Extended, and GPTBot.
These aren’t interchangeable: GPTBot handles training, while OAI-SearchBot handles retrieval and citations. Blocking one doesn’t block the other, and treating them as a single category is the single most frequent error we see in this layer.
A quick check: visit yourdomain.com/robots.txt and confirm indexing bots aren’t disallowed, and cross-check with the URL inspection tool in Google Search Console, though that only tests Googlebot. For AI bots specifically, #8.3’s approach or SUSO’s AI Search Visibility Checker is the more reliable read.
A simple example of a permissive, clearly scoped configuration:
# Allow indexing and retrieval crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Block training-only crawlers, if that's the intent
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /#2.2 Check CDN and Hosting-Level Blocks
A clean robots.txt is not a guarantee that bots are actually getting through. Server-level blocks are invisible to site owners because they happen upstream of your origin server, and robots.txt has no visibility into them.
This matters more now than it did even a year ago: Cloudflare began blocking AI crawlers by default in July 2025, and as of September 15, 2026, it’s moving to a more granular model that separates bot traffic into three categories: Search, Agent, and Training. Training and Agent are blocked by default on any page that carries advertising.
If your site runs on Cloudflare or a similar CDN (a network of servers that sits between visitors and your origin server), it’s worth checking the AI bot management settings directly rather than assuming your robots.txt file is the final word. A server log audit is the most reliable way to confirm what’s actually reaching your origin.
Find Out If AI Bots Can Actually Reach Your Site
Standard robots.txt reviews won’t surface CDN or hosting-level blocks. SUSO’s AI Search Visibility Checker tests whether key AI bots can actually reach your site, not just whether your directives say they should.
Check Your Domain#2.3 Consider llms.txt (Emerging Standard)
This is a voluntary signal file, similar in spirit to robots.txt but aimed at large language models.
It helps AI systems understand your site’s structure and which content you’d prefer they prioritize. It isn’t universally adopted or enforced yet, and it’s genuinely low priority, but it’s also so low-effort to implement that it’s worth doing now anyway, as one of the easiest future-proofing steps on this whole checklist. A reasonable next step is checking whether competitors in your space have already added one.
→ What Is llms.txt (and Is It Actually Important for AI Search?)
#2.4 Manage Crawl Rate and Server Capacity
Not every agentic crawler respects crawl-delay conventions (a polite, voluntary pace crawlers are expected to follow) the way established search crawlers do. Unmanaged scraper traffic can spike server load without warning.
A practical fix is setting WAF crawl-rate rules by user-agent. Go easy on known, well-behaved indexing bots, and throttle or block unrecognized scrapers. Regular log monitoring for unfamiliar user-agent strings catches most of this before it becomes a capacity problem.
Rendering & JavaScript
Content that doesn’t render isn’t indexed or parsed, whether the visitor is a search engine or an agent.
#3.1 Confirm Server-Side Rendering (SSR) for Content Pages
Server-side rendering (SSR) means your server builds the finished HTML before sending it to a visitor, rather than shipping a mostly-blank page and letting the browser fill it in with JavaScript afterward. Some AI crawlers, the retrieval bots specifically, don’t execute client-side JavaScript the way a modern browser does. That creates a problem for JavaScript-heavy pages: if the initial HTML contains little more than an empty app shell and the actual content only appears after JavaScript runs, a crawler may have little or nothing useful to retrieve.
In other words, what your browser eventually renders isn’t necessarily what an AI crawler can see. A page can look completely normal to you while delivering an almost empty HTML response to a bot.
The check is simple: Open the page’s View Source, rather than the rendered DOM in DevTools, and look for the page’s primary content in the initial HTML response.
If your main heading, copy, and other important content are already there, you’re giving crawlers something useful to work with before JavaScript kicks in.
Frameworks such as Next.js, Nuxt, and Astro make this relatively straightforward for content-heavy pages because they support server-side rendering or static generation.
#3.2 Don’t Hide Key Content Behind Interactions
Pricing, specs, and core value propositions shouldn’t live exclusively behind tabs, accordions, or JS-triggered modals.
Browser-based agents can execute JavaScript, but they handle complex, multi-step interactions less reliably than a human would, and retrieval crawlers, the kind that don’t run JavaScript at all, see nothing behind a hidden container.
A useful audit: confirm that anything commercially significant is present in visible, initial HTML.
#3.3 Respect Agent Patience with Page Speed
Agentic systems, including agentic browsers built on Chromium, can time out or give up partway through a task when a page is slow, the same way an impatient visitor might click away. Slow time-to-first-byte and render-blocking resources affect agent sessions, not just user experience metrics.
Core Web Vitals stay relevant here: agentic browsing tools built on Chromium inherit the same rendering constraints as any other browser.
Reasonable baseline targets are LCP under 2.5 seconds and TTFB under 600ms. Large JavaScript bundles that delay meaningful content render are a common, fixable culprit.
#3.4 Evaluate Cloudflare’s Markdown for Agents (Optional & Emerging)
Cloudflare introduced this feature in February 2026: it serves a clean Markdown version of page content directly to AI clients, stripping out navigation chrome, tracking scripts, and other UI noise that agents would otherwise have to filter through.
It’s available on Cloudflare-hosted sites and may be worth enabling for content-heavy pages.
However, in November 2025, Google’s John Mueller and Bing’s Fabrice Canel both warned against serving a separate Markdown or JSON version of a page specifically to AI crawlers, tying the practice to existing cloaking guidelines. Mueller followed up more bluntly in February 2026, calling the approach “a stupid idea” in a public discussion.
Converting pages to markdown is such a stupid idea.
John Mueller, Google, February 2026
The distinction that matters here: cloaking is about serving content that’s substantively different from what a human would see on the same page, not just a different format. If the Markdown version carries the same information as the HTML version, nothing added, nothing hidden, that’s standard HTTP content negotiation: a 27-year-old web standard where a page returns HTML to browsers, RSS to feed readers, and now Markdown to agents that ask for it via the Accept header, all from the same URL. That’s a meaningfully different situation from serving a separate, user-agent-gated version of a page that shows genuinely different content, which is the practice Mueller and Canel are actually warning against. Still, worth flagging: Google hasn’t issued an explicit statement exempting equivalent-content negotiation from its cloaking guidance, so verify your Markdown output matches your HTML output before relying on this distinction.
This isn’t exclusive to Cloudflare, either: the same content-negotiation approach can be self-hosted at the origin server level, and WordPress sites can add it via a plugin such as Joost de Valk’s Markdown Alternate. There’s no rigorous study yet on whether serving Markdown actually improves agent task success, but Cloudflare has reported up to 80% lower token usage for agents reading the Markdown version, and independent write-ups such as this overview of the current state of AI content negotiation are worth following as the practice matures. For SUSO specifically, that means we’re not implementing Markdown for Agents for our own clients yet either, so it’s still one to watch rather than act on.
Semantic HTML & Accessibility Tree
This part can be the most technically substantive of all in the checklist. The accessibility tree was built for screen readers, but AI agents have adopted it as their primary navigation map.
The important distinction is that agents aren’t using it because it’s an accessibility feature. They’re using it because it’s one of the cleanest and most efficient representations of what a page contains and what its elements can actually do.
#4.1 Use Native Semantic HTML Elements
Semantic HTML means using the tag that actually matches what an element does, rather than styling something to look like a button while it’s technically just a plain <div>. <button> for actions, <a href> for links, <select> for dropdowns.
A <div onclick> pattern doesn’t register with a button role in the accessibility tree, which means an agent may not recognize it as interactive at all.
A quick way to check: open DevTools, go to the Accessibility panel, and inspect the tree for your key conversion elements. Do they have roles and names, or are they invisible?
#4.2 Label Every Form Input
Every field in a form needs a clear, explicit label, not just a placeholder, so both people and AI agents know what to type where. Use <label for=”…”> tied to the input’s id attribute, and set autocomplete values using the standard vocabulary: name, email, tel, street-address, organization.
These tell an agent exactly what kind of data a field expects, which meaningfully reduces failed form completions. Placeholder text is not a substitute for a real label.
#4.3 Establish a Correct Heading Hierarchy
Headings work like a table of contents for a page. One H1 per page, with a logical H2 through H6 structure that doesn’t skip levels. Agents use headings to build a structural model of the page and locate specific sections, the same way a person scanning a page with their eyes would. A browser extension like HeadingsMap is a fast way to visualize the current structure.
#4.4 Use Landmark Region
Elements such as <nav>, <main>, <aside>, <header>, and <footer> create recognizable landmarks in the accessibility tree, giving an agent a clear way to orient itself within a page, much like they do for screen readers. In practice, that’s similar to how you’d visually scan a page for its header, main content, and footer.
When a page contains multiple landmarks of the same type, aria-label can help distinguish between them.
#4.5 Add ARIA States for Dynamic Components
ARIA (Accessible Rich Internet Applications) is a set of HTML attributes that describe a UI element’s role and state to assistive technologies, and increasingly to AI agents reading the accessibility tree. aria-expanded, aria-controls, and aria-hidden communicate state changes for menus, accordions, and toggles, so an agent knows whether something is open or closed without needing to take a screenshot to verify.
One caution worth flagging: ARIA is meant to supplement semantic HTML, not compensate for its absence. Misapplied ARIA roles tend to make things worse. Reach for semantic HTML first, and treat ARIA as the tool for edge cases.
#4.6 Write Descriptive, Specific Button and Link Text
This one is about wording, not code: “Book a demo” rather than “Learn more.” “Start free trial” instead of “Get started.” Vague labels create a real decision failure point for an agent, and it genuinely can’t determine what action a generic label leads to. This matters most on conversion-critical CTAs.
#4.7 Avoid Ghost Elements and Transparent Overlays
Visual overlays can prevent interactive elements from being detected via screenshot; a human visitor would just close these, but an agent that doesn’t know how often can’t. Pop-ups and modals are a particular problem, since they block page traversal, and agents often can’t dismiss them reliably.
This is a distinct failure mode from #3.2’s hidden-behind-interactions issue: there, content is architecturally absent until a tab or accordion is triggered; here, the content is already present on the page, but a visual overlay blocks an agent from detecting or reaching it.
Structured Data
Structured data gives an agent machine-readable, unambiguous facts about a page, which cuts down on interpretation errors.
#5.1 Prioritize Schema Markup Types
Organization schema is foundational for entity recognition. It gives machines structured information about who a business is, including its name, logo, contact details, and social profiles, making it easier to connect a website with the correct brand or entity.
FAQ schema explicitly associates questions with their answers, giving machines a clear representation of that content rather than forcing them to infer the relationship from page text. This can make relevant answers easier for AI systems to identify and interpret.
Worth flagging: Google fully deprecated FAQ rich results as of May 7, 2026, with Search Console reporting on them stopping between June and August 2026, completing a phase-out that began back in August 2023. FAQ schema no longer produces a visible rich result in Google Search.
It still has a real, narrower benefit: it gives AI systems and agents a clean, structured question-and-answer pair to extract directly, rather than having to infer the pairing from surrounding page text. That’s worth doing for extractability, not for any ranking or rich-result boost. See our guide to FAQ schema best practices for the full implementation details.
For ecommerce sites, Product schema provides structured details such as price, availability, ratings, and product specifications. These are exactly the kinds of facts an agent may need when comparing products or deciding whether one meets a user’s requirements.
For a closer look at which properties actually matter, see this clip from one of SUSO’s webinars:
In addition to these core types, Article/NewsArticle and BreadcrumbList schema can also be valuable depending on the content and structure of your site. The HowTo schema is subject to a similar caveat to FAQ: Google removed HowTo rich results back in September 2023, so it no longer appears as a visible result in Google Search, though the markup can still help an agent parse a sequence of steps.
The goal is to add the information an agent needs to understand, compare, and act on as explicit and machine-readable as possible.
#5.2 Use JSON-LD Format
Whichever of the types above you implement, write them in JSON-LD rather than Microdata or RDFa. JSON-LD is the specific code format structured data is usually written in, a small script block that search engines and agents can parse directly. It’s preferable to Microdata or RDFa for implementation simplicity and crawlability. Validate your markup with Google’s Rich Results Test and the Schema Markup Validator.
#5.3 Keep Accurate, Consistent Entity Data
This is largely a business-listings hygiene check, not a coding task: Schema data needs to match both what’s on the page and what appears on third-party profiles, such as Google Business Profile, Wikidata, Crunchbase, and similar sources.
Conflicting entity data makes an agent less confident it has the right business or entity, which can cause it to hesitate or abandon a task, like comparing offers, filling in a form, or completing a transaction, rather than simply making the brand less likely to be cited. (Entity consistency also supports citations in AI search and GEO, but that’s a different article’s territory.) NAP (name, address, phone) consistency across platforms is worth including as a standard part of any technical review.
Session Management & Agent Behavior
This is a less-discussed layer, but one most likely to trip up agentic interactions, particularly transactional ones.
#6.1 Review Bot Detection and CAPTCHA Sensitivity
CAPTCHA systems and aggressive bot detection can block legitimate agent sessions along with malicious traffic. Signals such as unusually fast interactions or the absence of mouse movement can trigger automated defenses even when an agent is legitimately trying to complete a task.
Audit your key transactional flows to make sure they can be completed without relying on human-only interaction signals. Where appropriate, consider rate-based and request-level controls alongside behavioral detection, rather than treating every non-human interaction as inherently suspicious.
Legitimate agents still need a clear path through forms, checkouts, and other critical workflows.
#6.2 Test Session Persistence Across Multi-Step Workflows
An agent completing a multi-step task, such as search > product page > add to cart > checkout, needs consistent session state the whole way through. A session reset mid-workflow causes the task to fail outright.
Test your key conversion paths with automation that simulates agent navigation, and check that session state persists across every step. This is one of the most reliable ways to uncover failures that a standard page-by-page SEO or UX audit can miss.
#6.3 Align Rate Limiting Policies
Overly aggressive rate limiting can throttle a legitimate agent before it has a chance to complete a task. The key is to distinguish between limits designed for human-facing traffic and those applied to APIs or crawlers, and make sure they don’t conflict.
A rule intended to protect an API, for example, shouldn’t accidentally prevent an agent from completing a legitimate multi-step interaction with the site.
#6.4 Avoid Gating Critical Actions Behind Login Walls
If key functionality sits behind a login wall, agents may be unable to access that entire layer of your site, and that’s as much a product decision as a technical one, worth raising with whoever owns the user experience, not just engineering. For sites where agent access to gated functionality is genuinely useful, MCP server implementations and WebMCP’s declarative actions are emerging ways to expose those capabilities more directly. More on both below.
Emerging Standards & Agentic Commerce
These aren’t “check the box and move on” items yet, but they’re developments worth understanding now, before they become standard practice.
#7.1 Model Context Protocol (MCP)
Anthropic introduced MCP in November 2024 as a standard for structured agent-to-service interactions, and it has already seen widespread adoption in B2B SaaS contexts. It lets agents take actions through structured tool calls instead of scraping or clicking through a UI.
Building your own MCP server isn’t a near-term to-do item for most companies reading this checklist. It’s most relevant to SaaS platforms that want to expose their own product or data layer directly to agents through structured tool calls, think HubSpot, Stripe, or Notion, each of which already runs a public MCP server, rather than a typical agency website.
It’s still worth understanding, though. SUSO is building its own internal MCP server, giving Claude direct access to relevant project files, campaign context, and our own prompt library instead of requiring manual setup at the start of each session. It’s for internal use only, but it’s a concrete example of what agentic readiness looks like in practice. This matters most if you rely on AI systems to act on your own data, for example pulling it and turning it into visualizations or other outputs, rather than just asking a chatbot questions.
#7.2 WebMCP
This is a joint proposal from Google and Microsoft, published as a W3C Community Group Draft in February 2026.
It extends the MCP concept into the browser itself, letting a page register its own actions, such as form submissions, bookings, and searches, as structured tools that an agent can call directly, rather than simulating clicks and typing.
As of mid-2026, it’s running as an origin trial in Chrome with experimental support in Edge; no major production agent calls WebMCP tools yet, and Firefox and Safari haven’t committed to implementation.
It’s not something to build a roadmap around today, but it’s worth piloting if you have the development resources and watching closely if you don’t.
#7.3 Agentic Commerce: UCP and ACP
The most immediate near-term implication of this whole protocol layer is for ecommerce, where agents are starting to complete purchases autonomously on a shopper’s behalf.
OpenAI and Stripe introduced the Agentic Commerce Protocol (ACP) in September 2025, initially powering Instant Checkout inside ChatGPT. Google introduced the Universal Commerce Protocol (UCP) in January 2026, developed with a coalition that includes Shopify, Target, and Walmart, and has been expanding its capabilities to include cart support, catalog access, and identity linking throughout 2026.
Both protocols aim to standardize how agents access product data, pricing, and availability as they navigate a purchase. But regardless of the implementation, both depend on the same foundation: clean, consistent product data that machines can reliably interpret.
Without it, even a well-built protocol integration has little to work with.
For a walkthrough of how UCP works in practice, see this clip from SUSO’s ecommerce GEO webinar:
See our full guide to AI search optimization for online stores for a deeper treatment of agentic commerce readiness, UCP, and what these protocols mean for product discoverability and conversion in AI-mediated shopping journeys.
Read the eCommerce GEO GuideMonitoring Agentic Traffic
None of this is useful if your team can’t measure whether it’s working.
#8.1 Run Server Log Analysis
Server logs are simply records of every visit to your site, humans and bots alike. This is the most reliable method for identifying which AI user agents are actually reaching your site and which pages they’re accessing.
Standard analytics tools struggle to track agent visits reliably, given consent requirements, direct-traffic attribution issues, and session spoofing.
A regular log review cadence, with unfamiliar user-agent strings flagged for investigation, closes most of this gap.
#8.2 Monitor AI Referral Traffic in GA4
As of May 13, 2026, Google added a native “AI Assistant” channel to GA4’s Default Channel Group, so a baseline view of AI referral traffic now shows up automatically, no manual setup required.
It has a couple of real gaps worth knowing about, though: it doesn’t backfill historical data, and AI-app or embedded-browser traffic that arrives without a referrer still lands in Direct.
A custom channel group is still worth building for one reason: it lets you backfill historical AI referral sessions from before May 2026, which the native channel doesn’t retroactively reclassify. Beyond that, the native channel already covers current AI referral traffic, including Perplexity.
→ How to Track AI Traffic in GA4: Agentic Web Optimization: Prepare Your Website for Agentic AI Traffic [Checklist]#8.3 Verify Actual Bot Access
As covered above, server-level CDN blocks and hosting defaults, along with WAF rules, can silently block bots that your robots.txt file allows through.
AI visibility tracking measures how often your content appears in LLM responses and captures something referral traffic can’t: brand exposure before the click.
A citation doesn’t need to generate a visit to have value. If your brand is being surfaced during AI-mediated discovery, that’s a signal worth tracking on its own.
Try SUSO’s AI Search Visibility Checker
Audit which AI bots can actually access your site and surface infrastructure-level blocks that standard technical audits tend to miss.
Try SUSO’s AI Visibility Checker#8.4 Check Third-Party Agent-Readiness Scores
Google’s Chrome DevTools Lighthouse now includes an experimental “Agentic Browsing” category (Chrome version 150+), which audits four things on the page as rendered in your own browser: an llms.txt file, WebMCP support, the accessibility tree, and cumulative layout shift. It reports a pass ratio rather than the usual 0–100 score.
Cloudflare runs a free public scanner at isitagentready.com, scoring five categories: discoverability (robots.txt, sitemap, DNS for AI discovery), content accessibility (markdown content negotiation), bot access control (AI bot rules, content signals, web bot auth), protocol discovery (MCP, WebMCP, and related agent-protocol signals), and commerce readiness (UCP, ACP, and similar standards).
SUSO’s AI Search Visibility Checker takes a different approach: rather than auditing declared policies or the rendered page, it sends live requests using real AI bot user agents to confirm what actually reaches your origin server. Worth running as a complement to the checks above, not a replacement.
Conclusion
The technical checklist for traditional SEO has stayed fairly stable for years: crawlability, indexability, page speed, and mobile usability. The agentic layer adds a new set of requirements that most agencies haven’t yet folded into their standard audit template. But it’s an addition, not a replacement.
Good technical SEO hygiene is already the foundation on which agent readiness is built on. Nothing in this checklist asks you to throw away what already works; it asks you to extend it a bit further.
If you’re prioritizing where to start, access control and rendering are the highest-leverage items on this list. If an agent can’t reach a page or can’t parse it, nothing else here matters yet.
SUSO’s technical SEO audits now include these checks as a standard part of the process, because the gap between “technically sound for Googlebot” and “technically sound for AI” is real, and it’s closing fast.
And for our non-technical partners, SUSO can handle the technical execution alongside you, so you don’t need to become an expert in technical optimization to know it’s being taken care of.
Auditing a client site and want a second opinion on agentic readiness?
Let’s discuss your caseFAQs
-
What is agentic engine optimization?
Agentic engine optimization is about making your website easy for AI agents to find, understand, navigate, and use. That means looking at things like crawl access, rendering, page structure, and whether key interactions actually work for non-human visitors.
-
What is agentic browsing?
Agentic browsing is when an AI system navigates the web and takes actions on your behalf. It might search for a product, compare options, fill out a form, or even complete a purchase.
That makes it different from AI search, which can give you an answer without you visiting or interacting with a website, and from traditional browsing, where a human has to make each decision and click.
-
Do AI agents see websites the same way as humans do?
AI agents don’t experience a website quite the way we do. Instead of simply looking at the page visually, they can work from screenshots, the HTML and DOM, and the accessibility tree to understand what’s on the page and what it can do.
The practical difference is important: a page can look perfectly clear to a person but still be confusing or unusable to an agent if its structure, labels, or interactive elements aren’t clear.
-
How do I know if AI agents can reach my website?
Checking robots.txt is a useful starting point, but it doesn’t tell you the whole story. Your CDN, hosting setup, firewall, or other infrastructure can block automated traffic before it ever reaches the site.
For a more reliable picture, check your server logs for the user-agent strings themselves: names like GPTBot, ClaudeBot, and Bytespider are training crawlers, OAI-SearchBot, PerplexityBot, and Claude-SearchBot are retrieval bots that power AI citations, and ChatGPT-User or Claude-User signal an agent visiting on a specific person’s behalf. Seeing these in your logs is what actually tells you an AI agent reached your site, and how it’s interacting with it.
A tool such as SUSO’s AI Search Visibility Checker can also be a practical first step before getting into log analysis.
The practical difference is important: a page can look perfectly clear to a person but still be confusing or unusable to an agent if its structure, labels, or interactive elements aren’t clear.
-
Does optimizing for AI agents affect my traditional SEO?
These two work together rather than compete. Clean semantic HTML, good crawlability, fast rendering, and well-structured data can make your site easier for both search engines and AI agents to understand.
-
What’s the difference between agentic web optimization and GEO?
The easiest way to think about it is visibility vs. usability.
GEO focuses on whether AI systems discover, understand, and cite your content. Agentic optimization focuses on whether those systems can actually navigate and use your website.
So GEO asks, “Does AI know about me?” Agentic engine optimization asks, “Can AI do something on my site?”
Both matter, but this article focuses on the technical side of making your website usable by AI agents.
