Search is changing faster than at any point in the last two decades. AI-powered tools like ChatGPT, Perplexity, Google AI Overviews, and Microsoft Copilot are now answering millions of queries directly — without sending users to a website at all. For businesses that rely on organic search traffic, this is both a threat and an opportunity. The businesses that adapt now will dominate AI-generated answers. Those that do not will become invisible. This guide covers every technical step you need to take to make your website visible to AI search engines — from llms.txt and structured data to entity optimisation and content architecture.
Key Takeaways
- Create a llms.txt file at your domain root — a plain-text AI discovery file that tells LLMs exactly who you are, what you do, and where your key content lives.
- Update robots.txt to explicitly allow GPTBot, PerplexityBot, ClaudeBot, and other AI crawlers, and reference your sitemap and llms.txt.
- Implement comprehensive JSON-LD structured data — especially Organization, FAQPage, and Service schemas — to give AI systems machine-readable facts about your business.
- Reduce your site's dynamic rendering percentage by adding static noscript content or moving to server-side rendering so AI crawlers that do not execute JavaScript can still read your content.
- Build consistent entity signals across Google Business Profile, LinkedIn, and authoritative directories — AI systems build their understanding of your business from dozens of sources, not just your website.
- Write content in a question-and-answer format with specific facts, inverted pyramid structure, and semantic HTML — this is the content architecture AI systems are most likely to cite.
What is Answer Engine Optimisation (AEO) and why does it matter?
Answer Engine Optimisation (AEO) is the practice of structuring your website's content, technical infrastructure, and entity signals so that AI systems — including large language models (LLMs), AI search engines, and generative AI assistants — can accurately understand, cite, and recommend your business in their responses.
Traditional SEO optimises for ranking positions on a search results page. AEO optimises for being the answer — the source that an AI cites when a user asks a relevant question. The distinction matters enormously: when ChatGPT recommends a lead generation agency to a UK business owner, it is not showing ten blue links. It is naming one or two specific businesses. If yours is not one of them, you do not exist in that interaction.
The scale of this shift is significant. As of 2026, ChatGPT has over 100 million weekly active users. Perplexity processes hundreds of millions of queries per month. Google's AI Overviews now appear on a significant proportion of UK search results. The traffic implications for businesses that are not optimised for these systems are already being felt — and will accelerate.
Step 1: Create and publish a llms.txt file
The llms.txt standard is a plain-text file placed at the root of your domain (e.g. https://yourdomain.com/llms.txt) that tells AI crawlers and large language models exactly what your website is, what it does, and where to find the most important content.
Think of it as a robots.txt for AI — but instead of restricting access, it actively guides AI systems towards your most valuable content and provides a structured summary of your business identity.
What to include in your llms.txt file:
- Business identity — your full legal name, trading name, what you do, who you serve, and your unique positioning
- Services or products — a clear, structured list of every service or product with a brief description of each
- Geographic coverage — where you operate (critical for local and national businesses)
- Key pages — direct URLs to your most important pages (homepage, services, contact, about)
- Content inventory — links to your blog posts, guides, and resources with brief descriptions
- Contact information — email, booking links, and any other direct contact methods
- Differentiators — what makes you different from competitors, in plain language
# Company Name, ## Services) and keep descriptions factual and specific. LLMs parse structured plain text extremely well. Avoid marketing fluff — write as if you are briefing an intelligent assistant.Once published, submit your llms.txt URL to llmstxt.org and directory.llmstxt.cloud to accelerate discovery. Also add a discovery link in your HTML <head>:
Step 2: Optimise your robots.txt for AI crawlers
Your robots.txt file (at yourdomain.com/robots.txt) controls which parts of your site automated crawlers can access. Most businesses have a basic robots.txt that allows all crawlers — but for AEO, you need to be more deliberate.
Key AI crawlers and their user-agent strings:
GPTBot— OpenAI's crawler (used to train and update ChatGPT)PerplexityBot— Perplexity AI's crawlerClaudeBot— Anthropic's Claude crawlerGooglebot— Google (also powers AI Overviews)Bingbot— Microsoft Bing (powers Copilot)YouBot— You.com AI search
The critical decision is whether to allow or block AI training crawlers. Allowing them means your content may be used to train future AI models — which increases the likelihood of your business being cited in AI-generated answers. Blocking them means your content is not used for training, but you may also reduce your visibility in AI search results.
For most businesses seeking AI search visibility, the recommended approach is to allow all AI crawlers and ensure your robots.txt explicitly references your sitemap and llms.txt:
Allow: /
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
AI-Content: https://yourdomain.com/llms.txt
Step 3: Implement comprehensive structured data (JSON-LD)
Structured data is machine-readable markup that tells AI systems and search engines exactly what your content means — not just what it says. It is the single most impactful technical change you can make for AEO, and it is implemented via JSON-LD scripts in your HTML <head>.
Essential schema types for AEO:
- Organization — your business identity, contact info, social profiles, and service areas
- LocalBusiness / ProfessionalService — geographic coverage, opening hours, and service categories
- FAQPage — questions and answers that AI systems can directly cite in responses
- HowTo — step-by-step processes that AI systems use to answer procedural queries
- Service — individual service descriptions with pricing ranges and delivery areas
- Article / BlogPosting — per-article metadata including author, date, and topic
- BreadcrumbList — site hierarchy that helps AI understand your content structure
- WebSite — site-level identity with a SearchAction for sitelinks search box
A minimal but effective Organization schema looks like this:
"@context": "https://schema.org",
"@type": ["Organization", "ProfessionalService"],
"name": "Your Business Name",
"url": "https://yourdomain.com",
"description": "Clear, factual description of what you do and who you serve.",
"areaServed": "United Kingdom",
"knowsAbout": ["Topic 1", "Topic 2", "Topic 3"],
"hasOfferCatalog": { "@type": "OfferCatalog", "itemListElement": [...] }
}
Step 4: Reduce dynamic rendering — serve content as static HTML
One of the most overlooked AEO issues is dynamic rendering. Most modern websites are built with JavaScript frameworks (React, Vue, Next.js) that render content in the browser rather than serving it as static HTML. This is a problem for AI crawlers because many LLM crawlers do not execute JavaScript — they only read the raw HTML response from the server.
If your page's key content (headlines, service descriptions, FAQs, contact information) is only visible after JavaScript executes, AI crawlers may never see it. This directly reduces your AEO visibility.
How to check your rendering percentage: Use a tool like Screaming Frog or a GEO audit tool to compare the raw HTML response of your page against the rendered DOM. A rendering percentage above 15–20% suggests that a significant portion of your content is invisible to non-JS crawlers.
Solutions to reduce dynamic rendering:
- Server-Side Rendering (SSR) — render pages on the server and send complete HTML to the browser. Next.js, Nuxt.js, and SvelteKit all support SSR natively.
- Static Site Generation (SSG) — pre-render all pages at build time. Gatsby, Astro, and Next.js static export are excellent options.
- Add a <noscript> block — include a static HTML summary of your key content inside a
<noscript>tag so crawlers that do not execute JS still read your core message. - Inline critical content in the HTML shell — place your most important text (headline, description, services, FAQs) directly in the
index.htmlfile rather than loading it via JavaScript.
Is your website invisible to AI search engines?
Book a free AEO audit call with APXL. We will review your site's structured data, rendering percentage, and llms.txt implementation and tell you exactly what to fix.
Book a Free Growth CallStep 5: Entity optimisation — build a clear, consistent identity across the web
AI systems do not just read your website. They build a model of your business from dozens of sources across the web — your Google Business Profile, LinkedIn company page, Trustpilot listing, industry directories, press mentions, and more. This collection of signals is called your entity.
The more consistent and authoritative your entity signals are, the more confidently AI systems will cite your business. Inconsistencies — different business names, addresses, or descriptions across different platforms — create ambiguity that reduces AI confidence in recommending you.
Entity optimisation checklist:
- Claim and fully complete your Google Business Profile — this is the single most important entity signal for UK businesses
- Ensure your business name, address, and phone number (NAP) are identical across all directories, social profiles, and your website
- Create and maintain a LinkedIn company page with a complete description, services, and regular posts
- Get listed in authoritative UK business directories: Yell, Yelp, Checkatrade (if relevant), FreeIndex, and industry-specific directories
- Add sameAs links in your Organization schema pointing to all your official profiles
- Earn mentions and links from authoritative UK publications and industry websites
- Publish a Wikipedia-style "About" page on your site with factual, third-person information about your business history, team, and credentials
Step 6: Content architecture — write for AI citation
AI systems prefer content that is structured, specific, and directly answers questions. The content architecture principles that drive AEO are different from traditional SEO copywriting — and understanding the difference is essential.
Write in a question-and-answer format. AI systems are trained on Q&A data. Content that explicitly poses a question and then answers it concisely is far more likely to be cited than content that buries the answer in long paragraphs. Use H2 headings that are phrased as questions your ideal customer would actually ask.
Use the inverted pyramid structure. Lead with the direct answer, then provide supporting detail. AI systems often extract just the first one or two sentences of a section — make sure those sentences contain the key information.
Include specific facts, statistics, and examples. AI systems weight content that contains verifiable, specific information more highly than generic claims. Cite sources where possible. Include real numbers, dates, and named examples.
Use clear, semantic HTML. Proper use of H1, H2, H3, ul, ol, and table elements helps AI systems understand the hierarchy and relationships within your content.
Publish a dedicated FAQ page. A well-structured FAQ page with 15–25 specific questions and direct answers is one of the highest-ROI AEO investments you can make. Mark it up with FAQPage JSON-LD schema for maximum impact.
Step 7: Submit your sitemap and monitor AI crawler activity
A well-structured XML sitemap at yourdomain.com/sitemap.xml is essential for ensuring AI crawlers can discover all your content. Your sitemap should include every important page with accurate lastmod dates so crawlers know which content is fresh.
Sitemap best practices for AEO:
- Include all blog posts, service pages, and landing pages — not just the homepage
- Set accurate
lastmoddates — AI crawlers prioritise recently updated content - Use
changefreqto signal how often content is updated - Submit your sitemap to Google Search Console and Bing Webmaster Tools
- Reference your sitemap in both
robots.txtand yourllms.txt
Monitoring AI crawler activity: Review your server access logs regularly for visits from GPTBot, PerplexityBot, and ClaudeBot. Increasing crawler frequency is a positive signal that your content is being indexed. If you see no AI crawler activity after implementing these steps, check that your robots.txt is not accidentally blocking them.
Test your AEO implementation: Search for your business name and key services in ChatGPT, Perplexity, and Google AI Overviews. Note whether you appear, what information is cited, and whether it is accurate. This is your baseline — track it monthly to measure the impact of your AEO work.
Ready to take action?
Book a free growth call with APXL
Find out how APXL can help your UK business generate better quality enquiries and turn them into real sales opportunities.
Book a Growth Call