LLMs.txt Not Working? How to Correctly Format Your Site for AI Answer Engines

If your llms.txt file is live but Perplexity, ChatGPT, or other answer engines still ignore your company, the file is probably not the only problem. This guide separates technical implementation issues from the bigger visibility issue: whether your site gives AI systems enough clean, verifiable, decision-ready information to cite.

Problem Summary

The symptom is simple: you created an llms.txt file, placed it on your domain, and expected better AI visibility. Nothing changed. Your pages are still not cited, your product is still summarized poorly, or your competitors still appear in AI answers while your brand is missing.

Here is the direct answer: llms.txt is not a ranking switch; it is a navigation file that only helps when the rest of your site is easy to crawl, summarize, verify, compare, and cite.

That is the practical stance Raze takes with B2B SaaS, AI, devtool, and technical product teams. Do not treat llms.txt as an AI SEO strategy. Treat it as one small part of answer-engine visibility, alongside page architecture, positioning clarity, structured content, crawlability, brand trust, and conversion paths.

The new funnel is no longer just impression to click to conversion. For many B2B buyers, the path is now:

Impression
AI answer inclusion
Citation
Click
Conversion

If the answer engine cannot understand what you do, who you serve, why you are credible, and which page proves the claim, the file will not save you.

The evidence supports that caution. SE Ranking reported research across 300,000 domains and found that llms.txt does not currently appear to increase AI visibility or citation frequency. Kai Spriestersbach also argued that adoption remains extremely low, citing approximately 0.1 percent adoption in his analysis of the standard. The file may become more useful over time, but in 2026 it should not be treated as the main growth lever.

For a SaaS website, the real question is not only whether llms.txt is working. The better question is whether your site gives AI systems a clean sales argument they can quote without guessing.

Symptoms

The most common symptoms show up in three places: server behavior, AI answers, and commercial performance.

The file exists, but nothing changes in AI answers

You can visit https://yourdomain.com/llms.txt, see the file, and still get no citations in AI tools. This is normal. Publishing the file does not force AI crawlers to read it, trust it, or cite the URLs inside it.

Community and technical discussions have repeatedly pointed to the same issue: many crawlers and agents do not check the file by default. Cyrus Shepard discussed Reboot experiment findings in a LinkedIn post on AI crawlers, and technical discussion on Hacker News described platform audits where agents ignored these files at the server level.

That does not make the file useless. It means expectations need to be grounded.

Your product is described inaccurately

This is often a positioning problem, not a file problem. AI answers may call your product a CRM when it is actually a revenue intelligence platform. They may describe you as a generic analytics tool when your strongest use case is developer observability. They may cite old blog posts instead of your current product pages.

That usually means the site is not consistently reinforcing category, audience, use cases, integrations, proof, and differentiation.

Competitors are cited for comparison prompts

This is one of the most painful signals. A buyer asks for the best tools for a use case, the best alternative to a known vendor, or which platforms support a workflow. Your competitor appears. You do not.

In many cases, the missing asset is not llms.txt. It is a clean comparison page, migration page, pricing explainer, integration page, sandbox page, or technical trust page. We see the same issue on conversion paths too: if your website does not help a human compare you quickly, AI systems will struggle to compare you accurately.

This is why technical AI visibility often intersects with conversion-focused web design, SaaS website redesign, and page architecture. A useful product can still lose if the site makes it hard to understand.

Likely Causes

There are five likely causes when llms.txt is not working. Start with the mechanical issues, but do not stop there.

The file is in the wrong location

The proposed convention expects the file at the root of the domain:

https://example.com/llms.txt

Common mistakes include placing it at:

https://example.com/.well-known/llms.txt
https://example.com/blog/llms.txt
https://www.example.com/llms.txt while the canonical site is non-www
A staging domain that is blocked from crawlers

If the file returns a redirect chain, login screen, 404, 403, or JavaScript-rendered page, it is not cleanly accessible.

The format is technically readable but strategically weak

Many files are built like a sitemap dump. That is the wrong mental model.

A good llms.txt should point answer engines toward the most useful, stable, high-signal pages. For a B2B SaaS company, that often means:

Homepage
Core product page
Use case pages
Pricing or packaging page
Comparison pages
Integration pages
Security or trust center
Documentation or API reference
Customer proof pages
Original research or benchmark pages

If the file contains every blog post, every tag page, every thin archive, and every campaign URL, it adds noise.

This is the same problem we see on bloated SaaS sites. A pricing page should help evaluators compare tiers and buying paths quickly, which is why we have covered similar issues in SaaS pricing page UX. AI systems need the same clarity buyers need: clean structure, stable claims, and proof attached to the right page.

Robots rules or server headers are blocking access

Your llms.txt can be public in a browser and still difficult for crawlers. Check:

robots.txt rules
CDN firewall rules
Bot protection settings
Geo restrictions
User-agent blocking
Unexpected noindex headers on important URLs
Canonical tags pointing elsewhere

Do not assume browser access equals crawler access.

Your CMS plugin created the file but not the content strategy

Some SEO plugins can help generate or expose the file. That does not mean the output is commercially useful.

For WordPress sites, Yoast documentation says the llms.txt setting can be enabled by going to Settings > General > Site features and locating the API section. The WordPress.org support thread also reflects a common user problem: teams cannot find the setting or assume the option is missing.

Even when the plugin works, the default file may not prioritize the pages that influence buying decisions.

The site does not contain enough citable evidence

This is the biggest issue for B2B teams.

AI answers pull from sources that feel trustworthy and uniquely useful. Your content should include a clear point of view, recognizable frameworks, definitions, comparison criteria, proof, and pages that make your claims easy to verify.

A weak site says: we help teams scale faster.

A stronger site says: we help Series A to Series C devtool teams reduce demo friction by clarifying technical use cases, surfacing integration proof, and routing buyers to docs, sandbox, or sales based on intent.

The second version is easier for humans to buy and easier for answer engines to cite.

How to Diagnose

Use the AI citation readiness check before changing the file again. It is a simple four-part model Raze uses when evaluating whether a SaaS website is ready for answer-engine visibility.

The AI citation readiness check

Access: Can crawlers reach the file and the URLs it references?
Clarity: Can a non-expert understand the category, audience, use case, and differentiation in under 30 seconds?
Evidence: Are claims backed by proof, documentation, customer examples, pricing logic, or technical detail?
Conversion path: If a buyer clicks from an AI citation, does the landing page move them toward a useful next step?

This model matters because answer-engine visibility without conversion is just another vanity metric.

Step 1: Test the root file

Open the file directly:

https://yourdomain.com/llms.txt

Then test the www and non-www versions. Check whether the result is a 200 status code. If it redirects, document the redirect chain. A single clean redirect is usually acceptable for users, but every extra hop adds uncertainty.

Use a technical SEO crawler, server logs, or a simple command-line request to confirm headers:

curl -I https://yourdomain.com/llms.txt

Look for:

200 OK
content-type: text/plain or another plain text compatible response
No authentication requirement
No unexpected cache issue
No blocking response from the CDN

Step 2: Compare the file against your buyer journey

Do not ask whether the file includes enough URLs. Ask whether it includes the right URLs.

For a B2B SaaS company, map URLs to the buying journey:

Problem-aware: use case pages and pain-specific landing pages
Solution-aware: product pages and category pages
Comparison-stage: alternative pages and competitor comparison pages
Validation-stage: security, documentation, customer proof, integrations
Buying-stage: pricing, demo, sandbox, procurement pages

If the file skips validation-stage pages, AI answers may understand what you sell but not why buyers should trust you.

This is also why a product sandbox can be more than a conversion asset. A clear sandbox page gives buyers and answer engines structured evidence about what the product actually does. We covered this pattern in our guide to product sandbox UX.

Step 3: Run answer tests manually

Build a prompt set before you make changes. Use the same prompts every week for four to six weeks.

Example prompt set:

Best platforms for [specific use case]
Best alternatives to [known competitor]
Does [your brand] support [integration or workflow]?
Compare [your brand] vs [competitor]
What is [your brand] used for?
Who is [your brand] best for?
Is [your brand] enterprise-ready?
What tools help [buyer role] solve [pain]?
Which vendors offer [technical capability]?
What are the pricing considerations for [category]?

Record whether your brand appears, whether it is cited, which page is cited, and whether the answer is accurate. The baseline can be simple: 10 prompts, 0 to 10 brand mentions, 0 to 10 citations, and a short accuracy note for each answer.

Step 4: Inspect the pages you want cited

A page that deserves citation usually has these traits:

A clear definition of the product or concept
Specific audience and use case language
Comparison criteria
Proof points or examples
Stable URLs
Descriptive headings
Internal links to related evidence
A clear next step for the buyer

If your homepage reads like a positioning fog machine, do not expect an answer engine to infer the sharp version of your story. Brand trust matters here. For early-stage SaaS companies selling into enterprise buyers, trust often comes from specific visual and content cues, not decoration. We have broken that down in our guide to SaaS brand identity.

Fix Steps

Fix the file, then fix the site around it. That order matters.

Step 1: Put llms.txt at the root and keep it plain

Use the root path:

https://yourdomain.com/llms.txt

Keep it plain text. Do not require JavaScript. Do not hide it behind a consent wall. Do not serve different content based on location unless there is a real compliance reason.

A simple structure is better than a clever one:

# Company Name

> One-sentence description of what the company does, who it serves, and the outcome it helps buyers evaluate.

## Key Pages

- Homepage: https://example.com/

- Product: https://example.com/product/

- Pricing: https://example.com/pricing/

- Security: https://example.com/security/

- Documentation: https://example.com/docs/

- Comparisons: https://example.com/compare/

- Integrations: https://example.com/integrations/

## Best Pages for AI Answers

- What the product does: https://example.com/product/

- Who it is for: https://example.com/use-cases/

- How it compares: https://example.com/compare/

- Technical trust: https://example.com/security/

This is not an official guarantee of crawler behavior. It is a clean way to make your preferred sources obvious if the file is consumed.

Step 2: Remove low-value URLs

Do not fill the file with archive pages, thin blog posts, expired campaigns, or parameter-heavy URLs.

Bad candidates:

Tag archives
Internal search pages
Old launch posts
Duplicate campaign pages
Paginated blog indexes
Unmaintained feature pages

Better candidates:

Core product page
Use case pages with strong buyer language
Integration pages with technical detail
Pricing page with clear packaging logic
Comparison pages with honest criteria
Trust center or security page
Docs and API pages
Customer proof pages

Contrarian stance: do not use llms.txt to submit more content. Use it to point AI systems toward the pages that make your company easiest to understand and verify.

Step 3: Rewrite weak page sections for extraction

AI answer engines need extractable claims. That means your pages should include direct, structured answers.

Replace vague copy like:

Scale your operations with a modern platform built for growing teams.

With specific copy like:

Acme helps B2B customer success teams identify expansion risk by combining product usage data, support signals, and renewal history in one account health view.

The second version gives an answer engine category, audience, workflow, data sources, and use case.

Apply this to:

Homepage hero
Product overview
Use case introductions
Pricing explanations
Comparison tables
Integration pages
Security pages
Demo page copy

For Raze, this is where AI SEO, AEO, SaaS web design, and conversion-focused web design meet. A clear answer for AI is often the same clear answer a buyer needed before booking a demo.

Step 4: Add evidence pages, not just opinion pages

AI systems need sources they can cite. Buyers need proof they can trust.

Build or improve pages that answer these questions:

What does the product do?
Who is it best for?
What problems does it not solve?
How does pricing work?
How does it compare to alternatives?
What integrations does it support?
What security standards or controls matter?
What technical docs prove the capability?
What customer examples support the claim?
What next step should a buyer take?

A practical measurement plan looks like this:

Baseline: 10 priority prompts, current brand mentions, current citations, current cited URLs, and accuracy notes.

Intervention: update llms.txt, rewrite the homepage and three priority pages for extractable claims, add internal links to proof pages, and publish one comparison or trust asset.

Expected outcome: more accurate brand descriptions and stronger citation eligibility over four to six weeks, measured by the same prompt set and server log review. This is not a ranking guarantee. It is a controlled way to see whether the site is becoming easier to understand and cite.

Step 5: Align the click path after the citation

A citation is not the finish line. If the AI answer sends a high-intent buyer to a confusing page, the opportunity leaks.

Every cited page should have:

A clear above-the-fold answer
Supporting proof within one scroll
Internal links to deeper validation
A CTA matched to the buyer stage
Fast load performance
No unnecessary form friction

For example, a comparison-stage visitor should not land on a generic homepage and be forced to hunt. They should land on a page that answers the comparison, explains tradeoffs, links to docs or proof, and routes them to demo, sandbox, or pricing.

That is why Raze treats answer-engine visibility as part of the revenue website, not a detached SEO task.

How to Verify the Fix

Verification should happen across the file, the crawl path, the answer output, and the conversion path.

Confirm technical access

Check the file after deployment:

Root URL returns 200
File is plain text
Important URLs inside the file return 200
Canonical tags match intended URLs
Robots rules do not block key pages
CDN rules do not block common crawler behavior
Internal links support the same URL priorities

If you use WordPress and Yoast, confirm the setting remains active after plugin updates. The Yoast help page is the source to use for the current UI path.

Re-run the same answer prompts

Do not change the prompt set every week. That makes the data useless.

Track:

Brand mention rate
Citation rate
Citation URL
Accuracy of product description
Competitor presence
Click-through behavior where measurable
Demo, sandbox, pricing, or contact engagement from cited pages

If the answer improves from inaccurate to accurate but citations do not increase, that is still progress. Accuracy often comes before visibility.

Review logs and analytics carefully

Server logs can show whether known crawlers request llms.txt, but absence of evidence does not prove failure. Some AI systems may use search indexes, licensed data, browser retrieval, or other retrieval paths.

Google Search Central discussions, including this support thread on not deploying llms.txt, also reflect the broader uncertainty around the file’s current importance for technical SEO governance.

The practical move is to measure more than file hits. Measure whether AI answers become more accurate, whether better pages are cited, and whether cited visitors find a strong conversion path.

When to Escalate

Escalate when the issue is no longer a simple file problem.

Escalate to technical SEO or engineering when access is inconsistent

Bring in technical support if:

The file returns different responses by user agent
CDN or firewall rules are blocking unknown bots
Important pages are blocked by robots rules
Canonicals are inconsistent
The site relies heavily on client-side rendering
Redirect chains are messy
The CMS keeps overwriting the file

For SaaS teams using Next.js, headless CMS setups, reverse proxies, or internationalized routing, the issue may sit in deployment architecture. This is where an embedded design and growth team needs technical credibility, not just content advice.

Escalate to positioning and conversion work when AI describes you poorly

If AI tools can access your pages but still summarize you badly, the site is sending weak signals.

That usually means you need:

Homepage repositioning
Product narrative cleanup
Better use case architecture
Comparison pages
Pricing clarity
Technical trust assets
Stronger internal linking
More extractable proof

This is where Raze fits. As a SaaS web design agency, B2B SaaS design agency, AI SEO agency, AEO agency, and conversion-focused web design partner, Raze helps technical companies turn unclear websites into clearer sales arguments for both buyers and answer engines.

The best marketing sites reduce buyer effort before sales ever gets involved. AI search rewards many of the same traits: clarity, specificity, trust, and evidence.

FAQ

Why is llms.txt not working even though the file is live?

Because live does not mean consumed, trusted, or cited. Research from SE Ranking found no current correlation between publishing llms.txt and higher AI citation frequency across its 300,000-domain analysis.

Does my site need an llms.txt file in 2026?

It can be useful as a low-effort governance and discovery signal, but it should not be your main AI visibility strategy. The higher-impact work is making your site crawlable, specific, verifiable, and easy to cite.

What should go inside an llms.txt file?

Include the pages that best explain your company, product, use cases, pricing, integrations, comparisons, documentation, and trust signals. Avoid dumping every blog post or archive URL into the file.

Can Yoast SEO generate llms.txt automatically?

Yoast provides an llms.txt feature for WordPress sites, and its documentation says users can enable it under Settings > General > Site features in the API section. Plugin output should still be reviewed because automatic generation does not replace buyer-focused page prioritization.

Will llms.txt help me get cited by Perplexity or ChatGPT?

It may help if those systems or their retrieval layers use the file, but there is no reliable guarantee. Better citation eligibility comes from clear pages, extractable claims, trusted evidence, crawl access, and consistent positioning across the site.

What should I fix first if AI answers describe my product incorrectly?

Start with your homepage, product page, and highest-intent use case pages. Rewrite them so a buyer and an answer engine can quickly identify your category, audience, core use case, proof, and next step.

If your AI visibility problem is really a positioning, page architecture, or conversion problem, book a working session with Raze and we will help you find the leak.