CrawlBrief for LLMs.txt

CrawlBrief for LLMs.txt is a WordPress llms.txt generator that creates a structured /llms.txt file and optional Markdown versions of public WordPress content for AI assistants, AI agents and large language models.

The plugin supports the llms.txt v2 proposal, including rel="describedby" discovery for /llms.txt, rel="alternate" type="text/markdown" links for eligible WordPress content, and matching HTTP Link metadata on Markdown responses.

CrawlBrief combines automatic generation with editorial control. Instead of publishing an uncontrolled list of every URL, you can choose the pages that matter, organize important resources, edit descriptions, feature or exclude content, and provide clean Markdown representations that compatible AI tools can retrieve when needed.

WordPress llms.txt Generator

CrawlBrief automatically publishes:

• /llms.txt – a structured, LLM-friendly overview of your WordPress website.
• /index.html.md – a Markdown representation of the homepage.
• /example-post/index.html.md – clean Markdown versions of eligible posts, pages and supported public post types.

The endpoints are served dynamically through WordPress. CrawlBrief does not create or overwrite physical llms.txt or Markdown files on your server.

llms.txt v2 Support

CrawlBrief includes discovery features from the current llms.txt v2 proposal:

• Adds rel="describedby" so public pages can identify the applicable /llms.txt file.
• Adds rel="alternate" type="text/markdown" for pages with a Markdown representation.
• Sends matching discovery information through HTTP Link headers on Markdown responses.
• Keeps the v2-compatible index.html.md URL format for WordPress URLs without filenames.
• Links directly to clean Markdown resources from generated llms.txt sections when available.

Automatic and Manual Page Selection

Choose how CrawlBrief builds the important page sections of your llms.txt file:

• Automatic – detect important WordPress pages automatically.
• Automatic + Custom – use automatic detection and add your own selected WordPress resources.
• Manual – control the selected resources yourself.

CrawlBrief can detect common site pages such as:

• Homepage
• About page
• Contact page
• Privacy Policy
• Terms page
• WordPress posts page
• WooCommerce shop page when available

Automatically detected resources can be excluded individually.

Searchable WordPress Resource Selection

Add important WordPress pages using searchable selectors instead of manually copying URLs.

Selected resources are stored by WordPress post ID, so changing a page slug or permalink later does not break the corresponding llms.txt entry.

You can create separate selections for:

• Start Here resources
• Optional resources

Additional external URLs, archives and documentation hosted outside WordPress can still be added through the Advanced additional-links section.

Editable llms.txt Descriptions

CrawlBrief lets you control how important resources are described to AI systems.

You can:

• Override descriptions for automatically detected pages.
• Add description overrides to selected Start Here resources.
• Add description overrides to selected Optional resources.
• Set a dedicated llms.txt description on individual posts or pages.

If no override is supplied, CrawlBrief can fall back to the available WordPress description, excerpt, content-derived text or built-in role description.

Editorial Control

Each supported WordPress resource can use CrawlBrief’s editorial controls:

• Automatic – let CrawlBrief include the resource normally.
• Featured – prioritize the resource in a dedicated Featured Resources section.
• Excluded – prevent the resource from appearing in generated CrawlBrief sections.

This makes CrawlBrief useful for curated llms.txt files rather than simple URL dumps.

Structured llms.txt Sections

Depending on your settings and content, CrawlBrief can generate sections for:

• Start Here
• Featured Resources
• WordPress categories
• Public post types
• Pages
• Site Indexes
• Optional resources

You can configure the number of links per section, choose supported public post types, exclude selected category slugs, include the WordPress sitemap, and control optional sections.

CrawlBrief also prevents curated Optional resources from being duplicated in general page sections.

Clean Markdown for AI Assistants

Optional Markdown representations remove much of the surrounding theme and navigation markup and expose useful page content in a simpler text format.

Markdown output can contain:

• Page or post title
• Short description
• Canonical URL
• Author
• Last-updated date
• Featured image
• Headings
• Paragraphs
• Lists
• Links
• Main page content

If a static WordPress homepage has no stored page body, CrawlBrief can use the configured site summary and detailed context so the homepage Markdown endpoint is still useful.

Caching and Automatic Regeneration

CrawlBrief includes an optional generated-output cache to reduce repeated database work.

It can:

• Cache generated llms.txt output.
• Regenerate cached content on a daily schedule.
• Clear generated output when relevant WordPress content changes.
• Invalidate stale output when CrawlBrief is upgraded.
• Clear cached output when settings are saved.
• Integrate with common WordPress caching plugins when available.

Privacy and Lightweight Operation

CrawlBrief performs generation inside your WordPress installation.

• No external API is required.
• No account is required.
• No visitor analytics are collected by CrawlBrief.
• Site content is not automatically sent to LLMs.txt Lab or another third-party service.
• No public footer credit or backlink is added.
• No physical llms.txt or Markdown files need to be written to the server.

Who Is CrawlBrief For?

CrawlBrief is useful for WordPress sites that want a maintained llms.txt file and clean Markdown content for AI assistants, AI agents, retrieval tools and other systems that support the llms.txt convention.

It can be used on:

• Blogs
• Educational websites
• Documentation sites
• Business websites
• News and publishing sites
• WooCommerce stores
• Knowledge bases
• Membership and learning websites
• Sites using public custom post types

What Is llms.txt?

llms.txt is an emerging Markdown-based convention for giving AI systems a concise overview of a website and links to important machine-readable resources.

It is complementary to existing web standards such as robots.txt and XML sitemaps. CrawlBrief focuses on helping compatible AI assistants and agents discover and retrieve useful site content; it does not promise higher Google rankings or guaranteed inclusion in AI-generated answers.

Works With AI Assistants and Agents

CrawlBrief produces standard web and Markdown resources that can be accessed by compatible AI assistants, agents and retrieval tools, including tools associated with ecosystems such as ChatGPT, Claude, Perplexity and other large language model applications.

Actual use of llms.txt depends on the individual AI system or crawler.

Optional LLMs.txt Lab Tools

CrawlBrief generates and manages llms.txt inside WordPress. For additional checks or manual workflows, LLMs.txt Lab provides optional browser-based tools:

• Validate llms.txt – check an existing llms.txt file or URL.
• Generate llms.txt – manually create an llms.txt file when needed.
• LLMs.txt Lab – access additional llms.txt tools and resources.

These tools are optional. CrawlBrief does not automatically send site content or generated llms.txt data to LLMs.txt Lab.

How to Verify the Plugin

After activation, open Settings > CrawlBrief for LLMs.txt. The settings page shows verification URLs for your website.

To check the generated llms.txt file, visit:

https://example.com/llms.txt

To check the homepage Markdown endpoint, visit:

https://example.com/index.html.md

To check a post or page Markdown endpoint, use the existing CrawlBrief pattern:

https://example.com/example-post/index.html.md

The /llms.txt URL should load as plain text. A post or page Markdown URL should load clean Markdown content with the title, canonical URL, last-updated date, author when available, and page body. The homepage Markdown endpoint contains the available homepage or configured site information.

For llms.txt v2 discovery, view the source of a public HTML page. CrawlBrief outputs:

<link rel="describedby" href="https://example.com/llms.txt">

For an eligible post or page with Markdown representations enabled, it also outputs:

<link rel="alternate" type="text/markdown" href="https://example.com/example-post/index.html.md">

Markdown endpoint responses also advertise the covering llms.txt file and canonical HTML page through the HTTP Link header.

LLMs.txt v2 Compatibility

CrawlBrief 1.2.5 follows the llms.txt v2 discovery model while retaining backward compatibility with existing CrawlBrief installations.

• /llms.txt remains at the site root and covers the site’s public URLs.
• Existing /index.html.md and /post/index.html.md representations are retained. llms.txt v2 explicitly permits index.html.md for URLs without file names.
• Agents can discover Markdown representations through rel=”alternate” type=”text/markdown” rather than guessing URL transformations.
• Public HTML pages point to the covering llms.txt with rel=”describedby”.
• Markdown responses expose the same relationship through an HTTP Link header.
• The Optional section remains available as a convention for secondary resources that agents may skip when a shorter context is needed; it is not treated as a special processing instruction.
• Category and custom-post-type archive links remain normal HTML links unless CrawlBrief can actually serve a Markdown representation for the URL.

The llms.txt proposal remains an evolving community specification. See llmstxt.org for the current proposal.

Cache Compatibility

The plugin clears its internal transient cache when configured content changes. It also attempts to purge common caching plugins when available, including LiteSpeed Cache, WP Rocket, W3 Total Cache, WP Super Cache and SiteGround Optimizer.

Other caching plugins can hook into:

crwbrf_txt_cache_purged

Privacy

This plugin does not collect analytics, track visitors or automatically send generated llms.txt content or site data to a third-party service. All generation happens inside WordPress using public site content and settings saved by the site administrator. Links to optional external tools, including LLMs.txt Lab, are opened only when an administrator chooses to visit them. CrawlBrief does not automatically transmit site content or generated llms.txt data through those links.