How the sitemap and llms.txt work together on WEVONE: a practical guide

A Googlebot crawler issues a GET /sitemap.xml request and receives a compressed XML feed containing 85,000 canonical URLs broken down by universe namespaces. Four hundred milliseconds later, an autonomous purchasing agent built on Anthropic's Claude requests GET /llms.txt to determine how transactional escrow operates on WEVONE Mission. Both systems seek to understand the structure of the exact same marketplace platform, yet their data consumption requirements are fundamentally inverted.

Traditional search engine crawlers require exhaustive, low-overhead path listings to discover pages, check modifications via HTTP headers, and assign page rank metrics. Large language models (LLMs) and retrieval-augmented generation (RAG) pipelines require structured, context-dense markdown summaries that fit cleanly into context windows without wasting token budgets on navigation chrome, visual layout code, or redundant link farms.

At WEVONE, we treat these two files not as redundant metadata, but as dual interfaces to our platform architecture. Here is how we build, publish, and maintain them together.

The Indexing Divide: Navigation vs Context

For two decades, web indexability was governed almost exclusively by the Sitemaps XML protocol. A sitemap tells a web crawler three primary facts: that a URL exists, when it was last modified, and where it sits within a site's visual or taxonomic path structure.

When an LLM attempts to crawl a site using only a traditional sitemap, it encounters a structural bottleneck. Processing thousands of deep URLs like /tutus/vintage-jackets/item-908231 forces the model to fetch individual HTML documents. Each page carries megabytes of DOM bloat, client-side scripts, and UI chrome. Extracting platform mechanics—such as how dispute resolution works or how shipping costs are calculated—requires parsing hundreds of unstructured pages and synthesizing the common rules manually.

This is where llms.txt steps in. Proposed as a standardized plain-text standard for AI agent discovery, /llms.txt provides a human-readable and machine-parsable map of a platform's core mechanics, primary endpoints, and domain boundaries. If the sitemap is a spatial survey of every leaf on a tree, llms.txt is the direct schematic of the tree's internal plumbing.

Anatomical Comparison: A WEVONE Tutus Listing

To see how these two standards complement each other, consider how WEVONE exposes a second-hand clothing listing on our Tutus universe.

The XML sitemap entry prioritizes route resolution and delta tracking:

<url>
  <loc>https://wevone.com/tutus/listings/wool-overcoat-4029</loc>
  <lastmod>2025-02-14T08:30:00Z</lastmod>
  <changefreq>daily</changefreq>
  <priority>0.8</priority>
</url>

The entry in llms.txt presents the operational parameters of the Tutus universe itself, allowing an AI model to evaluate whether and how to interact with that listing:

## Tutus Universe (Second-Hand Fashion)
- Operational Status: Live (EU-wide)
- Escrow Mechanism: Platform-managed, 48-hour post-delivery inspection window.
- Item Listing Schema: https://wevone.com/docs/api/schemas/tutus-item.json
- Core Route: /tutus/listings/[slug]
- Key Rules: Sellers must ship within 5 business days using WEVONE pre-paid labels. Condition grading follows WEVONE Standard v2.1.

The crawler uses the XML line to schedule its next indexing pass over that specific jacket listing. The LLM uses the llms.txt section to answer a user's prompt: "Can I return a coat bought on WEVONE Tutus if it doesn't fit?" without having to scrape 50 separate listing pages to deduce the 48-hour inspection policy.

How WEVONE's Context Engine Compiles Both Artifacts

Maintaining two separate metadata feeds by hand inevitably leads to drift. If an engineering team updates the escrow release window on WEVONE Nest from 24 hours to 48 hours, updating the documentation without updating llms.txt causes language models to hallucinate outdated platform guarantees.

WEVONE handles this via Mia's context memory ingest engine. Mia—our internal AI infrastructure layer—monitors changes across platform logic, route registries, and service level agreements.

When a service deployment modifies platform routes or transaction rules, the build pipeline triggers two parallel sync tasks:

1. The Sitemap Generator runs against PostgreSQL read-replicas, harvesting active canonical routes across all ten universes (Tutus, Nest, Mission, Event, Pilote, Pet, Trust, Skills, Tools, Invest). It generates partitioned XML files divided by universe namespace, capped at 50,000 URLs per file per the protocol spec.
2. The LLM Manifest Compiler extracts high-level universe declarations, API specs, and operational parameters from WEVONE's central domain service registry. It parses raw documentation markdown, evaluates token counts using the OpenAI tiktoken tokenizer, and outputs both a condensed /llms.txt (target size: under 4,000 tokens) and an extended /llms-full.txt (target size: under 30,000 tokens).

Because both feeds compile from the exact same source code annotations and domain definitions, structural changes in our routing table instantly propagate to the XML sitemap, while operational changes in our dispute windows propagate directly to llms.txt.

The Freshness Dilemma: Real-Time State vs Static Indexing

Neither XML sitemaps nor llms.txt solve the challenge of high-velocity dynamic state. A short-term rental property on WEVONE Nest may be booked for three specific nights in July while a web crawler is midway through reading the sitemap.

We explicitly distinguish between static platform rules and dynamic instance state.

• Static Rules (How Nest security deposits are handled, what currency conversions apply, how host identity is verified) are baked into /llms.txt.
• Dynamic Routes (Individual property pages, user-generated event listings) live in the XML sitemap hierarchy.
• Real-Time State (Instant availability, exact pricing calculations, current driver positions in WEVONE Pilote) is explicitly excluded from both. Instead, llms.txt provides direct OpenAPI endpoints that models can query for live state once they understand the platform's core mechanics.

An LLM reading our /llms.txt learns how to call GET /api/v1/nest/availability?property_id=X. It does not rely on static markdown files to guess whether property X is free tonight.

Honest Limitations and Open Questions

The dual-index approach is not without friction. Token budgets remain a strict constraint. As WEVONE expands its active features across all ten universes, keeping /llms.txt concise enough to fit inside small system prompt windows without dropping critical policy nuances requires aggressive prioritization.

Furthermore, while major search engines strictly respect XML sitemap directives, third-party AI scraper bots vary wildly in how they digest llms.txt. Some models read the manifest, follow linked references, and cache the system boundaries correctly. Others ignore /llms.txt entirely and continue executing brute-force HTML scrapes that burn server bandwidth and ingest raw layout noise.

Until web standards bodies formally codify llms.txt alongside robots.txt and sitemap.xml, maintaining these parallel discovery paths remains a necessary operational tax for platforms operating at scale.

Implementation Summary for Marketplace Architecture

If you are engineering search and model discovery for a multi-faceted platform, avoid forcing one standard to perform the job of the other. Use XML sitemaps to map path existence, update frequencies, and complete domain breadth for deterministic web indexers. Use llms.txt to provide concise, token-efficient, semantic operational rules for language models and autonomous agents.

Keeping both tied to a single source of truth in your domain codebase prevents factual drift, reduces scraping overhead, and ensures that both human-directed search engines and autonomous AI buyers see the exact same underlying operational reality.