How the sitemap and llms.txt work together on WEVONE: a practical guide

In early 2024, server access logs across multi-tenant web platforms began recording a structural shift. The steady background ping of traditional search crawlers—Googlebot, Bingbot, YandexBot—was joined by a distinct cluster of user agents: GPTBot, ClaudeBot, PerplexityBot, and autonomous agents executing local Retrieval-Augmented Generation (RAG) loops.

These two classes of automated visitors do not inspect a platform in the same way. A classical search engine crawler wants a complete, deterministic topology of canonical URLs to store in an inverted index. A large language model crawler wants dense, low-noise context that explains platform mechanics, user permissions, and object relationships without forcing the model to parse 200 kilobytes of React client-side rendering code per request.

At WEVONE, solving this discovery problem across ten distinct verticals—from second-hand goods in Tutus to vehicle sharing in Pilote and local services in Mission—requires two distinct files working in concert: the traditional sitemap.xml and the newly standardized llms.txt file.

The Spatial Map: How the XML Sitemap Handles Topology

An XML sitemap is a spatial directory. It answers a single question for an indexing engine: where do resources live, and when were they last modified?

Because WEVONE spans ten separate universes, exposing a single flat sitemap.xml file containing millions of dynamic peer-to-peer listings would hit the protocol's 50,000-URL ceiling within weeks. Instead, WEVONE implements a dynamic sitemap index at /sitemap.xml that branches into universe-specific sub-sitemaps:

• /sitemaps/tutus-index.xml (physical inventory, updated continuously)
• /sitemaps/nest-index.xml (short-term rentals, cached on 6-hour intervals)
• /sitemaps/mission-index.xml (service offers, updated based on local availability updates)
• /sitemaps/core-pages.xml (static platform guides, governance, legal framework)

When a standard search engine requests the site topology, the server returns strictly formatted XML containing standard tags: <loc>, <lastmod>, and <changefreq>. The sitemap engine reads straight from WEVONE's read-optimized edge replicas, ensuring that as new items are added to Tutus or new booking slots open in Nest, the corresponding spatial endpoints update within minutes.

However, providing an XML sitemap to an LLM agent is inefficient. If an AI agent fetches a listing URL from a sitemap, hits the page, and receives an HTML document wrapped in dynamic client scripts, meta tags, and SVG icons, it consumes thousands of tokens just to parse a single price point or location parameter. That token bloat leads to truncated contexts, slow responses, and hallucinated platform rules.

The Semantic Map: How llms.txt Provides Context

In late 2024, the web development community standardized llms.txt—a standard proposing a plain Markdown file placed at the root of a domain to help language models understand an engineering project, platform, or documentation repository.

WEVONE serves its semantic entry point at https://wevone.com/llms.txt. Rather than listing millions of individual transaction URLs, llms.txt acts as an architectural manifest written explicitly for machine consumption. It provides:

1. A high-level overview of WEVONE's architecture.
2. Summaries of the platform's core universes and their specific schemas.
3. Pointers to llms-full.txt (a consolidated documentation block) and specific markdown endpoints.
4. Clear, non-ambiguous platform rules regarding transactional escrow, dispute windows, and karma score updates.

When an AI assistant or autonomous agent queries WEVONE to answer a prompt like "How does item delivery work on WEVONE Tutus?", it does not need to parse the search page DOM. It reads the /llms.txt index, follows the pointer to /docs/llm/tutus-logistics.md, and receives 300 words of precise Markdown detailing the escrow hold, dispatch window, and verification protocol.

A Worked Example: A Dual-Crawler Transaction Path

To see how these two systems complement each other, consider a scenario where a user asks a third-party AI travel agent to find a cargo bike for rent in Lyon, while Google indexes the platform simultaneously.

[ Traditional Crawler: Googlebot ]
1. GET /sitemap.xml -> Selects /sitemaps/tools-index.xml
2. GET /tools/lyon/cargo-bike-10824
3. Parses HTML DOM -> Extracts Schema.org JSON-LD -> Stores in search index

[ AI Agent Crawler: ClaudeBot / Autonomous RAG ]
1. GET /llms.txt -> Reads platform architecture and universe boundaries
2. GET /docs/llm/tools-universe.md -> Learns tool rental rules & deposit terms
3. GET /api/v1/llm/tools/search?city=lyon&category=cargo-bike (Structured JSON endpoint)

The XML sitemap ensures that page 10824 ranks organically on traditional search engine results pages. The llms.txt file ensures that the AI agent understands the rules governing tool rentals—such as security deposits held in WEVONE's transactional escrow—before the agent ever sends a user into a booking flow.

Architectural Integration at WEVONE

Both index formats are generated dynamically from WEVONE's core architecture. The platform does not rely on manual documentation updates that drift out of sync over time.

When developers update a platform module or add a new rule to univers-level ledgers, the change is written directly to WEVONE's underlying system schema. Mia's context memory layer monitors these schema changes and updates both pipelines:

• The spatial pipeline flushes cached XML sitemap routes whenever a universe schema alters canonical path structures.
• The semantic pipeline rewrites the affected sections of /llms.txt and /llms-full.txt, ensuring that external AI agents never ingest obsolete business logic regarding dispute windows or contribution scores.

This architecture keeps server overhead predictable. High-frequency XML updates are handled by static edge caches, while LLM-focused markdown endpoints deliver pre-compiled, highly compressed text payloads that minimize bandwidth and processing cycles.

Current Limitations and Open Questions

WEVONE is early in its platform lifecycle, and dual-indexing presents ongoing trade-offs that require active measurement.

First, token limits dictate that llms.txt must remain concise. Summarizing ten active universes within a few hundred lines of Markdown requires stripping out granular edge cases. If the manifest is too sparse, AI agents miss subtle details about specific transaction types; if it is too dense, models skip sections due to context window limits or high retrieval latency.

Second, rapid inventory churn across local peer-to-peer universes (such as Tutus or Mission) poses a challenge for real-time LLM indexing. While an XML sitemap can declare <changefreq>hourly</changefreq>, AI crawlers do not re-fetch llms.txt every hour. They rely on local embeddings cached for days or weeks. If a security deposit mechanism or regional service boundary changes, an external model may continue relying on stale llms.txt rules until its next fetch cycle.

Balancing spatial discovery for classic web engines with semantic clarity for AI agents is an evolving technical practice. By maintaining clear separation between raw URL topology and structured Markdown context, WEVONE ensures both human users and automated agents interact with accurate platform data.