# Sitemap and robots.txt

SEO and AI visibility

A **sitemap** lists the pages of your store so search engines find them all; **robots.txt** tells crawlers what they may read. GrandNode provides both, per store.

### What the sitemap contains

In **Settings → General and common settings**, tab **Store information**:

![The sitemap settings: Sitemap enabled, include subcategories, include pictures, include brands and include products in sitemap](https://grandnode.com/Plugins/Theme.GrandNodeCom/Content/kb/seo/settings-sitemap.webp)

Sitemap settings.

- **Sitemap enabled** - shows the sitemap link and page.
- **Include subcategories / brands / products / pictures in sitemap** - what goes into the sitemap besides the main pages. Turn on products: they are the pages you most want found.

### sitemap.xml

The `sitemap.xml` file is written by the scheduled task **Generate sitemap XML file**. The task is not enabled after installation: open **System → Maintenance → Schedule tasks**, enable it (once a day is enough) or click **Run now**. Until it has run, `/sitemap.xml` returns "not found".

![The generated sitemap.xml opened in a browser, listing URLs with changefreq and lastmod](https://grandnode.com/Plugins/Theme.GrandNodeCom/Content/kb/seo/sitemap-xml.webp)

The generated sitemap.xml.

Submit `https://your-store/sitemap.xml` in Google Search Console and Bing Webmaster Tools. `/sitemap` (without .xml) is a different thing: a sitemap page for people.

With several instances (pods), the task runs on one of them. Set `Application:MediaPath` to a volume shared by all instances, so every instance serves the same `sitemap*.xml` - see the [application settings reference](https://grandnode.com/system-appsettings#media-path). On a multi-store or multi-language installation the files are named `sitemap-{store}-{language}.xml`; point robots.txt and Search Console at the right file for each store.

### robots.txt

Each store has its own robots.txt, edited under **System → Maintenance → Robot.txt**. The default points to the sitemap and keeps crawlers out of the admin, cart, checkout, account and other pages that should not be indexed.

![The Robots.txt editor with the default rules: User-agent, Sitemap, Host and many Disallow lines for admin, cart, checkout and account pages](https://grandnode.com/Plugins/Theme.GrandNodeCom/Content/kb/seo/robots-editor.webp)

System → Maintenance → Robot.txt.

#### Rules for AI crawlers

AI companies use their own crawlers. Some collect training data, others fetch pages to answer a user's question right now - blocking the latter makes your products invisible in AI answers. Add rules by crawler name, for example:

```
# allow assistants that fetch pages to answer users
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# no training on our content
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /
```

Check each company's documentation for the current names; they change.

### Tips

- Never disallow product or category pages "to save crawl budget"; disallowed pages cannot appear in results.
- After editing robots.txt, open `/robots.txt` on the store to check what crawlers actually get.

### Related

- [Scheduled tasks](https://grandnode.com/system-scheduled-tasks)
- [Visibility in AI assistants](https://grandnode.com/seo-ai-visibility)

- [Web page](https://grandnode.com/seo-sitemap-and-robots)