Get GrandNode

Sitemap and robots.txt

A sitemap lists the pages of your store so search engines find them all; robots.txt tells crawlers what they may read. GrandNode provides both, per store.

What the sitemap contains

In Settings → General and common settings, tab Store information:

The sitemap settings: Sitemap enabled, include subcategories, include pictures, include brands and include products in sitemap
Sitemap settings.
  • Sitemap enabled - shows the sitemap link and page.
  • Include subcategories / brands / products / pictures in sitemap - what goes into the sitemap besides the main pages. Turn on products: they are the pages you most want found.

sitemap.xml

The sitemap.xml file is written by the scheduled task Generate sitemap XML file. The task is not enabled after installation: open System → Maintenance → Schedule tasks, enable it (once a day is enough) or click Run now. Until it has run, /sitemap.xml returns "not found".

The generated sitemap.xml opened in a browser, listing URLs with changefreq and lastmod
The generated sitemap.xml.

Submit https://your-store/sitemap.xml in Google Search Console and Bing Webmaster Tools. /sitemap (without .xml) is a different thing: a sitemap page for people.

With several instances (pods), the task runs on one of them. Set Application:MediaPath to a volume shared by all instances, so every instance serves the same sitemap*.xml - see the application settings reference. On a multi-store or multi-language installation the files are named sitemap-{store}-{language}.xml; point robots.txt and Search Console at the right file for each store.

robots.txt

Each store has its own robots.txt, edited under System → Maintenance → Robot.txt. The default points to the sitemap and keeps crawlers out of the admin, cart, checkout, account and other pages that should not be indexed.

The Robots.txt editor with the default rules: User-agent, Sitemap, Host and many Disallow lines for admin, cart, checkout and account pages
System → Maintenance → Robot.txt.

Rules for AI crawlers

AI companies use their own crawlers. Some collect training data, others fetch pages to answer a user's question right now - blocking the latter makes your products invisible in AI answers. Add rules by crawler name, for example:

# allow assistants that fetch pages to answer users
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# no training on our content
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Check each company's documentation for the current names; they change.

Tips

  • Never disallow product or category pages "to save crawl budget"; disallowed pages cannot appear in results.
  • After editing robots.txt, open /robots.txt on the store to check what crawlers actually get.

Question not answered here? Ask the community on GitHub Discussions ↗