Robots.txt and sitemap.xml: correct 2026 SME setup

Robots.txt and sitemap.xml are the SEO "service files" most SMEs never check. Result: Google doesn't discover important pages or wastes crawl budget on useless ones. Correct setup takes 30 minutes.

Robots.txt: the file telling what NOT to crawl

Located at https://yourdomain.com/robots.txt. Standard example:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /private/
Disallow: /*?orderby=
Allow: /wp-admin/admin-ajax.php

Sitemap: https://gamerbit.it/sitemap.xml

What NOT to mistakenly disallow

• Never disallow / (blocks entire site)
• Never disallow CSS/JS (Google needs them for render)
• Never disallow images if you want them indexed (Google Images)
• Never use Disallow: /* with generic asterisk

Sitemap.xml: the map telling what exists

WordPress: SEO plugins (Yoast, Rank Math) generate automatically. Custom: generate dynamically including only URLs with 200 status.

Best practice:

• One main sitemap linking to secondary sitemaps (posts, pages, products)
• Max 50,000 URLs per sitemap (Google limit)
• Include <lastmod> last modified date
• Do NOT include URLs with noindex, redirect, 404 status

Submit to Google Search Console

1. GSC → Sitemap → "Add new sitemap"
2. Enter full URL (e.g. sitemap.xml)
3. Check errors after 24-48h

Properly configured robots.txt and sitemap give +30-50% indexing on large sites in 60 days. Included in Grow package €390/month.

Want to apply this to your SME?

If you'd like a concrete analysis of your situation, request the free audit from Gamerbit. In 48 hours we tell you exactly where you're losing leads, time or budget — no sales pitch.

For a complete system, the 2026 no-brainer offers are the most direct starting point.