# Ekonomidata.nu robots.txt # https://www.robotstxt.org # One shared group for every crawler. # # IMPORTANT: robots.txt groups are NOT additive. A crawler obeys only the # single most-specific User-agent group that matches its name (RFC 9309), # and ignores every other group — including "User-agent: *". So a named # "User-agent: Googlebot / Allow: /" group with no Disallow lines would make # that bot IGNORE all the Disallow rules below (which live in this "*" group), # letting it crawl /admin, /api/v1/, etc. We therefore keep a single # wildcard group instead of per-bot groups. # # All crawlers are welcome under these rules, including: # - Search engines: Googlebot, Bingbot # - AI answer engines: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, # Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, CCBot, # meta-externalagent # Ekonomidata.nu wants to be a citable source for AI assistants; the shared # "Allow: /" below welcomes them while still honoring the Disallow rules. User-agent: * Allow: / # Disallow admin and internal paths Disallow: /admin Disallow: /admin/ Disallow: /sidekiq Disallow: /sidekiq/ Disallow: /rails/ Disallow: /up # Disallow API broadly, then allow specific public endpoints Disallow: /api/v1/ Allow: /api/v1/articles Allow: /api/v1/statistics Allow: /api/v1/key_indicators Allow: /api/v1/subject_areas # DebtClockSidebar fetches its figures from here on statistic, budget, # polling and valresultat pages. Allow: /api/v1/debt_clock/sidebar # Block query parameter variations to prevent duplicate content Disallow: /*?*sort= Disallow: /*?*order= # Sitemap location Sitemap: https://ekonomidata.nu/sitemap.xml