# robots.txt for the OAP alias host (policy: SITE-SPEED-AND-CACHE-PLAN-20261030 section 5.2, step 1). # Internal search is not for crawlers: its result pages are noindex and cost PHP time on every hit. # Bulk metadata for AI and research use: see https://openaccesspub.org/llms.txt # Everyone not named below. Googlebot and bingbot have their own groups, so the search block can be added for them in step 2. User-agent: * Allow: / Disallow: /_* Disallow: /2020 Disallow: /ankita Disallow: /2019 Disallow: /support Disallow: /cdn-cgi/ Disallow: /qdpm/ Disallow: /admin Disallow: /mail/ Disallow: /my/ Disallow: /admin2/ Disallow: /journal-preview Disallow: /journal_html/ Disallow: /open-access-journals-search Disallow: /article/searchquery Disallow: /Article/searchquery Disallow: /article/journal_search # STEP 2 (not yet): add the search lines here only after the Search Console Pages report shows no indexed search URLs. # A block hides the noindex tag, so an already indexed search URL could stay in results without content. User-agent: Googlebot User-agent: Googlebot-IA User-agent: bingbot Allow: / Disallow: /_* Disallow: /2020 Disallow: /ankita Disallow: /2019 Disallow: /support Disallow: /cdn-cgi/ Disallow: /qdpm/ Disallow: /admin Disallow: /mail/ Disallow: /my/ Disallow: /admin2/ Disallow: /journal-preview Disallow: /journal_html/ User-agent: Applebot Allow: / Disallow: /_* Disallow: /2020 Disallow: /ankita Disallow: /2019 Disallow: /support Disallow: /cdn-cgi/ Disallow: /qdpm/ Disallow: /admin Disallow: /mail/ Disallow: /my/ Disallow: /admin2/ Disallow: /journal-preview Disallow: /journal_html/ Disallow: /open-access-journals-search Disallow: /article/searchquery Disallow: /Article/searchquery Disallow: /article/journal_search # Alias hosts: AI crawlers and SEO tools are asked to use https://openaccesspub.org instead (every alias page names it as canonical). User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: GPTBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: ClaudeBot User-agent: Perplexity-User User-agent: PerplexityBot User-agent: Google-Extended User-agent: meta-externalagent User-agent: FacebookBot User-agent: Amazonbot User-agent: CCBot User-agent: ElevenlabsCrawler User-agent: ElevenlabsBot User-agent: YouBot User-agent: Exa User-agent: Bytespider User-agent: AhrefsBot User-agent: SemrushBot User-agent: SiteAuditBot Disallow: / # SEO and link-index crawlers: not wanted. User-agent: MJ12bot User-agent: DotBot User-agent: serpstatbot User-agent: SERankingBacklinksBot User-agent: DataForSeoBot User-agent: Barkrowler User-agent: BLEXBot User-agent: SEOkicks User-agent: SEOkicks-Robot Disallow: / Sitemap: https://openaccesspub.org/sitemap.xml