best digital marketing agency
Let's Talk

Helping isn’t a choice.
It’s in our code.

A screenshot of a website with a form and text, likely a survey or questionnaire. The purpose of the image is to provide

Beware of the Better Robots.txt Plugin – Free Version

During a routine SEO audit, our team discovered that the free version of the Better Robots.txt plugin (v3) on WordPress websites was blocking every major AI crawler — ChatGPT, Perplexity, OpenAI’s search bot and Google-Extended — from an entire website. Worse, the setting that is supposed to control this is overridden inside the plugin’s own code, so changing it in the dashboard does nothing. If you use this plugin, check your robots.txt today.

What we found

While auditing a website’s crawlability, we noticed its AI search visibility had collapsed. The cause was sitting in robots.txt: a large “AI Bot Restrictions” block disallowing GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, Google-Extended, ClaudeBot and around thirty other crawlers — each with Disallow: /. That block was generated automatically by the Better Robots.txt plugin’s v3 “AI module.”

The real problem: a setting you can’t actually change

Here is the part that matters. In the free version, the plugin’s code forces the AI search policy back to “block all” every time the file is generated, regardless of what you select. We confirmed this directly in the plugin’s source: whenever the setting exists, it is reset to block_all before robots.txt is built. In plain terms — you cannot switch it off from the settings screen. Allowing AI search appears to be gated behind the paid tier. So a site owner can believe AI bots are allowed while the plugin quietly keeps them blocked.
Beware of the Better Robots.txt Plugin – Free Version 1

Why this matters in 2026

AI answer engines — ChatGPT, Perplexity, Google’s AI Overviews and Gemini — are now a genuine discovery channel. Blocking their crawlers means your brand cannot be cited or surfaced in AI answers. For any business investing in content and visibility, that is a silent leak you may never notice until your AI presence is gone.

How to check your own site (2 minutes)

  1. Visit yourdomain.com/robots.txt.
  2. Look for an “AI Bot Restrictions” section, or lines like User-agent: GPTBot followed by Disallow: /.
  3. If you see them and you did not deliberately add them, a plugin almost certainly did.

How to fix it

  • Decide your policy first. If you want AI engines to find and cite you (most businesses do), they should be allowed.
  • The reliable fix: stop relying on a plugin that overrides you. Replace the virtual robots.txt with a clean, static robots.txt you control — no forced AI blocks, with your sitemap line intact.
  • Re-test robots.txt, then request a re-crawl in Google Search Console.

Our take

Plugins that change critical SEO directives automatically, on update, in ways you cannot reverse without paying, do not belong on a serious website. Audit what your plugins write to robots.txt — especially after any major plugin update. A “helpful” recommendation should never become a silent threat to your visibility.

Need a hand auditing your site’s crawlability or AI-search readiness? Talk to our SEO team.

Digital Marketing Services

Join us

Be a part of our mission to make a difference. By supporting our CSR initiatives, you can help uplift communities and create meaningful change. Join us in spreading hope, fostering education, and empowering those in need. Fill out our form to learn more about how you can contribute and be involved in our efforts. Together, we can amplify the impact and bring smiles to many faces. Let’s inspire, support, and transform lives—one step at a time.

Let's Talk