Should You Block GPTBot?
See why you should allow or block the new GPTbot from OpenAI from crawling your site.
What is GPTbot?
All kinds of bots already crawl your site, including:
- Googlebot
- Bingbot
- Yandex
- Ahrefs
- SEMrush
- And more
Some of these bots are from search engines. And some are offering SEO services to paid members.
The GPTbot will be used to crawl the web for content to train OpenAI’s next LLM (Large Language Model), which will be called GPT5.
Why Tell Us About GPTbot?
Previous OpenAI models, like 3.5 and 4.0, were trained mainly on books and a bit on the internet, but only up to 2021.
None of the OpenAI models have access to current content on the internet.
That’s a problem for using ChatGPT to do research.
(Yes, you can get access to the internet via ChatGPT now, but the model was not trained on info after 2021, meaning that only the link you visit or give it is being considered in its reply.)
And note that OpenAI crawled the web without telling anyone or allowing their content to be blocked.
Plus, it has been shown that the previous bots could crawl info that was normally locked behind a paywall or was gathered via APIs attached to popular online forums, like Reddit.
In fact, so much info was grabbed from Reddit that the company decided to put an exorbitant price tag on use of their APIs and deny access to 3rd party APIs that ran end around ads and such. It has caused a LOT of controversy and hard feelings among those who run sub-Reddits and users of those forums.
And, there has been an enormous outcry from multiple content creators who are upset their content was “scraped” by AI bots and used to train current models without consent, even to the point of launching lawsuits.
GPTbot Can Be Blocked
So, this time around, OpenAI publicly announced the new GPTbot was being released.
And they gave a way for site owners to block it entirely, or allow it to only crawl specific directories.
See the article for how to do the block in your robots.txt file.
Or, join my DIY SEO course where you can see how to do this, and so much more.
To Block, or Not To Block, That Is the Question
Before you fly off the handle with righteous indignation that it is a violation for any AI bot to crawl your site without your consent, think about this.
Have you made any arrangements to block the other 20-30 SEO agency bots that are:
- crawling your site now
- chewing up your system resources doing it
- and giving your info to their paying customers to beat you at your SEO game?
Me and my site audit clients are blocking them – and not just in robots.txt either. We’re blocking them before they can ever hit the hosting much less our site.
By doing this we’ve also dropped heavy bot resource usage so that our sites are faster too.
And think about this.
Microsoft is heavily invested in OpenAI.
And the Microsoft Bing browser has already incorporated some ChatGPT functions.
When you do a search in Bing CoPilot, it shows search results with citations.
If you ever hope to have your site show up in such AI searches, the GPTbot will need to have crawled your site.
But, consider this too.
How much traffic are you getting from Bing search now?
Do you think your site will show up in the top 5 sites that are given citations?
If Bing is not a big part of your traffic now, then the AI integration into Bing is not likely to change that if your site is not in the top 5 cited.
The Impact of AI on Search Has Been Small
Neither me, nor any of my clients have seen a drop in search traffic due to AI search integration.
The fact is, very few folks are using Bing CoPilot or even Bard to aid them in search – few compared to how many other regular searches are being done everyday.
The main folks who are using ChatGPT are marketers and content creators, not the every-day folks doing a search for something online.
More folks rather ask Alexa or Siri a question than use ChatGPT and have to figure out how to properly prompt it.
We Don’t Know Where We’re Going, But We’re Moving Fast
AI use is already falling into certain niches:
- Enterprise – where they are training their own LLM models on their own data
- Automation – speeding up repetitive tasks
- Content Creation – to get help and inspiration with creating content to be consumed online and in products like books and videos
- Marketing – to get ideas plus automate promotion tasks, like generating sales pages and social media posts
- Art and Entertainment – from images to videos
Research and general search are the slowest moving aspects of AI.
The answers and sites it returns per query are just not all that helpful.
And Jane Q. Public is not going to use something that doesn’t give her the answers she seeks.
But, that could change, in just a few month’s time.
The fact is, we don’t know what’s coming. But we do know it’s coming fast.
Specialized AI Will Be the Ticket
Behind the scenes AI innovation is a flurry of activity.
AI startups are being funded like crazy.
What I believe we’ll see in the near future are apps that do one thing, and use AI to power their engine.
Need a weight loss plan?
There will be an AI app for that – recipes and all.
Need to plan a trip?
There will be an AI app for that – and it will book your flight, hotel, make dinner reservations for you, and create an itinerary of destinations for you too. In fact, we are on the verge of this right now with Google’s AI integrations.
Bing and ChatGPT can only do it by kluging together a bunch of plugins – nobody wants to learn how to do that. They want a single integrated solution.
So, Should You Block the GPTbot?
Well, that depends.
Can the content you create become useful to AI apps?
Then maybe you should consider blocking it.
With the constantly changing tech world I report on, plus how-to tutorials, and opinion pieces like this where folks follow me to hear a trusted source’s view on something, I don’t mind if the bots train on my content.
I very seriously doubt AI can replace what I do here on BlogAid or on the AI Discover Hub or on Heartwood Art.
Consider Doubling Down on Video
If anything, I believe AI will drive more folks to me to see how things are done.
AI may tell you the steps to build a workbench, but it can’t replace someone showing you how to do it.
When folks want to learn how to make money with AI images, they want how-to videos.
Heck, every plugin vendor gives you how-to steps, but most folks still want a video walkthrough like I do here on BlogAid. (See my Recommended Plugins, and several have tutorials.)
To me, making how-to videos and courses are my insurance again AI replacing me.
Do What Works Best For You
I think every site owner needs to take a good look at where AI is now and just how fast it is being incorporated into everything, and decide for themselves what they need to do moving forward with their content, and whether or not they want it to be available to AI bots.
