Google Search Console (formerly Google Webmaster Tools) is sending emails out warning site owners that some CSS and JS files are being blocked from their bot crawlers. They also state this will result in reduced ranking. Here’s what you need to know about this warning, how to check it, and ways to fix real issues.
False Reporting
First, be aware that the way some of these crawl errors are being reported by Google just aren’t true.
Google’s testers will tell you that the problem is your robots.txt file.
[clickToTweet tweet=”I’ve seen the Google CSS and JS crawl error reported on sites that don’t even have a robots.txt file” quote=”I’ve seen the Google CSS and JS crawl error reported on sites that don’t even have a robots.txt file”]
More on that in a moment.
What are CSS and JS?
CSS – Cascading Style Sheet – mostly used by themes and plugins to define the style parameters.
JS – JavaScript – mostly used by themes and plugins to define functional aspects.
Check it, Fix it
Following are helpful tips on how to find the real sources of the render blocking issues and fix them, if they can be fixed. Some cannot. But you need to know exactly what they are anyway.
Mobile Friendly Check
The first time I ran into a blocked crawl issue was when the Google Mobile Friendly checker first came out in early 2015. It crawls differently than GSC. You can see the rendering difference on one of my client’s sites.
Once I changed blocks to sensitive areas in the robots.txt file, the rendering difference fixed itself too.
[clickToTweet tweet=”The Google Mobile Friendly test is a fast way to see CSS and JS crawl blocks.” quote=”The Google Mobile Friendly test is a fast way to see CSS and JS crawl blocks.”]
Click the Show Resources tab to see exactly what Googlebots are having issues with.
Ads Causing Blocks
The above resource blocks are on one of my client’s sites that runs ads.
The site is Mobile Friendly, but it still shows 2 resources blocked.
And yet, it reports them as being blocked by robots.txt (in the first sentence).
That’s just not true. I know, I wrote that robots.txt file myself. There’s nothing in there that would block these two items.
Plugins Causing Blocks
Many plugins with widgets that display something on your site to the public come with their own CSS and JS files and may be blocking them from crawl bots.
If that’s the case, you may be able to see them in the same Resource list as shown above. (You’ll see another example of how to check for them in a moment.)
How to Check Your Site
If you get an email from Google Search Console about this, or do see crawl errors in the Mobile Friendly check, you’ll want quick ways to run down the source of the problems.
Test robots.txt
Finding your robots.txt file is easy.
And if you’ve already verified your site on Google Search Console (GSC), testing it is easy too.
Find your robots.txt file
- In your browser, type in the URL of your site followed by /robots.txt
- Here’s mine https://blogaid.net/robots.txt
- If you get a 404 error page, that means you don’t have a robots.txt file.
Test your robots.txt file
- Next, log into the Google Account you use for GSC.
- Go to Google’s Robots.txt Tester tool.
- Click the Please select a Property button.
You’ll see the test result. Errors and Warnings are displayed at the bottom.
As you can see, Google is not fond of my crawl delay. It, and all other bots, had rather sit and chew on my site unabated. Too bad. I don’t want my system resources eaten up that way by bots. Google will have to get over it. They are not the only ones this is for.
Check Google Page Speed Insights
Go to Google PageSpeed Insights and input your site’s URL.
Go to the Desktop view.
If there are any blocking issues, click that link for Show how to fix.
You may see something like mine, below:
Notice that all blocks are from plugins. There is nothing I can do about that. If you see blocks from ads, there may be nothing you can do about that either.
Check the Render
While you’re in PageSpeed Insights, check the render in the right sidebar for both Desktop and Mobile.
Check Mobile Friendly
As previously mentioned, a fast way to find blocks is via the Google Mobile Friendly Checker.
Also make note of how it renders and ensure that it matches the Mobile render version in PageSpeed Insights.
Verify Your Site on Google Search Console
You want to be notified immediately any time that Google makes a change which could affect your ranking with them.
If you haven’t already verified your site with GSC, do it today.
Submit your XML sitemap for better crawling.
Create a robots.txt for better crawling and site security and performance too. (See what your site looks like without a robots.txt file.)
Need Help?
I see issues like these in Site Audits every single day, along with major security and performance issues. Plus, lots of easy-to-fix SEO fundamentals being missed.
Contact me directly to get the help you need to put your site on a firm foundation with Google.
What Every Site Owner Should Know could save you hundreds of dollars and months of frustration. Free with subscription to BlogAid News.
Thanks for the heads up on this issue. I have similar issues with plugins & delay :). There are also a couple of theme related issues that were identified. So, do you recommend that we contact the relevant plugin creators with the issue to see if they can work some magic in their next updates?
Ingrid, I don’t think the plugin devs or ad folks are going to do too much about this. There are just things that Google won’t render. They’ve been skipping over ad sections for a long time now. I heard there was a blanket fix from one of the big security plugins, but my first instinct tells me that’s probably not a good idea. I’ll have to check the details to find out how it was done, and to see how other folks are tackling this.
I am also seeing a lot of forums recommending changes to robots.txt. We are in for an interesting few days!
Allow: /wp-content/themes/
Allow: /wp-content/plugins/
Allow: /wp-includes/js/
Allow: /wp-includes/css/