Control How Google Crawls and Indexes Your Site Pages

You may not want Google to know about, crawl, or index some of the pages on your site, like special sales pages or events. Learn about the different directives you can use, like noindex, nofollow and how to keep pages out of your XML Sitemap too.

Easily Set Directives

The on-page Yoast SEO plugin module (below your text editor) makes it super easy to control the directives you give Google for each page/post on your site.

The choices are:

• Follow
• Nofollow
• Index
• Noindex
• noarchive

Let’s have a look at the different combos and when to use them.

Default Settings

Normally you would want Google to discover, crawl, and index all of your site pages so they can be listed in SERPs (Search Engine Results Pages).

And that’s why the defaults in the Yoast SEO plugin are set to:

• Index and Follow

Index tells bots to include the page in search and to crawl everything on it, including all of the links.

Follow tells bots to pass link juice to all of the links on the page unless any of them are set to nofollow individually.

Read: Follow and NoFollow Links and SEO for more on passing link juice and PageRank with links.

NoIndex

You may have pages that you don’t want Google to index or crawl. That means you don’t want them to be listed in search results.

NoIndex is the directive to tell Google not to index or crawl this page and to keep it out of search results.

Examples of Pages to NoIndex

• Sales pages
• Limited or exclusive offers
• Thank you pages for optins or other sign ups or subscriptions

NoArchive

This is a little used directive to tell bots not only to skip indexing the page. It also tells them to not hold it in their archives.

In other words, take it off the radar completely.

Not all bots recognize this directive, but Google bots do.

I generally include this directive on all pages I mark as NoIndex.

NoFollow

You may have pages full of links that you don’t want Google bots to pass link juice to.

NoFollow is the directive that tells Google to skip following all links on the whole page and to not pass link juice to any of them.

Example of Pages to NoFollow

• Resource pages to external sites
• Vendor pages with affiliate links
• Sales pages with links to special/limited offers

NoIndex Follow

There will be times that you want pages to not display in SERPs (Search Engine Results Pages) but you will want all of the links followed.

For example, a thank you page for your contact form may have links to other pages on your site.

You don’t want the thank you page itself showing up in SERPs, but you do want Google to crawl the links because that’s like a vote for your own pages.

In other words, cross promoting your internal links helps Google understand what other pages are important on your site.

Remove From XML Sitemap

Unfortunately, marking a page as NoIndex in the on-page module of the Yoast SEO plugin does not remove it from your XML Sitemap.

From there, it could still be discovered by bots.

Yoast used to include a checkbox in this module to remove a site from the XML Sitemap, but he moved it.

Now it has it’s own tab.

But first, you’ll need to grab the page ID.

Look in your URL bar while you’re editing the page.

It will look something like this.

The highlighted number is the page ID.

Next, in your left admin sidebar, under SEO, click XML.
(If you don’t see XML, you need to turn on Advanced Settings in Dashboard > Features.)

Then click the Excluded Posts tab.

There you will see a single field to enter all page IDs.

If you have more than one, separate them by commas, but no spaces in between.

After you Save the settings, that page will no longer be listed in your XML Sitemap.

Want More SEO Tips?

My DIY SEO Course has 9 tutorials dedicated to the Yoast SEO plugin so you set everything up right. Plus, you’ll learn settings to lay a proper SEO foundation on your whole site.