• Skip to primary navigation
  • Skip to content
  • Skip to primary sidebar
  • Skip to footer
BlogAid Logo

BlogAid

Help for DIY Site Owners and Webmasters - WordPress, SEO, HTTPS, Security, and Performance

  • Home
  • Blog
    • Current Posts
    • Helpful Posts
  • Tips Tuesday
  • Site Services
    • Happy Clients
    • Setup, Backups, Fixes
    • Site Service Requests
    • Site Audits
      • What’s In the Audit
      • Audit Request Form
    • HTTPS Conversion
      • About the Service
      • HTTPS Request Form
  • Resources
    • Plugins
    • Helpful Posts
    • Site Resources
    • Start Here with BlogAid
  • Classes
    • Happy Clients
    • All Classes
    • DIY SEO Course
    • Webmaster Training
  • Videos
  • About
    • About MaAnna
    • Happy Clients
    • Privacy Policy and Terms of Use
  • Contact
    • General Contact
    • Site Service Requests

Control How Google Crawls and Indexes Your Site Pages

April 10, 2017 by MaAnna Stephenson

TwitterFacebookPinterestLinkedinRedditWhatsApp

Control How Google Crawls and Indexes Your Site Pages

You may not want Google to know about, crawl, or index some of the pages on your site, like special sales pages or events. Learn about the different directives you can use, like noindex, nofollow and how to keep pages out of your XML Sitemap too.

Easily Set Directives

The on-page Yoast SEO plugin module (below your text editor) makes it super easy to control the directives you give Google for each page/post on your site.

The choices are:

  • Follow
  • Nofollow
  • Index
  • Noindex
  • noarchive

Yoast SEO on page module advanced tab

Let’s have a look at the different combos and when to use them.

Default Settings

Normally you would want Google to discover, crawl, and index all of your site pages so they can be listed in SERPs (Search Engine Results Pages).

And that’s why the defaults in the Yoast SEO plugin are set to:

  • Index and Follow

Index tells bots to include the page in search and to crawl everything on it, including all of the links.

Follow tells bots to pass link juice to all of the links on the page unless any of them are set to nofollow individually.

Read: Follow and NoFollow Links and SEO for more on passing link juice and PageRank with links.

NoIndex

You may have pages that you don’t want Google to index or crawl. That means you don’t want them to be listed in search results.

NoIndex is the directive to tell Google not to index or crawl this page and to keep it out of search results.

Examples of Pages to NoIndex

  • Sales pages
  • Limited or exclusive offers
  • Thank you pages for optins or other sign ups or subscriptions

NoArchive

This is a little used directive to tell bots not only to skip indexing the page. It also tells them to not hold it in their archives.

In other words, take it off the radar completely.

Not all bots recognize this directive, but Google bots do.

I generally include this directive on all pages I mark as NoIndex.

NoFollow

You may have pages full of links that you don’t want Google bots to pass link juice to.

NoFollow is the directive that tells Google to skip following all links on the whole page and to not pass link juice to any of them.

Example of Pages to NoFollow

  • Resource pages to external sites
  • Vendor pages with affiliate links
  • Sales pages with links to special/limited offers

NoIndex Follow

There will be times that you want pages to not display in SERPs (Search Engine Results Pages) but you will want all of the links followed.

For example, a thank you page for your contact form may have links to other pages on your site.

You don’t want the thank you page itself showing up in SERPs, but you do want Google to crawl the links because that’s like a vote for your own pages.

In other words, cross promoting your internal links helps Google understand what other pages are important on your site.

Remove From XML Sitemap

Unfortunately, marking a page as NoIndex in the on-page module of the Yoast SEO plugin does not remove it from your XML Sitemap.

From there, it could still be discovered by bots.

Yoast used to include a checkbox in this module to remove a site from the XML Sitemap, but he moved it.

Now it has it’s own tab.

But first, you’ll need to grab the page ID.

Look in your URL bar while you’re editing the page.

It will look something like this.

Post ID in URL

The highlighted number is the page ID.

Next, in your left admin sidebar, under SEO, click XML.
(If you don’t see XML, you need to turn on Advanced Settings in Dashboard > Features.)

Then click the Excluded Posts tab.

There you will see a single field to enter all page IDs.

If you have more than one, separate them by commas, but no spaces in between.

After you Save the settings, that page will no longer be listed in your XML Sitemap.

Want More SEO Tips?

My DIY SEO Course has 9 tutorials dedicated to the Yoast SEO plugin so you set everything up right. Plus, you’ll learn settings to lay a proper SEO foundation on your whole site.

TwitterFacebookPinterestLinkedinRedditWhatsApp

Filed Under: SEO

About MaAnna Stephenson

MaAnna is a geek who can still speak in plain English. She helps DIY site owners plus webmasters and designers create sites that are secure, perform well, and get noticed by search engines and readers.

  • Facebook
  • Google+
  • LinkedIn
  • Pinterest
  • RSS
  • Twitter
  • YouTube

This book could save you hundreds of dollars and months of frustration. Get it free with your subscription to BlogAid News plus my blog posts.
Privacy Policy



Reader Interactions

Primary Sidebar

This book could save you hundreds of dollars and months of frustration. Get it free with your subscription to BlogAid News plus my blog posts.
Privacy Policy

Hi! I'm MaAnna, and a geek who can still speak in plain English. I help DIY site owners plus webmasters and designers create sites that are secure, perform well, and get noticed by search engines and readers. How May I Help You?

Let’s Connect

  • Facebook
  • Google+
  • LinkedIn
  • Pinterest
  • Twitter
  • YouTube

Looking for Something?

Search by Category

Footer

BlogAid News

This book could save you hundreds of dollars and months of frustration.

Get it free with your subscription to BlogAid News plus my blog posts. Privacy Policy

From the Blog

  • Free Social Share Button Plugins: The Best and Worst to Use
  • Scriptless Social Sharing Plugin: Best Settings for Speed and Looks
  • Tips Tuesday – Video is Huge, 2 Free Webinars, Social Share and Pinterest Case Study Update
  • Tips Tuesday – Sticky Bar Plugins, Gutenberg Phase 2, Safely Update Plugins
  • Tips Tuesday – Site Speed, Protect Images, Google and PDFs, WP 5.1 Security
  • Tips Tuesday – Site Security, MediaVine Research, Genesis 2.8, Gutenberg Phase 2
  • Collection #1 Data Breach and Your Site Security
  • Genesis 2.8 Promises a New Theme Experience

© 2019   Blog Aid · WordPress for Non-Geeks · All Rights Reserved

Disclosure: Some of the links on this website may be affiliate links. When you make a purchase from these links, I earn a small commission.
While commissions allow me to keep this site 100% free, I only endorse products I trust and use for myself and clients.