Indexability

What is Indexability?

Indexability is a web page’s ability to be indexed by search engines. Only indexable pages can show up in search results.

To index a web page, search engines, such as Google, must:

• Discover the page URL
• Crawl it (download its content)
• Process it (understand and extract key information from crawled pages)

Only then can the page appear in the search results.

If web crawlers can’t do their job and crawl a page on your site, then that page can’t be indexed appropriately. And without indexing, your website’s page can’t bring in any search traffic because it’ll remain invisible to search engines.

Why is Indexability Important?

Indexability allows search engines to display your website’s pages in SERPs. Without indexation, you can drive no organic search traffic to your site.

Sometimes, it makes more sense to make certain pages non-indexable. For example, non-indexable pages are preferred when it comes to landing pages, low-quality pages, or “Thank you for subscription” pages. After all, you don’t want these pages appearing in search results.

Assuming that you want the URLs to rank, though, you should ensure that those pages are both crawlable and indexable.

What Makes a Page Indexable?

To be viewed as “indexable,” the page must meet the following criteria:

1. The Page Must Be Crawlable

For a web page to be indexed, the search engine’s crawlers must be able to access and crawl its content first. One thing that directly impacts the “crawlability” of a web page is the robots.txt file.

A robots.txt file is a small file located in your website’s root directory. Its role is to provide instructions to web crawlers and tell them which pages on your site must not be crawled. Otherwise, bots will crawl any page they can find on the website; they do so by default.

There are instances where placing such “restrictions” makes sense. Still, it’s essential to ensure you’re not unknowingly blocking the pages you want to be indexed.

2. The Page Must Not Have a “Noindex” Tag

A “noindex” tag is an on-page directive that will instruct the search engine not to index a specific page. Basically, it prevents a page from appearing in SERPs.

Sometimes, it is advisable to “noindex” certain pages - such as landing pages, login pages, thin or low-quality pages, or “Thank you” pages - and keep them “private.”

However, if you misuse or, say, forget to remove this directive from the page you want to be indexed, you will lose organic search traffic.

You can crawl your website with Ranktracker’s Site Audit tool to make sure none of the pages you want to be indexed have a “noindex” tag.

3. The Page Must Be Canonical

A canonical tag (rel=“canonical”) in a page code defines the main version for duplicate, near-duplicate, and similar pages. This means that only canonical pages can be indexed while their non-canonical versions will be excluded.

Unless you add the canonical tag to a page, Google will rely on signals like internal and external backlinks, the XML sitemap, and redirects to decide which URL will “represent” that cluster in its index.

That means that if you want your page to show up in search results, it must not declare any other page as its canonical version. Good SEO practice is to have self-referencing canonical tags for indexable pages.

How to Help Google Index Your Page

Here’s what you can do to help Google index your site’s pages:

1. Submit Sitemaps to Google

A sitemap is essentially a list (typically in XML format) containing all the pages on your website.

The sitemap’s role is to tell Google about the pages you want to be indexed on your site.

Before you submit a sitemap, check if it reflects your website’s current state and fix any errors - such as broken links, orphaned pages, and the like.

You can submit your sitemaps to Google using the Google Search Console.

2. Submit Individual Pages via the URL Inspection Tool

If you’d like to request indexing or re-indexing of just a few individual URLs, you can use the URL Inspection Tool in Google Search Console.

The URL Inspection tool is a diagnostic tool found in the Google Search Console that offers you insight into Google’s indexed version of your web page and allows you to test the URLs live and view the rendered version of the page, among other things.

It also allows you to submit individual pages to Google.

If you want to request indexing of a new URL - or re-crawling and re-indexing of updated URLs - you can tell Google to do so by selecting the “Request Indexing” button. If there are no errors in your live URL, Google will add your page to its crawl queue.

Do note that there is a limit to how many URLs you can request indexing per day, though.

3. Provide Internal Links for Better/Faster Discoverability

Another way to encourage crawling and, in turn, speed up the website’s indexation is to employ internal linking.

As they scour the Internet, Google’s crawlers follow a path formed by links. So, when one page on your website links to another, these crawlers will follow the existing path - and discover new content.

Pages with no internal links are called “orphan pages.”

That’s why it’s crucial to ensure that you’ve provided internal links going to and from your most important pages. Granted, that typically occurs naturally when you create a well-organized site.

However, it doesn’t hurt to check if there are orphan pages - as in pages that don’t have internal links pointing to them - on your website. You can do so with the Ranktracker’s Site Audit tool.

On a related note, you should ensure that these internal links pointing to indexable pages don’t have a “nofollow” tag because Google won’t crawl nofollow links.

FAQs

What Is the Role of the Robots.txt File in Indexability?

The robots.txt file provides instructions to web crawlers on which pages or sections of your site should not be crawled, impacting the crawlability and, subsequently, the indexability of your pages.

How Can I Check If My Pages Are Indexed?

You can check if your pages are indexed by using the Google Search Console. The tool provides insights into the indexed status of your URLs.

What Is the Impact of Orphan Pages on Indexability?

Orphan pages, which have no internal links pointing to them, are less likely to be discovered and indexed by search engines. Ensuring all important pages are well-linked within your site improves their indexability.

By ensuring that your pages are crawlable, free of “noindex” tags, and properly canonicalized, you can improve their chances of being indexed and appearing in search engine results. Regular audits and strategic internal linking can further support this goal.