What is Crawl Budget in SEO?
All SEO specialists know that Google does not immediately crawl the pages of a website. Sometimes, crawling a page can take weeks. This delay can impact your website’s SEO.
For instance, you might optimize a landing page on your website, but despite waiting, the page remains unindexed. If you face such an issue, it’s time to optimize your crawl budget. In this article, we will explain what a crawl budget is and how you can optimize it.
What does Crawl Budget mean in SEO?
A crawl budget in seo refers to the number of pages that Google can crawl within a specific period (e.g., a day). The number of pages may vary slightly each day. For instance, Google might crawl 6 pages on your website in one day, and this figure could rise to 5,000 or even 4 million pages per day.
The crawl rate of a website generally depends on its size, the absence of bugs, the server’s efficiency (number of errors Google encounters), and the number of links on your website.
Why do search engines allocate a crawl rate for websites?
Unfortunately, search engines do not have unlimited resources, and they must distribute their focus among millions of websites. Therefore, they require a method to prioritize the crawling process. Allocating a crawl budget to each website helps search engines achieve this.
In summary: If Google does not index a page, it essentially does not exist!
Consequently, if your website’s number of pages exceeds its crawl budget, there will undoubtedly be pages on your website that remain unindexed and, therefore, unseen.
Thus, crawl budget plays a more significant role for larger websites, as Google’s bots can easily crawl and index smaller websites. The following situations necessitate special attention to your crawl budget:
- You own a very large website: If your website (e.g., an e-commerce website) has over 10,000 pages, Google might struggle to find all of these pages.
- You have added a batch of pages to your website: If you’ve recently added a new section with hundreds of pages, you need sufficient crawl budget to ensure these pages are indexed quickly.
- You have a large number of redirected pages: Redirects can consume your website’s crawl budget.
What is Google’s Perspective on Crawl Budget?
According to Google, there are three fundamental steps that a search engine follows to obtain relevant results from web pages:
- Crawling: Web crawlers access publicly available pages.
- Indexing: Crawlers analyze the content of each page and store the information they discover.
- Serving and Ranking: When a user types a query, Google presents the most relevant answers from the pages it has indexed.
Without crawling, your content will not be indexed and, therefore, will not appear on Google.
Google believes that crawl rate is not an issue that should cause concern. Most pages on the internet are crawled and indexed quickly after publication. If your website has only a few hundred pages, complete crawling of these pages is almost guaranteed. Determining what content to crawl and when becomes a challenge only for websites with a very large number of pages.
How is the Required Crawl Budget for Each Website Determined?
The crawl budget varies for each website and is automatically allocated by Google. Search engines consider various factors when determining your website’s crawl budget. In general, Google uses four key factors to allocate a website’s crawl budget:
- Website size: Larger websites require a higher crawl budget.
- Server performance: The performance and loading speed of your website can influence the budget assigned to it.
- Update frequency: How often do you update your content? Google prioritizes content that is regularly updated.
- Links: The structure of internal links and the presence of dead links.
It is important to note that increased crawling of your website does not necessarily improve its ranking. If your content does not meet the standards of your audience, your website will struggle to attract new users.
To better understand crawl budget, here are some key concepts:
Crawl limit / Host load
The crawl limit indicates how many crawl requests your website’s server can handle. Each time Google crawls a page, a request is sent to the server to access your website’s resources. If these requests are too numerous, the server resources may be unable to respond to all of them, causing the website to crash. Google determines this limit using “server error signals” and “the number of active websites on the host,” which are explained below.
Server Error Signals
Google’s crawling bots may encounter server issues multiple times when attempting to crawl your website.
Number of Active Websites on the Host
If your website operates on a shared host alongside hundreds of other websites and your website is relatively large, you will face significant crawl limitations. In such cases, it is essential to switch to a dedicated host to increase your crawl budget and improve the loading speed of your website’s pages.
Crawl Demand / Crawl Scheduling
Crawl demand determines which pages are worth crawling or re-crawling. This value is assessed based on the following factors:
- Page Popularity: URLs that are more popular on the internet are crawled or re-crawled sooner.
- Content Freshness: Pages that are regularly updated are more significant to Google’s bots.
- Page Type: Page type is another crucial factor in determining the value of pages. Compare a category page with a website policy page—which one is more likely to have changing content?
Why Should Crawl Budget Be Given Extra Attention?
You want search engines to find and understand many indexable pages on your website, ideally as quickly as possible. When you add new pages to your website or update existing ones, you want search engines to detect them promptly. The sooner Google’s bots index your pages, the sooner you can benefit from them.
If you waste your crawl rate or crawl budget, search engines will be unable to effectively scan your website. They may spend time on areas of your website that don’t matter to you, leaving important sections uncrawled. If Google’s bots lack information about certain pages, they won’t crawl or index them, and you won’t attract visitors to those pages through search engines.
Reasons That Lead to Wasting Crawl Budget
Optimizing crawl budget means ensuring no crawl budget goes to waste. Experts have examined crawl budgets for various websites and found that most of them struggle with similar issues.
These simple but significant problems can cause crawl budget shortages for your website. However, by addressing them, you can promptly optimize your website’s crawl budget and ensure your valuable pages are indexed faster. Factors that can waste your crawl budget include:
Presence of Product Filter Parameters in URLs
The URLs of most websites, especially e-commerce websites, often contain parameters that can be used to filter products or content.
For example: https://www.example.com/toys/cars?color=black is a website URL that uses filter parameters. When implementing product filters on e-commerce websites, parameterized URLs are commonly utilized. This approach is fine, but you must ensure that these parameters are inaccessible to search engines.
How can you make these parameters inaccessible to search engines?
Use your robots.txt file to instruct search engines not to crawl these pages. If this option is unavailable, configure the URL parameter settings in Google Search Console and Bing Webmaster Tools to specify which pages should not be crawled. Alternatively, you can add the nofollow attribute to your filter links.
Note: Since March 2020, Google has decided to ignore nofollow links. Hence, it’s recommended to prioritize the first method whenever possible.
Duplicate Content on the Website
Pages with entirely identical content are known as “duplicate content.” Examples include copied pages, internal search result pages, and tag pages.
Surely, you wouldn’t want search engines to spend their time on duplicate pages of your website, wasting your crawl budget. Therefore, it is crucial to avoid or minimize duplicate content on your website.
To address duplicate content on websites built with WordPress: Retain the content that is more complete compared to other duplicate content. Remove incomplete content. Redirect the old content to the new, consolidated content using the Redirection plugin or other redirect tools.
Low-Quality Content
Pages with very little content or pages that add no value to your website are considered low-quality content. Such pages are unattractive to search engines. Try to minimize their number or, if possible, remove them entirely. One example of low-quality content is an FAQ section where each question-and-answer pair is provided through a separate URL.
Broken or Redirected Links
Broken links are links that point to pages that no longer exist. Redirected links, on the other hand, are links that lead to URLs that redirect to other URLs.
Broken links and long chains of redirected links create dead-ends for search engines.
Whenever possible, minimize such links on your website.
By fixing broken and redirected links, you can quickly recover your website’s crawl budget. In addition to restoring your crawl budget, this action can significantly enhance the user experience for your website visitors. Redirects, especially redirect chains, increase page loading times, resulting in a poor user experience.
Incorrect URLs in the Website Sitemap
Google’s crawlers access your website through the sitemap. If your sitemap is filled with broken or redirected pages, Google will mistakenly crawl them. It is recommended to avoid including 3xx, 4xx, and 5xx redirects in your XML sitemap as much as possible. Always review your XML sitemap to ensure it does not contain irrelevant pages and that target pages are indeed included.
Pages with Slow Loading Speed
Pages that take a long time to load or fail to load altogether have a severely negative impact on your crawl budget. For search engines, this issue indicates that your website cannot handle user requests effectively, which could lead them to allocate a much lower crawl budget to your website.
When your website’s pages have high loading times, search engines will crawl fewer pages. Beyond this drawback, high page loading times significantly harm your website visitors’ user experience and reduce conversion rates.
If your page loading time exceeds two seconds, your website has a serious problem. Ideally, each page should load in less than one second.
Excessive No Indexable Pages
Every website contains a large number of non-indexable pages.
If your website has many non-indexable pages that are still accessible to search engines, you are essentially causing search engines to sift through irrelevant pages, which can deplete your crawl budget.
The following types of pages are non-indexable:
- Redirects (3xx)
- Pages not found (4xx)
- Pages with server errors (5xx)
- Pages marked as non-indexable (pages containing noindex directives)
Faulty Internal Linking Structure
If your website’s internal linking structure is not properly set up, search engines may overlook some pages.
How the pages on your website link to one another plays a crucial role in optimizing your crawl budget. This structure is referred to as the internal linking structure of the website.
Search engines are generally attracted to pages with well-planned and abundant internal linking. Distribute internal links throughout your website’s content. Ensure that your website’s most important pages have numerous internal links. Pages that have been recently crawled often rank better in search engines. Keep this in mind as you organize your internal linking structure.
Conclusion
This article has explored crawl budget and its optimization methods. Addressing the issues mentioned above not only improves your crawl budget but also enhances your website’s user experience, attracting more visitors.
