Introducing the new Crawl Errors criterion
As we all know too well, almost all websites have some kind crawling issues, whether due to broken links from internal pages or linking websites, problems with servers not responding properly, incorrect linking from sitemaps, or other reasons.
WooRank users have been able to sync their Google Analytics accounts to pull useful data into their WooRank reviews for some time now. But from today, it is now possible to sync your Google Search Console account (formerly Google Webmaster Tools) to extract crawl error data to feed into your WooRank Project, in addition to your Site Crawl data, giving you details of errors that could be causing problems for both your users and the search engines, thereby hampering your SEO efforts.
How to set it up
First, log into your WooRank account and head to your overview page. Click on the project that you want to sync, then select ‘Settings’. It’s important to note that you must have Google Search Console access for the website that you wish to sync.
Next, scroll down to ‘Google Search Console’ and click ‘Sync with Google Search Console’.
You will need to select the address associated with the account (add your account if it’s not listed here).
Next, click ‘Allow’ to give WooRank permission to access your account (please note, we do not store or share any of your data). Now select the profile for the website that relates to your review.
Once you get the confirmation message, close the window then click ‘Save’ at the bottom of the Settings page.
Next time you refresh this advanced review, you will notice the new ‘Crawl Errors’ section in the left menu, under ‘Optimize’.
Use the tabs to switch between error categories, and click ‘show more’ to see up to 2,000 errors (up to 1,000 desktop errors and 1,000 smartphone errors).
The mobile icon between the ‘Detected’ column and the ‘link from’ icon indicates that the crawl error appeared on a smartphone, but not a desktop.
Click on any of the URLs to view the error page, or click to the left of the table to open a pop-out with more information about each of the errors (this appears at the bottom of the table).
You can see the last 90 days of data in the crawl errors graph, which can be filtered by desktop errors or smartphone errors using the tabs, and by error type by selecting them in the legend below.
How can I find the pages or XML sitemaps that link to these broken URLs?
To find a list of web pages or XML sitemaps that reference the specific errors, simply click on the link icon to the right of the row. Click the links to open the linking source.
What do the error categories mean and how do I fix them?
We provide crawl errors from 4 different error categories:
These errors are usually discovered when Googlebot attempts to crawl a page that doesn’t exist. This is typically caused by broken links as a result of typos in URLs from links, or deleted pages have have no redirects in place.
Fixing ‘Not Found’ Errors
Fix internal links – use the ‘?’ icon beside each error to see any links/xml sitemaps that point to the broken page. If these links are internal, simply update them to point to the correct URL. If they are from another website, either contact the webmaster to request that they are update, or add 301 redirects to ensure that any users (and indeed link juice) are redirected to the true page.
These errors are often caused by issues that prevent Googlebot from properly accessing your pages.
Fixing Server Errors
Fixing Not Followed
Use The ‘Fetch as Google’ feature in Google Search Console to see how Google renders the webpages. This should help you to see if Googlebot is having problems crawling particular sections of your content. For more help, see the ‘Not followed’ section in Google’s Search Console Help.
When a page on your website cannot be found, your server should return a 404 (not found) error. However, if a website has not been properly configured, your server may fail to return the expected 404 status, despite the page not existing. Google is now getting better at identifying when this happens, which is where the Soft 404 error comes in. In this case, these crawl errors suggest that your pages should be returning the 404 error status.
They may also be triggered when irrelevant redirects have been put in place. For example, if you remove a piece of content and decide to redirect the old page to the home page, Google might see that the content on the two pages are not connected, so may treat the old page as a 404, rather than following the redirect.
Fixing Soft 404
First, make sure that your website returns the correct 404 HTTP status code when attempting to access a non-existent page. There are several online tools to check status codes, but a great extension for this is AYIMA Redirect Path. If you don’t have a custom 404 page, you should create one, as this will help to improve user experience.
If the URL redirects to an irrelevant page (for example, a number of old pages redirected to the home page), consider finding more suitable pages to redirect to; remove the redirects and allow the pages to return a 404 status; or if it makes sense you may want to recreate these pages.
How can I mark the the errors as fixed?
Once you have fixed each error, click the ‘Mark as fixed’ button to the left of each row. This will also remove the error from your GSC account.
How often are the errors updated?
We refresh the crawl errors every time you access your report, so you will always see the most recent data.
What happens when I fix an error?
It can take some time for the errors that you have fixed to disappear from Google’s list. We hope to launch an update in the coming months to allow you to mark the errors as fixed, which will speed up the removal process.
How many errors can I see?
Google limits the number of errors that we can report on to 1,000 per error category per device (desktop and smartphone). This means that you can potentially see up to 8,000 errors. If you have more than this limit then you will need to fix the existing errors before new ones appear.
Will the crawl errors appear in my PDF reports?
No. We hope to provide a more suitable way of downloading these errors in the coming updates.
Do you have any feedback for us? Leave your comments below if you have any questions or comments about this new feature.