Update: I feel really sad to inform that google has discontinued the Google Search Console API Explorer. Due to this, this method doesnāt work anymore.
The Google Search Console is of great help to Webmasters with several useful features like sitemap submission, search analytics, links to your site, crawl errors report. With the GSC, the SEO specialists have a powerful tool at their fingertips to direct the course of their SEO requests.
However, the Google Search Console also has several aspects that are somewhat frustrating. For eg; the āLinks to your siteā feature only shows a maximum of 1000 domains. Now, Imagine how many domains link to a site like Moz or Buzzfeed! Do you think the Google Search Console serves them well?
The sitemap tool shows how many URLs were indexed but does not distinguish between the non-indexed and indexed URLs. And well there are other similar annoyances. (Update: Google is testing a beta feature called Index Coverage report that will show indexed pages count and reasons why some pages could not be indexed. Read more on Google Webmaster Blog.)
Another point that annoys Google Search Console (GSC) users is the Crawl Errors report. The Search Console shows us the crawl errors, the HTTP status code of the error (404, 503 etc), and also the source of the error. The bothersome part is you have to click on each URL to view where the error originates from.
Another problem is the 1,000 errors limit. Youāll have to fix the existing ones to view newer ones.
The Google API explorer is a savvy tool to communicate with numerous Google APIs. But we need to concern ourselves only with the Search Console API.
Youāll need Full Access to use the API and Search Console account, so make sure you have them beforehand.
Once logged in, youāll notice the Search Console API offers several Services, 13 services to be exact. But our focus here is the one labeled webmasters.urlcrawlerrorssamples.list. So, go ahead and click on it.
The next screen will look like the image below. Beyond this point itās a simple 3 step process:
The fields you notice on the next screen read as follows:
Use the fields editor to select required fields. The available ones are:
I prefer selecting all the data and filtering it down later. You are free to choose your path.
Oh! Do turn on the Authorize requests using OAuth 2.0 button on top right (marked with an orange rectangle in the image)
or
Once all the fields are set, just click the Authorize and execute button.
Youād be greeted with a popup before the report runs, like in the image below:
Check both options and hit the Authorize and execute button.
If everything goes well, you should see a status 200 OK message with a JSON output similar to the image below
Copy the output after the first curly bracket { till the last } curly bracket in the end.
Now, head over to a JSON to CSV converter. I prefer https://konklone.io/json/ for the simple interface, a preview window, and the ability to download the CSV.
Note: I tried another converter too but the report wasnāt 100% accurate. Test your converter before you finalize on a report.
Paste the JSON output and convert it into a CSV.
This is pretty much self-explanatory! Download your CSV and start fixing those 404 errors.
Word to the wise: Google Search Console may not list sources or linking domains for all errors. So don’t worry if some rows returned are empty.
404 errors may not seem critical but fixing them helps saves crawl budget, improves user experience, and also helps retain link value.
You may use the same process to find and fix other errors. The API includes 9 types of errors in the Google Search Console that you can fix. You donāt have to wreck your brain to search them out; we have listed all error types below.
I hope you find this useful! Now head to your laptop and start fixing those 404 errors.
I am part of the CueBlocks Organic Search(SEO) team. I love playing football and reading about Technical SEO.
Committing to green claims as a business is a huge promise to deliver on. For ecommerce stores, Shopify is leading …
Continue reading “Evaluating the Carbon Emissions of Shopify Themes”
Introduction Dark mode, a feature that lets users switch the color scheme of an app or website to darker colors, …
Continue reading “Dark Mode: Accessibility vs Sustainable Web Design”
Given the major issues that our world is currently facing, such as pollution and climate change, sustainability becomes an inevitable …
For an eCommerce brand committed to good, success goes beyond creating excellent products; it extends to effectively communicating your values …
Continue reading “Show, Don’t Tell: Demonstrating Transparency in Your eCommerce Store”
In today’s market, sustainability has evolved from a passing trend to a pivotal consideration for both consumers and businesses. Globally, …
Continue reading “How to Market Sustainable Products Effectively”
Today, businesses place high importance on sustainability and ethical practices. For B2B and e-commerce leaders, being a certified B Corp. …
Continue reading “Decoding B Corp Marketing Challenges: Strategies for Success”
Thanks for this post, works excellent! Helped a lot and prevented me from writing my own script ;).
Thanks very much for this excellent post Rajiv! Just tried it and its very handy! I was just wondering when you think you’ll find a method to extract more than 1,000 URL’s as that would be ideal for larger websites! Thanks again and all the best! Alex
Thank you, Alex!
I am currently working on it. It appears a lot of us run into 404 errors in bulk. Will be posting the solution for 1000 plus URLs soon, hopefully.
Cheers!
Rajiv
Hi Rajiv,
Great walk-through of the API.
Using the Search Console interface I was able to download 1,325/5,929 errors while the API only returned 1,000. Any idea what I’ve missed?
Thanks again – JR
Hi John,
Thank you! Unfortunately, the API explorer seems to be limited to 1000 results only. I am working on a way around this. I’ll make it another blog post once it is complete.
Rajiv
Thanks Rajiv.
This worked a treat, first time.
Glad to know you found it useful, Ewan
Hi,
Its working!
Thanks for showing the easy way to download it.
Great stuff.
Regards,
Malisa
Thank you, Malisa!