While doing SEO of your website, and especially when you’re stuck!, it sometimes become important to sneak a peek in Google’s database to see which of your website’s URLs are there in the Google index. And you wish if you could get the list of your URLs indexed by Google. Unfortunately, Google does not give any direct access into its index database. Therefore, it is not possible to straightway ask Google to give you a list of URLs that is has indexed.
Google, however, provides a lot of search operators and facilities to show you the information you are seeking. One of these operators is site. This operator can be used to get a list of all the URLs of a particular domain from Google Index. For example, if you enter the following in Google search:
site:techwelkin.com
The results will contain all the TechWelkin pages from Google’s index. The first page, as usual, will have 10 results and then you can go through result pages to view more pages.
This method is okay if your website has a few pages. But if your website boasts of hundreds or thousands of pages —the above method will be difficult to use.
Moreover, some SEO tools require a URL list to operate upon. Preparing such a list by copying indexed addresses from returned results is going to be very taxing. So, I am presenting a simpler solution here.
STEP 1: Log into your Google account and then go to Google Drive.
STEP 2: Create a new spreadsheet.
STEP 3: Paste the following function in first cell:
=importXml("https://www.google.com/search?q=site:techwelkin.com&num=100&start=1","//cite")
We have noticed that this method no longer works and Google no longer allows easy access to the list of indexed URLs.
STEP 4: Now place the cursor in cell number A101 and paste the above function again after changing start param from 1 to 101
Repeat the fourth step for as long as you need to. Every time you’ll get 100 URLs. Don’t be greedy by changing num=100 to num=1000 … this will not work because Google does not return more than 100 results in one go.
We hope this will help you. Share with us your questions and suggestions.
in google webmaster there are display 7000 link index but when i saw in google (site: url) there are only 500 url indexed? how can get proper result of indexed url. i want all that 7000 url in google (site:) which display in webmaster tool.
It doesn’t seem like this query works anymore? can you confirm this?
Hi Jonathan, you’re right. This method no longer works. Google has blocked the access. They are no longer providing an easy access to the list of indexed URLs. We have updated the article. For top 10 indexed URLs, you can use our free tool and for the rest, you can use our professional services.
Wonderful! It is working! Thanks for it. But I have a question, if there is any formula that shows NOT indexed URLs from a sitemap?
Hi Mike, you can always find the difference between the list generated by the formula given in this article and your sitemap. You can use MS Excel for this. It’s easy to find URLs not indexed by Google.
I’m getting less URLs with every step: start=201 got 96 URLs, start=601 got only 62 URLs.. any advice?
Hi Tanya, probably, Google is omitting similar results from the fetch request. I would advice you to append “&filter=0” at the end of the request URL to ensure that no such filtration of similar results take place. I hope this would help. Please let me know the results! Thank you!
Fantastic! Thank you.
Follow up question – Do you know if it is possible to pull the page title and meta description for each URL as well?
Hi Stacy, you should pull page titles and meta description from your own website. Google does not always use meta description in the search results. It may show a different part of the page content as description depending on the keywords are used by the searcher.
Hi Paul,
here is the solution of your problem:
"In order to show you the most relevant results, we have omitted some entries very similar to the 239 already displayed.
If you like, you can repeat the search with the omitted results included."
Please go on last page of search result and you will find the above message.
this is the main reason of differences.
Regards,
Kunal Bhopal
Lookalike Matt Cuts
this formula i can use only upto <1000, this was a quicker way to get the indexed url, but i require to pull more indexed urls from google, google has indexed pages around 2 million, can anyone tell me how to pull out the list of 2 millions indexed urls list.
Awesome It’s working
thnks
Hi Lalit..
Thanks a lot, really helpful!!
First I did ‘F2’ and pasted the link in Google Sheets but it didn’t work and then I did direct copy paste and it worked like a champ!! Superb!!
Thanks,
Anson
Hi Anson! I am happy that this article could be of some help to you! Stay in touch with TechWelkin!
I can add the first 100 urls but when I try to add more I get the following error :
“Error: Wrong number of arguments to IMPORTXML. Expected 2 arguments, but got 1 arguments.”
Hi Ken, please let me know the complete formula that you are using after getting first 100 results.
After I added the first 100 I then changed the code to the code shown below and placed it into row 101, column A.
=importXml(“https://www.google.com/search?q=site:tubeamplifierparts.com&num=100&start=101″,”//cite”)
I am seeing a discrepancy in the number of URLs returned if I do a site search on Google (239 results) compared to using this function to return the URLs into a spreadsheet (193). Do you have any idea why these numbers would differ? This is an awesome function and tool! Thanks!
Hi Paul,
Thanks for the comment. I can think of no reason as to why there should be any difference. This formula simply gets the same index XML results which Google shows after formatting as HTML. I would suggest you double check if you’re using the formula in right way (you need to change the start value every time). I will let you know if a possible reason for such discrepancy would crop up in my mind. Glad you found this post useful! Stay tuned.
Excellent – just what I needed! Most other stuff I read online was too complex, or didn’t pick up all links, but with your snippet it was just a matter of popping into a spreadsheet, offsetting a couple of times and we got a the 250+ URLs we needed. Thanks!
I am glad to be of help, Karl. Thank you for taking time out to comment.