There are numerous good reasons you may perhaps have to have to seek out all of the URLs on an internet site, but your correct aim will identify Whatever you’re hunting for. As an illustration, you might want to:
Detect just about every indexed URL to research concerns like cannibalization or index bloat
Collect recent and historic URLs Google has observed, specifically for website migrations
Come across all 404 URLs to recover from put up-migration faults
In Each and every circumstance, one Resource gained’t Provide you with almost everything you require. Regrettably, Google Search Console isn’t exhaustive, in addition to a “internet site:case in point.com” lookup is restricted and tricky to extract info from.
In this put up, I’ll stroll you through some equipment to develop your URL record and before deduplicating the info utilizing a spreadsheet or Jupyter Notebook, determined by your web site’s dimension.
Aged sitemaps and crawl exports
For those who’re on the lookout for URLs that disappeared with the Reside site a short while ago, there’s a chance an individual on your own workforce could possibly have saved a sitemap file or perhaps a crawl export before the modifications were made. In the event you haven’t now, check for these documents; they might generally offer what you would like. But, in case you’re reading this, you most likely did not get so Fortunate.
Archive.org
Archive.org
Archive.org is an invaluable Instrument for SEO tasks, funded by donations. In the event you look for a site and select the “URLs” option, you can entry as much as 10,000 listed URLs.
Nonetheless, there are a few restrictions:
URL limit: It is possible to only retrieve nearly web designer kuala lumpur ten,000 URLs, that's inadequate for larger websites.
Top quality: A lot of URLs could possibly be malformed or reference resource data files (e.g., illustrations or photos or scripts).
No export alternative: There isn’t a designed-in strategy to export the record.
To bypass the lack of an export button, make use of a browser scraping plugin like Dataminer.io. Having said that, these limitations suggest Archive.org may not offer a complete solution for larger websites. Also, Archive.org doesn’t indicate no matter whether Google indexed a URL—but if Archive.org found it, there’s a very good probability Google did, way too.
Moz Professional
While you may perhaps normally make use of a backlink index to find exterior sites linking to you personally, these tools also discover URLs on your web site in the procedure.
Tips on how to use it:
Export your inbound one-way links in Moz Pro to get a speedy and straightforward list of focus on URLs from a web site. For those who’re managing a huge website, consider using the Moz API to export info outside of what’s workable in Excel or Google Sheets.
It’s important to Take note that Moz Pro doesn’t verify if URLs are indexed or found by Google. However, because most web pages implement the same robots.txt procedures to Moz’s bots as they do to Google’s, this method normally will work perfectly as a proxy for Googlebot’s discoverability.
Google Look for Console
Google Look for Console features several valuable resources for constructing your list of URLs.
Hyperlinks experiences:
Just like Moz Professional, the Back links section presents exportable lists of goal URLs. Sad to say, these exports are capped at 1,000 URLs Every. You could utilize filters for particular pages, but considering that filters don’t implement on the export, you may perhaps really need to rely upon browser scraping applications—limited to five hundred filtered URLs at any given time. Not best.
Efficiency → Search engine results:
This export gives you a list of webpages obtaining research impressions. Though the export is proscribed, You can utilize Google Lookup Console API for greater datasets. You can also find no cost Google Sheets plugins that simplify pulling additional intensive knowledge.
Indexing → Pages report:
This area provides exports filtered by challenge type, however they are also constrained in scope.
Google Analytics
Google Analytics
The Engagement → Internet pages and Screens default report in GA4 is a wonderful supply for gathering URLs, having a generous Restrict of 100,000 URLs.
A lot better, you may use filters to create unique URL lists, efficiently surpassing the 100k Restrict. For instance, if you want to export only website URLs, observe these steps:
Phase 1: Include a segment towards the report
Stage two: Click on “Create a new phase.”
Phase 3: Determine the segment which has a narrower URL pattern, such as URLs that contains /blog/
Observe: URLs located in Google Analytics may not be discoverable by Googlebot or indexed by Google, but they offer worthwhile insights.
Server log data files
Server or CDN log data files are Maybe the last word Device at your disposal. These logs capture an exhaustive list of each URL route queried by end users, Googlebot, or other bots over the recorded time period.
Concerns:
Data dimensions: Log files is usually substantial, numerous web pages only keep the final two weeks of knowledge.
Complexity: Examining log files could be hard, but various tools can be found to simplify the process.
Combine, and great luck
After you’ve collected URLs from these sources, it’s time to mix them. If your website is sufficiently small, use Excel or, for more substantial datasets, resources like Google Sheets or Jupyter Notebook. Be certain all URLs are persistently formatted, then deduplicate the record.
And voilà—you now have an extensive listing of current, previous, and archived URLs. Fantastic luck!