GLOBAL BRIDGE LABS
← All posts/Website Development

Crawl your website before a migration: what to export

Crawl your website before a migration: which crawler settings to use, what to export and how the crawl becomes the start of your redirect map.

By Dhanushka Pinto, Co-founder / DirectorPublished 3 min read
Crawl website before migration: key takeaways infographic by Global Bridge Labs
Key takeaways from this article. Share it with the link and credit Global Bridge Labs.
On this page

Key takeaways

Before anything changes, crawl the old site and keep the export. The crawl records every URL a crawler can reach, with its status code, title, main heading, canonical tag, word count and internal links. It becomes the backbone of your redirect map and the reference you compare the new site against. Once the old site is switched off, you cannot crawl it again.

  • Crawl the live site before any changes.
  • Export URLs, status codes, titles and headings.
  • Include images, PDFs and other files.
  • Keep the export safe; you cannot redo it later.

Need help running a pre-migration crawl? Message us on WhatsApp.

Chat on WhatsApp →

Why crawl before a migration?

A crawl gives you a full, dated picture of the site as it is. You need it for three jobs: to build the list of URLs that need redirects, to record what each page contained so content can be carried across, and to compare the new site with the old one after launch. Many teams only realise they need the old titles or headings once the old site has gone.

What settings should you use?

Most desktop crawlers have a free tier that covers small sites. Set them up to capture more than just pages.

  • Crawl the live domain, all subdomains you use, HTTP and HTTPS.
  • Include images, PDFs and other file types.
  • Follow links in the XML sitemap as well as the menu.
  • Record redirects and their targets, not just final pages.
  • Extract titles, descriptions, headings and canonical tags.
  • Save the crawl file and a spreadsheet export.

What do you do with the export?

Turn the export into the first column of the redirect map. Remove duplicates, such as the same page with and without a trailing slash, and note pages that already redirect. Then add URLs the crawler could not find, from Search Console, analytics and backlink reports. Keep a copy of titles, headings and word counts beside each URL, so you can check content parity on staging.

Want us to turn your crawl into a redirect map? Message us on WhatsApp.

Chat on WhatsApp →

What does this look like in practice?

A pattern we see: the crawler finds 180 pages, but Search Console lists over 300 URLs that received clicks in the last year, including old landing pages no longer linked from anywhere. Without combining both sources, those orphan pages would have been missed, and several of them brought steady enquiries.

Pre-migration crawl checklist

  • Crawl the live site before any change.
  • Include files, subdomains and both protocols.
  • Export titles, headings, canonicals and status codes.
  • Add URLs from Search Console and analytics.
  • Store the export where everyone can find it.

Next step

Run a crawl of your current site this week, even if the migration is months away. Save it with the date in the file name, and repeat it just before launch.

Message us on WhatsApp about your pre-migration crawl, or book a 30-minute consultation.

Chat on WhatsApp →

Sources and further reading

Frequently asked questions

What is a pre-migration crawl?

It is a full scan of your current website with a crawler tool, recording every reachable URL, its status code, title, headings, canonical tag and links. It is the starting point for your redirect map and the reference for checking that the new site carries across everything that matters.

Which tool should I use to crawl my website?

Any reputable desktop or cloud crawler will do. Many have a free tier suitable for small sites of a few hundred URLs. Choose one that can export to a spreadsheet, record redirects and extract titles and headings. The tool matters less than keeping the export safe.

Does a crawl find every page on my site?

No. A crawler only finds pages linked from somewhere it can reach. Orphan pages, old campaign pages and files linked only from emails or other sites are often missed. Add URLs from Search Console, analytics, your sitemap and backlink reports to fill the gaps.

Written by

Dhanushka Pinto
Dhanushka Pinto
Co-founder / Director

Global Bridge Labs (GBL) is a UK–Sri Lanka partner for social media, websites and BPO. Everything here comes from client delivery, not theory.

Share this article

Reading is good.
Fixing is better.

30 minutes with our team and you'll leave knowing which of the three problems to fix first.

Get a Free Quote →
Keep reading