A free, open source SEO crawler
What the free SEO crawler checks
A technical SEO audit, from one crawl
-
Live crawl results
Watch every URL stream into a sortable table - status code, title, meta description, first H1, indexability and effective canonical - while the crawl is still running.
-
Canonicals & indexability
See each page's effective canonical, spot mismatches and chains, and find out exactly why a page is or isn't indexable.
-
Redirects & site structure
Follow redirect chains and loops, measure crawl depth and internal links, and find orphan pages.
-
hreflang & structured data
Validate hreflang values and return tags, and check Schema.org markup and rich results eligibility.
-
Headings, links & ARIA
Audit heading structure, hyperlinks and other links, plus ARIA usage across your templates.
-
JavaScript rendering
Crawl raw HTML or render pages in a real browser, and compare the two to see what JavaScript changes.
-
Speed & site checks
Collect page speed signals, and probe robots.txt and sitemaps at the start of every crawl.
-
Content analysis
Compare crawls against keyword lists to identify gaps.
-
Content mapping
Match similar content across crawls to map redirects, or cluster similar content to identify duplication and interlinking opportunities.
-
Link equity mapping
Map the flow of link equity through your site.
-
An interface built for audits
A UX designed around the needs of an SEO carrying out a technical audit.
See Hardcrawl in action
A look at the views you get with every crawl. Click any screenshot to enlarge it.
-
Crawl setup
Discover pages from a start URL, or audit a fixed list of URLs. Choose a rendered (browser) or HTML-only crawl, set the crawl scope, and optionally crawl CSS, JS, images and fonts.
-
HTTP header
One row for every unique URL discovered, with its status code, content type, response size and page template - the foundation of any technical audit.
-
Hyperlinks
Every link on every page, with its type, internal or external status, and the status code of the page it sits on. Find broken and redirecting links fast.
-
Inlinks
See how many distinct pages link to each URL. Spot orphaned pages and important pages that are poorly linked.
-
Assets and other links
Audit images, stylesheets, scripts, canonical and alternate links, and other non-hyperlink references discovered during the crawl.
-
Metadata
Review titles, meta descriptions and robots directives across the whole site, and filter for missing, duplicate or over-length values.
-
Canonicals
Compare HTML and HTTP header canonicals for every page, and find missing, conflicting or non-indexable canonical targets.
-
Site structure
Check how many clicks each page sits from the start URL. Important pages buried deep get crawled less often and receive less internal link value.
-
Template summary
Pages are grouped by their HTML layout, so a problem that repeats across a template can be fixed once, in the template code.
-
Schema
See the structured data on every page, validate it, and spot which Schema.org types are eligible for Google rich results.
-
Content and headings
Audit heading structure across the site. Find missing or multiple H1s and subheadings that skip a level.
-
Rendered difference
Compare the raw HTML a non-JavaScript crawler sees with the fully rendered page, to find content, links, canonicals and schema that only exist after JavaScript runs.
-
Speed
Review response timings, page weight and HTTP version for each page, and find slow templates that need attention.
-
ARIA summary
A per-page summary of ARIA landmarks, widgets and links worth reviewing, to spot accessibility problems across whole page templates.
-
ARIA tags
Drill into the individual elements flagged on each page, such as buttons and navigation landmarks, and see exactly which ARIA attributes they carry.
-
WebMCP
Find the WebMCP tools that pages register for AI agents, and check that each one is valid and not duplicated.
-
Link equity mapping
A 1-100 estimate of the internal link equity each page accumulates from your site's own linking structure.
-
Similarity mapping
Match similar content across crawls to map redirects, or group near-duplicate pages to find duplication and interlinking opportunities.
-
Export
Choose the views and templates you need, then export your audit as an Excel workbook or a folder of CSV files.
Your crawl data stays on your machine
Hardcrawl runs as a desktop application for PC and Mac. No URL request limits, beoynd
Download Hardcrawl
COMING SOON
More tools from Archytas
-
AISpy
Track your brand's visibility across ChatGPT, Claude, Gemini, Perplexity and Google AI Mode.
-
ClusterDuck
Cluster keywords by meaning, right in your browser.
-
Bulk DR Checker
Check the Ahrefs Domain Rating for hundreds of domains at once.
-
Common Crawl Checker
Find out whether your site is in the Common Crawl dataset.
Free SEO crawler FAQ
Is Hardcrawl really a free SEO crawler?
Yes. Hardcrawl is free and open source.
What is an SEO crawler?
An SEO crawler (or SEO spider) visits the pages of a website by following links, just like a search engine bot, and records the technical data that affects how those pages are crawled, indexed and ranked.
Where is my crawl data stored?
Hardcrawl is a desktop app, and crawls are stored in a local database on your own computer.
Can Hardcrawl crawl JavaScript-rendered sites?
Yes. Hardcrawl can crawl raw HTML or render pages in a browser, and can compare the two.
How does the content gap analysis work?
Similar to Google, Hardcrawl takes a list of keywords, and uses a BM25 algorithm to identify relevant URLS from your crawl. If you use the included embedding functionality, it can also then identify the most relevant content block, and the semantic similarity. The goal is that you can take a large piece of keyword research, and identify exactly where the content gaps lie.
How does the duplicate content mapping work?
During a crawl, you have an option to embed each page - similar to how LLMs mathematically models language. Duplicate content mapping then compares each document to its nearest neighbors, scoring the 'distance' between the two to identify duplicated URLS, or potential interlinking opportunties.