Skip to main content

A free, open source SEO crawler

Hardcrawl is a desktop SEO spider and technical SEO auditing tool. Identify rendering issues, broken links, and map link equity flows. Find keyword gaps, map identical content and automate redirect mapping.

What the free SEO crawler checks

A technical SEO audit, from one crawl

A look at the views you get with every crawl. Click any screenshot to enlarge it.

Your crawl data stays on your machine

Hardcrawl runs as a desktop application for PC and Mac. No URL request limits, beoynd

Download Hardcrawl

COMING SOON

More tools from Archytas

Free SEO crawler FAQ

Is Hardcrawl really a free SEO crawler?

Yes. Hardcrawl is free and open source.

What is an SEO crawler?

An SEO crawler (or SEO spider) visits the pages of a website by following links, just like a search engine bot, and records the technical data that affects how those pages are crawled, indexed and ranked.

Where is my crawl data stored?

Hardcrawl is a desktop app, and crawls are stored in a local database on your own computer.

Can Hardcrawl crawl JavaScript-rendered sites?

Yes. Hardcrawl can crawl raw HTML or render pages in a browser, and can compare the two.

How does the content gap analysis work?

Similar to Google, Hardcrawl takes a list of keywords, and uses a BM25 algorithm to identify relevant URLS from your crawl. If you use the included embedding functionality, it can also then identify the most relevant content block, and the semantic similarity. The goal is that you can take a large piece of keyword research, and identify exactly where the content gaps lie.

How does the duplicate content mapping work?

During a crawl, you have an option to embed each page - similar to how LLMs mathematically models language. Duplicate content mapping then compares each document to its nearest neighbors, scoring the 'distance' between the two to identify duplicated URLS, or potential interlinking opportunties.