A website can look completely normal while still having broken links, missing metadata, invalid structured data, crawlability problems, or performance issues across pages you never opened.
As a frontend developer, I spend a lot of time checking websites from the browser.
I usually have several things to verify: Does the page render correctly? Are the SEO tags there? Is the canonical correct? Are Open Graph tags present? Are there broken links? Is structured data available? Are important pages actually discoverable? How does performance look?
Instead of opening a website and manually inspecting pages, you can run an audit from the terminal.
The crawler discovers pages and resources, then runs different checks against what it finds.
These are small pieces of HTML, but missing or incorrect values can have a noticeable impact on how pages are interpreted and shared.
This becomes particularly difficult to notice on larger websites where some pages may rarely be visited.
A crawler can follow links across the site and identify URLs that don't return the expected response.
Structured data is another area where manually checking every page becomes tedious.
sitebot can inspect structured data found in pages and help surface pages where it is missing or potentially problematic.
One of the interesting parts of building a crawler is realizing that the list of URLs you think a website has isn't necessarily the list a crawler can actually discover.
Core Web Vitals provide useful signals around the actual user experience of a page, so I wanted performance to be part of the same developer-oriented workflow rather than something completely disconnected from the rest of the audit.
The exact output is still evolving, but the idea is to make the result easy to understand and useful enough to act on.
Traditional crawling and modern frontend applications don't always see the web in the same way.
A page that depends heavily on client-side rendering can behave very differently depending on how it is accessed.
A tool can technically detect hundreds of problems and still be frustrating to use.
If you're interested in website crawling, SEO, frontend performance, or developer tooling, I'd genuinely appreciate feedback.
What is something you regularly check manually that you wish you could just run from the terminal?