About our crawler

What it is

MetaManagerBot fetches pages to read their metadata — the <title>, the meta description, Open Graph and Twitter Card tags — and reports how the page will look when somebody shares it. It identifies itself on every request:

MetaManagerBot/1.0 (+https://metamanager.dev/bot)

What it does not do

It does not execute JavaScript, submit forms, follow links marked rel="nofollow", or download images beyond the first few kilobytes needed to measure a social image's dimensions. It does not collect email addresses or personal data, and it does not index your content for search.

How often it visits

Single page checks happen when somebody pastes your URL into the checker, and only then. Site audits are started deliberately by a signed-in customer auditing a site, and are capped at 500 pages per run. There is no continuous or scheduled crawling of sites nobody asked us about.

Requests are made one at a time with a pause between them. If your robots.txt declares a Crawl-delay, we use it.

How to block it

We read robots.txt before anything else and obey it. To refuse us entirely:

User-agent: MetaManagerBot
Disallow: /

Or to keep one section private while allowing the rest:

User-agent: MetaManagerBot
Disallow: /private/

A disallowed page is never requested. It is reported to whoever ran the check as skipped, so they know your site declined rather than assuming the page was broken.

Getting in touch

If the bot is behaving badly — ignoring your rules, requesting too often, or anything else — that is a defect and we want to know. Open an issue on the project and include the timestamps from your logs.