Manifest V3 · Chrome Extension

Browse the web like
a robots.txt crawler

Enforces robots.txt rules for your own browser traffic. Every domain you visit is checked — disallowed paths are blocked before the request reaches the network.

Download latest release View source

Screenshots

Popup — toggle blocking, open live log or cache editor
Popup — toggle blocking, open live log or cache editor
Live log — blocked requests in real time with line-level links
Live log — blocked requests in real time with line-level links
robots.txt cache editor — annotated, inline-editable, with URL tester
robots.txt cache editor — annotated, inline-editable, with URL tester
Race condition banner — reload prompt when blocking was missed
Race condition banner — reload prompt when blocking was missed
Options — select user-agent, clear cache
Options — select user-agent, clear cache

Features

🚫

Real blocking

Uses Chrome's declarativeNetRequest API. Disallowed requests are cancelled before they reach the network — not just logged.

📋

Live log

See every blocked request in real time: URL, method, resource type, reason, and the exact robots.txt line that triggered the block.

✏️

Editable cache

Inspect and edit any cached robots.txt directly in the browser. Changes take effect immediately — no refetch needed.

🔍

URL tester

Type any URL to instantly see whether it would be allowed or blocked — and which rule is responsible.

🤖

Selectable user-agent

Test as *, Googlebot, or any custom agent. Switching triggers an immediate rule rebuild for all cached domains.

⚠️

Race condition detection

First visits can't be blocked (rules must be installed ahead of requests). The extension detects this, logs it, and shows a reload banner.

💾

Persistent rules

DNR rules and cache survive browser restarts and service worker sleep. Revisited domains are blocked immediately from the first request.

🔗

Deep links

Click a line number in the live log to jump directly to that line in the cache editor — host pre-selected and line highlighted.

How it works

Manifest V3 removed blocking webRequest, so blocking and observation are split across two mechanisms.

1

Navigation detected

webNavigation.onBeforeNavigate fires when you navigate to any HTTP(S) page.

2

robots.txt fetched & cached

The extension fetches /robots.txt for the domain and caches it for 24 hours. Server errors trigger a deny-all; 4xx or no file trigger allow-all.

3

DNR rules installed

Each Allow and Disallow path is translated into a declarativeNetRequest dynamic rule and installed in Chrome — before any sub-resources fire.

4

Requests blocked

Chrome evaluates the rules natively — no JavaScript involved. Longer, more specific patterns take priority; Allow beats Disallow at equal specificity.

5

Observer logs the verdict

A non-blocking webRequest listener re-runs Google's official RobotsMatcher for ground-truth logging. The exact robots.txt line number is resolved and sent to the live log.

⚠ First-visit race condition
On the very first visit to an uncached domain, requests may fire before rules are installed. The extension detects this retroactively, logs affected URLs as robots-race, changes the extension icon, and shows a reload banner on the tab — so you can force a clean page load with rules already in place.

Install

Not yet on the Chrome Web Store — install manually from the latest release.

  1. Download robotstxt-vX.Y.zip from the latest release
  2. Unzip the file into a folder on your machine
  3. Open chrome://extensions in Chrome
  4. Enable Developer mode (top right toggle)
  5. Click Load unpacked and select the unzipped folder
  6. Pin the extension to the toolbar — you're done
Download latest release