robots.txt Tester for Rails Apps

Fetch the robots.txt of any site, see how it is parsed, and find out which single rule decides whether a crawler may request a given path.

All tools
Just the host. The tool fetches /robots.txt from it over HTTPS.
The path as the crawler sees it, starting with a slash.
The crawler's token, not its full user-agent string. Matching is case-insensitive.

Fetching the other site can take a few seconds.

This tool fetches the file from our server rather than your browser, so it works without JavaScript and reads hosts your browser could not.

public/robots.txt or a Rails route

Rails offers two honest homes for this file and they are not interchangeable. `public/robots.txt` is served straight off disk by nginx, never boots Rails, costs nothing per request and turns up in code review like any other file - which is the right answer whenever the content is fixed. A route, `get "/robots.txt", to: "robots#show", defaults: { format: :text }`, is the right answer once the content has to be computed. One trap makes the route worthless: if the action renders through the application layout, or the format is left to content negotiation, the crawler is handed a doctype and a nav bar, and something that is not `text/plain` is not a robots.txt at all. Declare the format, render a layoutless text view, and assert the content type in a request spec rather than trusting it.

One file cannot serve staging and production

A file checked into public/ is the same file in every environment, which is how staging gets indexed. Production has to invite crawlers in and staging has to keep them out, and one static file cannot say both things at once. A route can: render a view that emits `User-agent: *` and `Disallow: /` unless `Rails.env.production?`, or key the answer on `request.host` when one deployment serves several domains. Keep the production copy in a view rather than a heredoc in the controller and it stays as reviewable as the static file was. Basic auth across the whole staging host is stronger still, because it also stops whatever never asks for robots.txt in the first place.

Which rule actually decides

A crawler reads the whole file, picks exactly one group, and ignores every other group in it. It picks the group whose `User-agent` token most specifically names it, and the wildcard group applies only when no group does: a file with a `Googlebot` section and a `*` section is telling Googlebot to read the first and nothing else, including the rules it liked in the second. Inside that group the longest matching pattern wins, and when an `Allow` and a `Disallow` of equal length both match, `Allow` takes it. That is why `Disallow: /rails/` alongside `Allow: /rails/active_storage/blobs/` still lets blobs through, why a bare `Disallow: /admin` also covers `/admin/users`, and why `Disallow: /users/sign_in` leaves the rest of `/users` open.

Questions Rails developers ask about robots.txt

Put it in public/robots.txt when the file is the same everywhere the app runs. Nginx serves it off disk, the request never reaches Ruby, and the file is reviewable in the diff like any other asset. Move it to a route when the answer has to differ per environment or per host - a staging deployment that must disallow everything, or a multi-tenant app where each tenant's file mentions a different sitemap. The moment the content is computed, a controller is the honest place for it.

Usually yes for the redirect and representation routes, because every variant of every attachment is its own URL and a crawler will happily walk all of them. Disallowing /rails/active_storage/ keeps that budget for your real pages. Public images you actually want indexed are better served from a route of your own that you allow explicitly, or from a CDN host with its own robots.txt, because an Allow inside a disallowed prefix only works if the crawler supports longest-match and you have tested that it does.

Almost always because staging deploys the same public/robots.txt as production, which invites the crawler in, and somebody linked to a staging URL. A checked-in static file cannot say two different things, so either serve robots.txt from a route that answers Disallow / unless Rails.env.production?, or put HTTP basic auth in front of the whole staging host. The second is stronger - it also keeps staging out of the training set of anything that ignores robots.txt entirely.

Yes. Sitemap is the one directive in the file that is not scoped to a group and not relative to the host - it takes a full URL, so Sitemap /sitemap.xml is silently useless. Generate it with the same helper that builds your other absolute URLs so the scheme and host come from configuration rather than from a string you typed once. It also does not need to sit on the host that serves robots.txt, which is how a sitemap on a CDN or a separate assets host gets declared.

No, and treating it as one is the usual mistake. robots.txt is a crawl instruction, not an access control, and the file is public, so a Disallow line advertises the path you are hiding. Authorisation is what keeps /admin private - the policy layer, not a text file. A disallowed URL can still be listed with no snippet if somebody links to it, so if you need a page kept out of an index and it is reachable without signing in, let the crawler fetch it and answer with a noindex header or meta tag instead.

Start creating your next app now

Get BlueprintRead docs