Auth Middleware Redirects robots.txt to the Login Page

Conclusion

Middleware that sends unauthenticated requests to /login tends to treat well-known files like robots.txt and sitemap.xml the same way. Crawlers receive a redirect to the login page instead of the file, and the Lighthouse SEO audit fails.

The cause is in two layers.

1. robots.txt and friends are missing from the middleware allow-list → treated as unauthenticated and 307’d to /login
2. Public pages added later inherit the root layout’s noindex → stopping the redirect still leaves them out of search results

Fix one and the other’s symptom remains.

Symptom

In a setup where public pages (an application form, say) were added later on top of an internal business system, running a Lighthouse audit drops the SEO score sharply.

Tracing it, requests to /robots.txt hit the middleware’s auth check and are 307’d to /login. What the crawler receives is not robots rules but a nudge towards the login screen.

manifest.webmanifest was caught the same way. The browser tries to parse it as JSON, so the returned login-page HTML is a syntax error, and the console fills with errors. Lighthouse’s “are there console errors” check loses points there too.

Cause

Layer one — well-known files missing from the allow-list

This app’s middleware inspects the path to decide “may this be accessed without logging in”. It is a whitelist, enumerating the public paths.

const isPublic =
  path.startsWith("/p/") ||
  path === "/apply" ||
  path.startsWith("/apply/") ||
  path === "/terms" ||
  // ...

It looks exhaustive, but /robots.txt, /sitemap.xml and /manifest.webmanifest are absent. These are not files you think of as pages. Enumerating “which paths should be public” naturally means listing the pages you actually open in a browser, and the well-known files that crawlers and browsers fetch implicitly fall outside that.

A path not on the allow-list falls into the auth branch and is redirected to /login.

Layer two — public pages inherit the root layout’s noindex

The root layout defaulted to robots: { index: false, follow: false }. With the internal business system making up most of the app, not wanting it in search is a sound judgement.

export const metadata: Metadata = {
  // ...
  robots: { index: false, follow: false },
};

The problem is that public pages added later inherit that default. Next.js merges metadata from nested layouts, but unless robots is explicitly overridden, the root’s noindex stays in effect on the page.

Fixing only the layer-one 307 makes robots.txt return correctly, but a public page that inherited noindex still will not appear in search even though robots now permits it. The two causes are independent, and fixing one does not fix the other.

The fix

Layer one — add the well-known files to the allow-list

const isPublic =
  path === "/robots.txt" ||
  path === "/sitemap.xml" ||
  path === "/manifest.webmanifest" ||
  path.startsWith("/p/") ||
  path === "/apply" ||
  // ...

Explicit robots.ts and sitemap.ts were also added, permitting only what should be public and explicitly denying authenticated paths and URLs containing tokens.

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: "*",
      allow: ["/apply", "/terms"],
      disallow: "/",
    },
    sitemap: `${APP_URL}/sitemap.xml`,
  };
}

Putting disallow: "/" first means that as paths grow, only what is explicitly allowed becomes public, so a missed allow-list entry fails closed rather than exposing something.

Layer two — opt public pages back into indexing

The root layout keeps noindex; the public page’s metadata overrides it.

export const metadata: Metadata = {
  title: "...",
  description: "...",
  robots: { index: true, follow: true },
};

That override has to be written every time a public page is added. Flipping the root default to index: true would instead risk exposing internal pages to search, so the default stays private with individual opt-ins.

Why it went unnoticed

Neither cause shows up in ordinary browsing. A logged-in developer opening /apply sees the page correctly, and opening /robots.txt by hand hides the 307 because the browser follows the redirect silently.

It only surfaces from a “logged out, does not follow redirects” viewpoint like Lighthouse’s. The middleware allow-list and the layout’s robots inheritance are both accumulations of changes that look reasonable line by line in review. Unless adding a public page comes with a per-page check for “do the well-known files pass” and “did I override noindex”, the same gap keeps reappearing.

For another case where a fix in one layer just revealed the next, see Astro 7 breaking content collections in three places at once.