Paulund
Subscribe
← Changelog

Lookout15 November 2026, 4 min read

Adding http, www and trailing-slash duplicate URL checks to Lookout

Find the http, www, slash and case forms of your URLs that don't settle on one address.

I added a URL form check to Lookout as a way to check my own existing sites. A page can answer as http:// or https://, or with or without www. It can also answer with or without a trailing slash, in two letter cases, or with tracking parameters such as utm_source on the end. Lookout now checks that a site settles on one of those forms. It spends at most 23 extra requests a scan doing it, and only one kind of finding, a home page form that doesn't redirect properly, counts in a site's health.

Lookout already followed redirect chains for up to 10 hops inside a site. It dropped links to another scheme or host before recording them, and it never requested the other scheme or www forms of the home page.

What a scan now tries

A scan now makes a few extra requests:

  • The home page in its other forms. Three requests: the other scheme, the other www form, and both together. Each should redirect with a 301 or 308 to the preferred form. That is the address you added the site with, upgraded to https:// if you added it as http://.
  • Trailing slashes. Up to 10 linked paths, tried in the other trailing-slash form.
  • Letter case. Up to 10 linked paths that contain capital letters, tried in lower case.

I capped it at 23 extra requests a scan, and only the 3 home page requests when a site has no such links. They don't count against the 7,500-page limit.

Here is one of those home page requests against paulund.co.uk, made by hand with curl:

Terminal
$ curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" http://www.paulund.co.uk/
301 https://paulund.co.uk/

A 301 to the preferred form like that is what Lookout wants to see. A 302, 303 or 307 gets a low "Temporary redirect to the preferred form" finding. A form that answers with a 200 instead of redirecting is also flagged as low. When a linked path answers in both trailing-slash forms, or in both letter cases, Lookout reports the pair as a duplicate.

The links on each page are checked too. When a page links to another scheme or host form of the site, or adds tracking or session parameters to an internal link, the finding names that page. The parameters it looks for are utm_*, gclid, fbclid and msclkid, and the session keys sid, sessionid, phpsessid and jsessionid. A key you already ignore in Lookout's URL parameter settings is skipped.

A request that fails, or answers with a 4xx, a 5xx or a rate limit, counts as not checked. It never raises a finding, because some sites have no www host at all.

Only the home page redirect counts in health

I made a home page form that doesn't redirect properly a low finding that counts in the site's health. The trailing-slash, letter-case and link findings are advisory. They appear in the scan, but they stay out of health, urgent counts and alerts.

They stay out of health because they would land on every customer's site at its next scan, and Google picks one canonical URL from a set of duplicates either way.

What's still rough

Each scan re-checks open trailing-slash and letter-case findings first, and those re-checks use up the 10 requests for that kind. A site with 10 or more open trailing-slash duplicates spends all 10 on them, so a new one isn't found until some are fixed. The findings past the cap stay open, so they aren't cleared by mistake.

Each of these extra requests only looks at the first response, so it doesn't follow a redirect any further. The list of tracking parameters is fixed. You can skip a key in the URL parameter settings, but there's no screen to add one.

Checking canonical tags is next.