“We can't be found on Google” is one of the most common sentences in conversations about a company website — and one of the vaguest. Two completely different situations hide behind it. Either the search engine does not know the page at all: then no better copy and no nicer headline will help, because nobody gets to see it. Or the search engine knows the page but shows it far down the list: then every technical fix is wasted effort, because technically everything is already fine. Mixing up these two cases costs weeks of work that misses the actual problem. This article therefore does not offer a pile of tips but a fixed order of checks: first the question of whether the page is indexed at all, then six technical blockers in a set sequence, and only afterwards the content side. Every step has a test that gives a clear answer within minutes.
Two problems that must be separated
“We can't be found” describes a feeling, not a finding. Troubleshooting needs two clearly separated terms. Indexing means a search engine has fetched the page, processed it and added it to its data set. Ranking means that among all the pages it has stored, this one appears in a certain position when somebody types a query. These are two levels with entirely different causes and remedies. A page that is not in the index has a technical problem or a discovery problem. A page that is in the index and still gets no visits has a content problem — or simply strong competition.
The scale helps to put things in perspective. Analysing roughly 14 billion pages from its own data set, Ahrefs found that 96.55 percent (Ahrefs) of all pages studied receive no visits at all from Google Search, while a further 1.94 percent (Ahrefs) get between one and ten visits per month. These figures are not evidence that search is broken. They show that “present, but without search visits” is the statistical norm on the web — and that the question “why aren't we at the top” rarely has a single defect as its answer.
Google gives no assurance that it will crawl, index or serve your page, even if your page follows the Google Search Essentials.
That statement belongs at the start because it sets expectations straight. There is no entitlement to being indexed and no entitlement to a particular position. What does exist is a manageable list of reasons why a page is very unlikely to be included — and that list can be worked through. The table below assigns the most common observations from practice to the level they belong to.
| Observation | Likely level | First test |
|---|---|---|
| The page does not appear even when searching for its exact title | Indexing | URL Inspection in Search Console |
| Only the home page appears, all subpages are missing | Indexing of individual pages | Open the page indexing report |
| The site appears for the company name but for no service-related query | Ranking | Is there a dedicated page for that query at all? |
| There are plenty of impressions but hardly any clicks | Presentation in the result | Check the title and first line of the result |
| Nothing from the domain appears and it is only a few weeks old | Discovery | Sitemap submitted, any inbound links? |
What this article deliberately leaves out
The first test: is the page in the index at all?
Before anything is changed, one question needs an answer: does the search engine know this address? The quickest stopgap is the site: search operator, entered as site:example.com or, for a section, as site:example.com/services/. If nothing shows up, that is an indication — but not proof, and the displayed number of results is certainly not.
The list of URLs returned is not always exhaustive. Bigger sites shouldn't expect to see all their URLs in the results.
A reliable answer only comes from the URL Inspection tool in Search Console. For a single address it answers four questions at once: whether the page is indexed, whether crawling is allowed, whether indexing is allowed, and which address Google has selected as the canonical version. Two values are reported side by side — the canonical declared by the site owner and the one selected by Google (Google Search Console Help). If those two differ, the cause has usually already been found.
A trap inside the inspection tool
- Search for the exact page title in quotation marks. If the page shows up, it is indexed and the problem sits at the ranking level.
- Check
site:your-domain.comand note which page types appear and which are missing — home page, service pages, guide articles. - Open URL Inspection for one missing address and read the four rows: index status, crawl allowed, indexing allowed, Google-selected canonical.
- Open the page indexing report and sort the reasons by count. The most frequent reason is the first work order.
The page indexing report is the single most useful tool in this whole article, because it names the cause instead of merely describing the state. Among other reasons it distinguishes “Blocked by robots.txt”, “Excluded by ‘noindex’ tag”, “Page with redirect”, “Alternate page with proper canonical tag”, “Duplicate without user-selected canonical”, “Crawled – currently not indexed” and “Discovered – currently not indexed” (Google Search Console Help). Each of those reasons has a different remedy, and the checklist below works through them in exactly that order.
Check 1: is robots.txt blocking the page?
The robots.txt file sits in the root directory of a domain and controls which areas may be fetched. It applies per host, protocol and port: the file at https://www.example.com/robots.txt does not apply to https://example.com/ (Google Search Central, robots.txt specification). That is one of the quiet sources of error, because in practice only one of the two variants usually gets checked. In the HTTP Archive analysis, 83.9 percent (HTTP Archive Web Almanac) of sites measured on mobile serve a robots.txt with status code 200, while 14.1 percent (HTTP Archive Web Almanac) return a 404 — which is harmless, because Google treats a missing file like a file without restrictions.
# What an unremarkable robots.txt looks like
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
# These two lines block the entire website.
# They usually date back to the build phase and
# were forgotten when the site went live:
#
# User-agent: *
# Disallow: /The technical limits are quickly told and rarely the actual problem: Google reads at most 500 kibibytes (Google Search Central, robots.txt specification) of a robots.txt, caches it for up to 24 hours (Google Search Central, robots.txt specification) and follows at least five (Google Search Central, robots.txt specification) redirect hops before treating the file as absent. Only 0.06 percent (HTTP Archive Web Almanac) of the measured files exceed the parsing limit, and 97.82 percent (HTTP Archive Web Almanac) stay below 100 kilobytes. The behaviour on server errors matters more: if robots.txt answers with a 5xx status, Google pauses crawling for 12 hours (Google Search Central, robots.txt specification) and then falls back to the last cached version for up to 30 days (Google Search Central, robots.txt specification).
A crawl block is not an exclusion from the index
- A
Disallow: /left over from the build phase — the classic total outage. - A robots.txt that exists only under the www variant while the non-www variant returns a server error.
- Blocked directories for scripts, fonts or images that are needed to render the page.
- A missing or wrong
Sitemap:line pointing at another host or at a file that no longer exists. - Rules written for a staging system that travelled along when the files were copied to the live domain.
Check 2: is there a noindex in the source?
The second check takes thirty seconds: open the page source and search for the word “noindex”. The rule can sit as a meta element in the head or be delivered as an HTTP response header — both have the same effect (Google Search Central, documentation on blocking indexing). It is by no means rare: 46.2 percent (HTTP Archive Web Almanac) of pages measured on mobile contain a robots meta element at all, and 3.9 percent (HTTP Archive Web Almanac) of them carry a noindex rule. On thank-you pages, internal search results or staging copies that is intentional. On a service page it is a defect.
<!-- Variant 1: in the head of the page -->
<head>
<meta name="robots" content="noindex, nofollow">
</head>
<!-- Variant 2: as a response header from the server -->
X-Robots-Tag: noindex
<!-- What you want on a page that should be found: -->
<meta name="robots" content="index, follow">
<link rel="canonical" href="https://www.example.com/services/roof-repair/">The double mistake that hides itself
A second cause is harder to spot because the source looks unremarkable: a broken head section. 10.9 percent (HTTP Archive Web Almanac) of pages measured on mobile contain elements that end the head section prematurely, and 22 percent (HTTP Archive Web Almanac) even place an image inside it. Everything after the break — robots meta, canonical, language annotations — slips into the body and is no longer treated as a directive. Even the web.dev notes on metadata make clear that such information belongs in the head and that search engine bots treat it as a request they are expected to honour (web.dev). The matching test is the live inspection in Search Console: it shows the rendered source as Google sees it, not as it looks in the editor.
Finally, every change here needs time. A removed noindex rule only takes effect once the page has been fetched again; if it still shows up in the results, it has probably not been recrawled since the change, and depending on the importance of the page that can take months (Google Search Central, documentation on blocking indexing). URL Inspection lets you request indexing again — with a daily limit and without any assurance that the page will then be included (Google Search Console Help).
Check 3: where does the canonical point?
A canonical names the authoritative address for content that is reachable under several addresses. 65 percent (HTTP Archive Web Almanac) of pages measured on mobile set such an element, and 69 percent (HTTP Archive Web Almanac) do so in the desktop crawl. The problem is rarely that it is missing — it is that it points at the wrong address. The nastiest case is a copied template or staging system whose canonical still refers to the address of the original: the page then tells the search engine itself that it is nothing but a copy of somebody else's page.
Indicating a canonical preference is a hint, not a rule.
That matters in both directions. A wrong canonical can push a page out of the results, while a correct canonical does not enforce a preferred address. Google weighs several signals: whether the page is served over HTTP or HTTPS, existing redirects, the presence of the address in a sitemap and the rel canonical annotation (Google Search Central, documentation on canonicalization). If those signals contradict each other, the search engine decides — and the page indexing report reports it as “Duplicate, Google chose different canonical than user” (Google Search Console Help).
| The canonical points to ... | What follows from it | Correction |
|---|---|---|
| the address of the page itself | the expected normal case | none |
| the home page, on every single subpage | all subpages count as duplicates of the home page | give each page its own address |
| an external domain from a template or staging system | the page is treated as a copy of somebody else's original | switch it to your own domain and request a fresh check |
| the http or the non-www variant | signals are split across two addresses of the same page | pick one variant and permanently redirect the rest |
| a page that itself carries a noindex | the rule effectively carries over to the whole group | check the target or remove the canonical |
Two subtleties come on top. First, a canonical can be set in the HTML and in the HTTP response header at the same time: 1 percent (HTTP Archive Web Almanac) of pages use the header, and in 0.8 percent (HTTP Archive Web Almanac) the two contradict each other. Second, on 2.1 percent (HTTP Archive Web Almanac) of pages measured on mobile the canonical changes during rendering because a script overwrites it afterwards. Both look unremarkable in the editor and only surface in a live inspection. Anyone doing a small regular review round spots such deviations early; what such a routine can look like is described in the article on keeping a website maintained.
Check 4: sitemap, status codes and soft 404s
A sitemap is a list of the addresses a website considers worth finding. It does not replace internal linking, but it speeds up discovery — especially for new or deeply buried pages. The requirements are unambiguous: at most 50,000 addresses (Google Search Central) and 50 megabytes (Google Search Central) uncompressed per file, UTF-8 encoded, with complete absolute addresses. Larger inventories get split across several files plus an index file. And here too Google's own framing applies: submitting a sitemap is a hint, not an instruction (Google Search Central).
Three mistakes turn up especially often. The sitemap is not reachable at the address given, because it moved after a change of system. It contains addresses of the wrong host variant — non-www addresses, say, while the site is served under www. Or it lists pages that carry a noindex or redirect elsewhere; then the signals contradict each other and the sitemap loses its value.
200 for valid pages
The success status. Google passes the content on to the next processing step — which on its own is not yet inclusion in the index (Google Search Central).
301 for moves
According to Google a permanent redirect is a strong signal that the target should be processed, while a 302 acts only as a weak signal (Google Search Central).
404 and 410 for removals
Addresses returning a 4xx status are not indexed, and addresses already included are removed from the index (Google Search Central).
Avoid soft 404s
An error message delivered with status 200 counts as a soft 404 — the page says “not found” while the server says “everything is fine” (Google Search Console Help).
The soft 404 is the case small firms are least aware of, because it stays invisible in the browser. A typical pattern: a deleted page redirects to the home page, or a system serves a pretty error page with status 200 for any unknown address. The search engine is then handed an apparently valid page for every possible typo. Also relevant for troubleshooting: server errors in the 5xx range as well as status 429 make Google temporarily slow down crawling (Google Search Central) — a server with occasional outages becomes a visibility problem without anything being wrong with the site itself.
Check 5: www and non-www, http and https
By default every website is reachable under four addresses: with and without www, each over http and https. Google explicitly names protocol variants as one of the causes of duplicate content (Google Search Central, documentation on canonicalization). If several of those variants answer with status 200, the same website exists several times over — the signals are split, and the search engine decides which version appears in the results. The clean target state is easy to describe: one variant answers with 200, the other three redirect permanently to it.
Address requested expected answer
---------------------------------------------------------------------
http://example.com/ 301 -> https://www.example.com/
http://www.example.com/ 301 -> https://www.example.com/
https://example.com/ 301 -> https://www.example.com/
https://www.example.com/ 200
https://www.example.com/services/ 200
https://www.example.com/does-not-exist 404 (not 200 with an error text)Two patterns show up regularly in this check. The first is the catch-all redirect: https://example.com/services/ does not land on https://www.example.com/services/ but on the home page. For the search engine, the individual page then no longer exists at its address. The second is the chain: http without www redirects to http with www, which redirects to https with www. It works, but it costs extra hops on every request; a direct redirect to the target address is cleaner. Where domain, DNS and server interact, the article on domain and business email basics helps with the groundwork.
- Pick one variant as the target address — the choice between www and non-www does not matter in substance, it only has to be consistent.
- Check that internal links use the target variant and not the redirected version.
- Switch the sitemap to the target variant, otherwise sitemap and redirect contradict each other.
- Set up a Search Console property for the whole domain so that all variants are evaluated together.
- Verify that the canonical on individual pages names the same variant as the redirect.
Check 6: is the domain simply still young?
If the five technical checks come back clean and the domain is only a few weeks old, the most likely answer is an uncomfortable one: it is still too early. Google describes three ways in which pages are found — addresses already known from earlier crawls, links from pages that are already known, and submitted sitemaps (Google Search Central, documentation on how Google Search works). A brand-new domain without a single inbound link meets none of those three conditions on its own.
For the expected duration there is a solid formulation in the Search Console Help: indexing typically takes about a day, but can take considerably longer in some cases, and submitting a request is no assurance that the page will end up in the index (Google Search Console Help). There is also a daily limit on how many such requests can be filed. Anyone installing a third tool after 48 hours is working against a process that simply needs time.
Four weeks of patience instead of four new tools
Indexed and still invisible: the content side
From here on the nature of the question changes. If the page is in the index, technology is no longer the cause, and every further hour spent on status codes is wasted. Now it is about whether a suitable page exists for the query at all, what people actually type, and who already occupies that position. The 96.55 percent (Ahrefs) without search visits mentioned at the start arise overwhelmingly here — not from defects, but from a lack of fit.
Is there a matching page?
Anyone who wants to be found for “gutter cleaning” needs a page that deals with exactly that service. A home page listing every service in a single paragraph answers no individual query in full.
Company name or actual need?
Searching for your own company name tests indexing, not visibility. New customers do not know the name and search for the problem, the location or the price instead.
Who is actually standing there?
For broad queries without a location, portals and directories often hold the top spots. Writing against them is laborious; the reachable ground usually lies in more specific queries tied to a place or a service.
The company-name test is worth doing because it separates both levels in a single step. If the website appears when you search for your own name, it is indexed — and the technical checklist is complete. If it then appears for no service-related query at all, the link between the language of your customers and the pages you have is missing. How to collect that language without any additional tools is described in detail in the article on finding the search terms your customers use.
| What gets typed | What the searcher expects | Which page fits |
|---|---|---|
| the company name | address, opening hours, phone number | home page and business profile |
| service plus location | a provider within reach | a service page that names the location in its text |
| “how much does ... cost” | a price range and examples | a page with a price range or a guide section |
| the service without a location | overview, choice, comparison | mostly portals and directories, rarely a single firm |
| a specific question | a direct answer in a few sentences | a guide section with the question as its heading |
A third cause on this level is named by Search Console directly: “Crawled – currently not indexed” means the page was fetched but not included, and that it may or may not be indexed later (Google Search Console Help). That is not a technical defect but a content signal — typically on very short pages, on several near-identical location pages, or on texts that do not make the purpose of the page clear. The effective lever here is plain language; which formulations help is shown in the article on writing website copy people actually read.
And a realistic look at the competition belongs here as well. A firm working in a district with four comparable businesses can realistically hold a top position for location-based queries. A firm competing with national portals for a broad term needs staying power and content those portals do not have. That assessment is not capitulation but the basis for priorities: first secure the reachable queries, then tackle the hard ones. Which pages should exist first is put in order in the article on the pages a business website needs.
The order of checks as a checklist
The list below is the short version of the whole article and is meant to be worked through in this order. Every open item is a concrete task. Anything further down only pays off once everything above it is done.
- The exact page title was searched in quotation marks — if the page appears, there is no indexing problem.
- URL Inspection was opened for at least one missing address and the four rows were read.
- The page indexing report was sorted by reason and the most frequent reason has been noted.
- robots.txt was requested under both host variants and contains no
Disallow: /. - The source of the affected page contains no noindex, neither as a meta element nor as a response header.
- The head section is intact — canonical and robots annotation sit before the first content element.
- The canonical points to the page itself and names the same host variant as the redirects.
- The sitemap is reachable at the address given, lists the target variant and no redirected addresses.
- Of the four address variants exactly one returns status 200, the other three redirect permanently to it.
- A non-existent address returns 404 and not an error page with status 200.
- For a domain under four weeks old, the sitemap has been submitted and at least one link to the domain exists.
- Every important service-related query has its own page that deals with exactly that query.
What a built-in delivery layer takes off your plate
What stands out about this list is how little of it is a matter of expertise. The six technical checks describe states that are either set up correctly from the start or can slip out of place with every change. In self-maintained systems the errors typically appear where several building blocks interact: one add-on sets the canonical, a second writes robots.txt, the redirects live in the server configuration, and a third block generates the sitemap. Each of those places can fail on its own without anything looking wrong in the browser.
Static delivery with clear status codes
Every page is served as a finished file. Existing addresses answer with 200, removed ones with 404, moved ones with a permanent redirect — without an add-on that has to reproduce that logic.
Sitemap and robots.txt from the actual inventory
Both files are generated at publish time from the pages that actually exist. They cannot point at an old structure, because they are not a separately maintained list.
One address variant, one canonical
The target variant is fixed, the others redirect to it, and every page carries its own address as the canonical — including translated versions with the matching language annotations.
The practical difference does not show on launch day but on the twentieth change request: when a page is renamed, a service added or a location page removed, sitemap, canonicals and status codes stay consistent without an extra step. What belongs to the scope is listed in the overview of features; how the path from briefing to a published website looks is described on the page how XICflow works. Finished results can be viewed in the example sites, and the cost is set out in the pricing overview.
Anyone currently weighing a self-maintained system against a managed delivery layer will find the differences in exactly these points — status codes, sitemap, robots.txt, canonicals, redirects — in the comparison of the approaches. And if the checklist from this article gets stuck at a point that cannot be classified, a short conversation about the specific case is usually faster than another night in the page source.
The short version
Sources and studies