Request
Status codes, headers, robots rules, redirects and server behavior determine whether a crawler can access the intended resource.
SEO / Technical SEO
Technical SEO is search infrastructure.
Diagnose the engineering conditions that determine whether important pages can be crawled, rendered, canonicalized, indexed, understood and retrieved reliably.
A technical SEO audit should do more than generate a list of warnings. KeenSight maps crawler behavior, URL states, internal architecture, rendering, performance and search-engine diagnostics into a prioritized implementation plan—then validates whether the fixes actually changed the system.
Guidance reviewed August 2026 · Current Google and Bing documentation
HTTP request
Render
Canonical
Links + sitemap
Index decision
Retrieval
200
robots
01 / Audit model
Audit the path from request to retrieval.
Technical SEO is a sequence of dependent systems. A page cannot be indexed if a crawler cannot access or render its important content. Canonical signals cannot consolidate duplicates if they contradict redirects and sitemaps. Internal links cannot help discovery if they point through broken chains or JavaScript states the crawler never sees.
Render
Critical content, links, metadata and structured data need to survive the rendering path used by the crawler.
Canonicalize
Duplicate and near-duplicate URLs need consistent signals so search systems can identify the preferred representative URL.
Discover
Internal links and sitemaps expose important URLs and their relationship to the rest of the site.
Index
Search systems decide whether the page is eligible and useful enough to retain in the index.
Retrieve
Indexed pages compete to answer real queries. Technical SEO removes infrastructure failures that prevent relevant pages from participating.
02 / Audit coverage
Ten technical domains. One prioritized system.
The right scope depends on the site. An ecommerce platform with millions of faceted URLs has a different risk profile from a small B2B site. International SEO and log analysis are conditional disciplines—not boxes every audit needs to check for appearance’s sake.
01high
Crawl controls
robots.txt, crawler access, firewall/CDN rules, status codes, crawl traps and blocked resources
02high
Index controls
meta robots, X-Robots-Tag, Page Indexing diagnostics, snippet eligibility and accidental exclusions
03high
Canonicalization
redirects, rel=canonical, sitemap consistency, duplicate clusters and parameterized URLs
04high
Architecture
crawl paths, internal links, orphan pages, breadcrumbs, hierarchy, URL conventions and faceted navigation
05medium
Rendering
server-rendered HTML, client-side rendering, hydration, JavaScript links, metadata and structured data
06high
Performance
LCP, INP, CLS, resource delivery, image strategy, fonts, caching and real-user field data
07medium
Structured data
eligible markup, JSON-LD validity, visible-content alignment and rich-result requirements
08conditional
International
hreflang clusters, return links, canonicals, language/region codes and x-default where needed
09high
Migrations
URL mapping, permanent redirects, canonical updates, sitemaps, Search Console and change monitoring
10conditional
Logs & monitoring
crawler requests, status distribution, crawl frequency, error patterns and important-URL coverage
03 / Current guidance
Technical SEO changes. Old checklists age badly.
Some widely repeated audit steps are now obsolete or misleading. KeenSight maintains the implementation standard against current platform documentation instead of preserving old tool names and rules indefinitely.
Older adviceUse Search Console Coverage report
Current practiceUse the Page Indexing report, Crawl Stats and URL Inspection.
Google’s current technical requirements explicitly point site owners to Page Indexing and Crawl Stats for inaccessible or non-indexed URLs.
Older adviceRun Google’s Mobile-Friendly Test
Current practiceUse responsive testing, Lighthouse/DevTools, real-device QA and URL Inspection.
Google retired the Mobile-Friendly Test and Mobile Usability report in December 2023. Mobile-first indexing remains the default.
Older adviceUse dynamic rendering for JavaScript SEO
Current practicePrefer server-side rendering, static rendering or hydration.
Google calls dynamic rendering a workaround rather than a recommended long-term solution.
Older adviceINP should be under 250 ms
Current practiceTarget INP ≤200 ms at the 75th percentile.
Current Core Web Vitals thresholds define good INP as 200 ms or less; poor is above 500 ms.
Older adviceUse nofollow to conserve link equity
Current practiceQualify links based on relationship: sponsored, ugc or nofollow where appropriate.
Google does not recommend blanket nofollow. Ordinary trusted editorial links do not need a rel qualifier.
Older adviceEvery key page should be within a fixed 3–4 click depth
Current practicePrioritize crawl paths, orphan prevention and contextually useful internal links.
There is no universal Google click-depth rule. Important pages should be discoverable and well connected without forcing an artificial number.
04 / Crawl vs index
robots.txt and noindex solve different problems.
A robots.txt rule controls whether a crawler may request a URL. It is not the normal mechanism for keeping an HTML page out of Google Search. If Googlebot cannot crawl the page, it cannot see a page-level noindex directive either.
For index control, a crawlable page can return a meta robots noindex rule or an X-Robots-Tag header. Confidential material should not rely on either mechanism—use authentication or remove public access entirely.
Crawling and indexing explained →robots.txt
Use robots rules to manage crawling—not as the mechanism for keeping HTML pages out of Search.
noindex
Use a meta robots rule or X-Robots-Tag when a crawlable resource should not be indexed.
HTTP status
Important indexable pages should resolve successfully. Diagnose 4xx, 5xx, soft 404s and redirect chains.
Resources
Do not block CSS, JavaScript, images or APIs required for the crawler to render and understand important content.
Header check
curl -I https://example.com/pageInspect status, redirects and X-Robots-Tag.05 / Canonicals & sitemaps
Align every canonical signal around the URL you actually want indexed.
Google currently describes permanent redirects and rel=canonical annotations as strong canonicalization signals, while sitemap inclusion is a weaker supporting signal. Contradictions across those systems create ambiguity and unnecessary crawling.
Sitemaps should use fully qualified URLs and list the canonical URLs you want search engines to consider. Google limits a single sitemap to 50,000 URLs or 50 MB uncompressed; larger inventories should be split into multiple sitemap files and, when useful, a sitemap index.
Build a stronger internal discovery graph →Permanent redirectStrong
Best when an old or duplicate URL should permanently resolve to another URL.
rel="canonical"Strong
Best when duplicate or very similar URLs need to remain accessible.
Sitemap inclusionWeak
Useful supporting signal; include URLs you actually want search engines to consider canonical.
06 / Site architecture
Internal linking is crawler routing and information architecture.
Important pages should be reachable through crawlable HTML links and connected in ways that explain context. We look for orphan pages, dead ends, navigation traps, over-deep templates, inconsistent canonical linking and page clusters that do not reinforce their parent topics.
We do not enforce a magic “three-click rule.” The goal is to make strategically important URLs easy to discover and to preserve a coherent hierarchy as the site grows.
Internal linking explained →07 / JavaScript & mobile-first indexing
Render critical search content before JavaScript becomes a dependency.
Google can process JavaScript, but JavaScript introduces additional rendering stages and failure modes. Current Google guidance recommends server-side rendering, static rendering or hydration over dynamic rendering, which Google now describes as a workaround rather than a preferred long-term solution.
Google also crawls and indexes sites with its smartphone crawler. Responsive design is the easiest pattern to maintain, and important mobile content, metadata, structured data and robots directives should remain equivalent to desktop.
Server / static HTML
request→HTML→hydrate
Important text, links and metadata are present early and remain available if client-side code fails.
Client-only critical content
request→shell→JS→content
Requires more careful rendering validation because failures can hide links, metadata or primary content from crawlers.
08 / Core Web Vitals
Measure real-user experience, then diagnose the bottleneck.
Lighthouse is useful laboratory data. Core Web Vitals are fundamentally field metrics, and Google evaluates the recommended thresholds at the 75th percentile of page loads. The current good thresholds are LCP ≤2.5 seconds, INP ≤200 milliseconds and CLS ≤0.1.
A technical audit should connect each failed metric to the responsible template, resource or runtime behavior—not just report the score.
LCP
≤ 2.5 sLargest Contentful Paint should be good at the 75th percentile of real-user page loads.
INP
≤ 200 msInteraction to Next Paint measures responsiveness across user interactions.
CLS
≤ 0.1Cumulative Layout Shift should stay low across the full page lifecycle.
09 / Structured data & international SEO
Use machine-readable signals where they accurately describe visible content.
Structured data can help search systems understand entities and qualify pages for supported search features, but markup must correspond to the content users can actually see. We validate syntax, property requirements and whether the search feature is still supported before treating schema as an opportunity.
International SEO is similarly conditional. When equivalent pages target different languages or regions, hreflang annotations need fully qualified URLs and return links. x-default can provide a fallback where appropriate. Canonicalization and hreflang must not contradict each other.
{
"@context": "https://schema.org",
"@type": "Service",
"name": "Technical SEO Audit"
}Illustrative JSON-LD. Production markup must match visible content and supported properties.<link rel="alternate" hreflang="en-us" href="https://example.com/us/page" />
<link rel="alternate" hreflang="en-gb" href="https://example.com/uk/page" />
<link rel="alternate" hreflang="x-default" href="https://example.com/page" />Only implement when true localized variants exist.10 / Redirects, migrations & HTTPS
Large technical changes need a migration system, not a launch-day redirect file.
Permanent server-side redirects such as 301 or 308 are the preferred signal when URLs have moved permanently. Temporary redirects such as 302 or 307 serve a different purpose. For domain or URL migrations, the work includes URL mapping, internal links, canonical tags, sitemaps, Search Console, monitoring and keeping redirects in place long enough for search systems and users to transition.
11 / Monitoring & server logs
Use platform data and logs to see what crawlers actually do.
Search Console’s Page Indexing, Crawl Stats and URL Inspection tools show different parts of Google’s view. Bing Webmaster Tools adds Site Explorer, crawl diagnostics, sitemap reporting, Site Scan and IndexNow visibility. For large or complex sites, raw server logs can show crawler requests that dashboards aggregate away.
Log analysis is most useful when there is a hypothesis: important sections are not being crawled, duplicate URLs consume crawl demand, bot requests hit error states, or changes are not being discovered as expected.
$ grep 'Googlebot' access.log | awk '{print $7}' | sort | uniq -c
$ grep ' 404 ' access.log | awk '{print $7}' | sort | uniq -c
$ curl -I https://example.com/page
HTTP/2 200
link: <https://example.com/page>; rel="canonical"Illustrative diagnostic commands. Real analysis should account for hosting, CDN logs, bot verification and privacy controls.12 / Prioritization
Severity without effort is not a roadmap.
An audit becomes useful when findings are sequenced by business importance, technical severity, confidence, implementation cost and dependencies. A single accidental noindex can outrank months of minor template cleanup.
IssueEffortImpactSequence
Accidental noindex / robots conflictLowCriticalShip immediately
Broken canonicals or migration redirectsMediumCriticalShip immediately
5xx / systemic crawl failuresMediumCriticalShip immediately
Orphaned commercial pagesMediumHighFirst sprint
Sitemap contaminationLowHighFirst sprint
Mobile content mismatchMediumHighFirst sprint
Poor LCP / INP / CLSMedium–HighHighPerformance sprint
JavaScript-only critical contentHighHighArchitecture sprint
hreflang errorsMediumConditionalInternational sites
Log-analysis automationMediumConditionalLarge / complex sites
13 / 30–60–90
Turn the audit into an implementation cadence.
The exact sequence changes by site, but the pattern is consistent: remove blockers first, improve architecture second, then operationalize monitoring so regressions do not quietly return.
Remove blockers
- Validate robots.txt, status codes, noindex rules and X-Robots-Tag headers.
- Confirm canonical targets and clean contradictory signals.
- Repair sitemap errors and submit clean canonical URL sets.
- Fix redirect loops, broken internal links and critical 4xx/5xx failures.
- Confirm Search Console and Bing Webmaster Tools ownership and monitoring.
Improve architecture
- Repair orphaned pages and strengthen contextual internal linking.
- Reduce unnecessary duplicate and faceted URL crawl paths.
- Improve server rendering or hydration where critical content depends on fragile JavaScript.
- Deploy valid, visible-content-aligned structured data on eligible page types.
- Resolve the highest-impact Core Web Vitals bottlenecks using field and lab data.
Operationalize quality
- Automate Lighthouse / browser checks and regression monitoring where practical.
- Use server-log analysis for crawler behavior on large or strategically important sites.
- Create migration and release checklists for redirects, canonicals, sitemaps and metadata.
- Establish recurring Page Indexing, Crawl Stats, CWV and Bing diagnostics reviews.
- Re-crawl the site and compare the new technical baseline against the original audit.
14 / Deliverables
A technical audit should end with something developers can ship.
The output is not a PDF full of tool exports. It is a shared implementation model: what is broken, why it matters, how to fix it, how to test the fix and what to monitor afterward.
Technical crawl model
A normalized URL inventory with status, indexability, canonical, depth, inlinks and template context.
Search-engine diagnostics
Search Console and Bing Webmaster findings organized by root cause rather than screenshot volume.
Rendering review
Source HTML vs rendered DOM comparison for pages that depend on JavaScript or hydration.
Priority backlog
Issues scored by severity, confidence, impact, implementation effort and dependency.
Implementation guidance
Concrete developer-ready recommendations, examples and acceptance criteria.
Validation plan
A repeatable post-fix crawl, browser, field-data and search-engine monitoring sequence.
15 / Start with the infrastructure
Find the technical constraints limiting search visibility.
KeenSight can audit the path from HTTP response through crawl, rendering, canonicalization, architecture, indexing and performance—then turn the findings into a prioritized developer-ready implementation plan.