Technical SEO Architecture: Crawlability, Sitemaps, Canonicalization & Redirects

Quick 3-Step Simple Fix for Everyday Users

1. Check your robots.txt file: Open your browser and navigate to yourdomain.com/robots.txt. Make sure it does not say Disallow: / for all user-agents, and confirm it lists the direct URL to your sitemap at the bottom.
2. Verify single-address loading: Type your website using different variations (such as http://, http://www., and https://). Confirm that every version smoothly redirects to your single, preferred secure HTTPS address without browser security warnings or multiple redirect jumps.
3. Submit your sitemap in Google Search Console: Log into Google Search Console, click on "Sitemaps" in the left sidebar, enter sitemap.xml, and click Submit to confirm Google receives green success confirmations.

Deep Technical Diagnostic: Crawl Pipeline & Bot Routing

Search bot traversal follows a deterministic state machine. When Googlebot or Bingbot initiates a crawl cycle, it checks access restrictions, verifies transport security, inspects canonical headers, and parses document graphs.

[Search Bot Ingress: Googlebot / Bingbot]
          |
          v
[Step 1: Check /robots.txt]
    |-- HTTP 4xx/5xx or Disallow: /? ----> Bot Aborts Crawl
    |-- Allowed: Read Crawl-Delay & Sitemaps
          |
          v
[Step 2: Edge CDN / Reverse Proxy (Nginx / Cloudflare)]
    |-- HTTP Insecure? ------------------> 301 Redirect to HTTPS
    |-- Non-Canonical Host (www vs apex)? -> 301 Redirect to Canonical Host
    |-- Trailing Slash Mismatch? ---------> 301 Standardized Route
          |
          v [HTTP 200 OK Response]
[Step 3: HTML Parser & DOM Evaluation]
    |-- Parse <link rel="canonical" href="...">
    |-- Inspect <meta name="robots" content="noindex, follow">
    |-- Extract <a href="..."> hypermedia graph
          |
          v
[Step 4: XML Sitemap Cross-Referencing & Indexing Storage]

Production Configuration Directives

1. Production robots.txt Template

# /public/robots.txt
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /_next/
Disallow: /*?*sort=
Disallow: /*?*filter=
Disallow: /*?*search=

# Host declaration and XML Sitemaps
Host: https://fixit.cybertechcomps.com
Sitemap: https://fixit.cybertechcomps.com/sitemap.xml
Sitemap: https://fixit.cybertechcomps.com/sitemap-categories.xml

2. Nginx SSL, Host Canonicalization & Trailing Slash Redirection

# Redirect HTTP to HTTPS
server {
    listen 80;
    listen [::]:80;
    server_name fixit.cybertechcomps.com www.fixit.cybertechcomps.com;
    return 301 https://fixit.cybertechcomps.com$request_uri;
}

# Redirect www to apex canonical host
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name www.fixit.cybertechcomps.com;

    ssl_certificate /etc/letsencrypt/live/fixit.cybertechcomps.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/fixit.cybertechcomps.com/privkey.pem;

    return 301 https://fixit.cybertechcomps.com$request_uri;
}

# Canonical application server block
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name fixit.cybertechcomps.com;

    ssl_certificate /etc/letsencrypt/live/fixit.cybertechcomps.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/fixit.cybertechcomps.com/privkey.pem;
    
    # Strip duplicate slashes
    merge_slashes on;

    # Canonical Link Header fallback for static assets
    location ~* \.(pdf|docx|xlsx)$ {
        add_header Link "<https://fixit.cybertechcomps.com$request_uri>; rel=\"canonical\"";
        try_files $uri =404;
    }

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
    }
}

3. Next.js App Router Dynamic Sitemap Generation (app/sitemap.ts)

import { MetadataRoute } from 'next';
import { getAllTopics, getAllCategories } from '@/lib/topics-data';

const BASE_URL = 'https://fixit.cybertechcomps.com';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const topics = getAllTopics();
  const categories = getAllCategories();

  const staticRoutes: MetadataRoute.Sitemap = [
    {
      url: `${BASE_URL}/`,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 1.0,
    },
    {
      url: `${BASE_URL}/docs/`,
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 0.9,
    },
  ];

  const categoryRoutes: MetadataRoute.Sitemap = categories.map((cat) => ({
    url: `${BASE_URL}/docs/${cat.slug}/`,
    lastModified: new Date(),
    changeFrequency: 'weekly',
    priority: 0.8,
  }));

  const topicRoutes: MetadataRoute.Sitemap = topics.map((topic) => ({
    url: `${BASE_URL}/docs/${topic.categorySlug}/${topic.slug}/`,
    lastModified: new Date(),
    changeFrequency: 'monthly',
    priority: 0.7,
  }));

  return [...staticRoutes, ...categoryRoutes, ...topicRoutes];
}