Monitoring & Observability Guide

Overview

This guide covers the production observability stack for the ScopeForged client portal, including correlation IDs, health checks, error tracking, and deployment verification.

Correlation IDs

Every HTTP request is assigned a unique correlation ID that flows through logs, error tracking, and response headers.

How It Works

The AddCorrelationId middleware runs early in both web and API middleware stacks:

1. Checks for an incoming X-Request-Id or X-Correlation-ID header (from load balancers/proxies)
2. Generates a UUID if no header is present
3. Stores the ID on the request attributes ($request->attributes->get('correlation_id'))
4. Shares it with Laravel's log context via Log::shareContext()
5. Tags the Sentry scope (if Sentry is installed)
6. Returns the ID in the X-Request-Id response header

Accessing the Correlation ID

// In a controller or middleware
$correlationId = $request->attributes->get('correlation_id');

// In log output (automatically included via Log::shareContext)
Log::info('Processing payment'); // {"message": "Processing payment", "correlation_id": "abc-123-..."}

Health Check Endpoints

Two public endpoints are available for load balancers and deployment scripts.

Liveness: GET /health

Checks that the application can serve requests (database + cache connectivity).

{
  "status": "healthy",
  "checks": {
    "database": { "status": "healthy", "response_time_ms": 1.2 },
    "cache": { "status": "healthy", "response_time_ms": 0.5 }
  },
  "timestamp": "2026-02-14T12:00:00+00:00"
}

• Returns 200 OK when healthy, 503 Service Unavailable when unhealthy
• Rate limited to 10 requests per minute
• No authentication required

Readiness: GET /health/ready

Full readiness check including queue and scheduler status.

{
  "status": "healthy",
  "checks": {
    "database": { "status": "healthy" },
    "cache": { "status": "healthy" },
    "queue": { "status": "healthy", "pending_jobs": 3, "failed_jobs": 0 },
    "scheduler": { "status": "healthy", "last_run": "2026-02-14T11:59:00+00:00" }
  },
  "timestamp": "2026-02-14T12:00:00+00:00"
}

Status Values

Error Tracking (Sentry)

Sentry integration provides real-time error tracking and performance monitoring.

Configuration

Set the DSN in your .env:

SENTRY_LARAVEL_DSN=https://your-dsn@sentry.io/project-id

Key settings in config/sentry.php:

• traces_sample_rate: Performance tracing sample rate (default: 0.1 = 10%)
• environment: Auto-detected from APP_ENV
• release: Set via SENTRY_RELEASE env var

Correlation with Logs

Sentry events are automatically tagged with the correlation ID, allowing you to trace from a Sentry error back to application logs.

Async API Logging

API usage and debug logs are written asynchronously via dispatch()->afterResponse() to avoid blocking the response:

• LogApiUsageJob — Writes api_usage_logs records
• LogApiDebugRequestJob — Writes api_debug_logs records (when debug mode enabled)

This moves database writes off the critical path, improving API response times.

Deployment Verification

Post-Deploy Health Check

The deploy script (scripts/deploy/deploy-linux.cjs) automatically verifies the application after deployment:

1. Curls /health with a 30-second timeout
2. Retries up to 3 times with 5-second delays
3. Logs success or warns on failure (does not auto-rollback)

Skip with --skip-verify flag:

node scripts/deploy/deploy-linux.cjs --skip-verify

Standalone Health Check

Use the standalone script for manual verification:

./scripts/deploy/health-check.sh [URL] [TIMEOUT] [RETRIES]

# Examples
./scripts/deploy/health-check.sh                                    # Default (scopeforged.com)
./scripts/deploy/health-check.sh https://staging.example.com/health  # Custom URL
./scripts/deploy/health-check.sh https://example.com/health 10 5    # 10s timeout, 5 retries

Deployment Notifications

When a deployment completes, admin users receive notifications via:

• Database (always) — Shows in the admin notification center
• Slack (conditional) — Sends to Slack when LOG_SLACK_WEBHOOK_URL is configured

Environment Variables