Background Jobs & Queue Processing Guide

Last Updated: 2026-01-10 Status: Enhanced Plan Reference: 018-background-jobs-queue-processing.md, 053-background-jobs-enhancement.md

Overview

The Background Jobs system handles asynchronous task processing using Laravel's queue system. It enables efficient handling of time-consuming operations like email sending, PDF generation, file processing, and scheduled tasks without blocking user requests. The enhanced monitoring dashboard provides real-time visibility into queue health, job metrics, and failed job management.

Table of Contents

1. Queue Configuration
2. Job Types
3. How to Use
4. Queue Monitoring Dashboard
5. Job Metrics & Analytics
6. Failed Job Management
7. Priority & Throttling
8. Technical Architecture
9. Related Features

Queue Configuration

Queue Drivers

Environment Setup

QUEUE_CONNECTION=database

# For Redis
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379

# For SQS
SQS_KEY=your-key
SQS_SECRET=your-secret
SQS_QUEUE=your-queue
SQS_REGION=us-east-1

# Monitored Queues (comma-separated)
QUEUE_MONITORED_QUEUES=default,high,low,emails

Queue Names

Job Types

Email Jobs

Processing Jobs

Scheduled Jobs

Notification Jobs

How to Use

Dispatching Jobs

Immediate Dispatch:

SendInvoiceEmail::dispatch($invoice);

Delayed Dispatch:

SendReminderEmail::dispatch($invoice)->delay(now()->addHours(24));

Specific Queue:

GenerateReport::dispatch($params)->onQueue('low');

Chained Jobs:

Bus::chain([
    new ProcessUpload($file),
    new GenerateThumbnail($file),
    new NotifyUser($user),
])->dispatch();

Using Job Metrics Middleware

Add the TrackJobMetrics middleware to jobs for automatic metrics collection:

use App\Jobs\Middleware\TrackJobMetrics;

class ProcessData implements ShouldQueue
{
    public function middleware(): array
    {
        return [new TrackJobMetrics()];
    }

    public function handle(): void
    {
        // Job logic
    }
}

Running Queue Workers

Development:

php artisan queue:work

Production (with options):

php artisan queue:work --queue=high,default,low --sleep=3 --tries=3

Process specific queue:

php artisan queue:work --queue=emails

Supervisor Configuration

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/artisan queue:work --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/path/to/worker.log
stopwaitsecs=3600

Queue Monitoring Dashboard

Navigate to Admin → Queues to access the comprehensive monitoring dashboard.

Dashboard Features

Queue Health Status

Real-Time Updates

The dashboard automatically refreshes every 30 seconds via AJAX polling. Access real-time metrics at:

GET /admin/queues/realtime

Job Metrics & Analytics

Analytics Dashboard

Navigate to Admin → Queues → Analytics for detailed job analytics.

Available Periods

Analytics Charts

Job Type Performance Table

View detailed performance metrics per job type:

• Total count
• Average duration (ms)
• Average memory (MB)
• Fail rate (%)

Accessing Metrics Programmatically

use App\Services\Queue\QueueMetricsService;

$metrics = app(QueueMetricsService::class);

// Get summary for dashboard
$summary = $metrics->getSummary();

// Get analytics for a period
$analytics = $metrics->getJobAnalytics('24h');

// Get queue-specific metrics
$queueMetrics = $metrics->getQueueMetrics('high');

// Get hourly throughput
$throughput = $metrics->getThroughput('default');

Failed Job Management

Navigate to Admin → Queues → Failed Jobs for comprehensive failed job management.

Failed Jobs Dashboard

Individual Job Details

View detailed information for each failed job:

• Job UUID, queue, connection
• Failure timestamp
• Full exception with stack trace
• Job payload (JSON formatted)
• Related failures for same job type

Actions

Artisan Commands

# View failed jobs
php artisan queue:failed

# Retry specific job
php artisan queue:retry {uuid}

# Retry all failed
php artisan queue:retry all

# Clear all failed
php artisan queue:flush

Priority & Throttling

Navigate to Admin → Queues → Priorities for queue and throttle configuration.

Queue Priority

• Drag-and-drop to reorder queue priority
• Higher priority queues are processed first
• Configure workers per queue
• Pause/resume individual queues

Rate Limiting

Configure rate limits for specific job classes:

Programmatic Configuration

use App\Models\QueueConfig;
use App\Models\JobThrottle;

// Set queue priority and workers
$config = QueueConfig::forQueue('high');
$config->update([
    'priority' => 10,
    'workers' => 3,
    'is_paused' => false,
]);

// Add job throttle
JobThrottle::create([
    'job_class' => 'App\Jobs\SendEmail',
    'per_minute' => 60,
    'concurrent' => 5,
]);

Technical Architecture

Service Classes

Models

Database Schema

Table: job_metrics

Table: queue_configs

Table: job_throttles

Routes

Middleware

The TrackJobMetrics middleware automatically records:

• Job start time and class
• Completion status and duration
• Memory usage
• Exception messages for failures

Best Practices

For Reliability

1. Set appropriate timeouts for job execution
2. Use unique job IDs to prevent duplicates
3. Implement idempotency for retryable jobs
4. Log job progress for debugging
5. Handle failures gracefully
6. Use metrics middleware for visibility

For Performance

1. Use appropriate queues for job priority
2. Batch similar operations together
3. Avoid serializing large objects
4. Set reasonable retry limits
5. Monitor queue depth and scale workers
6. Configure throttles for rate-limited APIs

For Monitoring

1. Check dashboard daily for queue health
2. Set up alerts for high failure rates
3. Review analytics for performance trends
4. Address failed jobs promptly
5. Tune throttles based on external API limits

Troubleshooting

Debug Mode

# Run single job for debugging
php artisan queue:work --once

# View job payload
php artisan queue:failed --json

Related Features

Dependencies

Complementary Features

See Also

• Notifications - Email sending
• Admin Tools - Queue management
• Performance Caching - Caching