Cookbook

Queue Infrastructure

Advanced queue operations — driver internals, built-in tasks, batches, failed-job analysis, glueful/queue-ops supervision and autoscaling, and production HA.

This page covers the infrastructure and operations layer of the queue system: driver internals, the built-in maintenance tasks, batches, failed-job analysis, worker supervision and autoscaling, and production HA.

Queue Drivers

Configure connections in config/queue.php and resolve the manager from the container — don't build a QueueManager by hand:

use Glueful\Queue\QueueManager;

$queue = app($context, QueueManager::class);

$queue->push(SendInvoiceJob::class, ['invoiceId' => 123]);                // default connection
$queue->push(SendInvoiceJob::class, ['invoiceId' => 123], 'emails', 'redis'); // queue, connection

$queue->size('emails');           // pending jobs
$queue->getStats('emails');       // driver statistics
$queue->testConnection('redis');  // connectivity check

Job names are always job classes: the worker refuses to instantiate a stored class name that does not implement Glueful\Queue\Contracts\JobInterface, so push('ProcessEmail', …) style string names never run.

Database Queue Driver

Lock-free claim: a worker reserves a job with a conditional update that only lands while the row is still unreserved; a worker that loses the race tries the next candidate. No row locks, so it behaves the same on SQLite, MySQL and PostgreSQL, and two workers never run the same job.
Priority and delay: ordered by priority and available_at.
Bulk insert: bulk() writes all rows in one insertBatch().
Failed-job isolation: exhausted jobs move to queue_failed_jobs.

Database Schema

From the framework's queue migration (installed when queue.default is database):

CREATE TABLE queue_jobs (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    uuid VARCHAR(12) NOT NULL UNIQUE,
    queue VARCHAR(255) NOT NULL DEFAULT 'default',
    payload TEXT NOT NULL,                 -- signed JSON
    attempts INTEGER DEFAULT 0,
    reserved_at TIMESTAMP NULL,
    available_at TIMESTAMP NOT NULL,
    priority INTEGER DEFAULT 0,
    batch_uuid VARCHAR(12) NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
    -- indexes: queue, reserved_at, available_at, priority, batch_uuid,
    --          (queue, reserved_at), (queue, available_at), (priority, available_at)
);

CREATE TABLE queue_failed_jobs (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    uuid VARCHAR(12) NOT NULL UNIQUE,
    connection VARCHAR(255) NOT NULL,
    queue VARCHAR(255) NOT NULL,
    payload TEXT NOT NULL,
    exception TEXT NOT NULL,
    batch_uuid VARCHAR(12) NULL,
    failed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE queue_batches (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    uuid VARCHAR(12) NOT NULL UNIQUE,
    name VARCHAR(255) NOT NULL,
    total_jobs INTEGER DEFAULT 0,
    pending_jobs INTEGER DEFAULT 0,
    processed_jobs INTEGER DEFAULT 0,
    failed_jobs INTEGER DEFAULT 0,
    cancelled_at TIMESTAMP NULL,
    finished_at TIMESTAMP NULL,
    options JSON NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NULL
);

A finished job's row is deleted — there is no completed-jobs table.

Redis Queue Driver

Uses the connection's prefix (REDIS_QUEUE_PREFIX, default glueful:queue:) on every key:

queue:{name}               # List — ready jobs
queue:{name}:delayed       # Sorted set — delayed jobs (score = available time)
queue:{name}:reserved      # Sorted set — reserved jobs
queue:{name}:failed        # List — failed jobs (read by queue:failed / queue:retry)
job:{uuid}                 # Hash — job data (expires after job_expiration, default 3600 s)
queues                     # Set — known queue names

Tasks vs Jobs

The framework separates business logic (Tasks, src/Tasks/) from queue execution (Jobs, src/Queue/Jobs/). A task is a plain class with handle(array $parameters = []): array; its job is a thin Glueful\Queue\Job that builds the task with $this->context, runs it and logs the result. Tasks can run directly (tests, CLI) or through the queue/scheduler.

Task	Job	Scheduled as
`Glueful\Tasks\CacheMaintenanceTask`	`Glueful\Queue\Jobs\CacheMaintenanceJob`	`cache_maintenance`
`Glueful\Tasks\DatabaseBackupTask`	`Glueful\Queue\Jobs\DatabaseBackupJob`	`database_backup`
`Glueful\Tasks\LogCleanupTask`	`Glueful\Queue\Jobs\LogCleanupJob`	`log_cleanup`
`Glueful\Tasks\SessionCleanupTask`	`Glueful\Queue\Jobs\SessionCleanupJob`	`session_cleaner`
`Glueful\Tasks\NotificationRetryTask`	`Glueful\Queue\Jobs\NotificationRetryJob`	`notification_retry_processor`

use Glueful\Queue\Jobs\CacheMaintenanceJob;
use Glueful\Tasks\LogCleanupTask;

// Direct execution
$result = (new LogCleanupTask($context))->handle(['retention_days' => 14]);

// Queued execution
app($context, QueueManager::class)->push(CacheMaintenanceJob::class, [
    'operation' => 'fullCleanup',
], 'maintenance');

The same split works for your own code:

final class RebuildSearchIndexTask
{
    public function __construct(private readonly ApplicationContext $context) {}

    public function handle(array $parameters = []): array
    {
        // ... business logic, testable without a queue
        return ['indexed' => 0];
    }
}

final class RebuildSearchIndexJob extends \Glueful\Queue\Job
{
    public function handle(): void
    {
        $result = (new RebuildSearchIndexTask($this->context))->handle($this->getData());

        app($this->context, \Psr\Log\LoggerInterface::class)
            ->info('Search index rebuilt', $result + ['job_uuid' => $this->getUuid()]);
    }
}

Batch Processing

There is no batch API (the queue_batches table is reserved). To group jobs, give each bulk() entry a shared batch_uuid; it is stored on the queue_jobs row (and on queue_failed_jobs if the job fails), and $this->getBatchUuid() is available inside the job:

use Glueful\Helpers\Utils;

$batchUuid = Utils::generateNanoID();

$jobs = array_map(fn (string $email) => [
    'job' => SendNewsletterJob::class,
    'data' => ['to' => $email],
    'batch_uuid' => $batchUuid,
], $recipients);

$uuids = app($context, QueueManager::class)->bulk($jobs, 'emails');

Track progress from the tables — completed jobs are deleted, so "done" is total − pending − failed:

$pending = db($context)->table('queue_jobs')->where('batch_uuid', $batchUuid)->count();
$failed = db($context)->table('queue_failed_jobs')->where('batch_uuid', $batchUuid)->count();
$done = count($uuids) - $pending - $failed;

To run something when a batch finishes, have each job check whether it is the last one left ($pending for its batch is 1 — itself) and push a completion job.

Failed Job Analysis

Beyond queue:failed / queue:retry / queue:forget / queue:flush, FailedJobProvider (database driver) offers analysis helpers:

use Glueful\Queue\Failed\FailedJobProvider;

$failures = app($context, FailedJobProvider::class);

$job = $failures->find($uuid);                       // one record, or null
$stats = $failures->getStats(['queue' => 'emails']);  // counts for the filters
$patterns = $failures->getFailurePatterns();          // grouped by error
$results = $failures->retryMultiple([$uuidA, $uuidB]);
$deleted = $failures->flushCount(['queue' => 'emails']);
$pruned = $failures->prune(30);                       // older than 30 days
$any = $failures->cleanup();                          // at the provider's retention age
$json = $failures->export(['from_date' => '2026-10-01'], 'json');

Filters are connection, queue, from_date and to_date; any other filter is refused. A retry re-verifies the stored payload's signature and requeues it as a new job with fresh attempts.

queue.failed.* configures the container-resolved provider and the one the database driver builds ($queue->connection('database')->failures()); retention_days (default 30) is the age cleanup() prunes at. The built-in queue_maintenance scheduled job prunes past that age daily at 04:15 (QUEUE_MAINTENANCE_ENABLED); an app with its own config/schedule.php must list it — see Failed Jobs. queue.failed.max_retries is stored but not enforced.

Custom Failure Handling

Failure policy lives in the job itself: decide in handle() whether to release($delay) (retry later) or swallow the error, and use failed() for the final-failure side effects.

class SyncCrmContactJob extends \Glueful\Queue\Job
{
    public function handle(): void
    {
        try {
            $this->sync($this->getData());
        } catch (CrmRateLimitedException $e) {
            $this->release(min(3600, (2 ** $this->getAttempts()) * 60)); // back off
        } catch (CrmValidationException $e) {
            // Not retryable: record and stop
            app($this->context, \Psr\Log\LoggerInterface::class)
                ->warning('CRM rejected contact', ['error' => $e->getMessage()]);
        }
        // Anything else is thrown: the worker retries it, then records it as failed
    }

    public function failed(\Exception $exception): void
    {
        // Called once, after the last attempt
    }
}

Process Management

composer require glueful/queue-ops
php glueful extensions:enable queue-ops   # runs its migrations

# Start the default fleet and monitor it
php glueful queue:supervise

# Actions: work | process | spawn | scale | status | stop | restart | health
php glueful queue:supervise spawn   --queue=high --count=3
php glueful queue:supervise scale   --queue=default --count=5
php glueful queue:supervise status  --watch=2          # or --json
php glueful queue:supervise health
php glueful queue:supervise stop    --all              # or --worker-id=ID
php glueful queue:supervise restart --all

Fleet settings live in config/queue_ops.php (queue_ops.*): process (QUEUE_DEFAULT_WORKERS, QUEUE_MAX_WORKERS_GLOBAL, QUEUE_MAX_WORKERS, QUEUE_RESTART_DELAY, QUEUE_HEALTH_CHECK_INTERVAL, QUEUE_WORKER_TIMEOUT, …), auto_scaling, and per-queue queues.{name}.workers|max_workers|auto_scale.

Auto-Scaling

php glueful queue:autoscale                              # = run (load + schedule + resource aware)
php glueful queue:autoscale run --interval=30 --streaming
php glueful queue:autoscale run --no-resource-checks --no-scheduling

# Actions: run | status | config | schedule | resources | stream
php glueful queue:autoscale status    --detailed         # or --json
php glueful queue:autoscale config    --show             # --validate | --reload
php glueful queue:autoscale schedule  --list             # or --preview --days=7
php glueful queue:autoscale resources --current          # --history | --trends

Thresholds come from queue_ops.auto_scaling (QUEUE_AUTO_SCALING, QUEUE_SCALE_CHECK_INTERVAL, QUEUE_SCALE_UP_THRESHOLD, QUEUE_SCALE_DOWN_THRESHOLD, QUEUE_SCALE_UP_STEP, QUEUE_SCALE_DOWN_STEP, QUEUE_SCALE_COOLDOWN); a queue is scaled only when its queue_ops.queues.{name}.auto_scale is on.

Production Deployment

High-Availability Setup

Run multiple lean core workers under a process manager (Supervisor, systemd). Each queue:work invocation is a single worker — use numprocs to scale.

# /etc/supervisor/conf.d/glueful-workers.conf

[program:glueful-worker-emails]
command=php /path/to/your-app/glueful queue:work --queue=emails --memory=256 --timeout=120 --max-runtime=3600
process_name=%(program_name)s_%(process_num)02d
numprocs=4
autostart=true
autorestart=true
user=www-data
redirect_stderr=true
stdout_logfile=/var/log/glueful-queue-emails.log

[program:glueful-worker-images]
command=php /path/to/your-app/glueful queue:work --queue=images --memory=512 --timeout=300 --max-runtime=3600
process_name=%(program_name)s_%(process_num)02d
numprocs=2
autostart=true
autorestart=true
user=www-data
redirect_stderr=true
stdout_logfile=/var/log/glueful-queue-images.log

End every deploy with php glueful queue:restart so workers pick up the new code (needs a shared cache store).

Performance Tuning

Driver: use QUEUE_CONNECTION=redis for high-throughput workloads.
Worker flags: tune --memory, --timeout, --max-jobs, --max-runtime and --sleep per queue (there is no worker_options config block).
Retry backoff: queue.workers.performance (QUEUE_BACKOFF_STRATEGY, QUEUE_BACKOFF_BASE, QUEUE_MAX_BACKOFF).
Visibility timeout: retry_after on the connection should exceed your longest job.

Monitoring and Alerting

Core has no alerting service. Watch the queue from a scheduled job and alert through your own channel:

final class QueueHealthCheckJob extends \Glueful\Queue\Job
{
    public function handle(): void
    {
        $queue = app($this->context, QueueManager::class);
        $failed = db($this->context)->table('queue_failed_jobs')
            ->where('failed_at', '>=', date('Y-m-d H:i:s', time() - 3600))
            ->count();

        if ($queue->size('default') > 1000 || $failed > 50) {
            app($this->context, \Psr\Log\LoggerInterface::class)
                ->critical('Queue unhealthy', ['pending' => $queue->size('default'), 'failed_last_hour' => $failed]);
            // ... or send a notification
        }
    }
}

With glueful/queue-ops, queue:supervise health and queue:autoscale status --json report worker health for external monitors.

Caching Recipes

Advanced caching — distributed coordination, advanced tagging, the edge/CDN seam, and production tuning.

Notification Channels & Templates

Channel internals — email providers, multi-channel selection, custom channels, and the template system.