Automation, Workers, and Event-Driven Workflows

The Constraint: Why Synchronous Processing Fails

Every HTTP request in a Laravel application runs inside a process with finite memory, a configured timeout, and a user waiting on the other end. For most requests (fetching a page, saving a form, returning JSON from an API), the lifecycle is fine. The work finishes in milliseconds. The problems start when the work is slow, unpredictable, or both.

PDF generation for a complex invoice might take 3 seconds or 30, depending on how many line items it contains. Sending a batch of 500 emails through an SMTP relay takes as long as the relay takes. Importing a CSV with 50,000 rows means 50,000 database writes, each with validation, event dispatch, and potential webhook callbacks to external systems.

Running any of these inside a web request creates three categories of failure.

The naive fix is to increase timeouts, raise memory limits, and add more workers. This delays the problem. It does not fix it. The fix is to move slow work out of the request lifecycle entirely: accept the request, dispatch a job to a queue, return a response immediately, and let a separate worker process handle the heavy lifting. This is background job processing. It is not a feature. It is infrastructure.

Queue Architecture and Driver Selection

Laravel's queue system abstracts the transport layer behind a consistent API. You dispatch jobs the same way regardless of whether they end up in Redis, Amazon SQS, a database table, or Beanstalkd. The driver choice matters for operations, not for application code.

Redis (the default for production)

Redis is the most common queue driver for Laravel applications, and for good reason. It is fast (operations measured in microseconds), supports blocking pops (workers do not poll), and integrates with Laravel Horizon for monitoring and metrics. We use Redis for the majority of our queue infrastructure.

The trade-off: Redis is an in-memory store. If the Redis instance restarts without persistence configured, queued jobs disappear. In production, this means either Redis with AOF persistence, Redis Sentinel for failover, or a managed service like AWS ElastiCache with Multi-AZ.

Amazon SQS

SQS is the right choice when durability matters more than latency. Messages are replicated across multiple availability zones. You do not manage the infrastructure. The trade-off is that SQS does not support blocking pops (workers poll), message ordering is best-effort (FIFO queues exist but add complexity), and Horizon does not support SQS. We use SQS for jobs where message loss is unacceptable, such as webhook-triggered jobs from payment providers: you cannot afford to lose a Stripe webhook because Redis restarted.

Database queues

The database driver stores jobs in a jobs table. No additional infrastructure is needed. For applications with low job volume (fewer than 1,000 jobs per day), this is a reasonable starting point. It fails at scale because every job dispatch and every job pickup is a database write, competing with your application's transactional queries for connection pool capacity.

Queue topology

A single default queue is where most applications start. It is also where most applications should stop pretending they have thought about queue design. In production, we typically configure three to five named queues with explicit priorities.

Workers are then assigned to queues with priority ordering. A password reset email will never wait behind a 50,000-row CSV import. The critical queue drains first. Always.

Job Dispatching Patterns

Laravel provides several dispatch mechanisms, each suited to different situations. The choice affects execution timing, failure isolation, and how jobs relate to each other.

Synchronous dispatch. dispatch_sync() runs the job inline, in the current process. Use this for testing, for local development, and for jobs that must complete before the response returns. Never use synchronous dispatch in production for slow work.

Standard async dispatch. The job serialises onto the configured queue and returns immediately. A worker picks it up when capacity is available. This is the default and the right choice for most jobs.

Delayed dispatch. The job sits on the queue but is not available for processing until the delay expires. Useful for scheduled follow-ups, rate-limited API calls, and retry windows. Be aware that delayed jobs on Redis use sorted sets, which consume memory proportional to the number of delayed jobs.

Job chaining

Each job in a chain runs only if the previous one succeeded. If the first step fails, the validation and notification jobs never execute. This is the correct pattern for multi-step workflows where later steps depend on earlier results. Compare this with workflow engines, which handle more complex branching and conditional logic.

Job batching

Batching runs jobs concurrently with aggregate callbacks. Use it for parallelisable work: processing rows in a CSV, generating thumbnails for uploaded images, or sending notifications to a list of recipients. The batch tracks progress, so you can show the user a percentage complete via a real-time dashboard.

Job middleware

Middleware wraps job execution with cross-cutting concerns. The two we use most frequently:

Designing Idempotent Jobs

A job is idempotent if running it twice with the same input produces the same result as running it once. This property is not optional. It is a requirement for any job that might be retried.

Retries happen constantly in production. A worker crashes mid-job. Redis fails over. A deployment restarts all workers. The queue system re-dispatches the job because it was never acknowledged as complete. If that job already wrote half its data to the database, the retry writes it again. Without idempotency, you get duplicate records, double-charged customers, or emails sent twice.

Idempotency is not a library you install. It is a design discipline applied to every job class. We review job idempotency during code review the same way we review database migrations: as infrastructure that must be correct.

Failed Job Handling and Dead Letter Queues

Jobs fail. Connections drop. External APIs return 500s. A CSV contains a row with a malformed date that crashes the parser. The question is not whether jobs will fail but what happens when they do.

Retry strategies

Laravel's default retry behaviour is configurable per job. You can set the number of attempts and a backoff schedule (for example, 10 seconds, then 60, then 300). The backoff prevents a job from hammering an external service that is already struggling.

For transient failures (network timeouts, rate limits), retries with backoff usually resolve the issue. For permanent failures (invalid data, missing dependencies), retries waste resources. The job needs to distinguish between the two.

Dead letter queues

When a job exhausts all retries, Laravel moves it to the failed_jobs table. This is your dead letter queue. These jobs need attention. They represent work the system could not complete. The dead letter queue pattern, borrowed from enterprise integration patterns, treats failed jobs as messages that require human or automated intervention.

In our systems, we implement three layers of handling.

The worst failure mode is silent failure. A job fails, nobody notices, and the customer never receives their invoice. Dead letter queue monitoring prevents this.

Monitoring, Horizon, and Operational Visibility

A queue without monitoring is a queue where problems go undetected until a customer reports them.

Laravel Horizon

Laravel Horizon provides a dashboard and configuration layer for Redis-based queues. It shows real-time metrics for every queue and worker: jobs per minute, job runtime with percentile breakdowns, failed jobs with full exception traces, wait time before a worker picks up a job, and worker status across all processes.

Horizon also manages worker processes through its supervisor configuration, automatically scaling workers up or down based on queue depth. This is more reliable than manually managing queue:work processes with Supervisor.

What to monitor and alert on

Beyond Horizon's dashboard, we set up alerts for specific conditions that require human intervention.

Process management

Queue workers are long-running PHP processes. They do not restart between jobs. This makes them susceptible to memory leaks, stale database connections, and accumulated state. In production, we configure workers to restart regularly: process up to 1,000 jobs or run for one hour, whichever comes first, then exit cleanly. Supervisor or systemd restarts the worker immediately. This limits the impact of memory leaks and ensures workers pick up code changes after deployments.

Production Failure Modes

Tutorials show you how to dispatch a job and process it. They do not show you what happens when things go wrong at scale. These are the failure modes we encounter repeatedly across the 50+ Laravel applications we maintain.

Webhook-triggered job patterns

External systems (Stripe, Xero, CRM platforms) send webhooks to your application. Each webhook should dispatch a job rather than processing inline. This ensures the webhook endpoint returns a 200 quickly and isolates the processing from the HTTP request.

The challenge with webhook jobs is idempotency. Stripe sends the same webhook event multiple times as a reliability measure. Your job must handle receiving the same event three times without creating three payment records. Store the webhook event ID and check it before processing, following the same idempotency patterns described above. This pattern connects to our broader approach to API integrations, where incoming data from external systems flows through queued jobs with validation, deduplication, and error handling at each stage.

When Background Jobs Change How a Business Operates

The technical patterns above are infrastructure. Their business impact is what makes them worth the engineering investment.

• ✓
  Reports that generate themselves Users click "Generate" and receive an email with the finished report. No waiting, no timeouts, no 504 errors.
• ✓
  Data imports with progress tracking Batch jobs with progress bars replace cron jobs that blocked other scheduled tasks and left operations guessing.
• ✓
  Payment webhooks that never go missing Dedicated queues with SQS durability, dead letter monitoring, and automatic retry. Revenue stops leaking through infrastructure gaps.
• ✓
  Responsive applications under load Heavy work happens in the background. Users never wait for email servers or report generation. Pages respond instantly.

These are patterns we have implemented across order management systems, financial operations platforms, and service delivery tools since 2005. Background job processing connects to workflow engines for complex multi-step processes, to real-time dashboards for operational visibility, and to infrastructure decisions about how workers are deployed and scaled.