Background jobs

A job queue in your own database: retries with backoff, scheduled and recurring jobs, and no job run by two workers at once.

This is an app part spec on Spekkr, published by vex. It is listed with the other specs. Paste the line to an agent that can open a web address, and the agent fetches the files and builds from them.

Spec
vex/background-jobs
Version
0.1.2
Published by
vex
Updated
15 Nov 2026

Build this spec

Build the spec at https://spekkr.com/vex/background-jobs/start.md.

Paste this to your AI agent. It opens the address, fetches the spec's files, records your choices and builds from them. Nothing to install, no account.

Published by vex.

What does the result look like?

Background jobs: dashboardBackground jobs: loginBackground jobs: forbiddenBackground jobs: dashboard mobile

What can I customize?

Database

Your choices go into the line you copy. Anything else is yours to edit, but it leaves the tested path.

What do I get?

  • spec.md Requirements and acceptance criteria.

    Background jobs

    What this builds

    A queue for work that should not happen inside a web request: sending email, calling other services, heavy calculation. Jobs live in the app's own database, so there is nothing else to run. Workers take jobs, retry the ones that fail, and never run one job in two places at once.

    Read spekkr.inputs.json first. A job is tried max_attempts times. One worker runs concurrency jobs at a time. recurring_jobs decides whether R6 exists and dashboard whether R8 does.

    Out of scope

    • Workflows made of several jobs that depend on each other.
    • Priorities beyond named queues.
    • Exactly-once execution. A job runs at least once, and handlers must cope with a repeat.

    Requirements

    Each requirement has an id. Tests name the id they cover.

    R1. Add a job
    • App code calls enqueue(kind, payload, options). The payload is JSON of at most 64 KB (64,000 bytes).
    • A job added inside a database transaction exists exactly when that transaction commits.
    • Options: run_at to start no earlier than a time, queue to name a queue other than default, and unique_key, covered by R5.
    • A kind with no registered handler is an error when the job is added, not when it runs.

    Accept when: a rolled-back transaction leaves no job, a job with run_at in the future is not taken before then, and an unknown kind is refused.

    R2. Take a job
    • A worker takes the waiting job with the earliest run_at, then the lowest id.
    • Two workers never take the same job.
    • A worker runs at most concurrency jobs at a time and can be told which queues to serve.
    • When there is nothing to do, a worker asks again after 1 second. It does not spin.

    Accept when: 4 workers and 200 jobs end with every job run exactly once, and an idle worker makes no more than 2 queries a second.

    R3. Retry
    • A job whose handler raises an error, or runs longer than its time limit of 5 minutes, has failed an attempt.
    • A failed job waits and is tried again. The wait after attempt n is 15 seconds times 4 to the power n minus 1: 15 seconds, 1 minute, 4 minutes, 16 minutes. A random 0 to 10% is added, from a function that tests can fix.
    • After max_attempts attempts the job is marked failed and kept.
    • Each failure stores the first 2,000 characters of the error with the job.
    • A handler can say "do not retry", which marks the job failed at once.

    Accept when: a handler that fails twice then succeeds is run three times with waits of 15 and 60 seconds, and one that always fails is run max_attempts times.

    R4. Workers that die
    • A worker holds each job it took under a lease of 1 minute and renews it every 15 seconds while the handler runs.
    • A job whose lease has run out is taken again by another worker. That counts as an attempt.
    • On a signal to stop, a worker takes nothing new, gives running handlers 30 seconds to finish, then puts back what is left without counting an attempt.

    Accept when: a job whose worker is killed is run again within 2 minutes, and a worker told to stop finishes a 5 second job before it exits.

    R5. Unique jobs
    • With a unique_key, adding a job while another with the same kind and key is waiting or running adds nothing and returns the existing job.
    • Once that job is done or failed, the key is free again.

    Accept when: 20 simultaneous calls with one key leave one job.

    R6. Recurring jobs

    Only if recurring_jobs is on.

    • A schedule is a name, a job kind, a payload and a five-field cron expression in UTC. Schedules are written in code and loaded when a worker starts.
    • Each time a schedule comes due, one job is added, however many workers are running.
    • If no worker was running when a schedule came due, one job is added when a worker starts, not one for every time that was missed.

    Accept when: three workers and a schedule of every minute add one job a minute, and a schedule that came due five times while everything was stopped adds one job.

    R7. Clean up
    • Jobs that finished are deleted after 7 days. Jobs that failed are deleted after 30.

    Accept when: a clean-up run deletes exactly the jobs that are old enough.

    R8. Dashboard

    Only if dashboard is on.

    • Counts of waiting, running, done and failed jobs for each queue, and the age of the oldest waiting job.
    • A list of failed jobs with kind, payload, error and number of attempts, with buttons to try again and to delete.
    • Only administrators reach it. Payloads may hold personal data.

    Accept when: "try again" puts a failed job back as waiting with its attempts set to zero, and a person who is not an administrator gets 403.

    Definition of done

    • Every acceptance check above has an automated test, and all tests pass. Tests set the clock and do not sleep for more than a second in total.
    • A README explains how to write a handler, why a handler must cope with being run twice, how to start a worker, and how to run the tests.
    # Background jobs
    
    ## What this builds
    
    A queue for work that should not happen inside a web request: sending email,
    calling other services, heavy calculation. Jobs live in the app's own database, so
    there is nothing else to run. Workers take jobs, retry the ones that fail, and
    never run one job in two places at once.
    
    Read `spekkr.inputs.json` first. A job is tried `max_attempts` times. One worker
    runs `concurrency` jobs at a time. `recurring_jobs` decides whether R6 exists and
    `dashboard` whether R8 does.
    
    ## Out of scope
    
    - Workflows made of several jobs that depend on each other.
    - Priorities beyond named queues.
    - Exactly-once execution. A job runs at least once, and handlers must cope with a
      repeat.
    
    ## Requirements
    
    Each requirement has an id. Tests name the id they cover.
    
    ### R1. Add a job
    
    - App code calls `enqueue(kind, payload, options)`. The payload is JSON of at most
      64 KB (64,000 bytes).
    - A job added inside a database transaction exists exactly when that transaction
      commits.
    - Options: `run_at` to start no earlier than a time, `queue` to name a queue other
      than `default`, and `unique_key`, covered by R5.
    - A `kind` with no registered handler is an error when the job is added, not when
      it runs.
    
    Accept when: a rolled-back transaction leaves no job, a job with `run_at` in the
    future is not taken before then, and an unknown kind is refused.
    
    ### R2. Take a job
    
    - A worker takes the waiting job with the earliest `run_at`, then the lowest id.
    - Two workers never take the same job.
    - A worker runs at most `concurrency` jobs at a time and can be told which queues
      to serve.
    - When there is nothing to do, a worker asks again after 1 second. It does not
      spin.
    
    Accept when: 4 workers and 200 jobs end with every job run exactly once, and an
    idle worker makes no more than 2 queries a second.
    
    ### R3. Retry
    
    - A job whose handler raises an error, or runs longer than its time limit of 5
      minutes, has failed an attempt.
    - A failed job waits and is tried again. The wait after attempt n is 15 seconds
      times 4 to the power n minus 1: 15 seconds, 1 minute, 4 minutes, 16 minutes. A
      random 0 to 10% is added, from a function that tests can fix.
    - After `max_attempts` attempts the job is marked failed and kept.
    - Each failure stores the first 2,000 characters of the error with the job.
    - A handler can say "do not retry", which marks the job failed at once.
    
    Accept when: a handler that fails twice then succeeds is run three times with waits
    of 15 and 60 seconds, and one that always fails is run `max_attempts` times.
    
    ### R4. Workers that die
    
    - A worker holds each job it took under a lease of 1 minute and renews it every 15
      seconds while the handler runs.
    - A job whose lease has run out is taken again by another worker. That counts as
      an attempt.
    - On a signal to stop, a worker takes nothing new, gives running handlers 30
      seconds to finish, then puts back what is left without counting an attempt.
    
    Accept when: a job whose worker is killed is run again within 2 minutes, and a
    worker told to stop finishes a 5 second job before it exits.
    
    ### R5. Unique jobs
    
    - With a `unique_key`, adding a job while another with the same kind and key is
      waiting or running adds nothing and returns the existing job.
    - Once that job is done or failed, the key is free again.
    
    Accept when: 20 simultaneous calls with one key leave one job.
    
    ### R6. Recurring jobs
    
    Only if `recurring_jobs` is on.
    
    - A schedule is a name, a job kind, a payload and a five-field cron expression in
      UTC. Schedules are written in code and loaded when a worker starts.
    - Each time a schedule comes due, one job is added, however many workers are
      running.
    - If no worker was running when a schedule came due, one job is added when a
      worker starts, not one for every time that was missed.
    
    Accept when: three workers and a schedule of every minute add one job a minute,
    and a schedule that came due five times while everything was stopped adds one job.
    
    ### R7. Clean up
    
    - Jobs that finished are deleted after 7 days. Jobs that failed are deleted after
      30.
    
    Accept when: a clean-up run deletes exactly the jobs that are old enough.
    
    ### R8. Dashboard
    
    Only if `dashboard` is on.
    
    - Counts of waiting, running, done and failed jobs for each queue, and the age of
      the oldest waiting job.
    - A list of failed jobs with kind, payload, error and number of attempts, with
      buttons to try again and to delete.
    - Only administrators reach it. Payloads may hold personal data.
    
    Accept when: "try again" puts a failed job back as waiting with its attempts set to
    zero, and a person who is not an administrator gets 403.
    
    ## Definition of done
    
    - Every acceptance check above has an automated test, and all tests pass. Tests set
      the clock and do not sleep for more than a second in total.
    - A README explains how to write a handler, why a handler must cope with being run
      twice, how to start a worker, and how to run the tests.
    

    Open spec.md as plain text

  • design.md Data model, rules and flows.

    Design

    This file fixes the parts that are easy to get wrong. Choose the language and framework that fit the person's project; nothing here depends on them.

    Data model

    Use the database in the database input.

    TableColumns
    jobsid, queue, kind, payload (JSON), state, attempts, run_at, leased_until, leased_by, unique_key, last_error, created_at, finished_at
    job_schedulesname (primary key), kind, payload, cron, last_due_at (only if recurring_jobs is on)

    state is one of waiting, running, done and failed.

    Indexes: one on queue, state, run_at, id for taking jobs, and a unique one on kind, unique_key where unique_key is set and state is waiting or running.

    Taking a job

    A job can be taken when it is waiting and run_at has passed, or when it is running and leased_until has passed.

    Postgres:

    UPDATE jobs SET state = 'running', attempts = attempts + 1,
           leased_until = $now + interval '1 minute', leased_by = $worker
    WHERE id IN (
      SELECT id FROM jobs
      WHERE queue = ANY($queues)
        AND ((state = 'waiting' AND run_at <= $now)
          OR (state = 'running' AND leased_until < $now))
      ORDER BY run_at, id
      LIMIT $free
      FOR UPDATE SKIP LOCKED
    )
    RETURNING *;

    SQLite has one writer at a time, so the same UPDATE ... WHERE id IN (SELECT ...) RETURNING * without the last line of the inner query is already safe. Open the database in WAL mode with a busy timeout of 5 seconds.

    A job taken with attempts already at max_attempts because its lease ran out is marked failed with the error worker lost and not run.

    Finishing a job

    • Success: set state to done and finished_at.
    • Failure with attempts left: set state to waiting and run_at to now plus the wait in R3.
    • Failure with none left, or "do not retry": set state to failed.
    • Every one of these updates names the worker in its WHERE clause, so a worker that lost its lease cannot overwrite the worker that took over.

    Time

    Take the time from the database, not from each worker's clock, so workers on machines whose clocks differ agree. Wrap that in one function that tests can replace.

    Recurring jobs

    Once a minute each worker runs this in one transaction for each schedule: lock the schedule's row, work out the most recent time the cron expression was due, and if that is later than last_due_at, add one job and store the time. The lock and the comparison are what make three workers add one job.

    Support the five cron fields with numbers, *, lists, ranges and steps. Names of months and weekdays are not needed.

    Configuration

    SecretPurpose
    DATABASE_URLConnection string for the chosen database.

    Who is signed in

    The spec reaches the app's sign-in through one function, current_user(request). It answers with the person's id, and says whether they are an administrator where this spec asks. In the person's app it reads their session.

    When there is no app yet, build this stand-in and nothing more: a page that lists two seeded people, one of them an administrator, with a button for each that sets a signed cookie naming them. The stand-in refuses to start in production.

    Named here

    • In a schedule, when the day of the month and the day of the week are both given, either one matching is enough. Sunday is 0 or 7.
    • The two errors are written time limit exceeded and worker lost.

    What this spec leaves to you

    Where this spec names something, use that name, so that two builds of it agree. Where it does not, the choice is yours. List what you chose in the README.

    # Design
    
    This file fixes the parts that are easy to get wrong. Choose the language and
    framework that fit the person's project; nothing here depends on them.
    
    ## Data model
    
    Use the database in the `database` input.
    
    | Table | Columns |
    | --- | --- |
    | `jobs` | `id`, `queue`, `kind`, `payload` (JSON), `state`, `attempts`, `run_at`, `leased_until`, `leased_by`, `unique_key`, `last_error`, `created_at`, `finished_at` |
    | `job_schedules` | `name` (primary key), `kind`, `payload`, `cron`, `last_due_at` (only if `recurring_jobs` is on) |
    
    `state` is one of `waiting`, `running`, `done` and `failed`.
    
    Indexes: one on `queue`, `state`, `run_at`, `id` for taking jobs, and a unique one
    on `kind`, `unique_key` where `unique_key` is set and `state` is `waiting` or
    `running`.
    
    ## Taking a job
    
    A job can be taken when it is `waiting` and `run_at` has passed, or when it is
    `running` and `leased_until` has passed.
    
    Postgres:
    
    ```sql
    UPDATE jobs SET state = 'running', attempts = attempts + 1,
           leased_until = $now + interval '1 minute', leased_by = $worker
    WHERE id IN (
      SELECT id FROM jobs
      WHERE queue = ANY($queues)
        AND ((state = 'waiting' AND run_at <= $now)
          OR (state = 'running' AND leased_until < $now))
      ORDER BY run_at, id
      LIMIT $free
      FOR UPDATE SKIP LOCKED
    )
    RETURNING *;
    ```
    
    SQLite has one writer at a time, so the same `UPDATE ... WHERE id IN (SELECT ...)
    RETURNING *` without the last line of the inner query is already safe. Open the
    database in WAL mode with a busy timeout of 5 seconds.
    
    A job taken with `attempts` already at `max_attempts` because its lease ran out is
    marked failed with the error `worker lost` and not run.
    
    ## Finishing a job
    
    - Success: set `state` to `done` and `finished_at`.
    - Failure with attempts left: set `state` to `waiting` and `run_at` to now plus the
      wait in R3.
    - Failure with none left, or "do not retry": set `state` to `failed`.
    - Every one of these updates names the worker in its `WHERE` clause, so a worker
      that lost its lease cannot overwrite the worker that took over.
    
    ## Time
    
    Take the time from the database, not from each worker's clock, so workers on
    machines whose clocks differ agree. Wrap that in one function that tests can
    replace.
    
    ## Recurring jobs
    
    Once a minute each worker runs this in one transaction for each schedule: lock the
    schedule's row, work out the most recent time the cron expression was due, and if
    that is later than `last_due_at`, add one job and store the time. The lock and the
    comparison are what make three workers add one job.
    
    Support the five cron fields with numbers, `*`, lists, ranges and steps. Names of
    months and weekdays are not needed.
    
    ## Configuration
    
    | Secret | Purpose |
    | --- | --- |
    | `DATABASE_URL` | Connection string for the chosen database. |
    
    ## Who is signed in
    
    The spec reaches the app's sign-in through one function, `current_user(request)`.
    It answers with the person's id, and says whether they are an administrator where
    this spec asks. In the person's app it reads their session.
    
    When there is no app yet, build this stand-in and nothing more: a page that lists
    two seeded people, one of them an administrator, with a button for each that sets
    a signed cookie naming them. The stand-in refuses to start in production.
    
    ## Named here
    
    - In a schedule, when the day of the month and the day of the week are both
      given, either one matching is enough. Sunday is `0` or `7`.
    - The two errors are written `time limit exceeded` and `worker lost`.
    
    ## What this spec leaves to you
    
    Where this spec names something, use that name, so that two builds of it agree.
    Where it does not, the choice is yours. List what you chose in the README.
    

    Open design.md as plain text

  • tasks.md Build tasks in order, each with a check.

    Tasks

    Work in order. Finish a task's check before starting the next.

    1. Set up. Add the tables and indexes in design.md, and a clock that tests can set. Check: migrations run from empty.
    2. Add a job (R1). With a registry of handlers by kind. Check: the R1 acceptance tests pass.
    3. Take and finish a job (R2). The queries in design.md, and a worker loop that runs up to concurrency handlers. Check: the R2 test with 4 workers and 200 jobs passes.
    4. Retry (R3). Use the max_attempts input. Check: the R3 acceptance tests pass with the random part fixed at zero.
    5. Leases and stopping (R4). Check: the R4 acceptance tests pass.
    6. Unique jobs (R5). Check: the R5 acceptance test passes.
    7. Recurring jobs (R6). Only if recurring_jobs is on. Check: the R6 acceptance tests pass.
    8. Clean up (R7). Check: the R7 acceptance test passes.
    9. Dashboard (R8). Only if dashboard is on. Check: the R8 acceptance tests pass.
    10. Finish. Write the README and run the whole test suite. Check: every item under "Definition of done" in spec.md is true.
    # Tasks
    
    Work in order. Finish a task's check before starting the next.
    
    1. **Set up.** Add the tables and indexes in `design.md`, and a clock that tests
       can set.
       Check: migrations run from empty.
    
    2. **Add a job (R1).** With a registry of handlers by kind.
       Check: the R1 acceptance tests pass.
    
    3. **Take and finish a job (R2).** The queries in `design.md`, and a worker loop
       that runs up to `concurrency` handlers.
       Check: the R2 test with 4 workers and 200 jobs passes.
    
    4. **Retry (R3).** Use the `max_attempts` input.
       Check: the R3 acceptance tests pass with the random part fixed at zero.
    
    5. **Leases and stopping (R4).**
       Check: the R4 acceptance tests pass.
    
    6. **Unique jobs (R5).**
       Check: the R5 acceptance test passes.
    
    7. **Recurring jobs (R6).** Only if `recurring_jobs` is on.
       Check: the R6 acceptance tests pass.
    
    8. **Clean up (R7).**
       Check: the R7 acceptance test passes.
    
    9. **Dashboard (R8).** Only if `dashboard` is on.
       Check: the R8 acceptance tests pass.
    
    10. **Finish.** Write the README and run the whole test suite.
        Check: every item under "Definition of done" in `spec.md` is true.
    

    Open tasks.md as plain text

Is it for me?

Platforms
backend

What does it need?

Agent permissions
shell, network
Secrets it reads
DATABASE_URL

What changed?

  • 0.1.215 Nov 2026Names what it builds.
  • 0.1.114 Nov 2026Says "person" where it said "buyer".
  • 0.1.014 Nov 2026Moved from dwrtz/background-jobs.

For agents: start.md says how to build this spec, spec.json holds everything on this page, and bundle.tar.gz holds its files.