Background jobs
A job queue in your own database: retries with backoff, scheduled and recurring jobs, and no job run by two workers at once.
This is an app part spec on Spekkr, published by vex. It is listed with the other specs. Paste the line to an agent that can open a web address, and the agent fetches the files and builds from them.
- Spec
vex/background-jobs- Version
- 0.1.2
- Published by
- vex
- Updated
- 15 Nov 2026
Build this spec
Build the spec at https://spekkr.com/vex/background-jobs/start.md.
Paste this to your AI agent. It opens the address, fetches the spec's files, records your choices and builds from them. Nothing to install, no account.
Published by vex.
What does the result look like?
What can I customize?
Your choices go into the line you copy. Anything else is yours to edit, but it leaves the tested path.
What do I get?
-
spec.mdRequirements and acceptance criteria.Background jobs
What this builds
A queue for work that should not happen inside a web request: sending email, calling other services, heavy calculation. Jobs live in the app's own database, so there is nothing else to run. Workers take jobs, retry the ones that fail, and never run one job in two places at once.
Read
spekkr.inputs.jsonfirst. A job is triedmax_attemptstimes. One worker runsconcurrencyjobs at a time.recurring_jobsdecides whether R6 exists anddashboardwhether R8 does.Out of scope
- Workflows made of several jobs that depend on each other.
- Priorities beyond named queues.
- Exactly-once execution. A job runs at least once, and handlers must cope with a repeat.
Requirements
Each requirement has an id. Tests name the id they cover.
R1. Add a job
- App code calls
enqueue(kind, payload, options). The payload is JSON of at most 64 KB (64,000 bytes). - A job added inside a database transaction exists exactly when that transaction commits.
- Options:
run_atto start no earlier than a time,queueto name a queue other thandefault, andunique_key, covered by R5. - A
kindwith no registered handler is an error when the job is added, not when it runs.
Accept when: a rolled-back transaction leaves no job, a job with
run_atin the future is not taken before then, and an unknown kind is refused.R2. Take a job
- A worker takes the waiting job with the earliest
run_at, then the lowest id. - Two workers never take the same job.
- A worker runs at most
concurrencyjobs at a time and can be told which queues to serve. - When there is nothing to do, a worker asks again after 1 second. It does not spin.
Accept when: 4 workers and 200 jobs end with every job run exactly once, and an idle worker makes no more than 2 queries a second.
R3. Retry
- A job whose handler raises an error, or runs longer than its time limit of 5 minutes, has failed an attempt.
- A failed job waits and is tried again. The wait after attempt n is 15 seconds times 4 to the power n minus 1: 15 seconds, 1 minute, 4 minutes, 16 minutes. A random 0 to 10% is added, from a function that tests can fix.
- After
max_attemptsattempts the job is marked failed and kept. - Each failure stores the first 2,000 characters of the error with the job.
- A handler can say "do not retry", which marks the job failed at once.
Accept when: a handler that fails twice then succeeds is run three times with waits of 15 and 60 seconds, and one that always fails is run
max_attemptstimes.R4. Workers that die
- A worker holds each job it took under a lease of 1 minute and renews it every 15 seconds while the handler runs.
- A job whose lease has run out is taken again by another worker. That counts as an attempt.
- On a signal to stop, a worker takes nothing new, gives running handlers 30 seconds to finish, then puts back what is left without counting an attempt.
Accept when: a job whose worker is killed is run again within 2 minutes, and a worker told to stop finishes a 5 second job before it exits.
R5. Unique jobs
- With a
unique_key, adding a job while another with the same kind and key is waiting or running adds nothing and returns the existing job. - Once that job is done or failed, the key is free again.
Accept when: 20 simultaneous calls with one key leave one job.
R6. Recurring jobs
Only if
recurring_jobsis on.- A schedule is a name, a job kind, a payload and a five-field cron expression in UTC. Schedules are written in code and loaded when a worker starts.
- Each time a schedule comes due, one job is added, however many workers are running.
- If no worker was running when a schedule came due, one job is added when a worker starts, not one for every time that was missed.
Accept when: three workers and a schedule of every minute add one job a minute, and a schedule that came due five times while everything was stopped adds one job.
R7. Clean up
- Jobs that finished are deleted after 7 days. Jobs that failed are deleted after 30.
Accept when: a clean-up run deletes exactly the jobs that are old enough.
R8. Dashboard
Only if
dashboardis on.- Counts of waiting, running, done and failed jobs for each queue, and the age of the oldest waiting job.
- A list of failed jobs with kind, payload, error and number of attempts, with buttons to try again and to delete.
- Only administrators reach it. Payloads may hold personal data.
Accept when: "try again" puts a failed job back as waiting with its attempts set to zero, and a person who is not an administrator gets 403.
Definition of done
- Every acceptance check above has an automated test, and all tests pass. Tests set the clock and do not sleep for more than a second in total.
- A README explains how to write a handler, why a handler must cope with being run twice, how to start a worker, and how to run the tests.
# Background jobs ## What this builds A queue for work that should not happen inside a web request: sending email, calling other services, heavy calculation. Jobs live in the app's own database, so there is nothing else to run. Workers take jobs, retry the ones that fail, and never run one job in two places at once. Read `spekkr.inputs.json` first. A job is tried `max_attempts` times. One worker runs `concurrency` jobs at a time. `recurring_jobs` decides whether R6 exists and `dashboard` whether R8 does. ## Out of scope - Workflows made of several jobs that depend on each other. - Priorities beyond named queues. - Exactly-once execution. A job runs at least once, and handlers must cope with a repeat. ## Requirements Each requirement has an id. Tests name the id they cover. ### R1. Add a job - App code calls `enqueue(kind, payload, options)`. The payload is JSON of at most 64 KB (64,000 bytes). - A job added inside a database transaction exists exactly when that transaction commits. - Options: `run_at` to start no earlier than a time, `queue` to name a queue other than `default`, and `unique_key`, covered by R5. - A `kind` with no registered handler is an error when the job is added, not when it runs. Accept when: a rolled-back transaction leaves no job, a job with `run_at` in the future is not taken before then, and an unknown kind is refused. ### R2. Take a job - A worker takes the waiting job with the earliest `run_at`, then the lowest id. - Two workers never take the same job. - A worker runs at most `concurrency` jobs at a time and can be told which queues to serve. - When there is nothing to do, a worker asks again after 1 second. It does not spin. Accept when: 4 workers and 200 jobs end with every job run exactly once, and an idle worker makes no more than 2 queries a second. ### R3. Retry - A job whose handler raises an error, or runs longer than its time limit of 5 minutes, has failed an attempt. - A failed job waits and is tried again. The wait after attempt n is 15 seconds times 4 to the power n minus 1: 15 seconds, 1 minute, 4 minutes, 16 minutes. A random 0 to 10% is added, from a function that tests can fix. - After `max_attempts` attempts the job is marked failed and kept. - Each failure stores the first 2,000 characters of the error with the job. - A handler can say "do not retry", which marks the job failed at once. Accept when: a handler that fails twice then succeeds is run three times with waits of 15 and 60 seconds, and one that always fails is run `max_attempts` times. ### R4. Workers that die - A worker holds each job it took under a lease of 1 minute and renews it every 15 seconds while the handler runs. - A job whose lease has run out is taken again by another worker. That counts as an attempt. - On a signal to stop, a worker takes nothing new, gives running handlers 30 seconds to finish, then puts back what is left without counting an attempt. Accept when: a job whose worker is killed is run again within 2 minutes, and a worker told to stop finishes a 5 second job before it exits. ### R5. Unique jobs - With a `unique_key`, adding a job while another with the same kind and key is waiting or running adds nothing and returns the existing job. - Once that job is done or failed, the key is free again. Accept when: 20 simultaneous calls with one key leave one job. ### R6. Recurring jobs Only if `recurring_jobs` is on. - A schedule is a name, a job kind, a payload and a five-field cron expression in UTC. Schedules are written in code and loaded when a worker starts. - Each time a schedule comes due, one job is added, however many workers are running. - If no worker was running when a schedule came due, one job is added when a worker starts, not one for every time that was missed. Accept when: three workers and a schedule of every minute add one job a minute, and a schedule that came due five times while everything was stopped adds one job. ### R7. Clean up - Jobs that finished are deleted after 7 days. Jobs that failed are deleted after 30. Accept when: a clean-up run deletes exactly the jobs that are old enough. ### R8. Dashboard Only if `dashboard` is on. - Counts of waiting, running, done and failed jobs for each queue, and the age of the oldest waiting job. - A list of failed jobs with kind, payload, error and number of attempts, with buttons to try again and to delete. - Only administrators reach it. Payloads may hold personal data. Accept when: "try again" puts a failed job back as waiting with its attempts set to zero, and a person who is not an administrator gets 403. ## Definition of done - Every acceptance check above has an automated test, and all tests pass. Tests set the clock and do not sleep for more than a second in total. - A README explains how to write a handler, why a handler must cope with being run twice, how to start a worker, and how to run the tests.
-
design.mdData model, rules and flows.Design
This file fixes the parts that are easy to get wrong. Choose the language and framework that fit the person's project; nothing here depends on them.
Data model
Use the database in the
databaseinput.Table Columns jobsid,queue,kind,payload(JSON),state,attempts,run_at,leased_until,leased_by,unique_key,last_error,created_at,finished_atjob_schedulesname(primary key),kind,payload,cron,last_due_at(only ifrecurring_jobsis on)stateis one ofwaiting,running,doneandfailed.Indexes: one on
queue,state,run_at,idfor taking jobs, and a unique one onkind,unique_keywhereunique_keyis set andstateiswaitingorrunning.Taking a job
A job can be taken when it is
waitingandrun_athas passed, or when it isrunningandleased_untilhas passed.Postgres:
UPDATE jobs SET state = 'running', attempts = attempts + 1, leased_until = $now + interval '1 minute', leased_by = $worker WHERE id IN ( SELECT id FROM jobs WHERE queue = ANY($queues) AND ((state = 'waiting' AND run_at <= $now) OR (state = 'running' AND leased_until < $now)) ORDER BY run_at, id LIMIT $free FOR UPDATE SKIP LOCKED ) RETURNING *;SQLite has one writer at a time, so the same
UPDATE ... WHERE id IN (SELECT ...) RETURNING *without the last line of the inner query is already safe. Open the database in WAL mode with a busy timeout of 5 seconds.A job taken with
attemptsalready atmax_attemptsbecause its lease ran out is marked failed with the errorworker lostand not run.Finishing a job
- Success: set
statetodoneandfinished_at. - Failure with attempts left: set
statetowaitingandrun_atto now plus the wait in R3. - Failure with none left, or "do not retry": set
statetofailed. - Every one of these updates names the worker in its
WHEREclause, so a worker that lost its lease cannot overwrite the worker that took over.
Time
Take the time from the database, not from each worker's clock, so workers on machines whose clocks differ agree. Wrap that in one function that tests can replace.
Recurring jobs
Once a minute each worker runs this in one transaction for each schedule: lock the schedule's row, work out the most recent time the cron expression was due, and if that is later than
last_due_at, add one job and store the time. The lock and the comparison are what make three workers add one job.Support the five cron fields with numbers,
*, lists, ranges and steps. Names of months and weekdays are not needed.Configuration
Secret Purpose DATABASE_URLConnection string for the chosen database. Who is signed in
The spec reaches the app's sign-in through one function,
current_user(request). It answers with the person's id, and says whether they are an administrator where this spec asks. In the person's app it reads their session.When there is no app yet, build this stand-in and nothing more: a page that lists two seeded people, one of them an administrator, with a button for each that sets a signed cookie naming them. The stand-in refuses to start in production.
Named here
- In a schedule, when the day of the month and the day of the week are both given, either one matching is enough. Sunday is
0or7. - The two errors are written
time limit exceededandworker lost.
What this spec leaves to you
Where this spec names something, use that name, so that two builds of it agree. Where it does not, the choice is yours. List what you chose in the README.
# Design This file fixes the parts that are easy to get wrong. Choose the language and framework that fit the person's project; nothing here depends on them. ## Data model Use the database in the `database` input. | Table | Columns | | --- | --- | | `jobs` | `id`, `queue`, `kind`, `payload` (JSON), `state`, `attempts`, `run_at`, `leased_until`, `leased_by`, `unique_key`, `last_error`, `created_at`, `finished_at` | | `job_schedules` | `name` (primary key), `kind`, `payload`, `cron`, `last_due_at` (only if `recurring_jobs` is on) | `state` is one of `waiting`, `running`, `done` and `failed`. Indexes: one on `queue`, `state`, `run_at`, `id` for taking jobs, and a unique one on `kind`, `unique_key` where `unique_key` is set and `state` is `waiting` or `running`. ## Taking a job A job can be taken when it is `waiting` and `run_at` has passed, or when it is `running` and `leased_until` has passed. Postgres: ```sql UPDATE jobs SET state = 'running', attempts = attempts + 1, leased_until = $now + interval '1 minute', leased_by = $worker WHERE id IN ( SELECT id FROM jobs WHERE queue = ANY($queues) AND ((state = 'waiting' AND run_at <= $now) OR (state = 'running' AND leased_until < $now)) ORDER BY run_at, id LIMIT $free FOR UPDATE SKIP LOCKED ) RETURNING *; ``` SQLite has one writer at a time, so the same `UPDATE ... WHERE id IN (SELECT ...) RETURNING *` without the last line of the inner query is already safe. Open the database in WAL mode with a busy timeout of 5 seconds. A job taken with `attempts` already at `max_attempts` because its lease ran out is marked failed with the error `worker lost` and not run. ## Finishing a job - Success: set `state` to `done` and `finished_at`. - Failure with attempts left: set `state` to `waiting` and `run_at` to now plus the wait in R3. - Failure with none left, or "do not retry": set `state` to `failed`. - Every one of these updates names the worker in its `WHERE` clause, so a worker that lost its lease cannot overwrite the worker that took over. ## Time Take the time from the database, not from each worker's clock, so workers on machines whose clocks differ agree. Wrap that in one function that tests can replace. ## Recurring jobs Once a minute each worker runs this in one transaction for each schedule: lock the schedule's row, work out the most recent time the cron expression was due, and if that is later than `last_due_at`, add one job and store the time. The lock and the comparison are what make three workers add one job. Support the five cron fields with numbers, `*`, lists, ranges and steps. Names of months and weekdays are not needed. ## Configuration | Secret | Purpose | | --- | --- | | `DATABASE_URL` | Connection string for the chosen database. | ## Who is signed in The spec reaches the app's sign-in through one function, `current_user(request)`. It answers with the person's id, and says whether they are an administrator where this spec asks. In the person's app it reads their session. When there is no app yet, build this stand-in and nothing more: a page that lists two seeded people, one of them an administrator, with a button for each that sets a signed cookie naming them. The stand-in refuses to start in production. ## Named here - In a schedule, when the day of the month and the day of the week are both given, either one matching is enough. Sunday is `0` or `7`. - The two errors are written `time limit exceeded` and `worker lost`. ## What this spec leaves to you Where this spec names something, use that name, so that two builds of it agree. Where it does not, the choice is yours. List what you chose in the README. - Success: set
-
tasks.mdBuild tasks in order, each with a check.Tasks
Work in order. Finish a task's check before starting the next.
- Set up. Add the tables and indexes in
design.md, and a clock that tests can set. Check: migrations run from empty. - Add a job (R1). With a registry of handlers by kind. Check: the R1 acceptance tests pass.
- Take and finish a job (R2). The queries in
design.md, and a worker loop that runs up toconcurrencyhandlers. Check: the R2 test with 4 workers and 200 jobs passes. - Retry (R3). Use the
max_attemptsinput. Check: the R3 acceptance tests pass with the random part fixed at zero. - Leases and stopping (R4). Check: the R4 acceptance tests pass.
- Unique jobs (R5). Check: the R5 acceptance test passes.
- Recurring jobs (R6). Only if
recurring_jobsis on. Check: the R6 acceptance tests pass. - Clean up (R7). Check: the R7 acceptance test passes.
- Dashboard (R8). Only if
dashboardis on. Check: the R8 acceptance tests pass. - Finish. Write the README and run the whole test suite. Check: every item under "Definition of done" in
spec.mdis true.
# Tasks Work in order. Finish a task's check before starting the next. 1. **Set up.** Add the tables and indexes in `design.md`, and a clock that tests can set. Check: migrations run from empty. 2. **Add a job (R1).** With a registry of handlers by kind. Check: the R1 acceptance tests pass. 3. **Take and finish a job (R2).** The queries in `design.md`, and a worker loop that runs up to `concurrency` handlers. Check: the R2 test with 4 workers and 200 jobs passes. 4. **Retry (R3).** Use the `max_attempts` input. Check: the R3 acceptance tests pass with the random part fixed at zero. 5. **Leases and stopping (R4).** Check: the R4 acceptance tests pass. 6. **Unique jobs (R5).** Check: the R5 acceptance test passes. 7. **Recurring jobs (R6).** Only if `recurring_jobs` is on. Check: the R6 acceptance tests pass. 8. **Clean up (R7).** Check: the R7 acceptance test passes. 9. **Dashboard (R8).** Only if `dashboard` is on. Check: the R8 acceptance tests pass. 10. **Finish.** Write the README and run the whole test suite. Check: every item under "Definition of done" in `spec.md` is true. - Set up. Add the tables and indexes in
Is it for me?
- Kind
- app part
- Platforms
- backend
What does it need?
- Agent permissions
- shell, network
- Secrets it reads
DATABASE_URL
What changed?
- 0.1.215 Nov 2026Names what it builds.
- 0.1.114 Nov 2026Says "person" where it said "buyer".
- 0.1.014 Nov 2026Moved from dwrtz/background-jobs.
For agents: start.md says how to build this spec, spec.json holds everything on this page, and bundle.tar.gz holds its files.