# Background jobs ## What this builds A queue for work that should not happen inside a web request: sending email, calling other services, heavy calculation. Jobs live in the app's own database, so there is nothing else to run. Workers take jobs, retry the ones that fail, and never run one job in two places at once. Read `spekkr.inputs.json` first. A job is tried `max_attempts` times. One worker runs `concurrency` jobs at a time. `recurring_jobs` decides whether R6 exists and `dashboard` whether R8 does. ## Out of scope - Workflows made of several jobs that depend on each other. - Priorities beyond named queues. - Exactly-once execution. A job runs at least once, and handlers must cope with a repeat. ## Requirements Each requirement has an id. Tests name the id they cover. ### R1. Add a job - App code calls `enqueue(kind, payload, options)`. The payload is JSON of at most 64 KB (64,000 bytes). - A job added inside a database transaction exists exactly when that transaction commits. - Options: `run_at` to start no earlier than a time, `queue` to name a queue other than `default`, and `unique_key`, covered by R5. - A `kind` with no registered handler is an error when the job is added, not when it runs. Accept when: a rolled-back transaction leaves no job, a job with `run_at` in the future is not taken before then, and an unknown kind is refused. ### R2. Take a job - A worker takes the waiting job with the earliest `run_at`, then the lowest id. - Two workers never take the same job. - A worker runs at most `concurrency` jobs at a time and can be told which queues to serve. - When there is nothing to do, a worker asks again after 1 second. It does not spin. Accept when: 4 workers and 200 jobs end with every job run exactly once, and an idle worker makes no more than 2 queries a second. ### R3. Retry - A job whose handler raises an error, or runs longer than its time limit of 5 minutes, has failed an attempt. - A failed job waits and is tried again. The wait after attempt n is 15 seconds times 4 to the power n minus 1: 15 seconds, 1 minute, 4 minutes, 16 minutes. A random 0 to 10% is added, from a function that tests can fix. - After `max_attempts` attempts the job is marked failed and kept. - Each failure stores the first 2,000 characters of the error with the job. - A handler can say "do not retry", which marks the job failed at once. Accept when: a handler that fails twice then succeeds is run three times with waits of 15 and 60 seconds, and one that always fails is run `max_attempts` times. ### R4. Workers that die - A worker holds each job it took under a lease of 1 minute and renews it every 15 seconds while the handler runs. - A job whose lease has run out is taken again by another worker. That counts as an attempt. - On a signal to stop, a worker takes nothing new, gives running handlers 30 seconds to finish, then puts back what is left without counting an attempt. Accept when: a job whose worker is killed is run again within 2 minutes, and a worker told to stop finishes a 5 second job before it exits. ### R5. Unique jobs - With a `unique_key`, adding a job while another with the same kind and key is waiting or running adds nothing and returns the existing job. - Once that job is done or failed, the key is free again. Accept when: 20 simultaneous calls with one key leave one job. ### R6. Recurring jobs Only if `recurring_jobs` is on. - A schedule is a name, a job kind, a payload and a five-field cron expression in UTC. Schedules are written in code and loaded when a worker starts. - Each time a schedule comes due, one job is added, however many workers are running. - If no worker was running when a schedule came due, one job is added when a worker starts, not one for every time that was missed. Accept when: three workers and a schedule of every minute add one job a minute, and a schedule that came due five times while everything was stopped adds one job. ### R7. Clean up - Jobs that finished are deleted after 7 days. Jobs that failed are deleted after 30. Accept when: a clean-up run deletes exactly the jobs that are old enough. ### R8. Dashboard Only if `dashboard` is on. - Counts of waiting, running, done and failed jobs for each queue, and the age of the oldest waiting job. - A list of failed jobs with kind, payload, error and number of attempts, with buttons to try again and to delete. - Only administrators reach it. Payloads may hold personal data. Accept when: "try again" puts a failed job back as waiting with its attempts set to zero, and a person who is not an administrator gets 403. ## Definition of done - Every acceptance check above has an automated test, and all tests pass. Tests set the clock and do not sleep for more than a second in total. - A README explains how to write a handler, why a handler must cope with being run twice, how to start a worker, and how to run the tests.