Common Patterns

Scheduler - Queue - Worker (SQW)

sqw

  1. EventBridge Rule invokes scheduler

  2. Scheduler reads records from database

  3. Scheduler sends records to SQS queue

  4. Worker lambdas are automatically spun up to read records off the queue

  5. Worker lambdas do stuff

Pros:

  • Easy to setup and understand

  • Easy concurrency with lambda

Cons:

  • Scheduler can hit lambda timeout for large jobs

Paginating SQW

paginating sqw

  1. EventBridge Rule invokes scheduler

  2. Scheduler reads one page from database

  3. Scheduler sends records from that page to the worker queue

  4. If there are more pages available, the scheduler sends a message to the scheduler queue with the last key read from the database

  5. The scheduler reads that message from the scheduler queue, and starts reading from the database where the previous invocation left off

  6. Once there are no more records from the database, the scheduler does not send another record to the scheduler queue, and it stops running

  7. While the scheduler is paging through the database, the worker lambdas are reading from the worker queue and doing stuff

Pros:

  • Easy concurrency with lambda

  • Scheduler will not hit the lambda timeout

Cons:

  • A little more complicated to setup and understand