Last Updated: 2024-11-21

Imagine you had an SQS (Simple Queue System) worker for processing an image. It has the following code

(Background info: The process can have state either Created or Uploade. Also assume this queue may process the same thing twice due to reasons)

uploaded(s3Key) {
  process = DynamoDB.getItem(processId)

  if (process.state !== 'Created') {
    throw new Error('transition not allowed')
  }

  DynamoDB.updateItem(processId, {'state': 'Uploaded', 'rawS3Key': s3Key})
  SQS.sendMessage({'processId': processId, 'action': 'process'});

}

Because it can throw an error if the state transition is not in the expected order ('Created' first, then 'Uploaded'), the code is not idempotent. Double processing will cause exceptions.

You can avoid this by modifying the code to allow multiple updates to Dynamo (does not hurt), opening things up to the odd bit of repetitive work but avoiding exceptions and exception reports.

uploaded(s3Key) {
  process = DynamoDB.getItem(processId)

  if (process.state !== 'Created' && process.state !== 'Uploaded') {
    throw new Error('transition not allowed')
  }

  DynamoDB.updateItem(processId, {'state': 'Uploaded', 'rawS3Key': s3Key})
  SQS.sendMessage({'processId': processId, 'action': 'process'});
}

If you retry now, you’ll make multiple updates to Dynamo, which doesn’t hurt

Lesson

Prefer to do small bits of work twice without complaining in queue systems to throwing exceptions.