> but somehow this feels like you've only lowered the likelihood of multiple job executions
This is basically the best you can do. Imagine the context of a strongly consistent store you use for queues, and N workers. The store is strictly CP. One worker gets task "A", executes it, but does not provide the acknowledge, because it crashes before to ack. In order to guarantee at-least-once-execution property the system has to re-issue the task to another worker after a timeout, causing multiple job execution.
If you want to avoid this, you have as a side effect a failure mode where it is possible that jobs are lost forever, which is a lot worse, since the original problem can be solved by making all jobs idempotent.
Basically Redis-based queues should try to provide guarantees about durability of jobs, and try to provide just best-effort mechanisms to avoid duplication of jobs when possible.
This is basically the best you can do. Imagine the context of a strongly consistent store you use for queues, and N workers. The store is strictly CP. One worker gets task "A", executes it, but does not provide the acknowledge, because it crashes before to ack. In order to guarantee at-least-once-execution property the system has to re-issue the task to another worker after a timeout, causing multiple job execution.
If you want to avoid this, you have as a side effect a failure mode where it is possible that jobs are lost forever, which is a lot worse, since the original problem can be solved by making all jobs idempotent.
Basically Redis-based queues should try to provide guarantees about durability of jobs, and try to provide just best-effort mechanisms to avoid duplication of jobs when possible.