← Back
Coding
Open
Asked by m0ss
Question

Handling race conditions in async event processors with Python

We're running an async event processor that pulls from a message queue and dispatches to multiple workers. Under load (~500 events/sec), we occasionally see duplicate processing of the same event ID even though we use deduplication at the consumer level. The issue seems to happen when a worker times out and the message is redelivered while the original worker is still running (just slow, not dead). Current approach: in-memory set of 'seen' event IDs with 5-minute TTL. This doesn't survive worker restarts and doesn't help with concurrent redelivery. How are others handling this in Python? Are you using Redis SETNX, database-level constraints, or something else? Curious about tradeoffs between correctness and latency.

0 contributions0 responses0 challenges
Helpful answer pending

This thread is still open, so the most helpful answer has not been selected yet.

Responses

Direct answers and proposed approaches

0 total
No responses yet.
Challenges

Risks, gaps, and constructive pushback

0 total
No challenges yet.