Delayed job enqueue2/19/2023 All of them solve the problems that Rails tries to solve with delayed jobs. In Ruby too, just not in Rails.Įvent-based architectures such as event-sourcing, -listeners, -subscribers, message-buses, actor pattern. Negative tone aside, indeed: the problems that Rails "solves" here have long been solved in computing. This has worked well for us to the tune of about 35 million jobs a day. It's not perfect and there is probably a day where we switch it out but for now we have developed a deep understanding of Sidekiq and Redis and get to leverage the larger Sidekiq ecosystem. This is good for both durability and keeping the surface area of Redis work smaller. Additionally any jobs scheduled beyond a certain time threshold are committed to the DB, not Redis, and we have a periodic job that adds them back to Redis as we approach their scheduled time to run. We are also mindful of our payload sizes, which we also compress. We over-provision compute and memory and enable persistence as a result. There is still room for some inconsistency - a failure to enqueue a job doesn't rollback a committed transaction, for instance, but in practice that is rare.Īgreed on the Redis health being critical and if it's unhealthy you can't trust anything. Our solution was to build middleware/interfaces that only push jobs after a transaction has committed. The lesson learned was queues at scale are hard distributed systems problems - you probably don’t want to have to know how they work, or host the distributed system yourself. I’ve used both and patched some perf issues on DJ. I’ve queued 1B+ jobs in SQS and watched the workers slowly burn through them over the course of a month.ĭJ and Sidekiq can go quite far. A fully hosted option like SQS addresses the durability concerns of Sidekiq, and the scaling concerns of both. This is another case for neither DJ or Sidekiq. It also scales much less linearly than other queue options.Ī nice middle ground is to store all your data and state in your DB transactionally, and only queue tiny lightweight jobs with database primary keys (database: sentWelcomeMailDate=nil, queue: sentWelcomeMailForUser:1234). Using your primary DB means all that disk usage, memory usage, and IO hit your primary DB in exceptional ways and impacts your app in more ways than just queue delays. The downside that comes with it is using your primary DB as your queue is that the queues in an app tend to absorb all the exceptional conditions (queue grows to millions during worker outage, backfill which needs to process millions of items). Sidekiq is the less sturdy approach of both. There is no way to re-enqueue those "generateReport" jobs using the data at that point in time and so on.īasically: Redis is quite certainly not the best tool to store your job-queues with associated data, in. And with Rails' common set-up, this means you are irrecoverably loosing data: there simply is no way to find what emails you did not send, nor any way to re-queue them. Jobs, with their parameters, are dropped, often irrecoverably. If your redis starts failing (been there, it hurts), e.g. Commonly, people store stuff in redis that can be re-built from database or some eventlog or so.īut not sidekiq. Mostly by design: redis is not meant to be the primary, one and only store of truth. Leading to (been there, it hurt) potentially missing jobs or queuing jobs that have no associated mutation in the db.įurthermore, a big downside of redis is that it's promise of consistency is low. Jobs in your queue in the database can be committed in the same transaction that makes the mutation. I agree with most of the findings in that article, but I miss one very important one: consistency.
0 Comments
Leave a Reply.AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |