I don't quite understand how the jobs are "dispatched" from Redis to the workers. Wouldn't workers need to poll the Redis cache in-order to figure out the available jobs? Can someone please explain? And also it is mentioned that it is debatable whether cron was the right choice for Slack. What other possible options Slack could have considered?
I don't quite understand how the jobs are "dispatched" from Redis to the workers. Wouldn't workers need to poll the Redis cache in-order to figure out the available jobs? Can someone please explain? And also it is mentioned that it is debatable whether cron was the right choice for Slack. What other possible options Slack could have considered?
I don't quite understand how the jobs are "dispatched" from Redis to the workers. Wouldn't workers need to poll the Redis cache in-order to figure out the available jobs? Can someone please explain? And also it is mentioned that it is debatable whether cron was the right choice for Slack. What other possible options Slack could have considered?
I had the same question in mind, I wonder if they went for pub/sub with Redis (https://redis.io/docs/latest/develop/interact/pubsub/) but yeah a clarification would be nice.
it because historical reason of evolving thir job queue system from Redis to Kafka+Redis
you can check this - https://slack.engineering/scaling-slacks-job-queue/
They might have used something like redisGears to fulfill this
Yes that might be possible. Thanks for the pointer!