In section "Lag-Aware Producer" it is mentioned that there is lag data for partitions of a topic. As the processing and thereby lag is a consumer side property i.e. different consumer groups can have different lag in partitions, how is this lag mapped to producer side lag?
It seems the Lag-Aware Producer is used for events without a key. For events with keys, this strategy may not be suitable due to the risk of out-of-order processing.
The Lag-Aware Consumer, however, raises some concerns. According to the concept, "if a particular consumer instance detects that it has fallen significantly behind in processing messages (i.e., it has high lag), it can voluntarily unsubscribe from the topic."
Typically, when lag increases, it’s often not due to slower hardware but rather an insufficient number of consumers in the group to handle the workload in parallel. If a lagging consumer unsubscribes in such cases, it could make the problem worse by placing even more load on the remaining consumers.
Thanks for the very informative post.
In section "Lag-Aware Producer" it is mentioned that there is lag data for partitions of a topic. As the processing and thereby lag is a consumer side property i.e. different consumer groups can have different lag in partitions, how is this lag mapped to producer side lag?
Thanks for the post!
It seems the Lag-Aware Producer is used for events without a key. For events with keys, this strategy may not be suitable due to the risk of out-of-order processing.
The Lag-Aware Consumer, however, raises some concerns. According to the concept, "if a particular consumer instance detects that it has fallen significantly behind in processing messages (i.e., it has high lag), it can voluntarily unsubscribe from the topic."
Typically, when lag increases, it’s often not due to slower hardware but rather an insufficient number of consumers in the group to handle the workload in parallel. If a lagging consumer unsubscribes in such cases, it could make the problem worse by placing even more load on the remaining consumers.
I was thinking the same thing. Their solution is probably suited to non-order demanding messages
Informative blog.
https://www.febspot.com/video/2533989