"LinkedIn prioritized consistency (C) and availability (A) over partition tolerance (P). The decision was driven by their latency and reliability goals."
Please correct me if I misunderstand anything --
I don't think this is an option. As long as it is a distributed system, network partition can't be avoided. In the context of distributed system, CAP basically mean, when partition happens, a system can choose either high consistency or high availability.
Or does LinkedIn not actually using any distributed storage/cache?
In this particular scenario, looks like they have not used distributed storage/cache but went with a different approach but having all the data stored in-memory of the hosts. I know based on other blogs, that they use Distributed storage/cache.
yea, then it seems there is a significant optimization on memory efficiency -- to store all users restriction data within a host (let's say a single high-end machine have 200GB of memory ) ,1B user restriction would require each user's restriction data smaller than 200B.
Yes, they have referenced FastUtil here as Data structures using low memory footprint. Also they could have used BitSet like approach too. Not sure how that will further improve it. We too use FastUtil in my company, it is good alternative.
"LinkedIn prioritized consistency (C) and availability (A) over partition tolerance (P). The decision was driven by their latency and reliability goals."
Please correct me if I misunderstand anything --
I don't think this is an option. As long as it is a distributed system, network partition can't be avoided. In the context of distributed system, CAP basically mean, when partition happens, a system can choose either high consistency or high availability.
Or does LinkedIn not actually using any distributed storage/cache?
In this particular scenario, looks like they have not used distributed storage/cache but went with a different approach but having all the data stored in-memory of the hosts. I know based on other blogs, that they use Distributed storage/cache.
yea, then it seems there is a significant optimization on memory efficiency -- to store all users restriction data within a host (let's say a single high-end machine have 200GB of memory ) ,1B user restriction would require each user's restriction data smaller than 200B.
Yes, they have referenced FastUtil here as Data structures using low memory footprint. Also they could have used BitSet like approach too. Not sure how that will further improve it. We too use FastUtil in my company, it is good alternative.