This week’s system design interview: Linux Crash Course - Understanding File Permissions (Youtube video) 9 Essential Components of a Production Microservice Application Iterative, Agile, Waterfall, Spiral Model, RAD Model... What are the differences?
In my opinion this architecture is also valid for most of the software these days. Not just microservices, but also web applications, distributed monolith and so on. Think Spotify, Netflix, Your bank web application and pretty much everything.
I believe also it deserves some extra discussion about the logs and metric collection.
- Pushing logs to Logstash (which seems to be suggested by the direction of the arrows) was the recommended way until a combination of Kubernetes cluster monitoring and Elastic Agent changed the paradigm for good few years ago. Logs are now written by the application running on K8s to local files on the k8s nodes and can be easily collected by Elastic Agents running on each K8s node and pushed directly to Elasticsearch. Logstash has almost become obsolete, except for some very specific use cases. Doing log aggregation this way has tremendous benefits for the application since it doesn't need to deal with pushing logs directly to Logstash, retries or other Logstash failures.
- Similarly to the point above. Applications exposes Prometheus-format metrics at a HTTP endpoints, Prometheus collects those metrics (aka it pulls from that endpoint) and pushes them to its storage.
- Actually Prometheus can be taken out of the picture as well as Logstash, since Elastic Agent can collect Prometheus-format metrics directly from the applications and push them to Elasticsearch.
Why should you trust me on what I said above?
I have worked for 2 years at Elastic in the cloud-native monitoring team and I have seen countless customers implement that exact pattern.
I'm still at Elastic but in a different department.
This week article on my newsletter Cloud Native Engineer will discuss in detail logs collection in Kubernetes with Elastic Agent.
How to design these diagrams
Great diagram about the Microservices application architecture at https://blog.bytebytego.com/i/146792961/essential-components-of-a-production-microservice-application.
In my opinion this architecture is also valid for most of the software these days. Not just microservices, but also web applications, distributed monolith and so on. Think Spotify, Netflix, Your bank web application and pretty much everything.
I believe also it deserves some extra discussion about the logs and metric collection.
- Pushing logs to Logstash (which seems to be suggested by the direction of the arrows) was the recommended way until a combination of Kubernetes cluster monitoring and Elastic Agent changed the paradigm for good few years ago. Logs are now written by the application running on K8s to local files on the k8s nodes and can be easily collected by Elastic Agents running on each K8s node and pushed directly to Elasticsearch. Logstash has almost become obsolete, except for some very specific use cases. Doing log aggregation this way has tremendous benefits for the application since it doesn't need to deal with pushing logs directly to Logstash, retries or other Logstash failures.
- Similarly to the point above. Applications exposes Prometheus-format metrics at a HTTP endpoints, Prometheus collects those metrics (aka it pulls from that endpoint) and pushes them to its storage.
- Actually Prometheus can be taken out of the picture as well as Logstash, since Elastic Agent can collect Prometheus-format metrics directly from the applications and push them to Elasticsearch.
Why should you trust me on what I said above?
I have worked for 2 years at Elastic in the cloud-native monitoring team and I have seen countless customers implement that exact pattern.
I'm still at Elastic but in a different department.
This week article on my newsletter Cloud Native Engineer will discuss in detail logs collection in Kubernetes with Elastic Agent.
I've just published an article that expand on logging in Kubernetes athttps://cloudnativeengineer.substack.com/p/log-collection-in-kubernetes