EP36: Types of Databases and Use Cases
This week’s system design refresher:
How does HTTPS work? (Youtube video)
Types of databases and use cases
ElasticSearch
REST API Design
Ilum the Spark cluster manager and monitoring tool (Sponsored)
With Ilum's solution, everyone can now quickly and easily deploy Apache Spark on any Kubernetes cluster. Our software eliminates the need for tedious configuration and reduces the time needed for deployment from days to minutes.
By leveraging the power of container orchestration and Apache Spark's scalability and reliability, we are making it easier than ever to stay ahead of the curve and explore the future of Big Data.
Ilum provides an all-in-one solution for:
Apache Spark Cluster management and monitoring
Managed Spark service
Hadoop replacement
Apache Livy alternative
Spark Session control over REST API
Real-time interaction with Spark jobs
And the best part? It's free! Unlock the power of Big Data today with Ilum.
How does HTTPS work?
How do you decide which type of database to use?
There are hundreds or even thousands of databases available today, such as Oracle, MySQL, MariaDB, SQLite, PostgreSQL, Redis, ClickHouse, MongoDB, S3, Ceph, etc. How do you select the architecture for your system? My short summary is as follows:
Relational database. Almost anything could be solved by them.
In-memory store. Their speed and limited data size make them ideal for fast operations.
Time-series database. Store and manage time-stamped data.
Graph database. It is suitable for complex relationships between unstructured objects.
Document store. They are good for large immutable data.
Wide column store. They are usually used for big data, analytics, reporting, etc., which needs denormalized data.
How do we learn ElasticSearch?
Based on the Lucene library, Elasticsearch provides search capabilities. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. The diagram below shows the outline.
Features of ElasticSearch:
Real-time full-text search
Analytics engine
Distributed Lucene
ElasticSearch use cases:
Product search on an eCommerce website
Log analysis
Auto completer, spell checker
Business intelligence analysis
Full-text search on Wikipedia
Full-text search on StackOverflow
The core of ElasticSearch lies in the data structure and indexing. It is important to understand how ES builds the term dictionary using LSM Tree (Log-Strucutured Merge Tree).
👉 Over to you: Have you used ElasticSearch in your project, and what is it for?
How does REST API work?
What are its principles, methods, constraints, and best practices? I hope the diagram below gives you a quick overview.
Blog post by Love Sharma.
Featured job openings
Openedges: Chief Architect (San Jose, Austin, Remote)
Heir: Senior Software Engineer, Full Stack (United States)
Great article as always, I just want to point out a couple of things:
Elasticsearch is so much more than just search. Today it also offers Security and Observability (think Logs, Metrics and Traces). That's why Elastic (the company behind Elasticsearch) was recently re-branded to "The Search Analytics Company". Spoiler, I work at Elastic.
If you (or any of your readers) want to learn more about SQLite and how it can be used to solve your data analysis problems take a look at my article https://cloudnativeengineer.substack.com/p/easy-data-analysis-for-busy-software-engineers.
Alternatively if you want to learn more about Elasticsearch I wrote the following articles:
- https://cloudnativeengineer.substack.com/p/ep-11-elasticsearch-development-environment
- https://cloudnativeengineer.substack.com/p/elasticsearch-ignore-malformed-the-antidote-for-mapping-exceptions-ddfa446c2ce5
- https://cloudnativeengineer.substack.com/p/ep-1-reroute-processor-for-elasticsearch
Thanks again for your insights.
Hey Alex! I love your posts and how you write. In this one specifically, I understand that the idea was to leave something summarized, but as a suggestion for the next one, you could go deeper into non-functional requirements that are important for this decision, as well as the CAP theorem