This week’s system design refresher:

• How does HTTPS work? (Youtube video)

• Types of databases and use cases

• ElasticSearch

• REST API Design

Ilum the Spark cluster manager and monitoring tool (Sponsored)

With Ilum's solution, everyone can now quickly and easily deploy Apache Spark on any Kubernetes cluster. Our software eliminates the need for tedious configuration and reduces the time needed for deployment from days to minutes.

By leveraging the power of container orchestration and Apache Spark's scalability and reliability, we are making it easier than ever to stay ahead of the curve and explore the future of Big Data.

Ilum provides an all-in-one solution for:

• Apache Spark Cluster management and monitoring

• Managed Spark service

• Hadoop replacement

• Apache Livy alternative

• Spark Session control over REST API

• Real-time interaction with Spark jobs

And the best part? It's free! Unlock the power of Big Data today with Ilum.

Learn More about Ilum!

How does HTTPS work?

How do you decide which type of database to use?

There are hundreds or even thousands of databases available today, such as Oracle, MySQL, MariaDB, SQLite, PostgreSQL, Redis, ClickHouse, MongoDB, S3, Ceph, etc. How do you select the architecture for your system? My short summary is as follows:

• Relational database. Almost anything could be solved by them. 

• In-memory store. Their speed and limited data size make them ideal for fast operations.

• Time-series database. Store and manage time-stamped data.

• Graph database. It is suitable for complex relationships between unstructured objects.

• Document store. They are good for large immutable data.

• Wide column store. They are usually used for big data, analytics, reporting, etc., which needs denormalized data.

How do we learn ElasticSearch?

Based on the Lucene library, Elasticsearch provides search capabilities. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. The diagram below shows the outline.

Features of ElasticSearch:

• Real-time full-text search

• Analytics engine

• Distributed Lucene

ElasticSearch use cases:

• Product search on an eCommerce website

• Log analysis

• Auto completer, spell checker

• Business intelligence analysis

• Full-text search on Wikipedia

• Full-text search on StackOverflow

The core of ElasticSearch lies in the data structure and indexing. It is important to understand how ES builds the term dictionary using LSM Tree (Log-Strucutured Merge Tree).

👉 Over to you: Have you used ElasticSearch in your project, and what is it for?

How does REST API work?

What are its principles, methods, constraints, and best practices? I hope the diagram below gives you a quick overview.

Blog post by Love Sharma.

Featured job openings

Openedges: Chief Architect (San Jose, Austin, Remote)

Heir: Senior Software Engineer, Full Stack (United States)