EP71: CI/CD Pipeline Explained in Simple Terms
This week’s system design refresher:
Why Google and Meta Put Billion Lines of Code In 1 Repository? (Youtube video)
CI/CD Pipeline Explained in Simple Terms
What does API gateway do?
Docker vs. Kubernetes. Which one should we use?
Cloud Native Anti Patterns
QA Wolf gets you to 80% automated test coverage in 4 months (Sponsored)
Manually end-to-end testing? Here's why you should switch to automation:
Test all your user flows in 3 minutes
Freedom to run your test suite whenever you want
Increased confidence in releases
Time to focus on other priorities
The list goes on...
So why wouldn't you? Time and resources. In-house teams typically take 2 years to reach high coverage. And you need at least a few automation engineers to build, run, and maintain a test suite.
Not anymore.
QA Wolf offers a cost-effective approach to getting 80% test coverage in just 4 months — even for the most complex web apps.
And, they include unlimited parallel runs on their testing infrastructure + 24-hour maintenance and triage. Schedule a demo to learn more.
PS: QA Wolf has a 4.8/5 ⭐️ rating on G2 - reviewed by companies with 51-1000+ employees.
Why Google and Meta Put Billion Lines of Code In 1 Repository?
CI/CD Pipeline Explained in Simple Terms
Section 1 - SDLC with CI/CD
The software development life cycle (SDLC) consists of several key stages: development, testing, deployment, and maintenance. CI/CD automates and integrates these stages to enable faster, more reliable releases.
When code is pushed to a git repository, it triggers an automated build and test process. End-to-end (e2e) test cases are run to validate the code. If tests pass, the code can be automatically deployed to staging/production. If issues are found, the code is sent back to development for bug fixing. This automation provides fast feedback to developers and reduces risk of bugs in production.
Section 2 - Difference between CI and CD
Continuous Integration (CI) automates the build, test, and merge process. It runs tests whenever code is committed to detect integration issues early. This encourages frequent code commits and rapid feedback.
Continuous Delivery (CD) automates release processes like infrastructure changes and deployment. It ensures software can be released reliably at any time through automated workflows. CD may also automate the manual testing and approval steps required before production deployment.
Section 3 - CI/CD Pipeline
A typical CI/CD pipeline has several connected stages:
- Developer commits code changes to source control
- CI server detects changes and triggers build
- Code is compiled, tested (unit, integration tests)
- Test results reported to developer
- On success, artifacts are deployed to staging environments
- Further testing may be done on staging before release
- CD system deploys approved changes to production
Latest articles
If you’re not a subscriber, here’s what you missed this month.
"I Was Under Leveled!" — Avoiding the Tragedy of Making Only $500k a Year
Network Protocols behind Server Push, Online Gaming, and Emails
To receive all the full articles and support ByteByteGo, consider subscribing:
What does API gateway do?
The diagram below shows the detail.
Step 1 - The client sends an HTTP request to the API gateway.
Step 2 - The API gateway parses and validates the attributes in the HTTP request.
Step 3 - The API gateway performs allow-list/deny-list checks.
Step 4 - The API gateway talks to an identity provider for authentication and authorization.
Step 5 - The rate limiting rules are applied to the request. If it is over the limit, the request is rejected.
Steps 6 and 7 - Now that the request has passed basic checks, the API gateway finds the relevant service to route to by path matching.
Step 8 - The API gateway transforms the request into the appropriate protocol and sends it to backend microservices.
Steps 9-12: The API gateway can handle errors properly, and deals with faults if the error takes a longer time to recover (circuit break). It can also leverage ELK (Elastic-Logstash-Kibana) stack for logging and monitoring. We sometimes cache data in the API gateway.
Over to you:
What’s the difference between a load balancer and an API gateway?
Do we need to use different API gateways for PC, mobile and browser separately?
Docker vs. Kubernetes. Which one should we use?
What is Docker ?
Docker is an open-source platform that allows you to package, distribute, and run applications in isolated containers. It focuses on containerization, providing lightweight environments that encapsulate applications and their dependencies.
What is Kubernetes ?
Kubernetes, often referred to as K8s, is an open-source container orchestration platform. It provides a framework for automating the deployment, scaling, and management of containerized applications across a cluster of nodes.
How are both different from each other ?
Docker: Docker operates at the individual container level on a single operating system host.
You must manually manage each host and setting up networks, security policies, and storage for multiple related containers can be complex.
Kubernetes: Kubernetes operates at the cluster level. It manages multiple containerized applications across multiple hosts, providing automation for tasks like load balancing, scaling, and ensuring the desired state of applications.
In short, Docker focuses on containerization and running containers on individual hosts, while Kubernetes specializes in managing and orchestrating containers at scale across a cluster of hosts.
Over to you: What challenges prompted you to switch from Docker to Kubernetes for managing containerized applications?
Guest post by Govardhana Miriyala Kannaiah.
Cloud Native Anti Patterns
By being aware of these anti-patterns and following cloud-native best practices, you can design, build, and operate more robust, scalable, and cost-efficient cloud-native applications.
Monolithic Architecture:
One large, tightly coupled application running on the cloud, hindering scalability and agilityIgnoring Cost Optimization:
Cloud services can be expensive, and not optimizing costs can result in budget overrunsMutable Infrastructure:
- Infrastructure components are to be treated as disposable and are never modified in place
- Failing to embrace this approach can lead to configuration drift, increased maintenance, and decreased reliabilityInefficient DB Access Patterns:
Use of overly complex queries or lacking database indexing, can lead to performance degradation and database bottlenecksLarge Containers or Bloated Images:
Creating large containers or using bloated images can increase deployment times, consume more resources, and slow down application scalingIgnoring CI/CD Pipelines:
Deployments become manual and error-prone, impeding the speed and frequency of software releasesShared Resources Dependency:
Applications relying on shared resources like databases can create contention and bottlenecks, affecting overall performanceUsing Too Many Cloud Services Without a Strategy:
While cloud providers offer a vast array of services, using too many of them without a clear strategy can create complexity and make it harder to manage the application.Stateful Components:
Relying on persistent state in applications can introduce complexity, hinder scalability, and limit fault tolerance
Over to you:
What anti-patterns have you faced in your cloud-native journey? How did you conquer them?