EP146: The Open Source AI Stack
This week’s system design refresher:
API Vs SDK! What's the difference? (Youtube video)
The Open Source AI Stack
Cookies Vs Sessions Vs JWT Vs PASETO
Algorithms you should know before taking System Design Interviews
Top 6 Load Balancing Algorithms
How TikTok Manages a 200K File Frontend MonoRepo?
SPONSOR US
API Vs SDK! What's the difference?
The Open Source AI Stack
You don’t need to spend a fortune to build an AI application. The best AI developer tools are open-source, and an excellent ecosystem is evolving that can make AI accessible to everyone.
The key components of this open-source AI stack are as follows:
Frontend
To build beautiful AI UIs, frameworks like NextJS and Streamlit are extremely useful. Also, Vercel can help with deployment.Embeddings and RAG libraries
Embedding models and RAG libraries like Nomic, JinaAI, Cognito, and LLMAware help developers build accurate search and RAG features.Backend and Model Access
For backend development, developers can rely on frameworks like FastAPI, Langchain, and Netflix Metaflow. Options like Ollama and Huggingface are available for model access.Data and Retrieval
For data storage and retrieval, several options like Postgres, Milvus, Weaviate, PGVector, and FAISS are available.Large-Language Models
Based on performance benchmarks, open-source models like Llama, Mistral, Qwen, Phi, and Gemma are great alternatives to proprietary LLMs like GPT and Claude.
Over to you: Which other tool will you add to the Open Source AI Stack?
Cookies Vs Sessions Vs JWT Vs PASETO
Authentication ensures that only authorized users gain access to an application’s resources. It answers the question of the user’s identity i.e. “Who are you?”
The modern authentication landscape has multiple approaches: Cookies, Sessions, JWTs, and PASETO. Here’s what they mean:
Cookies and Sessions
Cookies and sessions are authentication mechanisms where session data is stored on the server and referenced via a client-side cookie.
Sessions are ideal for applications requiring strict server-side control over user data. On the downside, sessions may face scalability challenges in distributed systems.JWT
JSON Web Token (JWT) is a stateless, self-contained authentication method that stores all user data within the token.
JWTs are highly scalable but require careful handling to mitigate the chances of token theft and manage token expiration.PASETO
Platform-Agnostic Security Tokens or PASETO improve upon JWT by enforcing stronger cryptographic defaults and eliminating algorithmic vulnerabilities.
PASETO simplifies token implementation by avoiding the risks associated with misconfiguration.
Over to you: Which authentication approach have you used?
Algorithms you should know before taking System Design Interviews
What are some of the algorithms you should know before taking system design interviews?
I put together a list and explained why they are important. Those algorithms are not only useful for interviews but good to understand for any software engineer.
One thing to keep in mind is that understanding “how those algorithms are used in real-world systems” is generally more important than the implementation details in a system design interview.
What do the stars mean in the diagram?
It’s very difficult to rank algorithms by importance objectively. I’m open to suggestions and making adjustments.
Five-star: Very important. Try to understand how it works and why.
Three-star: Important to some extent. You may not need to know the implementation details.
One-star: Advanced. Good to know for senior candidates.
Over to you: did I miss anything important on the list? Which ones do you know and which you don’t?
Top 6 Load Balancing Algorithms
Static Algorithms
1. Round robin
The client requests are sent to different service instances in sequential order. The services are usually required to be stateless.
2. Sticky round-robin
This is an improvement of the round-robin algorithm. If Alice’s first request goes to service A, the following requests go to service A as well.
3. Weighted round-robin
The admin can specify the weight for each service. The ones with a higher weight handle more requests than others.
4. Hash
This algorithm applies a hash function on the incoming requests’ IP or URL. The requests are routed to relevant instances based on the hash function result.Dynamic Algorithms
5. Least connections
A new request is sent to the service instance with the least concurrent connections.
6. Least response time
A new request is sent to the service instance with the fastest response time.
How TikTok Manages a 200K File Frontend MonoRepo?
A MonoRepo, short for a monolithic repository, is a software development strategy where a single repository contains multiple projects, libraries, and services.
The good parts of a MonoRepo are:
Better code sharing
Simplified dependency management
A unified view of the code base
However, the bigger the MonoRepo gets, the slower the various Git operations.
TikTok faced a similar change with its frontend TypeScript MonoRepo with 200K files.
To deal with this, TikTok built a tool named Sparo that optimizes the performance of Git operations for large frontend MonoRepos.
Sparo dramatically improved the performance of Git operations. Some stats are as follows
Git clone time went from 40 mins to just 2 mins.
Checkout went from 1.5 minutes to 30 seconds.
Status went from 7 seconds to 1 second.
Git commit time went from 15 seconds to 11 seconds.
Over to you: Have you used MonoRepos in your projects?
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com.
Thank you