How TikTok Manages A 200K File MonoRepo with Sparo
Learn to monitor Azure OpenAI (Sponsored)
Learn how you can track Azure OpenAI usage and performance, from monitoring your API requests and token usage, to tracking performance in aggregate or by language model.
Disclaimer: The details in this post have been derived from the TikTok Developer Blog. All credit for the technical details goes to the TikTok engineering team. The links to the original articles are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them.
TikTok, the popular short-form video-sharing platform, has a large and rapidly growing codebase for its web front end. This front-end app is built using TypeScript and is organized as a monorepo - a single Git repository containing code for multiple projects and libraries.
As TikTok's frontend engineering team and feature set grew, so did the size and complexity of this monorepo.
Over time, it expanded to contain over 1,000 separate projects and more than 200,000 source files. While the monorepo architecture provided benefits like shared code and tooling, it also started to cause significant performance issues.
Developers began to notice slowness with common Git operations that are essential to their everyday workflows. These slowdowns were a major drain on productivity, wasting valuable engineering time and ruining the development experience.
The issues stemmed from the sheer size of the codebase that Git had to process for each operation. To address this problem, TikTok's frontend infrastructure team began exploring solutions. They experimented with Git's built-in performance features like partial clones, shallow clones, and Git LFS, but found that they were insufficient for the monorepo's scale and growth rate.
Ultimately, the team developed an in-house tool called Sparo to tackle the monorepo performance challenges. They also eventually made Sparo open-source.
In this article, we'll explore how Sparo works and the benefits it has provided for TikTok's front-end engineering team.
The Problem of Git Slowness in Large Monorepos
A monorepo, short for a monolithic repository, is a software development strategy where a single repository contains multiple projects, libraries, and services, often maintained by different teams. This contrasts with the multi-repo approach, where each project has its separate repository.
The diagram below shows the difference between monorepos, multi-repos, and monolith code bases.
Monorepos have gained popularity among large tech companies like Google, Facebook, and Microsoft for several reasons.
Firstly, monorepos enable better code sharing and reuse across projects, reducing duplication and promoting standardization.
Secondly, they simplify dependency management, as projects within the monorepo can easily reference and use each other.
Thirdly, monorepos provide a unified view of the entire codebase, making it easier to perform cross-cutting changes and maintain a consistent build and test infrastructure.
However, as monorepos grow in size and complexity, Git operations become increasingly slow for everyone.
This is exactly what happened at TikTok. Commands like git clone to download a copy of the repository, git checkout to switch between branches, git status to see current changes, and git commit to save new changes took much longer than they had previously.
Here are some interesting stats from their observations:
Cloning the repository to start working on it could take over 40 minutes for developers with slower network connections. Even on fast connections, a full clone took over 20 minutes.
Checking out a different branch took over a minute and a half.
Just running git status to see the current state of the working copy took 7 seconds, interrupting the flow of developers.
Committing code changes was also painful at around 15 seconds per commit.
How Sparo Improves MonoRepo Performance?
At its core, Sparo leverages two advanced features of Git to dramatically speed up common operations on large monorepos: sparse checkout and partial clone.
Let’s look at both in detail.
Sparse Checkout
Git's sparse checkout feature allows specifying a subset of files or directories to check out from a repository, rather than fetching the entire codebase. Sparo uses this to only check out the files needed to build a given application - namely, the project and its dependencies.
In a monorepo with hundreds or thousands of projects, the files needed for any one project are a relatively small and slow-growing subset compared to the repo as a whole.
By using sparse checkout to limit the working copy to just this subset, Sparo significantly reduces the amount of data that needs to be fetched and the number of files that Git operations need to process. This results in much faster checkout times.
See the diagram below that explains the concept of sparse checkout in Git in which a developer only has to work on the Android Client Development.
Partial Clones
While sparse checkout reduces the number of files in the working copy, a standard Git clone still fetches the contents of every file in the repo and their complete history.
For large repositories, this still means a sizable amount of data transfer and disk usage, even if much of it isn't needed locally.
Git's partial clone feature, enabled by passing the --filter=blob:none option to git clone, optimizes this by only fetching file contents on-demand as they are needed, and excluding objects that aren't reachable by any reference. This reduces the size of the initial clone and speeds up subsequent fetches.
See the diagram below for a visual representation of the same.
Unlike a shallow clone, the full history is still available if needed, just not eagerly downloaded. Also, unlike Git LFS, partial clone works automatically for all files without a separate storage system.
Additional Sparo Enhancements
Sparo adds some additional enhancements beyond leveraging those two key Git features.
Checkout Profiles
Sparo introduces the concept of checkout profiles, which are predefined sets of directories to include in a sparse checkout. Profiles act as an easy starting point for new hires or contributors to discover what part of the codebase is relevant to a given team.
For example, a frontend team might define a profile that checks out their five most actively developed projects, the dependencies of those projects, and a few additional directories like docs and config.
These profiles are defined in a JSON file and checked into the repo, making it easy for developers to share profiles and quickly set up their working copy to match their team's standard environment.
See the example of the profile file:
{
"selections": [
{
"selector": "--to",
"argument": "project-a"
},
{
"selector": "--to",
"argument": "project-b"
},
{
"selector": "folder",
"argument": "docs"
}
]
}This profile would help the developers checkout project-a, project-b, and the docs folder.
With the profile created, developers can use sparo checkout --profile to checkout the repository according to the profile. This checks out only the specified projects and folders, significantly reducing the data fetched and the time taken.
Mirrored Commands
To make adoption frictionless, Sparo provides its command line interface that aims to be a drop-in replacement for the standard Git CLI. In other words, Sparo is fully compatible with Git, so teams can incrementally adopt it while still interoperating with standard Git usage.
By mirroring the Git interface, Sparo minimizes the learning curve and allows it to be gradually adopted into existing workflows. Familiar commands like clone, checkout, status, add, commit, etc. are all provided with the same syntax as their Git equivalents.
Developers can use the Sparo versions of commands to take advantage of the optimized performance, while still falling back to the regular Git commands if needed for advanced cases.
Under the hood, the Sparo versions of these commands automatically set the appropriate configurations to enable partial clone and sparse checkout based on the active profile. They also allow collecting anonymous usage telemetry to power dashboards for monitoring the adoption and performance of the tool.
Sparo Performance Gains
The TikTok team saw dramatic performance improvements after adopting Sparo:
clone time dropped from 23 minutes to just 2 minutes
checkout time went from 1.5 minutes to 30 seconds
status command time reduced from 7 seconds to 1 second
commit time improved from 15 seconds to 11 seconds
These improvements have a huge impact on developer productivity and experience. Instead of waiting minutes for Git operations to complete, developers can iterate and commit changes quickly, maintaining their flow and focus.
Conclusion
As monorepos grow in size and complexity, maintaining good performance for common Git operations becomes increasingly challenging. TikTok's frontend team experienced this firsthand as their TypeScript monorepo surpassed 1,000 projects and 200,000 source files, leading to significant slowdowns in commands like clone, checkout, status, and commit.
To address this problem, the TikTok team developed Sparo, an open-source tool that leverages Git's sparse checkout and partial clone features to speed up Git operations on large monorepos.
The TikTok engineering team's journey with Sparo highlights the importance of proactively addressing performance issues in rapidly growing codebases.
Sparo's roadmap includes a telemetry plugin system for monitoring and support for additional front-end build systems. As more companies encounter performance challenges with huge monorepos, tools like Sparo can become increasingly important for maintaining developer productivity.
References:
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com
Thanks 🙏👍 for writing this 😁🙂🙃😊😁😄😌🤠😁😁