Two kinds of large
An 80 GB game repository is large in two independent ways, and they need different answers.
Large in objects
Hundreds of thousands of files, deep directory trees, tens of thousands of commits, many branches. The cost drivers are status, add, commit, listing and history walks. The answer is an object model where the cost of an operation is proportional to the change, not to the repository.
Large in bytes
Individual files of hundreds of megabytes or several gigabytes: baked lightmaps, cinematics, packed asset bundles, ML checkpoints. The cost drivers are hashing, compression, transfer and disk. The answer is content addressing with dedup, cheap hashing, segment storage and, ahead, chunking and tiered blob storage.
Storage model
agit is content-addressed like git, with a different engine underneath.
- BLAKE3 ids built. An object id is the BLAKE3-256 hash of a one-byte kind tag plus the payload. BLAKE3 is several times faster than SHA-1 and SHA-256 on one core and agit switches to multi-threaded hashing above 128 KiB, so a 4 GB file hashes at memory bandwidth on a laptop. Hashing stops being the bottleneck.
- One copy per content built. A blob referenced from five hundred commits is stored once. Renaming or moving a 2 GB asset stores a new tree entry and nothing else. Identical files in different directories share one blob.
- Trees with sizes built. Each tree entry carries mode, id, size and name in a binary
AGT1record.status,ls-tree, the web UI and thecontextendpoint never open a blob to learn how big it is. - Append-only pack segments built. Objects are zstd-compressed and appended to a segment owned by the writing process; a segment is sealed at 1,024 objects or 256 MiB and a checksummed index (
AGX1) is published atomically on flush. Writing a million objects does not create a million files, and nothing needs a repack to stay fast. - Verify on read built. Every object is re-hashed when loaded. A flipped bit on a 50 GB volume is reported as
hash_mismatch, never served as content.agit fsckchecks a repository end to end. - Index with stat cache and tree cache built. The
AGI2index remembers mtime, size and inode per path and the id of every directory tree. A commit that changes one file re-encodes the three directories on its path and reuses every other cached tree id.write_treeon a 2,000-file tree costs 0.3 ms when nothing changed. - Delta compression planned. Segments compress each object on its own today. Content-defined chunking for large blobs (below) is the planned path to small deltas for small edits of big binaries; git-style delta chains between versions of text files are a later optimisation.
Transfer: only the delta travels
Push and fetch are two-step. The client sends the ids reachable from what it wants to push; the server answers with the subset it is missing; only those objects are streamed. Fetch works the same way with wants and haves. For an 80 GB tree where one texture and one script changed, a push moves one blob, the trees on its path, one commit and a compare-and-swap ref update. built
Objects on the wire use the same record shape as pack segments, so the server appends what it receives without re-encoding. built
What is still ahead for transfer:
- Resumable uploads and idempotency keys planned: a push retried after a dropped connection continues where it stopped and applies exactly once. Roadmap acceptance test: a 1 GiB clone resumes without re-transferring completed segments.
- Partial clone and on-demand blobs planned: clone the commit graph and trees, fetch blobs when a path is checked out. An artist's agent working on
assets/characters/never downloadsassets/cinematics/. - Streaming bodies in progress: today a fetch response is read into memory on the client before it is unpacked, which bounds a single transfer by available RAM (see the limits below).
Large files
Content addressing handles large files correctly today; the planned work makes them cheap.
| Concern | Today (0.1.0) | Planned |
|---|---|---|
| Hashing a 4 GB file | BLAKE3, multi-threaded; limited by disk read speed built | unchanged |
| Storing it twice in history | stored once by content built | unchanged |
| Editing 1 MB inside it | a second full copy, zstd-compressed built | content-defined chunking: only the changed chunks are new planned |
| Maximum object size | 4 GiB per object (32-bit length fields in segment records and wire frames) | chunked objects remove the cap planned |
| Memory while adding or pushing | the whole object is held in memory while hashing, compressing and streaming | streaming hash and compress; chunked transfer planned |
| Where blobs live on the server | local pack segments under the data directory built | hot NVMe tier plus S3-compatible blob tier with cross-repository dedup planned |
| Binary merge conflicts | detected (binary conflict kind), ours kept, reported in JSON built | lease-first workflow makes them rare (below) |
Workflows that keep a large team out of trouble
Most pain in big repositories is not storage, it is coordination. These are built today.
- Leases on asset directories built. Binary files do not merge. An agent (or a human through an agent) claims
assets/levels/forest/**for an hour; every other commit into it is refused withlease_conflictbefore any work is wasted. Leases are advisory, time-limited and visible on the lease board. - Workspaces per agent built. Twenty agents, twenty working directories, one object store on the build machine. No duplicated 80 GB checkouts per agent.
- Checkpoints instead of WIP commits built. Snapshot the whole tree before a destructive build step and restore it in one command, without polluting history with half-baked states.
- Submodules to split the monorepo built. Engine, tools and content as separate repositories pinned by commit through
.agitmodules. Agents working on tools never hash the content tree. - Policies on protected branches built.
max_files_per_commitkeeps an agent from committing a 40,000-file regeneration by accident; required intent and reviews keepmainexplainable. - Server-side compaction planned. Runs of micro-commits are collapsed into human-sized commits with provenance kept in
meta, so a year of agent activity does not become a million-commit history walk.
Honest limits in 0.1.0
The design above is the target. The 0.1.0 implementation has the following concrete limits; each is tracked in the roadmap, and none of them changes the on-disk object model.
| Limit | Why | What to do today |
|---|---|---|
| Objects up to 4 GiB each | segment records and wire frames use 32-bit lengths | split archives above 4 GiB; keep cinematics as per-shot files |
Whole object in memory on add, push, fetch | blobs are read, hashed, compressed and streamed as one buffer; a fetch response is buffered before unpacking | size the agent's machine for the largest asset plus the largest expected fetch; push often so each transfer stays small |
| No chunking or delta compression | each object version is a full zstd copy | fine for assets that change rarely; for frequently edited large binaries expect history to grow by the file size per change until chunking lands |
| No partial clone or sparse checkout | a clone fetches every object reachable from the requested branches | split by submodule; give agents clones of the repositories they work in, not the whole engine |
| Single server node | everything is files under one data directory; replication, sharding and blob tiers are Phase 2 | fast NVMe for the data directory, nightly backups of the directory, a reverse proxy with no body-size limit |
status cost grows with file count | warm stat cache measured at about 4 µs per file (8 ms for 2,000 files); the million-file target (< 500 ms) needs the parallel walk and a persisted walk cache from Phase 1 | ignore build output and caches in .agitignore; agents call status --json on the workspace they own |
| No LFS-style pointer files | agit's answer to LFS is tiers and partial clone, not pointers | keep truly enormous raw sources (uncompressed captures) outside version control until blob tiers land |
Roadmap items that matter for scale
| Phase | Deliverable | Target (not yet measured) |
|---|---|---|
| 1 · Performance | parallel status and add with a work-stealing walk; journal with group commit; append-only pack index; agit squash | status on 1,000,000 files, warm cache, < 500 ms; add of 100,000 files totalling 1 GiB within 1.2× of git |
| 2 · Protocol and server | streaming transfer, resumable uploads, idempotency keys, local pack tier plus S3-compatible blob tier with cross-repository dedup, sharding by repository, Raft log | clone of a 1 GiB repository over 1 Gbit/s at > 80% of link bandwidth; 10,000 to 20,000 commits/s per hot repository |
| 3 · Platform | server-side compaction of micro-commit runs; git bridge round trip on large projects | git import of a 100k-commit repository within 2× of git clone wall time |
| 5 · 1.0 | format freeze with a compatibility policy; the full benchmark suite re-run and published | a 1.0 repository opens unchanged in every 1.x release |
Measured numbers live only in docs/BENCHMARKS.md, with hardware and commands. Everything in the table above is a target until it appears there.
Sizing a server for a large project
- disk
- NVMe for
AGIT_DATA_DIR; plan for the deduplicated size of all repositories plus history growth. Pack segments are append-only, so sequential write throughput matters more than IOPS. - memory
- the server holds objects in memory while receiving and serving them; give it headroom for the largest objects times concurrent transfers.
- durability
AGIT_FSYNC=refs(default) fsyncs ref updates and the journal;allalso fsyncs every object write at a throughput cost;noneonly for throwaway servers.- reverse proxy
- no request body limit (
client_max_body_size 0in nginx), buffering off and long read timeouts for the SSE event stream, TLS terminated there. - backups
- the data directory is the only state; snapshot it nightly and test a restore. Encrypt backups: webhook secrets and notes are inside.
- network
- keep agents close to the server; a push of large assets is one HTTP request per batch, so latency matters less than bandwidth.
Try it on a real asset tree
agit git import /path/to/engine --into engine --json # full history, trailers become intent/task cd engine agit fsck --json # every object re-hashed agit status --json # sizes from the tree, no blob reads agit lease claim 'assets/levels/**' --ttl 2h --note "relight forest" agit remote add origin https://agit.example.com/api/v1/repos/engine agit push --create # only objects the server lacks
Then watch agit events --follow on the server side while a second agent tries to commit into assets/levels/ and is refused in milliseconds.