Very large projects

Repositories the size of a game engine.

Tens of gigabytes of textures, meshes, audio and generated entities next to a few million lines of code, touched by dozens of agents at once. This page explains how agit's storage and transfer are designed for that workload, which parts are built in 0.1.0, which are on the roadmap, and what to do today while they land.

built ships in 0.1.0 in progress on main planned designed in the RFCs, not implemented

Two kinds of large

An 80 GB game repository is large in two independent ways, and they need different answers.

Large in objects

Hundreds of thousands of files, deep directory trees, tens of thousands of commits, many branches. The cost drivers are status, add, commit, listing and history walks. The answer is an object model where the cost of an operation is proportional to the change, not to the repository.

Large in bytes

Individual files of hundreds of megabytes or several gigabytes: baked lightmaps, cinematics, packed asset bundles, ML checkpoints. The cost drivers are hashing, compression, transfer and disk. The answer is content addressing with dedup, cheap hashing, segment storage and, ahead, chunking and tiered blob storage.

Storage model

agit is content-addressed like git, with a different engine underneath.

  • BLAKE3 ids built. An object id is the BLAKE3-256 hash of a one-byte kind tag plus the payload. BLAKE3 is several times faster than SHA-1 and SHA-256 on one core and agit switches to multi-threaded hashing above 128 KiB, so a 4 GB file hashes at memory bandwidth on a laptop. Hashing stops being the bottleneck.
  • One copy per content built. A blob referenced from five hundred commits is stored once. Renaming or moving a 2 GB asset stores a new tree entry and nothing else. Identical files in different directories share one blob.
  • Trees with sizes built. Each tree entry carries mode, id, size and name in a binary AGT1 record. status, ls-tree, the web UI and the context endpoint never open a blob to learn how big it is.
  • Append-only pack segments built. Objects are zstd-compressed and appended to a segment owned by the writing process; a segment is sealed at 1,024 objects or 256 MiB and a checksummed index (AGX1) is published atomically on flush. Writing a million objects does not create a million files, and nothing needs a repack to stay fast.
  • Verify on read built. Every object is re-hashed when loaded. A flipped bit on a 50 GB volume is reported as hash_mismatch, never served as content. agit fsck checks a repository end to end.
  • Index with stat cache and tree cache built. The AGI2 index remembers mtime, size and inode per path and the id of every directory tree. A commit that changes one file re-encodes the three directories on its path and reuses every other cached tree id. write_tree on a 2,000-file tree costs 0.3 ms when nothing changed.
  • Delta compression planned. Segments compress each object on its own today. Content-defined chunking for large blobs (below) is the planned path to small deltas for small edits of big binaries; git-style delta chains between versions of text files are a later optimisation.
Storage path: working tree to index to pack segments, with planned blob tiers on the server client Working tree 500k files · 80 GB .agitignore honored Index (AGI2) stat cache per path cached tree ids Pack segments zstd · BLAKE3 ids 256 MiB roll · AGX1 index lstat changed transfer want / have negotiation POST objects/missing → only ids the other side lacks → object stream (kind | len | payload) server agit-server same segment format CAS refs · events · PRs Hot tier (NVMe) recent objects · planned Blob tier (S3) all objects · dedup · planned built: solid border · planned: dashed border

Transfer: only the delta travels

Push and fetch are two-step. The client sends the ids reachable from what it wants to push; the server answers with the subset it is missing; only those objects are streamed. Fetch works the same way with wants and haves. For an 80 GB tree where one texture and one script changed, a push moves one blob, the trees on its path, one commit and a compare-and-swap ref update. built

Objects on the wire use the same record shape as pack segments, so the server appends what it receives without re-encoding. built

What is still ahead for transfer:

  • Resumable uploads and idempotency keys planned: a push retried after a dropped connection continues where it stopped and applies exactly once. Roadmap acceptance test: a 1 GiB clone resumes without re-transferring completed segments.
  • Partial clone and on-demand blobs planned: clone the commit graph and trees, fetch blobs when a path is checked out. An artist's agent working on assets/characters/ never downloads assets/cinematics/.
  • Streaming bodies in progress: today a fetch response is read into memory on the client before it is unpacked, which bounds a single transfer by available RAM (see the limits below).

Large files

Content addressing handles large files correctly today; the planned work makes them cheap.

ConcernToday (0.1.0)Planned
Hashing a 4 GB fileBLAKE3, multi-threaded; limited by disk read speed builtunchanged
Storing it twice in historystored once by content builtunchanged
Editing 1 MB inside ita second full copy, zstd-compressed builtcontent-defined chunking: only the changed chunks are new planned
Maximum object size4 GiB per object (32-bit length fields in segment records and wire frames)chunked objects remove the cap planned
Memory while adding or pushingthe whole object is held in memory while hashing, compressing and streamingstreaming hash and compress; chunked transfer planned
Where blobs live on the serverlocal pack segments under the data directory builthot NVMe tier plus S3-compatible blob tier with cross-repository dedup planned
Binary merge conflictsdetected (binary conflict kind), ours kept, reported in JSON builtlease-first workflow makes them rare (below)

Workflows that keep a large team out of trouble

Most pain in big repositories is not storage, it is coordination. These are built today.

  • Leases on asset directories built. Binary files do not merge. An agent (or a human through an agent) claims assets/levels/forest/** for an hour; every other commit into it is refused with lease_conflict before any work is wasted. Leases are advisory, time-limited and visible on the lease board.
  • Workspaces per agent built. Twenty agents, twenty working directories, one object store on the build machine. No duplicated 80 GB checkouts per agent.
  • Checkpoints instead of WIP commits built. Snapshot the whole tree before a destructive build step and restore it in one command, without polluting history with half-baked states.
  • Submodules to split the monorepo built. Engine, tools and content as separate repositories pinned by commit through .agitmodules. Agents working on tools never hash the content tree.
  • Policies on protected branches built. max_files_per_commit keeps an agent from committing a 40,000-file regeneration by accident; required intent and reviews keep main explainable.
  • Server-side compaction planned. Runs of micro-commits are collapsed into human-sized commits with provenance kept in meta, so a year of agent activity does not become a million-commit history walk.

Honest limits in 0.1.0

Read this before pointing agit at 80 GB today.

The design above is the target. The 0.1.0 implementation has the following concrete limits; each is tracked in the roadmap, and none of them changes the on-disk object model.

LimitWhyWhat to do today
Objects up to 4 GiB eachsegment records and wire frames use 32-bit lengthssplit archives above 4 GiB; keep cinematics as per-shot files
Whole object in memory on add, push, fetchblobs are read, hashed, compressed and streamed as one buffer; a fetch response is buffered before unpackingsize the agent's machine for the largest asset plus the largest expected fetch; push often so each transfer stays small
No chunking or delta compressioneach object version is a full zstd copyfine for assets that change rarely; for frequently edited large binaries expect history to grow by the file size per change until chunking lands
No partial clone or sparse checkouta clone fetches every object reachable from the requested branchessplit by submodule; give agents clones of the repositories they work in, not the whole engine
Single server nodeeverything is files under one data directory; replication, sharding and blob tiers are Phase 2fast NVMe for the data directory, nightly backups of the directory, a reverse proxy with no body-size limit
status cost grows with file countwarm stat cache measured at about 4 µs per file (8 ms for 2,000 files); the million-file target (< 500 ms) needs the parallel walk and a persisted walk cache from Phase 1ignore build output and caches in .agitignore; agents call status --json on the workspace they own
No LFS-style pointer filesagit's answer to LFS is tiers and partial clone, not pointerskeep truly enormous raw sources (uncompressed captures) outside version control until blob tiers land

Roadmap items that matter for scale

PhaseDeliverableTarget (not yet measured)
1 · Performanceparallel status and add with a work-stealing walk; journal with group commit; append-only pack index; agit squashstatus on 1,000,000 files, warm cache, < 500 ms; add of 100,000 files totalling 1 GiB within 1.2× of git
2 · Protocol and serverstreaming transfer, resumable uploads, idempotency keys, local pack tier plus S3-compatible blob tier with cross-repository dedup, sharding by repository, Raft logclone of a 1 GiB repository over 1 Gbit/s at > 80% of link bandwidth; 10,000 to 20,000 commits/s per hot repository
3 · Platformserver-side compaction of micro-commit runs; git bridge round trip on large projectsgit import of a 100k-commit repository within 2× of git clone wall time
5 · 1.0format freeze with a compatibility policy; the full benchmark suite re-run and publisheda 1.0 repository opens unchanged in every 1.x release

Measured numbers live only in docs/BENCHMARKS.md, with hardware and commands. Everything in the table above is a target until it appears there.

Sizing a server for a large project

disk
NVMe for AGIT_DATA_DIR; plan for the deduplicated size of all repositories plus history growth. Pack segments are append-only, so sequential write throughput matters more than IOPS.
memory
the server holds objects in memory while receiving and serving them; give it headroom for the largest objects times concurrent transfers.
durability
AGIT_FSYNC=refs (default) fsyncs ref updates and the journal; all also fsyncs every object write at a throughput cost; none only for throwaway servers.
reverse proxy
no request body limit (client_max_body_size 0 in nginx), buffering off and long read timeouts for the SSE event stream, TLS terminated there.
backups
the data directory is the only state; snapshot it nightly and test a restore. Encrypt backups: webhook secrets and notes are inside.
network
keep agents close to the server; a push of large assets is one HTTP request per batch, so latency matters less than bandwidth.

Try it on a real asset tree

agit git import /path/to/engine --into engine --json     # full history, trailers become intent/task
cd engine
agit fsck --json                                        # every object re-hashed
agit status --json                                      # sizes from the tree, no blob reads
agit lease claim 'assets/levels/**' --ttl 2h --note "relight forest"
agit remote add origin https://agit.example.com/api/v1/repos/engine
agit push --create                                      # only objects the server lacks

Then watch agit events --follow on the server side while a second agent tries to commit into assets/levels/ and is refused in milliseconds.

Full installation and server guide → All features →