XTX Markets has announced that its in-house filesystem, TernFS, is now available as open-source software, under the GPL-2.0-or-later license for its core filesystem components and Apache-2.0 with LLVM exception for protocol definitions and client libraries
Originally built to handle the company’s massive data needs, TernFS is designed for exabyte-scale storage, supporting trillions of files and millions of clients at once.
And now, these capabilities are publicly available to everyone, as TernFS enters the landscape, joining projects like Ceph, Lustre, and GlusterFS that have long dominated the field. But before we continue, let’s shed some more light on the filesystem itself.
TernFS has been in use at XTX since mid-2023, managing over 500 petabytes of data across three data centers, backed by approximately 30,000 hard disks and 10,000 flash drives. The system is built to scale far beyond that—up to 10 exabytes of logical storage, trillions of files, and millions of concurrent clients.
The design of TernFS emphasizes immutability, durability, and multi-region operation. Files are immutable once written, which simplifies replication and consistency at scale. Metadata is distributed across multiple shards, removing single points of failure, while replication and Reed–Solomon coding provide redundancy.
Key components include:
- Registry for cluster metadata and service tracking
- Metadata shards using RocksDB with consensus via LogsDB
- Block services to store file contents redundantly across nodes
- Cross-Directory Coordinator for operations that span directories
Access is supported through a Linux kernel module, a FUSE client, and a minimal S3 API implementation. The system is optimized for large, immutable files—typically larger than a few megabytes. That means it is not well-suited for small files, frequent updates, or directory operations that require rapid changes.
For more information, see the announcement or visit the newly published GitHub repository.
