Open protocol for decentralized exchange and transformation of data

Website | Reference Implementation | Original Whitepaper | Chat

Open Data Fabric is an open protocol specification for decentralized exchange and transformation of semi-structured data, that aims to holistically address many shortcomings of the modern data management systems and workflows.

The goal of this specification is to develop a method of data exchange that would:

• Enable worldwide collaboration around data cleaning, enrichment, and derivation
• Create an environment of verifiable trust between participants without the need for a central authority
• Enable high degree of data reuse, making quality data more readily available
• Improve liquidity of data by speeding up the data propagation times from publishers to consumers
• Create a feedback loop between data consumers and publishers, allowing them to collaborate on better data availability, recency, and design

ODF protocol is a Web 3.0 technology that powers a distributed structured data supply chain for providing timely, high-quality, and verifiable data for data science, smart contracts, web and applications.

• Original Whitepaper (July 2020)
• Kamu Blog: Introducing Open Data Fabric
• Talk: Open Data Fabric for Research Data Management
• PyData Global 2021 Talk: Time: The most misunderstood dimension in data modelling
• Data+AI Summit 2020 Talk: Building a Distributed Collaborative Data Pipeline

More tutorials and articles can be found in kamu-cli documentation.

The specification is currently in actively evolving and welcomes feedback.

See also our Roadmap for future direction and RFC archive for the record of changes.

Coordinator implementations:

• kamu-cli - data management tool that serves as the reference implementation.

Engine implementations:

• kamu-engine-spark - engine based on Apache Spark.
• kamu-engine-flink - engine based on Apache Flink.

The specification was originally developed by Kamu as part of the kamu-cli data management tool. While developing it, we quickly realized that the very essence of what we're trying to build - a collaborative open data processing pipeline based on verifiable trust - requires full transparency and openness on our part. We strongly believe in the potential of our ideas to bring data management to the next level, to provide better quality data faster to the people who need it to innovate, fight deceases, build better businesses, and make informed political decisions. Therefore, we saw it as our duty to share these ideas with the community and make the system as inclusive as possible for the existing technologies and future innovations, and work together to build momentum needed to achieve such radical change.

See Contribution Guidelines

• RFC-000: RFC Template
• RFC-001: Record Offsets
• RFC-002: Logical Data Hashes
• RFC-003: Content Addressability
• RFC-004: Metadata Extensibility
• RFC-005: New Annotation Metadata Events
• RFC-006: Store Checkpoints as Files
• RFC-007: Simple Transfer Protocol
• RFC-008: Smart Transfer Protocol
• RFC-009: Ingest Source State
• RFC-010: Data Schema in Metadata
• RFC-011: Push Ingest Sources
• RFC-012: Recommend base16 encoding for textual representation of hashes and DIDs
• RFC-013: Enum representation in YAML encoding
• RFC-014: Minimizing scanning for last offset and block
• RFC-015: Unified changelog stream schema