Openmesh API

In active development

Openmesh API

Overview & Position in Openmesh Cloud

Openmesh API serves as the data access layer within the broader Openmesh Cloud ecosystem. While Openmesh Cloud provides decentralized infrastructure for compute and storage resources, the API layer enables standardized access to blockchain and market data through this infrastructure. This integration ensures that data access remains decentralized while maintaining high performance and reliability.

Technical Architecture

The system implements multiple access methods tailored to different use cases. WebSocket services deliver real-time market events with minimal latency, critical for applications requiring immediate data updates. Historical data access operates through scalable object storage served via CDN, optimized for applications requiring extensive historical analysis. For targeted analytics, the system integrates directly with Pythia's visualization capabilities.

Universal Data Collector (UDC)

The Universal Data Collector functions as a cornerstone component within the Openmesh ecosystem. When network nodes validate a new block, the UDC connects to source APIs through WebSocket connections, segmenting incoming data into discrete chunks. Each chunk receives a unique IPFS Content Identifier (CID), enabling peer-to-peer distribution through the Boxo protocol.

Blockchain data processing occurs through direct node connections via JSON RPC calls. This architecture processes raw blockchain data into human-readable events without relying on external APIs. After collection, data undergoes normalization using industry standards, receiving validation through consensus before indexing in the Openmesh Ledger.

Smart contract events undergo specialized processing, particularly for DeFi protocols where event data exists outside block headers. The OpenmeshDAO governs the selection of valuable data sources, determining which contract events merit collection and indexing under specific topics.

Data Flow & Processing

The infrastructure implements a sophisticated multi-stage processing pipeline designed for horizontal scalability and high throughput. Exchange data enters the system as semi-structured JSON, partitioned by exchange-symbol pairs, while blockchain data utilizes the Apache Avro format for efficient schema preservation.

Stream processors operate in parallel, maintaining event order integrity for exchange-specific data while enabling massive scalability. Raw data archives persist in compressed JSON format within object storage, while processed data converts to Apache Parquet for optimal column-based storage efficiency.

The datetime-based partitioning structure organizes events by year, month, day, and hour, facilitating efficient data retrieval. A PostgreSQL database provides advanced querying capabilities, while a scalable broadcaster network enables real-time event distribution.

Connector Infrastructure

Data connectors form the entry point for the Openmesh data pipeline, facilitating real-time ingestion from both on-chain and off-chain sources. The system employs containerized architecture managed through Kubernetes, ensuring stateless operation for rapid scaling and isolated failure domains.

Error handling encompasses sophisticated retry mechanisms and connection monitoring. The modular codebase supports extension to any arbitrary data source, enabling community contribution while maintaining system integrity.

Supported Data Types & Schema

Symbol notation follows the format <base>.<quote>-[PERP], accommodating both spot pairs and perpetual futures markets. Off-chain data coverage encompasses comprehensive exchange data, including order books, trades, market metrics, futures data, and real-time indicators.

On-chain data coverage provides extensive Ethereum blockchain data, including raw blocks, transactions, logs, and token transfers. DeFi protocol coverage includes the top thousand trading pairs by volume across major platforms.

Integration with Cloud & Pythia

The API layer integrates seamlessly with both Openmesh Cloud's decentralized infrastructure and Pythia's analytics capabilities. Cloud integration enables distributed data storage and processing, while Pythia integration provides sophisticated data visualization and analysis tools.

Future developments will expand blockchain coverage and enhance data processing capabilities, leveraging Openmesh Cloud's growing network of decentralized resources.