Universal Data Collector (UDC)

*Unified APIs technology is undergoing an architectural shift at the time of writing this. (June 2024)

The UDC is one of a core components of Openmesh. It facilitates the collection, enhancement, and standardization of a variety of data sources, extracting atomic events and packaging them into indexed data sets that can be used for live-streamed and historical data analytics.

The process begins with the raw data collection by Openmesh Core nodes through the collector, which computes the sources it is eligible to listen to each block. When a block is finalized, they connect to the next sources APIs via a websocket and pull data into chunks which it then identifies by an IPFS CID (Content Identifier). Using Boxo, this CID is then available to peers on the network allowing data to spread in a torrent-like fashion.

These processes are designed to be fault tolerant and to handle any unexpected event from the exchange or data source that could cause an outage or drop in data.

Blockchain data collectors connect directly to blockchain nodes, making JSON RPC calls to pull full, granular data to be processed and transformed into human-readable on-chain events. Openmesh has been designed to work with any connection to these nodes – it doesn’t rely on existing APIs. If users wish to run a connector themselves, they can run their own nodes or use any provider without worries for incompatibility.

After the raw data is collected, it is normalized using the standards developed by the CCXT project and if consensus is reached on the validity (so-say-we-all) of that data then it is indexed on the blockchain in Openmesh Ledger, making it available as a resource to the network.

Blockchain data requires special treatment, as there are smart contract events which do not exist in the block header such as those relating to DeFi protocols which will need to be processed and stored under an index relating to the selected data topic (eg. Uniswap). Not all contract event data is necessarily collected, only that which is decided to be a valuable source according to the OpenmeshDAO.

Being a community-focused open source project, the Openmesh Core has been designed to make it as easy as possible for community members to expand Openmesh data coverage, with highly modularised and containerised code that can easily extend to any arbitrary data source. The high-level steps to add an additional data source are:

1. Define the endpoint, data topics and methods in sourcedefinitions.go

2. Define the normalization procedures if they do not fit into existing functions in sources.go

3. Submit a pull request, and initiate the , the data source will be introduced to Openmesh as another data source in the network.