The CME Group Data Marketplace

Why Revelate?

Revelate was an easy choice, being the experts in implementing big data technology for capital markets. “The CME Group has always offered customers a robust set of historical data, and working with Revelate allows us to deliver it in a way that is as easy as shopping online,” says Tim Wheeler, Product Manager at CME Group.

“We don’t need to touch the actual data all that much, we mostly touch everything around the data,” says Nicolas Doyen, Product Manager at Revelate. “Clients can enhance their data sales processes and they can choose specific pieces to offer on the system. One of our big value props is that we enable clients to offer a full range of products, our technology scales infinitely, and we are constantly building new features to improve it. Clients are able to build a platform that fits their needs, exactly as they need it.”

Revelate deployed the enterprise data web store solution that powers CME DataMine for selling data. The platform’s technology is modular, which makes it simple for CME Group to scale and configure solutions for various use cases. The open flexibility in configurations along with the ability to add customized modules, would not only help CME Group organize their data for monetization, but would enable them to develop and redefine new product offerings. Building the infrastructure on-premises was not feasible, as the ability to scale made for a strong case in deploying the platform on cloud technology.

AWS Deployment

CME Group is one of several capital markets to choose AWS as their cloud provider. AWS is outstanding in their ability to be flexible, cost-effective and to meet high security standards and encryption protocols. Amazon has one of the largest ecosystems of security partners and solutions, resulting in the highest measures for privacy and data security. The numbers are convincing, with an impressive statistic showing 0.00001% faults.

TickSmith is an Advanced AWS Partner in Financial Services Competency, with members on staff certified by AWS. As a result, the platform’s cloud infrastructure follows an accredited and secure workflow from ingestion to delivery, with entitlement and monitoring capabilities throughout the process. “The sophistication of their cloud offering, specifically regarding the range of services available and the flexibility of these services, makes AWS easily the best choice,” adds Doyen. “For actual clients, there’s proven success with Amazon regarding security. In terms of capabilities alone in enabling us to accomplish our goals, you couldn’t have a better cloud. They have everything we need.”

"The CME Group has always offered customers a robust set of historical data, and working with Revelate allows us to deliver it in a way that is as easy as shopping online. Since the launch of DataMine, our modern data web store, we have increased our active customer base by more than 50%"

- Tim Wheeler, Product Manager, CME Group

The Results

The CME DataMine web store is the official source of the most comprehensive price information available for CME Group markets, with more than 750 terabytes of historical market data and more than 75,000 distinct data products. It provides a broad array of data types including Market Depth, End-of-Day, and Block Trades among others, which can help customers discover insights to capture market opportunities. CME Group has also integrated third-party datasets that provide data around the instruments traded on their exchange, such as crop health analytics and oil tanker fill estimates. A new use case for data monetization came along the way for CME CF Bitcoin Reference Rate (BRR) and CME CF Bitcoin Real-Time Index (BRTI), a standardized reference rate and spot price index for online bitcoin trading.

One of the main challenges of this project was the general management of big data distribution, which is a huge pain point for any company with massive amounts of data looking to monetize it. This required a revolutionary solution to the traditional SFTP (Secure File Transfer Protocol) model. Traditional SFTP approaches would require the data distributor to set up client-specific SFTP servers and continuously copy data into those servers. The original method was slow and accrued additional storage costs. Revelate addressed this issue by devising a new method, dubbed Virtual SFTP, that removes the need for the creation of client-specific SFTP servers and data duplication.

Another difficulty is finding a solution to deliver specific time ranges of data. The raw datasets, as inputted by third-party providers, include files with long date ranges and multiple parts. Revelate is currently working on an ETL that will break up the historical data file into smaller files- even down to the hour, so customers can access relevant data in pertaining time ranges. This removes the barrier of entry for end-users and enables them to access relevant data at a lower price point.

“It makes it that your big data problem becomes a very, very small data problem for the end consumer. For instance, some financial institutions don’t have big data technology expertise, so dealing with a 40 terabyte historical dataset is hard for them,” says Doyen. “So if they’re able to get smaller slices of data, they can more easily leverage them. They can actually load these files on their computers, which they couldn’t do before.”

CME Group jumped on the big data bandwagon pretty early compared to other financial institutions and has since reaped major benefits. DataMine is essentially an online web store or data catalog, powered by Revelate’s data platform. Customers can subscribe to and access data instantaneously via web interfaces and APIs. “Since the launch of DataMine, our modern data web store, we have increased our active customer base by more than 50%,” adds Wheeler.