Coping With Large and Diverse Quantities of Data


Coping With Large Quantities of Data

We hosted our first breakfast event on Wednesday, February 21st in Montreal with the contribution from Finance Montreal and AWS Amazon. The event’s purpose was to shine light on an issue that many financial institutions face which is: coping with large and diverse sets of data. The room was filled with eager-eyed professionals from banks such as Desjardins, BNP Paribas, Banque Nationale, and many others. The pain that many of these banks face is that they are extremely siloed or in other words, their data is dispersed over multiple departments and it becomes very difficult for them to centralize it in a common format for further analysis.

Another major concern in the room was security. As banks often deal with private data and have strict regulations on who has access to it, it is very important that any software solution or chosen cloud provider be of utmost security. AWS Amazon was able to address many of those concerns with impressive statistics showing 0.00001% faults.


Francis Wenzel, TickSmith CEO, kicked off the show pulling up a slide which featured a photo of the “Great Wave off Kanagawa” and declared “data is a lot like water.” You might be thinking: huh?! But (of course) he was on to something. Raw data is a lot like a water from the ocean, in the sense that when you are thirsty, you cannot drink ocean water even though there is a ton of it! Similarly, many financial institutions have unused data which needs to be processed in order for them to extract insightful analytics to further develop operations and grow revenue.

This immediately grabbed the crowd’s attention. More revenue? Tell us more. Francis continued by highlighting how far we’ve come since floppy discs but stressed that an uphill journey still exists for many financial institutions who are using legacy technology. He said that while data science applications are a hot topic and data visualization tools are easy to experiment with, what powers all of that is a strong data management solution.

That is where our GOLD Platform can help. As we are not data providers and we simply licence out the platform, it is easily integrated on premise or on cloud. In addition, our platform is modular which makes it simple for clients/partners to build and configure solutions for various use cases. One module might help them meet compliance requirements while another module can help them distribute their data and sell it to their customers.

Francis, knowing his audience’s concern about security, also added that the whole platform follows a secure workflow from ingestion to delivery: with entitlement and monitoring capabilities available throughout the process.


Francis passed the mic to Tim Wheeler who is the product manager of DataMine at the CME Group. Tim started off by reaffirming Francis’ statement that many institutions are still using outdated technology. He explained that until 2011, the CME Group, aka the second largest exchange group in the world, used hard disk drives to ship data to their clients.

It seems so comical and yet it still remains a reality for many Capital Markets. Although the CME Group, has always offered customers a robust set of historical data, Tim explained, that working with TickSmith allowed them and continues to allow them to deliver it efficiently and cost-effectively. Datamine is essentially an online “web store” or data catalog, powered by GOLD, which customers can subscribe to and access data instantaneously via web interfaces and APIs.

CME has more than 450 Terabytes of historical market data, some of it going back to the 1970s and CME DataMine is the official source of the most comprehensive price information available for CME Group markets. It provides a broad array of data types including Market Depth, End-of-Day and Block Trades among others, which can help customers discover insights to capture market opportunity.


Antoine Genereux, AWS Amazon Solutions Architect, took the stage to focus on security. The CME Group is one of many Capital Markets who chose AWS as their cloud provider; Antoine explained and showed specific statistics of why that is the case.

AWS is renowned for their ability to remain cost-effective but also meets high security standards and encryption protocols. Antoine broke down the security flow at AWS which consists of six main parts: identity management, detective control, infrastructure security, data protection, and incident response.

Multiple slides upheld the statement that AWS Amazon has one of the largest ecosystems of security partners and solutions. Some of those including: CloudCheckr, Armor, and Alert Logic.


These topics are extremely current. As a testament to that, the event was sold out and additional attendees were standing in the back of the room. During the Q & A session, many of the business professionals wanted to know more about the data lake functionalities and modules while some others had more questions on security.

Having a future-proof data lake platform will soon be the ‘norm’ as it opens up possibilities for firms to accomplish so much more. When we launched our GOLD Platform five years ago, many people did not understand “Big data technology.” It is now a word we hear every day. CME Group jumped on the big data bandwagon pretty early compared to other financial institutions and has since reaped major benefits.

Anything new is scary and of course, security will always be a concern when data leaves your four walls. However, both TickSmith and AWS have extremely reputable clients, such as the CME Group and National Bank of Canada, to advocate for their security standards.

To bring it back to Francis’ provocative statement: data, like water, allows us to survive. Data sustains daily operations and when managed appropriately could help run predictive analytics, backtesting, risk management, etc. The possibilities are nearly endless.