Data Access: A Quick Guide to Governing Users and Data in the Marketplace

From an administrative standpoint, data access is about who can see which data. But from a data product and marketplace standpoint, data access is all about creating a personalized experience.

The way access is configured for a user (or a role) is the primary driver of their user experience in the marketplace. After all, why should users see what they cannot access? It’s of no use to them and has no consumability. 

Though “super users” like data administrators or company executives may want more sophisticated methods of data access, less sophisticated users are likely to want something that plugs straight into PowerBI. Some marketplace users may not want to do anything technical; just give them a file to download, and they’re happy.

On the back end, access is all about entitlements and permissions. Marketplaces must honor role-based access controls (RBAC) and data-level controls. But on the front end, these controls should inform their web- or app-based experience with the marketplace.

Users Expect Personalized Experiences

The world’s leading retail marketplaces are personalized. Users who visit Amazon and eBay see products relevant to their needs or purchase history. Even if the user interface doesn’t change, the presented products do. This is how a marketplace leads an acquirer (i.e., a data consumer in the market) to avoid the 10,000 data products that won’t matter.

Data marketplaces should be adaptable and personalized. Data buyers are no different from detergent buyers on Amazon or retro gaming consoles on eBay. They expect an impersonal experience. The exact expectations apply to internal marketplaces; there’s no reason to short-change the customer experience.

User experiences and data access capabilities should scale with the market. We can extend this product experience expectation to enterprise features like single sign-on. If it’s part of the user experience, it’s data access.

Data access methods

Data marketplace consumers should be able to access their purchased products in various ways. The most prevalent methods are:

• Subscription model: Regular access to a changing dataset or stream in exchange for a recurring fee
• One-time purchase: Access to a static dataset that is unlikely to change (i.e., historical data)
• On-demand: Data that are retrieved at the moment it’s needed and stored in memory or temporary cache, not to be stored permanently
• Services model: Data as a service or layer above the raw data, like analytics, without the raw data itself

Of course, each access method will be subject to the licensing and restrictions the data provider sets. It’s up to the marketplace provider to facilitate (or dictate) who can access data products and how. 

Subscription model

The subscription delivery model typically allows the consumer to take ownership of the data, ingesting it into an internal system and using it however they see fit, subject to the licensing agreement. This approach requires substantial data maturity. The buyer needs to understand how the data is delivered, what transformation processes are necessary in order to make the data usable, how to establish multiple internal delivery pipelines where required, and how to ensure appropriate levels of redundancy, latency, and security.

This requirement for technical sophistication of the buying organization often leads sellers of subscription-based services to question the need for investment in making their data easy to consume. As a result, many subscription services—particularly in the capital markets segment—are delivered in proprietary or semi-proprietary formats, with little or no instruction on how to integrate the data with the buyer’s internal systems. And in many cases, the data is so critical that demand is high, giving little incentive for the seller to change its approach.

Subscription-based streaming services, in particular, are vulnerable to this situation. The emphasis is on the buyer to land the data onsite in real-time without losing updates, which could require expensive backfill and replay capabilities to restore later.

All of this makes it relatively difficult for smaller organizations, which often lack the requisite technical resources, to consume subscription-based services. As a result, subscription-based services—including streaming services—are often targeted at enterprise clients that are more likely to have in place the data engineering resource needed.

One-time Purchase

One-time purchases of datasets, meanwhile, present consumer organizations with the same technical challenges as subscriptions, except that they need to perform the required data transformation only once. These purchases often involve historical datasets with no updates, except corrections, for the buyer to accommodate. But while the one-time purchase model puts less onus on the buyer’s engineering sophistication than continuously delivered subscriptions, the need for consistent processing and pipelining still presents a barrier to entry for consumers.

On-Demand Delivery

On-Demand Delivery, alternatively, is a model where the data provider offers a service through which the data buyer does not store the data in their own systems. On-demand data access requires the buyer to receive the data into memory at the time of processing. If the buyer needs to prepare the data for processing at all, they are required to transform it every time it is used. While data consumers can still join the data with other internal datasets, it is not stored by client systems, which can limit the flexibility in how it is used. 

At high volumes, this setup may introduce cost considerations, too. On-demand processing constantly makes calls to externally hosted datasets, inviting high networking costs, as well as, the cost of additional memory required to process the incoming data. And the more complex the clusters of processors needed to handle the data become, the less ‘on-demand’ the service becomes, restricting its usage and value. As a result, on-demand services are better suited at lower volumes for smaller consuming organizations. Even with modern data sharing capabilities, this approach can incur high networking costs for the data providers.

This switches responsibility for the usability of the data entirely to the data provider, which means organizations seeking to supply datasets on demand must invest in a more robust infrastructure that ensures the data is delivered in a consumable way to relatively smaller and less sophisticated buyers. In short, on-demand delivery needs to be as simple and easy as possible for the consumer to have a solid incentive to remain within the on-demand platform.

Services delivery model

The services model approach is typically used to deliver a data service, such as analytics, rather than the raw underlying data itself. In essence, the data buyer asks a question of the dataset and the provider delivers an answer to that question in the form of a report, aggregated display, or other analytical response. In more sophisticated instances, the consumer can modify the question’s parameters, allowing a query of the host database, perhaps by time frame, instrument type, or some other characteristic.

Providers of services model offerings may enable consumers to access the underlying raw data through a so-called sandbox hosted by the seller. This approach can allow buyers to manipulate the raw data while protecting the IP of the provider. But buyers may be reluctant for competitive reasons to blend their data within the sandbox environment, limiting the facility’s value. In reality, the sandbox facility is most commonly used for testing the usefulness of data sets in a controlled way before buying.

As can be seen from the nuances of the various data access and distribution models, the buyer and seller may take on more or less responsibility for rendering the datasets in helpful questions. This has implications for the original data products’ design and licensing agreements and other commercial considerations, which we’ll consider in our upcoming blog. 

Takeaways

• Data access is all about personalized marketplace user experiences.
• From a technological standpoint, data access is all about entitlements and permissions (e.g., RBAC).
• Just because users buy data doesn’t mean they want a boring data marketplace experience.
• There are four data access methods in a data marketplace: subscription, one-time purchase, on-demand, and services.
• The more data products in a marketplace, the more a user should be exposed to those different products.