The Top Data Catalog Tools in 2023

A data catalog is the difference between an ocean of data and an ocean of value. When companies get serious about extracting value and insights from their data, they look to data catalogs.

Given the number of options available today, it can be difficult to choose one catalog over another. You can go open source and save yourself licensing fees, but then you’re responsible for the implementation, infrastructure, maintenance, and support. Or you can go with a high-end, fully-managed platform. It’s expensive, but for many companies, it’s well worth the cost.

Revelate works directly with catalogs every day and we have a variety of customers from SMBs to Fortune 100 enterprises. Everyone has different requirements and constraints, which further complicates the question of which data catalog is “best.”

Let’s take a look at some of the most popular open-source products on the market, discuss their strengths and weaknesses, and try to help you make the right decision for your business.

Understanding data cataloging and its importance

Data has become important enough in recent years to warrant its own data management lifecycle. Put simply, the data management lifecycle is the tracking and maintenance of data from its creation through its use, long-term storage, and/or deletion. Depending on the data maturity of an organization, a lot can happen after data is created.

This process is where we get into concepts like data lineage, data integrity and quality, metadata governance, ELT and ETL tools, data modeling, and data pipelining. Many of these concepts are quite complex and have their own frameworks, like data governance frameworks and metadata management frameworks. 

Data catalogs cover all of these concepts in one way or another. Of course, they specifically track what data exists within a data ecosystem, where it is, who owns it, and some basic metadata. Some more premium catalogs also track lineage and governance. Others might integrate with tools specifically dedicated to these functions.

The ability to manage these aspects of your data lifecycle is a great benefit of data catalogs. There are downsides to catalogs, too; for example, catalogs can be helpful for exposing what data is available in an organization, but they don’t necessarily make the data discoverable or consumable for the people who need it. For those use cases, you might consider something like Revelate, which is built for data product creation and fulfillment.

Open-source data catalogs

Because of the high demand for data catalogs, dependable open source alternatives are entering the market. Some of the more popular alternatives include Apache Atlas, Amundsen (developed by Lyft), LinkedIn DataHub, Netflix Metacat, and OpenMetadata (developed by the Linux Foundation). Obviously, with sponsors like Lyft, LinkedIn, and Netflix, these aren’t small-time projects lacking in support and meaningful feature development.

Many of these open-source options are cloud-native, support a variety of data types, sources, and destinations, and are user-friendly. In other words, the data catalog market is getting more competitive and interesting. While there are many strengths of these systems, there are many questions to consider, such as:

• Will the data catalog connect to the data sources that matter most for your organization? Data connector support varies between each open-source catalog
• Is there built-in metadata support and does it meet the needs of your organization? If you have a wide variety of data sources producing many types of data, metadata may be more important than you think
• Does it support data governance? For some organizations, a lack of data governance integration is a non-starter
• Is it easy to use? If you’re planning on having non-technical users in your catalog, ease of use may be a strong consideration for your decision

Open-source data catalogs are great, but they’re not quite the same as a paid option, which likely comes with SLA-driven support, bespoke feature development, and professional services for implementation.

Popular data catalog tools

The following popular data catalog solutions are diverse in their features and capabilities. Nearly all of them are capable data governance products as well as data catalog products.

• Apache Atlas (open source)
  • Apache Atlas is a free and open source data catalog that is well-suited for organizations that need to store and manage a large amount of metadata
  • “Apache Atlas provides open metadata management and governance capabilities for organizations to build a catalog of their data assets, classify and govern these assets and provide collaboration capabilities around these data assets for data scientists, analysts and the data governance team.” (Source)
  • Some find the user interface to be unintuitive and technical
• Atlan
  • Atlan is closed source and enhances several features of a typical data catalog, particularly metadata management and embedded collaboration 
  • Cloud-based and easy to use, but not as feature-rich
• Collibra
  • Collibra is a closed source, comprehensive data catalog that offers a wide range of features
  • Costs may be a concern for some organizations
• Dremio
  • Dremio is a data catalog that is focused on making data discovery and exploration fast and easy
  • It does not have many features for data governance
• Open Metadata (open source)
  • Open Metadata is an open source, cloud-native data catalog designed for interoperability with other catalogs
  • It is still under development and does not have many features
• Talend Data Catalog 
  • Talend is a closed source data catalog focused on helping organizations integrate and manage the entire data lifecycle
  • It is more feature-limited because Talend has a broad suite of individual products and services

Evaluating data catalog products

Like any other major piece of technology, there are factors to consider when evaluating and determining the best data catalog tools.

Understand organizational requirements

When looking for a data catalog, it is crucial to first understand your organization’s requirements for such a tool. You may require data governance solutions, data quality management tools, data lake support, or support for unstructured and structured data. 

If your organization is sufficiently large, it may not be possible to document the entirety of your corporate needs. If that’s the case, start with your department or line of business. The point is to get a large enough sense of what matters for a successful data catalog implementation. 

Important considerations include:

• Pre-existing software, like data governance tools, data quality tools, data transformation and other data platforms
• Your data scalability and performance needs for both today and five years from now
• A map of your data ecosystem—including your most important data sources, destinations, and use cases—and how you move data around (e.g. ETL processes, pipelines)
• What level of support you’ll need; this will depend on your organizational data maturity and number of data-savvy teams who will use the catalog
• Your existing and foreseeable future data governance models, including regulatory compliance
• Budget, not just for licensing but also for support, which should include your ability to support a self-hosted model if you’re looking at open source options

Core features & functionalities of open source data catalogs

Once you know your basic business requirements, you need to think about the core features of data catalogs, especially if they’re open source. Can it ingest data from your most important data sources? Is the ingestion method different from how you’re doing it today? If so, can it capture all the data from all the tables you need? Are the ingestion methods secure?

Beyond ingestion, you should also consider the search and discovery functionalities of the catalog. For example, consider whether a catalog’s data search capabilities can support natural language searching and whether it can interpret a query well enough to find data in tables or columns with cryptic names.

The third important consideration is data governance. We’ve covered some of these factors already, but some of the more notable capabilities to look into are:

• Data lineage and quality
• Data security and documentation of access
• Data compliance and adherence to privacy regulations

Integration capabilities & ecosystem compatibility

You might think that data ingestion requirements cover integrations, but data catalogs can do much more via integrations than ingestion. Let’s look at some of the differences.

Furthermore, a data catalog can have deeper integrations with other systems, enabling more powerful visualization capabilities, more visibility into metadata, and analytics. Such integrations may include APIs or custom connectors to work with existing tools and automated workflows. You’ll want to consider interoperability with popular data cataloging standards & frameworks, such as Data Catalog Vocabulary (DCAT), Resource Description and Access (RDA), Common Data Model (CDM), or Data Catalog Toolkit (DCT).

User interface & ease of use

Early on, your catalog users will likely be technically-proficient data professionals. You might set up some use cases for non-technical users, but a majority of your early users will be qualifying and assessing the deeper technical capabilities of the data catalogs you try. As we’ve seen time and again, the more an organization leans into its data, the broader the demand for data across non-technical teams.

As such, it’s critical that the user interface and ease-of-use factor into your decision for a data catalog. You’ll want to look at how technical features are exposed to an end user, whether the built-in navigation is intuitive, and what you’re able to customize or even white-label. Here are some considerations for each of the catalogs we mentioned earlier.

• Apache Atlas: Apache Atlas has a command-line interface (CLI) and a web user interface (UI). The CLI is more complex to use, but offers more flexibility. The web UI is easier to use, but is not as feature-rich as the CLI
• Amundsen: Amundsen has a web UI that is user-friendly. The UI is easy to navigate and provides a variety of features that make it easy to discover and understand data
• LinkedIn DataHub: LinkedIn DataHub has a web UI that is designed to be federated. This design means that the UI can connect to a wide range of data sources and can show data from all of these sources in a single view
• Netflix Metacat: Netflix Metacat has a web UI that is designed to be search-driven. This design means that users can find data by searching for keywords or phrases. The UI also provides a variety of features that make it easy to explore data
• OpenMetadata: OpenMetadata has a web UI that is designed to be cloud-native. This design means that you can deploy the UI on-premises or in the cloud. The UI is also interoperable, which means you can connect it to other data catalog systems

Data security & privacy considerations

When selecting a data catalog, dig into the details of what’s listed on the website. Websites generally list high-level information for marketing and sales purposes, but when it comes down to actual selection, you’ll want to get into the finer details. Here are some of our recommendations for data security and privacy:

• Data encryption: The data catalog should encrypt all data stored in the catalog. This feature protects data from unauthorized access
• Data access control: The data catalog should have a robust data access control system to ensure that only authorized users can access the data, preferably via RBAC or IAM integration
• Data lineage: The data catalog should track the lineage of all data that is stored in the catalog. This feature helps identify and mitigate data security risks
• Data auditing: The data catalog should have a data auditing system to track who has accessed the data and what they have done with it
• Data compliance: The data catalog should be compliant with all applicable data privacy and security regulations to ensure you protect your data in accordance with the law

Your security and privacy requirements will depend on your geography, your users’ geography, and your industry. Be sure that the catalog you choose has you covered, and that you have a plan to manage the gaps. For example, you’ll want to know how authentication and authorization work, which will likely depend on what technology you already have or are planning to migrate to.

Finally, it can never hurt to have a third-party assessment of the security and privacy capabilities of the catalog and how it plugs into your organization.

Performance & scalability

Every data catalog will tout performance and scalability as core features, but there’s only one way to understand whether they’ll meet your needs: try it yourself. Furthermore, it’s important to note that the catalog’s architecture can have an impact on performance and scalability. For example, a centralized data catalog will be more performant than a federated data catalog.

As your datasets grow larger, how will the catalog’s performance change? Before you make a selection, see if you can get a performance model, especially if you’re purchasing from a major vendor. Make sure you understand both the scalability and costs. Larger catalogs will always require more resources, but the cost curves will differ by catalog.

Also consider performance benchmarking, deployment options, and overall compatibility within your data ecosystem. A catalog built and designed for AWS may perform differently—probably worse—in Azure or GCP, especially with large data volumes. Consider that some of your data integrations may perform variably depending on how and where they’re hosted, the types of queries you’re running, the number of concurrent users, whether caching is a factor, and how the systems are load balanced.

Community support & documentation

Support and documentation can make or break a successful implementation. Just because a new catalog works the first year doesn’t mean you’ll enjoy using and troubleshooting it in the third year. Here are some of the community support and documentation considerations for a data catalog:

• The size and availability of community support: Community support availability is important for users who need help with the data catalog. A good community support system should have a forum where users can ask questions and get help from other users
• The quality of the documentation: Documentation quality is also important for users who want to learn how to use the data catalog. The documentation should be clear and concise, and information should be easy to find
• The frequency of updates: Update frequency is also important for users who want to stay up-to-date with the latest features and changes to the data catalog. Update documentation regularly so that users have access to the latest information

Try to get a sense of whether there are tutorials, how often the software is updated, how many of the updates are for bug fixes, and how many user support requests get resolved.

Potential limitation & challenges

As always, you’re likely to run into some limitations and challenges with a major technology initiative. Data catalogs are no exception. You may face technological challenges, or you may face political ones. Potential challenges of implementing a data catalog include:

• Data quality: The quality of the data in the data catalog is critical. If the data is not accurate or up-to-date, the data catalog will be of limited value
• Time and resources: Implementing a data catalog can be a time-consuming and resource-intensive process, especially for large organizations with complex data environments
• User adoption: Getting users to adopt and use the data catalog can be a challenge, especially if the data catalog is not well-designed or not integrated with other tools
• Technical expertise: Implementing and managing a data catalog requires technical expertise, which is a challenge for organizations that do not have in-house technical resources
• Cost: The cost of implementing and maintaining a data catalog can be a challenge for organizations with limited budgets
• Stakeholder buy-in: Make sure that key stakeholders are on board with the data catalog project to ensure the project is successful

How to successfully choose and implement a data catalog

Choosing and implementing a data catalog can be a complex endeavor, but don’t let it scare you. The information in this article should get you most of the way there; additionally, it’s vital to make sure you have a use case.

No one will want to use a data catalog if they don’t have a good reason to. A data catalog isn’t a solution in search of a problem. You must be able to make a case for the catalog and the required investment. Find a high-visibility, high-value use case and go for it. Be successful, then show your company the value of using a data catalog.

If you need help with implementation, Revelate has a rockstar professional services team. We’re deeply familiar with data catalogs, complex implementations, and use case definition. Let’s collaborate.

Unlock Your Data's Potential with Revelate

Revelate provides a suite of capabilities for data sharing and data commercialization for our customers to fully realize the value of their data. Harness the power of your data today!

Get Started