Who would build a house on mud using Mikado sticks as a foundation?
Even if the structure is solid, if no one speaks the same language, it seems difficult to complete the work; a great mythological tower learned this the hard way a few thousand years ago.
Today, data is the new wealth of businesses. Whether stored in a traditional data warehouse, a lake, or any other storage system, it is this data that will enable companies to create value and stand out from the competition.
In this article, we will explain the key importance of establishing a clear data dictionary that is shared by all stakeholders in a data warehouse project.
1. The definition
Let’s start with a quick review of what a data dictionary is. It is a list of data fields that will be included in our data warehouse. This data can take various forms—text, dates, numeric values—and cover the full range of functional requirements necessary for our business.
The value lists for certain data fields must be specified when creating the dictionary to simplify future retrieval.
There are two types of value lists: those that are fixed, such as the list of countries (updates are very infrequent), and open-ended lists, such as comment fields, which will be much more complex to reuse later.
The first step in creating a data dictionary is to establish a clear definition agreed upon by the data owners.
This shared definition is crucial. Some data definitions are uncontroversial, but if we’re discussing outstanding amounts with someone from risk, accounting, or liquidity, definitions may vary because everyone will have their own perspective.
In this specific case, a good data dictionary will not mix definitions but will duplicate the data so that everyone has their exact definition.
These shared definitions will simplify communication within the organization and ensure higher quality. Furthermore, in some cases, this may reduce operational or regulatory risks.
This clear data dictionary will make it easier to comply with regulatory or legal requirements (GDPR CNIL).
2. Data Governance
To ensure that data remains high-quality over time, several rules must be followed.
First, a team must be designated as the data owner to ensure that only that team can modify the data and that others can contact that team if any anomalies are found.
Next, it is important that the data be used by systems or users. Regular use will help ensure its reliability, as errors will be detected immediately by those who know the data best, and corrections can be made.
Finally, as part of a continuous improvement process, it is useful to implement automatic checks whenever an anomaly is found to try to resolve a potential future problem proactively. Regular cross-checking of data to identify anomalies is one of the most effective controls for maintaining a high level of quality.
3. Changes in the data dictionary over time
The data dictionary is an integral part of the data model that will be created in IT systems.
This model will need to avoid several pitfalls:
- Never store the same data in multiple locations: storing identical data in different places will likely lead to data inconsistencies down the line
- Failure to validate all business use cases, particularly regarding table cardinality: this crucial step validates the model’s structure, and if the functional concepts work well together, it will also prevent data duplication
- Being unable to upgrade the system without affecting existing data: historical data must always remain accessible, if only for auditing purposes
To return to our original analogy with construction: if the foundation is solid and the drainage system is properly installed, it will be possible to add an extension to the side or onto the house; it’s exactly the same with a data dictionary.
A data dictionary is therefore a key factor in the success of a data warehouse project.
Creating a precise data dictionary and sharing it early on will prevent major problems down the line. This is an important task that must be completed before development begins.
Short-term benefits include simplified communication with teams and easier testing on the development side.
In the long term, this will lead to improved quality (and thus greater value) as well as reduced development costs for future requirements.
Ultimately, a well-structured data dictionary is a long-term investment in effective data management and a high-quality data warehouse.
an article written by…
Eric LAPINA,
Data Practice Director at Quanteam
