Understanding the Story of your Data leads to Efficient Decision Making
It’s a fact. Organisations are generating enormous amounts of information through their data. Factor in multiple systems, various functions, and different departments, and you quickly see how even the smallest business is now producing vast amounts of complex and interconnected facts, figures, records and statistics. When we start collecting data that provides information about other data, it is called ‘metadata’.
However, even when an organisation has followed best practice, there is still no guarantee that someone seeking information can easily locate it.
You may think your data is well organised and maintained and that your processes are well documented. But even if this information is readily available and regularly updated, you will still need to remember every system update, all changes in business practices, and the comings and goings of people within your business to be able to navigate the information overload effectively to find your answers. For any business, any change to the status quo, at any number of touch points, could impact your ability to access your data efficiently.
In a world where transactions execute in nanoseconds, retrieving accurate data efficiently, and being able to trace and track your information effectively is now essential for business continuity. This is called Metadata Management and accessing this information is one of the many challenges organizations face today.
To put this into context, it is worth understanding how metadata sits in the technical world of a typical Datawarehouse or Datamart.
This flow of information could also be described like this:
Source Systems -> Data gets extracted and loaded from source systems into DW or DM-> Datamart modelling for reporting -> Final communication of the data via reports or visualizations.
Case for Metadata:
Picture kicking off a project involving one of your systems.
1. You focus on your deliverables.
2. You document each phase in your process such as requirements gathering or process implementation.
3. You test cases for UAT.
Concentrating on the future and the desired outcomes, it is easy to forget how the system was initially built and what changes have taken place since its inception.
Maybe you hit a block.
Typically, you would open the artifact -> study the logic -> trace it back to the database (if the logic is stored in the database) -> find out the dependencies, then investigate further.
But there is no guarantee this will resolve the issue and may not provide the full history of all changes, nor show you all the dependencies that caused the block in the first place.
However, if you had maintained the metadata, the block would be very easy to solve, because the metadata would…
a) Create a roadmap to your Datawarehouse
b) Provide a Directory to locate the contents, and
c) Provide a Definition of all your warehouse items
