The Challenge
Our client faced an unprecedented situation and immediately had to deal with event data from hundreds of sources. They were forced to work quickly to find and create information to assess the situation and guide the response. Due to the immediate nature of the situation, a huge amount of data was created with varying structure. Our client needed a Master Data Management (MDM) solution to correlate, search, and understand the entire data set collected during the response.
The Solution
As none of the current out-of-the-box MDM solutions met the requirements, a decision was made to develop a custom MDM solution. The solution extended the Microsoft analytics technology stack already in use within the organization.
A series of Dimensional Models were developed for each of the 14 types of entities (people, organizations, vendors, etc.) included in the scope. A common Extract, Transform, and Load (ETL) layer was developed to orchestrate the flow of data from the different sources of information into the MDM repository.
A Fuzzy Logic Match and Merge capability was developed, using SSIS native components. This feature was critical to ensure data governance, allowing the Data Stewards to quickly identify related and duplicated data among domains.
The Result
In the wake of this unprecedented situation, we empowered our client with an accurate and consistent set of event data to analyze the situation and guide their response. This solution was implemented successfully and included multiple domains. We were able to provide the client with a consistent, consolidated view of their data. After the MDM repository was completed, it was possible to link related data across domains, enabling the Search component of the project. The Data-stewards from each domain were able to review proposed duplicate and matched data to allow them to make an informed decision over data validity. This information was published to SharePoint, allowing it to be consumed internally and externally.