Collaborating on Master Tables That Speak the Business Language
Master data tables enable our business teams to report on, analyze, and understand the people, places, and things that influence the growth of our organizations. These important business entities must be well defined in the master tables by data modelers, consistently updated by new transactional records, and readily understood by our analysts who do not necessarily know (or even care to know) the nature of the raw data that arrive as new events from said transactions systems.
Putting in place a process to have master tables consistently perform for the business is one complex and delicate data orchestration. Get it right, and our internal customers will be eternally grateful. But getting it right is often easier said than done.
It’s all in the details
One of the key issues with master data tables, which we just alluded to, has to do with the shape of data. Data is shaped differently when it's manufactured versus when it’s ultimately consumed. Put it differently, master data tables that are fed transactional data speak a different language than the one source systems use when generating transaction data.
As we noted, master data tables report on people, places, and things — entities that are foreign to transactional systems and often to the data engineers responsible for them. Transactional systems produce raw event data that our analysts cannot readily decipher. Event data may be schema-less and contain encrypted or otherwise unrecognizable values. And event data may need to be paired with other source data to make sense.
And so, in addition to modeling our master tables effectively, we also need to transform source data from raw mumbo jumbo to business speak. We do this by applying business rules that, like a cooking recipe, dictate the series of steps that are needed to take to convert a bunch of raw ingredients to a delicious dish.
Collaborating on master tables
To overcome the language barriers we just described, the Lore IO platform offers a single framework where business analysts, data modelers, and data engineers collaborate to address simultaneously both data modeling and data transformation needs.
Lore IO takes a top-down approach where business analysts and their data modeling counterparts get to go first. They define their desired master tables and the columns populating them.
Next, they enter information that describes the business logic behind the requested master columns. A business analyst might indicate, for instance, “an active customer is a user id that purchased at least one product in the past three years.”
The project team then converts these data requests and semantic insights into a to-do list. Each task on the list calls out a transformational logic that the team must define. Anyone on the team can take ownership of a step. For instance, if a data modeler understands the upstream systems, then she can go ahead and encode a transformation rule. Otherwise, she can assign the task to a data engineer who knows the source system.
Once the logic is defined in Lore IO, data engineers connect their source data into the system and map them into to the transformation logic. Since data engineers understand the source systems, they can quickly plug them into the transformation logic without blocking the entire process.
At this point, Lore IO converts the logic into query code, runs it, and publishes the resulting data views that business analysts can materialize in their data warehouse.
Getting master table done right and done fast
By collaborating in Lore IO, data modelers can progressively enhance the master tables and either manage the transformations themselves or assign tasks to data engineers. Instead of serializing data modeling and data transformation, Lore IO customers progressively evolve their data fabric by bringing their teams together to collaborate on data requests.
This collaborative approach enables the business to create trusted master tables that business users can consume continuously, as additional transformation work is done behind the scenes. This empowers business teams to seek new answers quickly and be ready to take on new threats or opportunities.
But the benefits don’t stop there. Lore IO keeps watch as transaction data gets mapped to master tables and columns. The platform’s AI core progressively learns the mappings and understands the relationships between source data and the target schema. When new data sources brought into Lore IO, the platform makes intelligent source-to-target mapping recommendations that speed up the data preparation process.