Lore IO customers use our data transformation platform to develop their universal data fabrics, which they use to answer key operational questions. At the core of Lore IO is a sophisticated collaboration system. It enables business and technical teams to jointly convert new data requests into a series of micro-tasks. These micro-tasks involve data definition, modeling, and transformation. The teams work together on their assigned micro-tasks to progressively avail new tables and columns in their data fabrics.
Once all modeling and transformation micro-tasks are completed, data engineers bring source data into the system and map them to the transformation logic. Lore IO converts the transformation logic into query code, runs it, and publishes the resulting data views that customers can materialize to their data warehouses.
By democratizing data preparation across the organization through collaboration, Lore IO customers benefit from having deep visibility into the state of their data projects. They can readily understand what the next steps are, and what, if any, is blocking a given project. This increases team member accountability and trust.
But collaboration via micro-tasks does not benefit only the system users; it also makes the platform’s AI better, which, in turn, further accelerates data preparation projects. Lore IO uses AI to make intelligent data preparation recommendations. The system observes the micro-tasks that team members carry out, and then uses the insights it generates to recommend transformation steps when new data sources are onboarded.
More power to the machine
When customers break new data requests into a series of micro-tasks, each micro-task becomes a discrete activity that the AI can easily process to relate the input of the activity to its outputs. Lore IO can later identify new micro-tasks that are similar in nature to the old one, and recommend transformations based on prior observations. Therefore, micro-tasks enable the platform to better guide its users.
Another advantage of micro-tasks is that they enable greater reusability. Each micro-task results in a modular component —whether a table relationship, a data wrangling rule, or a data validation rule — that is stored in the universal data fabric and can be used for future data preparations. This enables the AI to easily identify and suggest the right component for subsequent micro-tasks.
Finally, Lore IO’s AI makes recommendations not just for similar micro-tasks but also for related ones. For instance, when a platform user completes one micro-task by, say, entering a description for a table column, the AI can make a better recommendation regarding a different micro-task that’s related to that column, such as how to map the column to a data source.
The unique way that Lore IO integrates user collaboration and AI implicitly facilitates collaboration between humans and the platform. By dividing their work into a set of discrete steps, users essentially participate in training the platform, which, in turn, simplifies their data preparation work.