How is Lore IO different?
Data platforms, ETL tools, data wranglers, ELT solutions, and data visualization tools represent hundreds of marketplace participants. One question we always get is how we are different from our all of them. This blog post attempts to address this question from a non-traditional angle: Instead of comparing speeds and feeds with each market category – which will make this post very long and repetitive – we will focus on three aspects of data transformations that are important to businesses today.
In case you’ve forgotten, Lore IO takes on the tough mission of handling complex data transformations elegantly and at scale. Customers of all sizes and varieties rely on our solution and team of data transformation experts to handle their most demanding data requests. And so this post focuses on data transformation at large. Here goes.
Eliminate Data Pipelines and ETL Code
Much of the enterprise data world is governed by data pipelines and ETL code. Data platforms, for instance, provide an end-to-end solution for data storage, processing, and consumption, where data pipelines play a central role. These become convoluted and unmanageable over time, as data engineers progressively add more functionality to support new data requests, thus introducing drag on the business. IT becomes worried that any changes or additions might bring the system to a halt, so new projects slow down.
Because of their broad solution approach, data platforms typically lack strong data transformation capabilities, although they do have some in place. For instance, AWS offers AWS Glue for data ETL. Others, such as Hortonworks prior to its merger with Cloudera relied on open-source solutions for data transformations, such as Apache Hive, Apache Spark, Apache Pig, and Apache NiFi.
Lore IO doesn’t compete with data platforms, but rather augments them. Indeed, many of our customers deploy our solution within their existing data platform, in their own private cloud.
The single most important benefit that Lore IO provides is the elimination of data pipelines and ETL code. Lore IO maintains all transformation logic declaratively, in SQL-like language that business analysts can understand and work with. As soon as data views are needed, Lore IO automatically converts the data definitions into ETL code, which it runs on the source data. Lore IO then avail the data views so that teams can materialize them in their data warehouse.
This way, there are no data pipelines and code to maintain manually.
Eliminate Prolonged Data Requirement Gathering
Availing new or modified data to the business requires substantial BI business requirements gathering. For instance, before data engineers can use ETL tools to build new (and cumbersome) data pipelines, they must spend significant time and resources getting the data requirements right. ETL tools can be used only when the data engineer has perfect understanding of the source data structure and the desired data schema. Since business and IT do not think or talk about data similarly, requirement gathering is a laborious and error-prone process.
Lore IO frees the organization from this predicament by offering centralized team collaboration and task management capabilities. Business customers initiate data requests on their own, by specifying their desired data views and the elements they should include. Lore IO facilitates the process of converting these requirements into a set of tasks for the business and technical teams to clarify the requirements and the necessary transformations.
What Lore IO essentially does is replace the traditional requirement gathering process with a collaborative work among business and IT. It provides all stakeholders visibility into the state of the projects and any potential blockers. With these capabilities in place, Lore IO accelerates time to value and data availability.
Accelerate Data Transformations
Procedural logic is a well-established approach in the enterprise data space. Data engineers who know how source data needs to be handled and reshaped must provide explicit instructions (ETL code) to the machines that carry out these jobs. For instance, data wrangling tools that take on cleaning up and normalizing source data can become difficult to use when changes to procedural logic must be implemented in several places of the code.
Lore IO accelerates data transformations with declarative logic.
Lore IO accelerates data transformations with declarative logic. Instead of specifying how transformations should function, Lore IO customers specify what the desired end-state of the data should be. They do so declaratively, in a SQL-like manner. This makes it easy for teams to maintain their transformations as data requirements increase or change, since the implementation details are abstracted away; when customers change their desired end-state, Lore IO automatically modifies the ETL code as needed.
Lore IO offers a library of transformation functions to handle complex transformations, such as serializing time-series data. Business and technical teams access this library via a graphic user interface. In addition, Lore IO offers to build custom transformations in SQL-like code. This further accelerates data transformations because customers build out their transformation in an intuitive manner, as if defining functions in Excel.
Augment Rather Than Displace
Analytics platforms are dynamic environments that continuously evolve to meet new data needs. It’s clear that no single provider can offer an end-to-end technology stack that best addresses all data analytics needs. Customers should instead take a different approach by building out their stacks with best-of-breed providers that successfully take on critical steps in the stack.
Data transformation – especially in large organizations or in organizations that depend on a large set of heterogeneous data sources – can be difficult to handle and scale. It’s therefore best to partner with a solution provider who singularly focuses on data transformations; one that can accelerate current transformations and support future needs.