For companies, it is increasingly important to make business decisions based on data, which is key to achieving the desired success.

This capability can be enhanced by incorporating new technologies that allow organizations to transform the millions of data points they possess into reliable information for strategic use. This enables them to more accurately understand their customers' preferences, purchasing trends, and behavior, allowing them to offer the products and services that consumers truly need.

Considering this context, one of our clients required a consolidated and standardized data repository that could be used by Data Scientists and BI Analysts to generate new initiatives.

“Our mission was to design and implement a Data Lake with the goal of storing, integrating and processing all the data that was scattered across the company’s various sources and silos on a single platform.”

By using this tool, our client could make much more efficient and agile use of these data, being able to cross-reference them to identify the relationship between them, interpret and analyze them in less time, and obtain more accurate information to guide decision-making, being able to project a more accurate value proposition towards the target market in the future.

The project started in October 2019 and is being run by our Datalakers team, which includes Jaime Catalán, Branko Cotoras, Yeferson Mendoza, Joseilys Vásquez, Sebastián Soto, Alejandro Álvarez, Leonardo Silva and Pedro Garcia, among others.

“From the beginning, our task was to handle the design of the architecture, as well as the internal structure and organization of the data for the Data Lake, which is being built in the cloud. This has been a very particular and unprecedented challenge for those of us who have had the opportunity to participate, requiring us to grow rapidly in knowledge and also adapt to deliver value and results in all the work done,” says Data Engineer Alejandro López.

In January of this year, after presenting the design, the implementation of the project began, which has become key for our client's management, because it has provided more value and has achieved improvements in the production processes.

“During this time, various technologies and methodologies have been applied to consolidate the tool, which has an on-premise Kafka cluster that collects data from sources in Chile and abroad, moving millions of messages daily to the Cloud Pub/Sub service. DataFlow jobs take messages and apply the logic to store them in the correct tables within BigQuery or BigTable, making the data available for consumption in less than 10 seconds on average from the source to the cloud,” he explains.

It also features Azure DevOps pipelines for both on-premises and cloud components, which can be deployed in a controlled manner across test and production environments. Finally, there's a Grafana monitoring system for the entire architecture, configured with email alerts to ensure the system is always running smoothly and efficiently.

“As a team, we continue working and making progress on this Zenta-branded project. We feel it is an important success story, where we have had the opportunity to participate from the areas of Data, Analysis, and Architecture,” Alejandro points out.

At Zenta, we want to highlight the contribution our AstronautTaaS team is making to implement a platform that is highly strategic for our client in business decision-making.