Fivetran launches Managed Data Lake Service for enterprises

Fivetran has unveiled its Managed Data Lake Service, designed to automate and simplify data lake management for enterprises of varied sizes. The service integrates over 500 pre-built and custom data sources seamlessly into major data lake destinations such as Amazon S3, Azure Data Lake Storage (ADLS), and Microsoft OneLake. This initiative aims to enhance enterprise data quality, completeness, and timeliness, while reducing the complexities associated with data integration.

By automatically converting customer data into popular open formats like Apache Iceberg or Delta Lake, Fivetran's Managed Data Lake Service provides the ease and queryability of a cloud data warehouse combined with the flexibility and scale of a data lake. This unique approach allows companies to enjoy the cost benefits of data lakes while harnessing the structural and reporting capabilities of data warehouses. The service's design eliminates the need for data records to be moved or duplicated across multiple locations, thereby supporting analytical, operational, and generative AI workloads efficiently.

George Fraser, CEO of Fivetran, stated, "Fivetran does the heavy lifting of change data management, PII detection, deduplication and other low-level table maintenance so that developers don't waste time on work that can be automated. We hope to make business users and data scientists alike more productive by providing clean, centralised, optimised data from any source."

Nick Chmura, Head of Data at Luma Financial Technologies, expressed similar sentiments, highlighting that automated table maintenance is a crucial feature as it avoids the high costs associated with managing diverse data source connectors manually.

The Managed Data Lake Service transforms traditionally ungoverned data lakes into organised, governed, and continuously optimised data stores. Native integrations with data catalogues like AWS Glue, Databricks Unity Catalog, and Microsoft Purview allow users to discover, access, and govern datasets with ease. Additionally, users can query and modify data using Python, SQL, or other supported languages via compatible compute engines such as Databricks, Snowflake, Starburst, or Redshift. The service also supports data transformation with dbt, visualisation with Power BI, and AI/ML model building with AWS Sagemaker, Azure Machine Learning, or Databricks Mosaic AI.

Fivetran's new service supports a wide array of data sources, including on-premises and cloud databases like Postgres, MySQL, Oracle, and SAP, as well as SaaS applications, data warehouses, events, and files. Custom connectors can be created to assure that virtually any data source is supported, reducing the need for additional engineering resources for pipeline management or connector development. This extensive compatibility helps businesses unify their data in the data lake efficiently.

Himanshu Raja, Director of Product at Databricks, commented on the excitement surrounding Fivetran's support for Delta Lake as a direct destination. He affirmed that the new capability allows customers to build an open lakehouse with Delta Lake powered by the Databricks Data Intelligence Platform, underscoring enhanced governance and security through integration with Unity Catalog.

The new Fivetran Managed Data Lake Service promises several benefits, including empowering business users and data scientists with centralised, query-ready data, enhancing operational efficiency by converting data to open table formats with robust cataloguing and governance, and reducing developer workload through automated table maintenance tasks. It also offers cost reduction by moving away from legacy data warehouses and ensuring clean and complete data replication.

The introduction of the Managed Data Lake Service comes at a time when demand for advanced AI capabilities is surging, particularly among large enterprises seeking cost-effective, flexible data architectures. Anders Holden, Director of Product Management at Starburst Data, praised the service for simplifying the ingestion process and reducing complexities around data lake management, which he believes will save businesses time and costs while accelerating their data pipeline processes.

With the Fivetran Managed Data Lake Service now available, businesses are poised to find innovative ways to leverage data, supercharge AI initiatives, and unlock significant business value.