Fivetran, the leading provider of automated data integration, today announced advancements in its integration with Databricks, the Data and AI Company, extending the companies’ strategic partnership to power the modern data stack.
Product enhancements addressing Delta Lake and the Lakehouse architecture with SQL Analytics accelerate time-to-value for customers. Updates include new ease-of-use features for setup, dbt packages and orchestration integration for advanced transformations, and history mode to ease historical data analysis for Databricks.
Fivetran, part of Databricks’ Data Ingestion Network, automatically loads and maintains the pipeline of raw data into ready-to-query formats allowing data analysts, data engineers and data scientists to focus more time and resources on analytics, data science and machine-learning applications.
With SQL Analytics, Databricks empowers data analysts to perform workloads on a data lake that were previously only addressable with a data warehouse. Fivetran’s support for SQL Analytics will enable data analysts and BI users to build end-to-end data pipelines on their lakehouse architecture in Databricks.
Fivetran plans to extend its data writer and transformations to include SQL Analytics. Importantly, SQL Analytics is expected to provide superior concurrency for Delta Lake, a critical capability for enterprises serving a large number of BI users.
Read More: Role of Innovation Teams During and Post COVID-19
“As two critical providers in the modern data stack, Fivetran and Databricks are delivering a more comprehensive data analytics environment for customers, while eliminating the complexity of repetitive configuration and maintenance,” said Fraser Harris, vice president of product for Fivetran. “These latest improvements further strengthen our commitment to make access to data as simple and reliable as electricity.”
Fivetran automates data integration with a fully managed extract-load-transform (ELT) architecture while Databricks, a unified data analytics platform, separates storage and compute which provides cost efficiencies at scale for companies to unlock a wide array of data analytics, date science and machine-learning use cases. Key new features and benefits include:
- Ease-of-use setup and configuration – Customers no longer have to set up a staging area. Fivetran does that in a secure environment trusted by 1,600+ customers.
- dbt support – From Fishtown Analytics, dbt is the most popular open-source SQL data transformation tool. Upcoming support will include:
-
- dbt packages – Fivetran delivers dbt packages for the most popular SaaS applications and cloud services, such as Salesforce, NetSuite, and Marketo, giving customers ready-made, customizable aggregations.
- dbt orchestration integration – For advanced transformations, the end-to-end data pipeline can be scheduled and controlled within Fivetran.
- History mode for Databricks – Designing and maintaining schemas for versioning historical analysis is time-consuming and error prone. Fivetran makes it easy by configuring the synchronisation settings, and automating the generation and maintenance of the schema.
Read More: Adjusting to the New Normal at the Workplace
The combination of Fivetran and Databricks helps customers, including Red Ventures, Prodigy Games and Slice, improve collaboration and allows them to scale big data analytics at a faster pace. Customers regularly report that the Fivetran zero-maintenance data integration service saves the equivalent of 2-3 full-time data engineers per year.
“When Databricks recommended Fivetran, I knew it was the right choice having used it at a previous company,” said Jason Ordway, chief technology officer at Slice. “With Fivetran, we have all of our data centralised and syncing in near-real-time so we can do advanced data science, create machine-learning models and conduct predictive analysis. The two solutions together have improved our data analytics.”
“We are delighted to work with Fivetran, both as a strategic partner and as a happy customer,” said Michael Hoff, SVP of Business Development and Partners at Databricks. “As a partner, Fivetran helps our joint customers accelerate time-to-insights by removing the time-consuming hassles of building and maintaining pipelines. With SQL Analytics, we want to extend this benefit to SQL and BI users so they can easily manage data and perform analytics on their data lakes.”