Airbyte Commoditizes Database Replication by Open-Sourcing Log-Based Change Data Capture

Fast, efficient replication of data from MySQL and Postgres databases to data warehouses, lakes and other databases

SAN FRANCISCO--(BUSINESS WIRE)--Airbyte, creators of the fastest-growing open source data integration platform, today announced the release of the Airbyte log-based Change Data Capture (CDC) open source software. This enables data stored in MySQL and Postgres databases to be replicated quickly and efficiently anywhere. In July, Airbyte plans to add support for Microsoft SQL Server, Oracle Database, MongoDB and DynamoDB.

Now, any authorized database user or administrator is able to perform database replication without requiring additional work from their IT (information technology) team.

Change Data Capture allows recording events of all the data and metadata changes that the source database has registered. These events can then be propagated to a downstream system thus enabling data replication in the destination data repository. The benefits of CDC include the following.

1. Support incremental updates by design. It is only recording changes.
2. Reliable detection of data deletion.
3. Minimizes disruption on databases as it doesn’t need to scan and read full tables.
4. Reduces data transfer costs by only transferring data by sending only incremental changes.

Many common databases support CDC, writing all record changes to log files. Airbyte now supports CDC to replicate data from Postgres and MySQL to data warehouses, lakes and databases. And Airbyte’s CDC is open source, so it’s possible to replicate databases while maintaining control of the data without having to go through a third-party service and risk additional privacy and security concerns.

“Some open-source solutions for CDC exist, but all of them are very hard to implement,” said Michel Tricot, co-founder and CEO of Airbyte. “Until now, there was no easy, no-code way of doing database replication with log-based change data capture technology. With Airbyte, any authorized user is able to start replicating databases within minutes without writing a line of code -- not days later after significant work by engineering teams.”

By commoditizing data integration, Airbyte is establishing the new standard of moving and consolidating data from different sources to data warehouses, data lakes, or databases in a process referred to as extract, load, and, when desired, transform (ELT). Businesses can create data pipelines from sources such as PostgreSQL, MySQL, Facebook Ads, Salesforce, and Stripe, and connect to destinations that include Redshift, Snowflake, and BigQuery.

Airbyte is already fostering a community of more than 2,500 users to build and maintain open source connectors that are available to anyone. To date, there are 75 connectors that Airbyte is certifying to ensure they are production ready. By the end of this year, the company anticipates it will reach 200 connectors, which would be the most pre-built connectors in the market. It recently introduced its Connector Development Kit (CDK) in order to enable its user community to accelerate development and quickly address the long tail of connectors.

The Airbyte connectors run in Docker containers, which means they can be deployed in minutes on any cloud platform. It also enables connectors to be built in any programming language.

To learn more, go here to view documentation for the Airbyte CDC and corresponding connectors. Also, this video demonstrates CDC for Postgres.

About Airbyte

Airbyte is the open-source data integration alternative running in the safety of your cloud and syncing data from applications, APIs, and databases to data warehouses, lakes, and other destinations. Airbyte was co-founded by Michel Tricot (former director of engineering and head of integrations at Liveramp and RideOS) and John Lafleur (serial entrepreneur of dev tools and B2B). The company is based in San Francisco. To learn more, visit airbyte.io.