// HACKER NEWS — CYBERSECURITY
MySQL CDC to BigQuery: what periodic syncs miss, and how binlog avoids it
MySQL CDC syncs miss deletes and intermediate updates. Learn how binlog-based Change Data Capture works, what MySQL settings it requires, and how to land it reliably in BigQuery.
Most MySQL-to-warehouse pipelines run on the same pattern: a scheduled job selects rows, compares them to what was there before, and writes the difference. It works, until it doesn't.
A SELECT-based sync only sees what exists right now. It has no way to know a row existed and was deleted between two runs, no way to see intermediate states of a row that changed more than once, and it puts real load on your production database every time it scans a large table just to find a handful of changed rows.
Change Data Capture reads directly from MySQL's binary log (binlog), the same mechanism MySQL uses internally for replication. Every INSERT, UPDATE, and DELETE is captured as it's written to the log, in order, with the complete row state. Nothing is inferred by comparison. Nothing depends on when a batch job happens to run.
This isn't about speed. A CDC pipeline that runs once an hour is still fundamentally more reliable than a batch sync that runs once a minute, because it captures everything that happened, not just the latest snapshot.
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'your_user';FLUSH PRIVILEGES;
SHOW VARIABLES LIKE 'log_bin';SHOW VARIABLES LIKE 'binlog_format';SHOW VARIABLES LIKE 'binlog_row_image';
If any of those aren't set correctly, they go in my.cnf, and MySQL needs a restart to apply them.
The full connector documentation, including every prerequisite and troubleshooting step, is at docs.erathos.com/connectors/databases/mysql#cdc-setup.
This is where a managed platform earns its keep. Erathos and similar tools handle the snapshot mode selection (full initial snapshot vs. binlog-only), the server-id assignment and collision avoidance, and the recovery logic when a binlog gets purged before the connection catches up, so the person running the pipeline doesn't have to rebuild that logic by hand every time a new source gets connected.