What PeerDB-io/peerdb shipped
Written by FoxPlug from the repository's own releases; not affiliated with Peerdb. Paste your own repository in the box below and FoxPlug writes the same weekly update for your product. This page is an automatic summary of the releases, pull requests and commits in github.com/PeerDB-io/peerdb; Peerdb did not write it and does not use or endorse FoxPlug. Every line links to the change it describes.
Get a weekly update like this for your product, free
Week of October 5, 2026
What shipped
- Added GCP Cloud SQL IAM database authentication for Postgres and MySQL peers through the Cloud SQL Go connector. Pull request #4881
- Released v0.37.11 with Cloud SQL IAM auth, MongoDB structured ingestion performance work and cron schedules for query-based CDC polls. Release
- Added a
sync_cronoption to query-based CDC so polls can run on a standard cron schedule instead of a fixed interval. Pull request #4883 - Reduced memory and CPU use in MongoDB structured ingestion based on profiling. Pull request #4890
- Fixed handling of Postgres domain types during CDC. Pull request #4903
- Fixed query-based CDC so records are staged without opening a ClickHouse connection first, which previously left mirrors stuck. Pull request #4884
- Upgraded pglogrepl to support zero-column inserts. Pull request #4908
- Stopped issuing a binlog dump KILL when the syncer closes over SSH, removing intermittent tunnel dial errors in MySQL CDC over SSH. Pull request #4878
- Classified transient BigQuery errors as retry-recoverable so they no longer fire critical alerts. Pull request #4887
- Classified the MySQL handshake error caused by a server without TLS support so users are told they need to enable TLS. Pull request #4907
Why it matters
This week adds a new way to authenticate: Cloud SQL IAM auth for Postgres and MySQL peers. It also fixes several source-side issues that could stall or alert on pipelines, including Postgres domain types, query-based CDC stuck without ClickHouse, MySQL binlog dumps over SSH and transient BigQuery errors. Query-based CDC can now poll on a cron schedule, and MongoDB structured ingestion uses less memory and CPU.
Changelog entry
- Support GCP Cloud SQL IAM auth for Postgres and MySQL peers through the Cloud SQL connector. Pull request #4881
- Add
sync_cronto query-based CDC for cron-scheduled polls. Pull request #4883 - Optimize MongoDB structured ingestion for memory and CPU use. Pull request #4890
- Fix Postgres domain type handling during CDC. Pull request #4903
- Fix query-based CDC to stage records without connecting to ClickHouse. Pull request #4884
- Upgrade pglogrepl to support zero-column inserts. Pull request #4908
- Skip binlog dump KILL on syncer close over SSH. Pull request #4878
- Classify transient BigQuery errors as retry-recoverable. Pull request #4887
- Classify MySQL handshake errors caused by missing server TLS support. Pull request #4907
- Classify the MongoDB pool connection acquisition timeout error. Pull request #4899
- Align MongoDB structured ingestion types and add an initial load plus CDC test. Pull request #4896
- Release v0.37.11. Release
V0.37.11 is out. Cloud SQL IAM auth for Postgres and MySQL peers, cron schedules for query-based CDC polls, MongoDB structured ingestion performance work, plus fixes for Postgres domain types, query-CDC staging and MySQL binlog over SSH.
PeerDB v0.37.11 is released. This week adds GCP Cloud SQL IAM database authentication for Postgres and MySQL peers through the Cloud SQL connector, and query-based CDC can now poll on a cron schedule instead of a fixed interval. MongoDB structured ingestion got memory and CPU optimizations from profiling. Fixes include Postgres domain type handling during CDC, query-based CDC staging records without a ClickHouse connection, no binlog dump KILL on syncer close over SSH, transient BigQuery errors treated as retry-recoverable, and clearer classification for MySQL servers without TLS support.
Week of September 28, 2026
What shipped
- MongoDB structured ingestion now supports native array types instead of landing them as JSON columns, with uniform element types. Pull request #4858
- BSON DateTime values in MongoDB structured ingestion now convert to timestamps correctly instead of being treated as type mismatches. Pull request #4876
- Column exclusions in MongoDB structured ingestion now propagate to projection logic, preventing excluded columns from appearing as unexpected fields. Pull request #4879
- BigQuery mirrors now display CDC table counts for inserts, updates, and deletes tracked from the source. Pull request #4871
- MySQL pipe creation is faster by avoiding unnecessary KEY_COLUMN_USAGE joins that caused full table scans on MySQL 5.7. Pull request #4861
- Query CDC for BigQuery now respects safety lag windows to avoid polling during unsafe time periods. Pull request #4832
- Test selection speed improved by batching package discovery and running concurrent operations instead of sequential per-package checks. Pull request #4854
- Query CDC pipes now emit metrics for batch fetches and current batch IDs to improve observability. Pull request #4859
- Cloud SQL IAM authentication refactored to align with RDS authentication patterns and enforce TLS configuration. Pull request #4872
- CI crashes from concurrent map writes are prevented with an RWMutex protecting dynconf maps. Pull request #4869
Why it matters
MongoDB structured ingestion improvements make arrays and dates work correctly out of the box, reducing data quality issues. BigQuery CDC gets better operational visibility and safety controls. Performance improvements across MySQL validation and test selection speed up development cycles.
Changelog entry
- MongoDB structured ingestion supports native array types with uniform element detection Pull request #4858
- MongoDB structured ingestion converts BSON DateTime to timestamps instead of string representations Pull request #4876
- MongoDB structured ingestion excludes columns in projection logic Pull request #4879
- BigQuery mirrors display CDC table counts for insert, update, and delete operations Pull request #4871
- MySQL pipe creation faster by skipping KEY_COLUMN_USAGE join on MySQL 5.7 Pull request #4861
- Query CDC skips polling during safety lag window Pull request #4832
- Test selection runs faster with concurrent package discovery Pull request #4854
- Query CDC emits batch fetch and batch ID metrics Pull request #4859
- Cloud SQL IAM authentication aligned with RDS auth and enforces TLS Pull request #4872
- CI stability improved with RWMutex protecting concurrent map access Pull request #4869
MongoDB structured ingestion now handles arrays, dates, and column exclusions correctly. BigQuery mirrors show CDC table counts. MySQL pipes validate faster. Query CDC respects safety windows.
This week's updates improve data accuracy and operational safety across multiple connectors. MongoDB structured ingestion now correctly handles native array types, converts BSON dates to timestamps, and respects column exclusions—eliminating data quality issues. BigQuery mirrors now display CDC table counts for better visibility. MySQL validation is faster by avoiding unnecessary table scans. Query CDC for BigQuery respects safety lag windows to prevent unsafe operations. These changes reduce troubleshooting and deliver more reliable replication.
Week of September 21, 2026
What shipped
- Added structured ingestion processing for MongoDB with QRepValueIterator for BSON documents and SchemaProjector integration for CDC and QRep workflows. Pull request #4845
- Introduced
structured_ingestionanddrop_unexpected_valuesconfiguration options for MongoDB table mappings with validation throughout the mirror configuration pipeline. Pull request #4818 - BigQuery query CDC settings can now be updated live when resuming paused mirrors through optional
query_cdcpayload. Pull request #4846 - BigQuery CDC now uses maximum watermark column value as checkpoint and scans subsequent non-overlapping windows when bounded query windows are empty. Pull request #4829
- BigQuery query CDC poll windows are now configurable per mirror with
query_cdc_safety_lag_secondsandquery_cdc_max_query_window_secondssettings. Pull request #4819 - BigQuery CDC Avro staging files are now chunked to handle extremely large objects from long sync intervals. Pull request #4801
- Database migrations now run in a separate flow-migrate container on startup rather than automatically within Nexus. Pull request #4826
- Flow CI was reorganized into granular jobs with build tag support and strict unit test selection at package level. Pull request #4849
- MongoDB changestream errors are now retried to handle optimized execution path toggles in MongoDB 9.0. Pull request #4841
- BigQuery storage read API now works with appends and changes methods by projecting change stream columns with _PEERDB prefix. Pull request #4824
Why it matters
MongoDB structured ingestion is now fully integrated with CDC and QRep workflows, enabling more sophisticated data pipeline configurations. BigQuery CDC improvements include configurable poll windows, better checkpoint handling, and chunked file staging for large-scale syncs. Infrastructure changes to database migrations improve reliability and modularity.
Changelog entry
- Feat(structured-ingestion-mongodb): DBI-1096 structured ingestion processing for MongoDB with QRepValueIterator and SchemaProjector integration Pull request #4845
- Feat(structured-ingestion-mongodb): DBI-1096 structured ingestion settings through table mappings with validation Pull request #4818
- Feat(bigquery): support live updates to query CDC settings when resuming paused mirrors Pull request #4846
- Feat(bigquery): use maximum watermark column value as checkpoint with subsequent window scanning Pull request #4829
- Feat(bigquery): make query CDC poll windows configurable per mirror Pull request #4819
- Feat(bigquery): chunk BigQuery CDC Avro staging files for large sync intervals Pull request #4801
- Feat(database): migrate to goose and run migrations in separate flow-migrate container Pull request #4826
- Chore(ci): reorganize flow CI into granular jobs with build tag support and strict unit test selection Pull request #4849
- Fix(mongodb): retry changestream errors for optimized execution path toggles Pull request #4841
- Fix(bigquery): enable storage read API for appends and changes methods by projecting change stream columns Pull request #4824
- Fix(clickhouse): skip table capacity check when getServerSetting access is denied Pull request #4838
- Test(clickhouse): add explicit test cases for missing permissions validation Pull request #4842
- Test(bigquery): add more e2e test scenarios for different ingestion methods Pull request #4831
MongoDB structured ingestion is live. BigQuery CDC gets configurable poll windows and smarter checkpoint handling. Database migrations now run in their own container for better reliability.
Three major areas shipped this week: MongoDB structured ingestion is now fully integrated with CDC and QRep workflows for more sophisticated configurations. BigQuery CDC received several improvements including live-updatable query settings, configurable per-mirror poll windows, and intelligent checkpoint management using maximum watermark values. Infrastructure upgrades moved database migrations to a separate container for improved reliability and modularity.
Week of September 14, 2026
What shipped
- BigQuery connector now uses storage read API client, improving throughput from 1.8 MB/s to ~18 MB/s for sources with heavy data volumes. Pull request #4797
- BigQuery CDC scheduling now uses
last_synced_atto determine when a sync completed successfully, improving reliability of scheduled pulls. Pull request #4810 - BigQuery connector now selects only configured columns via table mappings instead of SELECT * EXCEPT, reducing unnecessary data transfer. Pull request #4740
- Normalization now properly handles Nullable(JSON) columns, required for structured ingestion workflows. Pull request #4813
- MongoDB BSON to JSON conversion moved to flow/pkg, enabling ClickPipes' Discovery MongoDB schema inference. Pull request #4825
- Structured ingestion now accepts NaN and Inf float values, recording them as strings in intermediate raw events. Pull request #4798
- PostgreSQL CDC no longer performs unnecessary JSON marshal roundtrips for JSON columns, reducing CDC latency for large documents. Pull request #4759
- DropFlowSource cleanup retry alerts downgraded from errors to warnings to prevent alert spam during stuck cleanups. Pull request #4807
- MongoDB BSON to QValue converters refactored behind BsonToQValueConverter interface for better type dispatch. Pull request #4802
- ClickHouse allowed domains configuration now trims whitespace and ignores empty entries in comma-separated lists. Pull request #4795
Why it matters
This week includes significant performance improvements for BigQuery and PostgreSQL connectors that reduce latency and increase throughput for high-volume syncs. Structured ingestion support expanded with better JSON handling and MongoDB schema inference. Several reliability fixes address edge cases in CDC scheduling and error reporting.
Changelog entry
- BigQuery: Storage read API client now enabled by default, improving throughput approximately 10x for large datasets Pull request #4797
- BigQuery: CDC scheduling now uses
last_synced_atto detect successful syncs instead of justlast_attempt_atPull request #4810 - BigQuery: Table pulls now select only configured columns via mappings instead of SELECT * EXCEPT Pull request #4740
- PostgreSQL: CDC JSON column handling no longer performs unnecessary marshal/unmarshal roundtrips Pull request #4759
- Normalization: Nullable(JSON) columns now properly handled in normalize queries Pull request #4813
- Structured ingestion: NaN and Inf float values now accepted and recorded as strings Pull request #4798
- MongoDB: BSON to JSON conversion moved to flow/pkg for ClickPipes Discovery schema inference Pull request #4825
- MongoDB: BSON to QValue conversion refactored with per-type converters behind BsonToQValueConverter Pull request #4802
- Alerts: DropFlowSource cleanup retry failures downgraded from error to warning level Pull request #4807
- ClickHouse: Allowed domains configuration now trims whitespace and ignores empty entries Pull request #4795
- ClickHouse: Type combinations with Nullable and LowCardinality no longer require explicit listing Pull request #4799
- MongoDB: CDC now reports receive/process/addRecord timing metrics Pull request #4788
This week: BigQuery storage read API now live (18MB/s throughput), CDC scheduling reliability improved with last_synced_at tracking, PostgreSQL JSON handling optimized, and structured ingestion gains Nullable(JSON) support.
This week's improvements focus on performance and reliability. BigQuery connector throughput increased 10x with storage read API integration. PostgreSQL CDC latency reduced by eliminating unnecessary JSON parsing. BigQuery CDC scheduling now tracks successful syncs separately from attempts. Structured ingestion gains support for Nullable(JSON) columns and NaN/Inf float values. MongoDB schema inference enabled via BSON conversion refactoring.