What PeerDB-io/peerdb shipped
Written by FoxPlug from the repository's own releases; not affiliated with Peerdb. Paste your own repository in the box below and FoxPlug writes the same weekly update for your product. This page is an automatic summary of the releases, pull requests and commits in github.com/PeerDB-io/peerdb; Peerdb did not write it and does not use or endorse FoxPlug. Every line links to the change it describes.
Get a weekly update like this for your product, free
Week of November 2, 2026
What shipped
- Released v0.37.15 with fixes for snapshot columns, GUC pinning, and reverted ClickHouse JSON check. Release
- Classified ClickHouse Avro Date out-of-range errors as destination modified for better error handling. Pull request #5021
- Added support for server-side changeStreamPreAndPostImages to improve CDC throughput on sharded clusters. Pull request #4990
- Wrapped BigQuery source time and export errors for better classification. Pull request #5018
- Improved ClickHouse normalize performance by loading multiple staged batches per raw table insert. Pull request #4996
- Fixed snapshot to only include columns that existed at the snapshot time, preventing failures when columns are added during load. Pull request #5004
- Pinned GUCs that affect text transfer to prevent silent data changes during snapshot and CDC. Pull request #5002
- Fixed snapshot to add NULL partition when partition key is excluded, preventing silent row skipping. Pull request #5032
- Made normalize start on pending batches after a SyncFlow restart, so batches are not left unnormalized. Pull request #5034
- Classified DocumentDB too many cursors error as notify user to improve error handling. Pull request #5030
Why it matters
This week's release and fixes improve reliability and data correctness across connectors, especially for snapshot and CDC scenarios. Users benefit from fewer silent failures and better error classification, making pipelines more robust.
Changelog entry
- Release v0.37.15 with fixes for snapshot columns, GUC pinning, and reverted ClickHouse JSON check. Release
- Classify ClickHouse Avro Date out-of-range as destination modified. Pull request #5021
- Support server-side changeStreamPreAndPostImages for better CDC throughput. Pull request #4990
- Wrap BigQuery source time and export errors for classification. Pull request #5018
- Improve ClickHouse normalize performance by loading multiple staged batches per raw table insert. Pull request #4996
- Snapshot only columns that existed at the snapshot. Pull request #5004
- Pin GUCs that change values in text transfer. Pull request #5002
- Add NULL partition when partition key is excluded in snapshot. Pull request #5032
- Make normalize start on pending batches after a SyncFlow restart. Pull request #5034
- Classify DocumentDB too many cursors error as notify user. Pull request #5030
- Batch hot-path metric updates in MySQL connector to reduce CPU usage. Pull request #5012
- Add docs/tricky-areas.md to highlight code areas needing extra care. Pull request #5013
PeerDB v0.37.15 released with fixes for snapshot columns, GUC pinning, and ClickHouse JSON check. Also improved error classification and performance for ClickHouse and BigQuery.
PeerDB v0.37.15 is out, including fixes for snapshot column handling, GUC pinning to prevent silent data changes, and a reverted ClickHouse JSON check. This week also brought better error classification for ClickHouse and BigQuery, improved normalize performance, and support for server-side changeStreamPreAndPostImages. These changes make pipelines more reliable and data transfer more accurate.
Week of October 26, 2026
What shipped
- MySQL connector session
sql_modeno longer includes NO_BACKSLASH_ESCAPES, so tables with generated columns containing quotes can be queried again. Pull request #4985 - With ClickHouse native JSON enabled, Postgres and MySQL readers now wrap non-object JSON values so scalar, array and null documents sync instead of failing. Pull request #4962
- An optional client-side update lookup was added so MongoDB sharded cluster change streams are not limited by sequential updateLookup when update rates are high. Pull request #4974
- Sampling-based snapshot partitioning now covers ObjectID and is the default option, extending the approach already used for string type. Pull request #4970
- Shopspring/decimal was updated to v1.5.0 and the decode exponent limit raised to 131072 so large numeric values PostgreSQL supports are not rejected. Pull request #4976
- Pgx was updated to v5.11.0 with UTC timestamp codecs centralized across shared connection and standalone type-map constructors, including arrays, ranges and multiranges. Pull request #4975
- Version v0.37.14 was released with security fixes from dependency upgrades, the pgx and decimal updates, and the MongoDB ObjectID sampling range fix. Release
- The ClickHouse JSON readback test helper passes a setting to restore the older output format so synced e2e value checks work on ClickHouse 26.10 and later. Pull request #4978
Why it matters
This week fixes several blockers people hit in real pipelines: MySQL generated columns with quotes, scalar and array JSON into ClickHouse native JSON, and large numerics after the decimal upgrade. MongoDB sharded users get an option to lift CDC read throughput and better default ObjectID snapshot partitioning. The v0.37.14 release packages the dependency security fixes and the timestamp decoding preservation.
Changelog entry
- Release v0.37.14 with security fixes from dependency upgrades. Release
- MySQL connector: removed NO_BACKSLASH_ESCAPES from session
sql_modeso generated columns with quotes no longer break queries on the table. Pull request #4985 - Postgres/MySQL readers: wrap non-object JSON as objects so ClickHouse native JSON accepts scalar, array and null values. Pull request #4962
- MongoDB: added optional client-side update lookup to avoid sequential updateLookup limiting CDC read throughput on sharded clusters. Pull request #4974
- MongoDB: sampling-based snapshot partitioning now supports ObjectID and is the default option. Pull request #4970
- Updated shopspring/decimal to v1.5.0 and raised MaxDecodeExponent to 131072 to keep large numeric values. Pull request #4976
- Updated pgx to v5.11.0 and centralized UTC timestamp codecs for arrays, ranges and multiranges. Pull request #4975
- ClickHouse JSON test readback: pass a setting to restore the old output format on ClickHouse 26.10 and later. Pull request #4978
PeerDB v0.37.14 is out with security fixes from dependency upgrades, preserving timestamp and large numeric decoding, and a MongoDB ObjectID sampling range fix. Also this week: MySQL no longer hits NO_BACKSLASH_ESCAPES errors on generated columns with quotes, non-object JSON sync
PeerDB v0.37.14 is released, including security fixes from dependency upgrades, updated pgx and decimal with preserved timestamp and large numeric decoding, and a MongoDB ObjectID sampling range fix. Other changes this week: the MySQL connector no longer sets NO_BACKSLASH_ESCAPES in its session sql_mode, so tables with generated columns containing quotes work again. With ClickHouse native JSON enabled, Postgres and MySQL non-object JSON values are wrapped so scalar, array and null documents sync rather than fail on the first one. MongoDB sharded clusters get an optional client-side update lookup to avoid sequential updateLookup limiting CDC read throughput at high update rates, and sampling-based snapshot partitioning now supports ObjectID as the default. A test helper change restores Clic
Week of October 12, 2026
What shipped
- Released v0.37.12, which includes optional pre-image on MongoDB delete events, IAM role auth for Amazon DocumentDB, and an updated aws-rds-bundle-shasum in the Dockerfile. Release
- MongoDB CDC delete events can now optionally carry the change stream pre-image, so destination columns other than _id are no longer left empty. Pull request #4844
- Mongo peers can now use an AWS IAM role for Amazon DocumentDB instead of a stored username and password, tested on 5.0, with existing peers unchanged. Pull request #4919
- Table additions to running query-based BigQuery CDC mirrors now get a snapshot and a CDC checkpoint instead of failing to set up replication and reading stale table mappings. Pull request #4769
- Postgres inheritance children with their own column layout are now decoded correctly when handling inheritance for non-partitioned tables is enabled. Pull request #4930
- BigQuery CDC time travel exceeded errors are now classified as NOTIFY_BIGQUERY_TIME_TRAVEL_EXCEEDED so users are told a fresh initial load is needed. Pull request #4895
- S3 upload errors with HTTP 500 to 503 are now classified as retryable, since observed failures recovered when retried at the Temporal level. Pull request #4910
- MySQL lock wait timeout errors are now classified as retry recoverable instead of paging on-call. Pull request #4916
- A parse JSON error is now reported to the user when the exception is encountered. Pull request #4912
- MySQL no space left on device errors are now classified as connectivity notifications rather than other errors that page on-call. Pull request #4915
Why it matters
This week adds options MongoDB users have asked for: pre-image data on deletes and IAM role auth for DocumentDB, both in v0.37.12. Error classification changed across S3, GCS, MySQL and BigQuery so transient failures retry and only customer-fixable problems notify. BigQuery mirrors also gained fixes for added tables and time travel limits.
Changelog entry
- Released flow-api-client v0.9.5. Release
- Added org id to the template. Pull request #4928
- Released v0.37.12 with MongoDB pre-image on delete, DocumentDB IAM role auth, and Dockerfile aws-rds-bundle-shasum update. Release
- MongoDB CDC delete events can optionally carry the change stream pre-image. Pull request #4844
- Mongo peers can use AWS IAM role auth for Amazon DocumentDB. Pull request #4919
- Fixed Postgres inheritance children with their own column layout being decoded incorrectly. Pull request #4930
- Added tables in running query-based BigQuery CDC mirrors now get a snapshot and CDC checkpoint. Pull request #4769
- Classified BigQuery time travel exceeded as NOTIFY_BIGQUERY_TIME_TRAVEL_EXCEEDED. Pull request #4895
- Classified S3 HTTP 5xx upload errors as retryable. Pull request #4910
- Classified MySQL lock wait timeout as retry recoverable. Pull request #4916
- Expanded transient GCS error classification cases. Pull request #4911
- Notify users when a parse JSON exception is encountered. Pull request #4912
PeerDB v0.37.12 is out: MongoDB delete events can carry the change stream pre-image, DocumentDB peers can use IAM role auth, and BigQuery mirror table additions now get a snapshot and CDC checkpoint.
PeerDB v0.37.12 is out. MongoDB CDC delete events can now optionally carry the change stream pre-image, so destination tables keyed on a document field no longer receive empty values. DocumentDB peers can now use an AWS IAM role instead of a stored password. Running query-based BigQuery CDC mirrors now give added tables a snapshot and a CDC checkpoint. Error handling also improved: S3 and GCS transient errors retry, MySQL lock wait timeouts and no-space errors are classified more accurately, and BigQuery time travel exceeded surfaces as its own notification.
Week of October 5, 2026
What shipped
- Added GCP Cloud SQL IAM database authentication for Postgres and MySQL peers through the Cloud SQL Go connector. Pull request #4881
- Released v0.37.11 with Cloud SQL IAM auth, MongoDB structured ingestion performance work and cron schedules for query-based CDC polls. Release
- Added a
sync_cronoption to query-based CDC so polls can run on a standard cron schedule instead of a fixed interval. Pull request #4883 - Reduced memory and CPU use in MongoDB structured ingestion based on profiling. Pull request #4890
- Fixed handling of Postgres domain types during CDC. Pull request #4903
- Fixed query-based CDC so records are staged without opening a ClickHouse connection first, which previously left mirrors stuck. Pull request #4884
- Upgraded pglogrepl to support zero-column inserts. Pull request #4908
- Stopped issuing a binlog dump KILL when the syncer closes over SSH, removing intermittent tunnel dial errors in MySQL CDC over SSH. Pull request #4878
- Classified transient BigQuery errors as retry-recoverable so they no longer fire critical alerts. Pull request #4887
- Classified the MySQL handshake error caused by a server without TLS support so users are told they need to enable TLS. Pull request #4907
Why it matters
This week adds a new way to authenticate: Cloud SQL IAM auth for Postgres and MySQL peers. It also fixes several source-side issues that could stall or alert on pipelines, including Postgres domain types, query-based CDC stuck without ClickHouse, MySQL binlog dumps over SSH and transient BigQuery errors. Query-based CDC can now poll on a cron schedule, and MongoDB structured ingestion uses less memory and CPU.
Changelog entry
- Support GCP Cloud SQL IAM auth for Postgres and MySQL peers through the Cloud SQL connector. Pull request #4881
- Add
sync_cronto query-based CDC for cron-scheduled polls. Pull request #4883 - Optimize MongoDB structured ingestion for memory and CPU use. Pull request #4890
- Fix Postgres domain type handling during CDC. Pull request #4903
- Fix query-based CDC to stage records without connecting to ClickHouse. Pull request #4884
- Upgrade pglogrepl to support zero-column inserts. Pull request #4908
- Skip binlog dump KILL on syncer close over SSH. Pull request #4878
- Classify transient BigQuery errors as retry-recoverable. Pull request #4887
- Classify MySQL handshake errors caused by missing server TLS support. Pull request #4907
- Classify the MongoDB pool connection acquisition timeout error. Pull request #4899
- Align MongoDB structured ingestion types and add an initial load plus CDC test. Pull request #4896
- Release v0.37.11. Release
V0.37.11 is out. Cloud SQL IAM auth for Postgres and MySQL peers, cron schedules for query-based CDC polls, MongoDB structured ingestion performance work, plus fixes for Postgres domain types, query-CDC staging and MySQL binlog over SSH.
PeerDB v0.37.11 is released. This week adds GCP Cloud SQL IAM database authentication for Postgres and MySQL peers through the Cloud SQL connector, and query-based CDC can now poll on a cron schedule instead of a fixed interval. MongoDB structured ingestion got memory and CPU optimizations from profiling. Fixes include Postgres domain type handling during CDC, query-based CDC staging records without a ClickHouse connection, no binlog dump KILL on syncer close over SSH, transient BigQuery errors treated as retry-recoverable, and clearer classification for MySQL servers without TLS support.
Week of September 28, 2026
What shipped
- MongoDB structured ingestion now supports native array types instead of landing them as JSON columns, with uniform element types. Pull request #4858
- BSON DateTime values in MongoDB structured ingestion now convert to timestamps correctly instead of being treated as type mismatches. Pull request #4876
- Column exclusions in MongoDB structured ingestion now propagate to projection logic, preventing excluded columns from appearing as unexpected fields. Pull request #4879
- BigQuery mirrors now display CDC table counts for inserts, updates, and deletes tracked from the source. Pull request #4871
- MySQL pipe creation is faster by avoiding unnecessary KEY_COLUMN_USAGE joins that caused full table scans on MySQL 5.7. Pull request #4861
- Query CDC for BigQuery now respects safety lag windows to avoid polling during unsafe time periods. Pull request #4832
- Test selection speed improved by batching package discovery and running concurrent operations instead of sequential per-package checks. Pull request #4854
- Query CDC pipes now emit metrics for batch fetches and current batch IDs to improve observability. Pull request #4859
- Cloud SQL IAM authentication refactored to align with RDS authentication patterns and enforce TLS configuration. Pull request #4872
- CI crashes from concurrent map writes are prevented with an RWMutex protecting dynconf maps. Pull request #4869
Why it matters
MongoDB structured ingestion improvements make arrays and dates work correctly out of the box, reducing data quality issues. BigQuery CDC gets better operational visibility and safety controls. Performance improvements across MySQL validation and test selection speed up development cycles.
Changelog entry
- MongoDB structured ingestion supports native array types with uniform element detection Pull request #4858
- MongoDB structured ingestion converts BSON DateTime to timestamps instead of string representations Pull request #4876
- MongoDB structured ingestion excludes columns in projection logic Pull request #4879
- BigQuery mirrors display CDC table counts for insert, update, and delete operations Pull request #4871
- MySQL pipe creation faster by skipping KEY_COLUMN_USAGE join on MySQL 5.7 Pull request #4861
- Query CDC skips polling during safety lag window Pull request #4832
- Test selection runs faster with concurrent package discovery Pull request #4854
- Query CDC emits batch fetch and batch ID metrics Pull request #4859
- Cloud SQL IAM authentication aligned with RDS auth and enforces TLS Pull request #4872
- CI stability improved with RWMutex protecting concurrent map access Pull request #4869
MongoDB structured ingestion now handles arrays, dates, and column exclusions correctly. BigQuery mirrors show CDC table counts. MySQL pipes validate faster. Query CDC respects safety windows.
This week's updates improve data accuracy and operational safety across multiple connectors. MongoDB structured ingestion now correctly handles native array types, converts BSON dates to timestamps, and respects column exclusions—eliminating data quality issues. BigQuery mirrors now display CDC table counts for better visibility. MySQL validation is faster by avoiding unnecessary table scans. Query CDC for BigQuery respects safety lag windows to prevent unsafe operations. These changes reduce troubleshooting and deliver more reliable replication.
Week of September 21, 2026
What shipped
- Added structured ingestion processing for MongoDB with QRepValueIterator for BSON documents and SchemaProjector integration for CDC and QRep workflows. Pull request #4845
- Introduced
structured_ingestionanddrop_unexpected_valuesconfiguration options for MongoDB table mappings with validation throughout the mirror configuration pipeline. Pull request #4818 - BigQuery query CDC settings can now be updated live when resuming paused mirrors through optional
query_cdcpayload. Pull request #4846 - BigQuery CDC now uses maximum watermark column value as checkpoint and scans subsequent non-overlapping windows when bounded query windows are empty. Pull request #4829
- BigQuery query CDC poll windows are now configurable per mirror with
query_cdc_safety_lag_secondsandquery_cdc_max_query_window_secondssettings. Pull request #4819 - BigQuery CDC Avro staging files are now chunked to handle extremely large objects from long sync intervals. Pull request #4801
- Database migrations now run in a separate flow-migrate container on startup rather than automatically within Nexus. Pull request #4826
- Flow CI was reorganized into granular jobs with build tag support and strict unit test selection at package level. Pull request #4849
- MongoDB changestream errors are now retried to handle optimized execution path toggles in MongoDB 9.0. Pull request #4841
- BigQuery storage read API now works with appends and changes methods by projecting change stream columns with _PEERDB prefix. Pull request #4824
Why it matters
MongoDB structured ingestion is now fully integrated with CDC and QRep workflows, enabling more sophisticated data pipeline configurations. BigQuery CDC improvements include configurable poll windows, better checkpoint handling, and chunked file staging for large-scale syncs. Infrastructure changes to database migrations improve reliability and modularity.
Changelog entry
- Feat(structured-ingestion-mongodb): DBI-1096 structured ingestion processing for MongoDB with QRepValueIterator and SchemaProjector integration Pull request #4845
- Feat(structured-ingestion-mongodb): DBI-1096 structured ingestion settings through table mappings with validation Pull request #4818
- Feat(bigquery): support live updates to query CDC settings when resuming paused mirrors Pull request #4846
- Feat(bigquery): use maximum watermark column value as checkpoint with subsequent window scanning Pull request #4829
- Feat(bigquery): make query CDC poll windows configurable per mirror Pull request #4819
- Feat(bigquery): chunk BigQuery CDC Avro staging files for large sync intervals Pull request #4801
- Feat(database): migrate to goose and run migrations in separate flow-migrate container Pull request #4826
- Chore(ci): reorganize flow CI into granular jobs with build tag support and strict unit test selection Pull request #4849
- Fix(mongodb): retry changestream errors for optimized execution path toggles Pull request #4841
- Fix(bigquery): enable storage read API for appends and changes methods by projecting change stream columns Pull request #4824
- Fix(clickhouse): skip table capacity check when getServerSetting access is denied Pull request #4838
- Test(clickhouse): add explicit test cases for missing permissions validation Pull request #4842
- Test(bigquery): add more e2e test scenarios for different ingestion methods Pull request #4831
MongoDB structured ingestion is live. BigQuery CDC gets configurable poll windows and smarter checkpoint handling. Database migrations now run in their own container for better reliability.
Three major areas shipped this week: MongoDB structured ingestion is now fully integrated with CDC and QRep workflows for more sophisticated configurations. BigQuery CDC received several improvements including live-updatable query settings, configurable per-mirror poll windows, and intelligent checkpoint management using maximum watermark values. Infrastructure upgrades moved database migrations to a separate container for improved reliability and modularity.
Week of September 14, 2026
What shipped
- BigQuery connector now uses storage read API client, improving throughput from 1.8 MB/s to ~18 MB/s for sources with heavy data volumes. Pull request #4797
- BigQuery CDC scheduling now uses
last_synced_atto determine when a sync completed successfully, improving reliability of scheduled pulls. Pull request #4810 - BigQuery connector now selects only configured columns via table mappings instead of SELECT * EXCEPT, reducing unnecessary data transfer. Pull request #4740
- Normalization now properly handles Nullable(JSON) columns, required for structured ingestion workflows. Pull request #4813
- MongoDB BSON to JSON conversion moved to flow/pkg, enabling ClickPipes' Discovery MongoDB schema inference. Pull request #4825
- Structured ingestion now accepts NaN and Inf float values, recording them as strings in intermediate raw events. Pull request #4798
- PostgreSQL CDC no longer performs unnecessary JSON marshal roundtrips for JSON columns, reducing CDC latency for large documents. Pull request #4759
- DropFlowSource cleanup retry alerts downgraded from errors to warnings to prevent alert spam during stuck cleanups. Pull request #4807
- MongoDB BSON to QValue converters refactored behind BsonToQValueConverter interface for better type dispatch. Pull request #4802
- ClickHouse allowed domains configuration now trims whitespace and ignores empty entries in comma-separated lists. Pull request #4795
Why it matters
This week includes significant performance improvements for BigQuery and PostgreSQL connectors that reduce latency and increase throughput for high-volume syncs. Structured ingestion support expanded with better JSON handling and MongoDB schema inference. Several reliability fixes address edge cases in CDC scheduling and error reporting.
Changelog entry
- BigQuery: Storage read API client now enabled by default, improving throughput approximately 10x for large datasets Pull request #4797
- BigQuery: CDC scheduling now uses
last_synced_atto detect successful syncs instead of justlast_attempt_atPull request #4810 - BigQuery: Table pulls now select only configured columns via mappings instead of SELECT * EXCEPT Pull request #4740
- PostgreSQL: CDC JSON column handling no longer performs unnecessary marshal/unmarshal roundtrips Pull request #4759
- Normalization: Nullable(JSON) columns now properly handled in normalize queries Pull request #4813
- Structured ingestion: NaN and Inf float values now accepted and recorded as strings Pull request #4798
- MongoDB: BSON to JSON conversion moved to flow/pkg for ClickPipes Discovery schema inference Pull request #4825
- MongoDB: BSON to QValue conversion refactored with per-type converters behind BsonToQValueConverter Pull request #4802
- Alerts: DropFlowSource cleanup retry failures downgraded from error to warning level Pull request #4807
- ClickHouse: Allowed domains configuration now trims whitespace and ignores empty entries Pull request #4795
- ClickHouse: Type combinations with Nullable and LowCardinality no longer require explicit listing Pull request #4799
- MongoDB: CDC now reports receive/process/addRecord timing metrics Pull request #4788
This week: BigQuery storage read API now live (18MB/s throughput), CDC scheduling reliability improved with last_synced_at tracking, PostgreSQL JSON handling optimized, and structured ingestion gains Nullable(JSON) support.
This week's improvements focus on performance and reliability. BigQuery connector throughput increased 10x with storage read API integration. PostgreSQL CDC latency reduced by eliminating unnecessary JSON parsing. BigQuery CDC scheduling now tracks successful syncs separately from attempts. Structured ingestion gains support for Nullable(JSON) columns and NaN/Inf float values. MongoDB schema inference enabled via BSON conversion refactoring.