Releases: xataio/pgstream
Release list
v1.2.0
⚡ Highlights
🩺 New check command: epxerimental preflight validation before you run pgstream
🔐 New encrypted_aes_siv transformer
🗂 Support for "schema only" tables
🚀 Parallelized COPY workers for faster snapshots
🎯 Configurable transformer error policy
🛡 Strict mode: fail snapshots on dropped Postgres writes
🌙 Automated nightly & container image releases
What's Changed
New features
- New command:
checkby @kvch in #901 - Add schema compatibility validation for pgstream by @kv1sidisi in #896
- Add source and target postgres connectivity checks by @kvch in #902
- Add replication checks by @kvch in #904
- Add replica identity check by @kvch in #911
- Add preflight check for source SELECT privileges by @lghuy05 in #912
- Add preflight check for source sequence SELECT privileges by @lghuy05 in #947
- Move schema validation to preflight checks by @kvch in #951
- [Preflight] Check source and target extensions compatibility by @kvch in #952
- [Preflight] New category:
resourcesby @kvch in #971 - Quote schema/table/role identifiers in access check messages by @kvch in #944
- Refactoring of check category connections by @kvch in #905
- Add encrypted_aes_siv transformer by @geo-ocean in #998
- Add support for "schema only" tables by @tsg in #999
- Add new
TableSelectionfor filtering tables by @kvch in #949 - [Transformers] Configurable error policy by @kvch in #958
- Add strict mode to
WALWriterand counter for dropped messages by @kvch in #959 - Fail snapshots on dropped Postgres writes if in strict mode by @danddanddand in #976
- Parallelize COPY workers by @kvch in #995
- Exclude PostgreSQL temp schemas from snapshot discovery by @danddanddand in #954
- Skip legacy public PL/pgSQL handler functions on snapshot restore by @kvch in #906
- Skip pgstream metadata DML for Postgres targets by @danddanddand in #977
- Improve SQL parameter binding and escaping by @kvch in #965
- validate: Exit code 1 if validation fails by @fiws in #948
- Reload log level on SIGHUP by @pollychen-lab in #945
- Resolve column types & timestamp once per result set by @kvch in #950
- Check log level before logging in hot path by @kvch in #969
- Call
WalDataToDDLEventonce per event in the processor chain by @kvch in #970
Bug fixes
- Correctly treat top level JSONB null value by @tsg in #991
- Fix exclude_tables bug with multiple schemas by @tsg in #992
- Fix snapshot CLI never auto-enabling bulk ingest with flags by @kvch in #993
- Fix batch sender startup race between NewSender and Close by @kvch in #994
- Fix data race in batch-bytes auto-tuner by @kvch in #957
- Fix backoff
MaxIntervalwired toMaxElapsedTimeby @kvch in #956 - [Snapshot] Prevent
GetTablesshadowing outer result slice by @kvch in #960
CI / release
- Automated container image release with attestation by @kvch in #955
- Publish nightly releases by @kvch in #980
Dependency updates
- Bump github.com/testcontainers/testcontainers-go/modules/elasticsearch from 0.42.0 to 0.43.0 by @dependabot[bot] in #913
- Bump github.com/testcontainers/testcontainers-go/modules/postgres from 0.42.0 to 0.43.0 by @dependabot[bot] in #915
- Bump github.com/testcontainers/testcontainers-go/modules/opensearch from 0.42.0 to 0.43.0 by @dependabot[bot] in #916
- Bump github.com/testcontainers/testcontainers-go/modules/kafka from 0.42.0 to 0.43.0 by @dependabot[bot] in #917
- Bump github.com/labstack/echo/v4 from 4.15.2 to 4.15.4 by @dependabot[bot] in #899
- Bump actions/checkout from 6 to 7 by @dependabot[bot] in #898
- Bump github.com/schollz/progressbar/v3 from 3.19.0 to 3.19.1 by @dependabot[bot] in #962
- Bump clickhouse/clickhouse-server from 26.5.1-alpine to 26.6.1.1193-alpine in /build/docker by @dependabot[bot] in #961
- Bump golang.org/x/term from 0.44.0 to 0.45.0 by @dependabot[bot] in #987
- Bump golang.org/x/sync from 0.21.0 to 0.22.0 by @dependabot[bot] in #988
- Bump actions/attest-sbom from 2 to 4 by @dependabot[bot] in #985
- Bump actions/attest-build-provenance from 2 to 4 by @dependabot[bot] in #986
New Contributors
- @kv1sidisi made their first contribution in #896
- @lghuy05 made their first contribution in #912
- @pollychen-lab made their first contribution in #945
- @geo-ocean made their first contribution in #998
Full Changelog: v1.1.1...v1.2.0
Nightly build
Automated nightly build from main (7d863ad). These artifacts are unstable and intended for testing only.
v1.1.1
⚡ Highlights
🔒 SQL injection hardening in the replication path (identifier quoting, slot names, plugin options)
🔄 Materialized views refreshed after snapshot restore
🌳 Inherited / partition child rows excluded from CTID snapshots (ONLY)
🧩 Snapshot restore correctness: CLUSTER ON ordering, int4range/int8range typing, legacy handler functions
What's Changed
Security
- Improve SQL parameter binding and escaping by @kvch in #965
- WAL identifier quoting now validates a single complete quoted token instead of trusting leading/trailing ", so a crafted source object name can't break out onto the target.
- Replication slot
CREATE/DROPbind the slot name as a parameter. START_REPLICATIONescapes table-filter plugin options and validates the slot name before use.
Bug fixes
- Refresh materialized views after data restore by @danddanddand (#908)
- Use ONLY for CTID snapshot queries to exclude inherited child rows by @danddanddand (#909)
- Restore
CLUSTER ONstatements after their referenced indexes by @danddanddand (#907) - Type
int4range/int8rangevalues beforeINSERT/COPYby @danddanddand (#910) - Skip legacy public PL/pgSQL handler functions on snapshot restore by @danddanddand (#906)
New Contributors
- @danddanddand made their first contribution in #906
Full Changelog: v1.1.0...v1.1.1
v0.9.15
What's Changed
Security
- Improve SQL parameter binding and escaping by @kvch in #965
- WAL identifier quoting now validates a single complete quoted token instead of trusting leading/trailing ", so a crafted source object name can't break out onto the target.
- Replication slot
CREATE/DROPbind the slot name as a parameter. START_REPLICATIONescapes table-filter plugin options and validates the slot name before use.
v1.1.0
⚡ Highlights
📜 Environment variables in YAML configuration
📏 Human-readable byte size settings
🪵 Configurable logging format (console / JSON)
🩺 New /health and /ready endpoints
🆕 New stdout target for debugging pipelines
🧹 Optional null-byte sanitizer processor
🌳 ltree and cube type support
🔢 Large integer / bigint precision fix
🔒 Custom TLS for search stores
🛠 --upgrade flag for v0.9.x → v1.x migration
🐛 Bug fixes (composite PKs, constraint ordering, transformer panics, …)
Dependency updates
What's Changed
New features
- feat: support environment variables in YAML configuration by @mvanhorn in #790
- Add support for human readable byte size settings by @kvch in #823
- Add support for changing logging format by @kvch in #880
- Add new healthcheck endpoints:
/healthand/readyby @kvch in #882 - New target:
stdoutby @kvch in #883 - Add optional processor to strip null bytes from string columns by @AasheeshLikePanner in #871
- Add support for ltree by @kvch in #834
- Add support for cube data type by @kvch in #831
- Add support for custom TLS settings for the search stores by @kvch in #832
- Add upgrade flag that cleanups 0.9 objects and reinits by @tsg in #788
Bug fixes
- Fix precision loss on large integers and bigint by @kvch in #833
- Support composite primary keys in WAL metadata by @blurskye in #805
- Restore constraints before snapshot conflict updates by @blurskye in #806
- Eliminate panics in transformers when multiple workers are configured by @kvch in #884
- Register pgvector types by @rieg-ec in #809
- Collect all error messages before stopping pgstream by @kvch in #841
- Make includeGlobalDBObjects work for schema all by @tsg in #760
- Update jackc/pgx to v5.9.2 to resolve CVEs by @fiws in #810
Dependency updates
- Bump go.opentelemetry.io/otel/trace from 1.41.0 to 1.42.0 by @dependabot[bot] in #765
- Bump docker/setup-buildx-action from 3 to 4 by @dependabot[bot] in #766
- Bump opensearchproject/opensearch from 3.5.0 to 3.6.0 in /build/docker by @dependabot[bot] in #794
- Bump github.com/testcontainers/testcontainers-go/modules/kafka from 0.40.0 to 0.42.0 by @dependabot[bot] in #813
- Bump go.opentelemetry.io/contrib/instrumentation/runtime by @dependabot[bot] in #825
- Bump github.com/testcontainers/testcontainers-go/modules/postgres by @dependabot[bot] in #826
- Bump clickhouse/clickhouse-server in /build/docker by @dependabot[bot] in #828
- Bump github.com/segmentio/kafka-go from 0.4.50 to 0.4.51 by @dependabot[bot] in #835
- Bump github.com/elastic/go-elasticsearch/v8 from 8.19.3 to 8.19.6 by @dependabot[bot] in #836
- Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc by @dependabot[bot] in #837
- Bump github.com/labstack/echo/v4 from 4.15.1 to 4.15.2 by @dependabot[bot] in #838
- Bump github.com/lib/pq from 1.11.2 to 1.12.3 by @dependabot[bot] in #839
- Bump dependabot/fetch-metadata from 2 to 3 by @dependabot[bot] in #849
- Bump pgregory.net/rapid from 1.2.0 to 1.3.0 by @dependabot[bot] in #856
- Bump github.com/go-viper/mapstructure/v2 from 2.4.0 to 2.5.0 by @dependabot[bot] in #859
- Bump github.com/jackc/pgx/v5 from 5.9.2 to 5.10.0 by @dependabot[bot] in #861
- Bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc by @dependabot[bot] in #862
- Bump github.com/rs/zerolog from 1.34.0 to 1.35.1 by @dependabot[bot] in #863
- Bump github.com/pgvector/pgvector-go from 0.3.0 to 0.4.0 by @dependabot[bot] in #865
- Update golang/x/crypto to v0.52.0 by @dependabot[bot] in #872
- Bump golang.org/x/sync from 0.20.0 to 0.21.0 by @dependabot[bot] in #873
- Bump opensearchproject/opensearch from 3.6.0 to 3.7.0 in /build/docker by @dependabot[bot] in #888
- Bump opensearchproject/opensearch-dashboards from 3.6.0 to 3.7.0 in /build/docker by @dependabot[bot] in #889
- Bump golang.org/x/term from 0.43.0 to 0.44.0 by @dependabot[bot] in #890
New Contributors
- @mvanhorn made their first contribution in #790
- @SAY-5 made their first contribution in #802
- @blurskye made their first contribution in #805
- @rieg-ec made their first contribution in #809
- @fiws made their first contribution in #810
- @AasheeshLikePanner made their first contribution in #871
Full Changelog: v1.0.3...v1.1.0
v0.9.14
⚡ Highlights
🆕 tsvector type support
🛠 CLI docs/errors improvements for snapshot and run
🐛 Bug fixes: transformer panics with multiple workers, error collection on shutdown, pgvector type registration (all backported)
🔒 pgx upgraded to v5.9.2 to resolve CVEs
Dependency updates
What's Changed
Bug fixes
- Add tsvector type support in DML adapter (#796) (#845) by @xata-bot in #845
- cli: improve docs/errors around snapshot/run commands (#807) (#846) by @xata-bot in #846
- Eliminate panics in transformers when multiple workers are configured by @xata-bot in #886
- Collect all error messages before stopping pgstream by @xata-bot in #887
- fix(postgres): register pgvector types (#809) (#844) by @kvch in #844
- update jackc/pgx to v5.9.2 to resolve CVEs (#810) (#842) by @xata-bot in #842
Dependency updates
- Bump github.com/segmentio/kafka-go from 0.4.50 to 0.4.51 (#835) (#843) by @xata-bot in #843
- Bump fgrosse/go-coverage-report from 1.2.0 to 1.3.0 by @dependabot[bot] in #850
- Bump clickhouse/clickhouse-server in /build/docker by @dependabot[bot] in #851
- Bump opensearchproject/opensearch from 3.4.0 to 3.6.0 in /build/docker by @dependabot[bot] in #852
- Bump opensearchproject/opensearch-dashboards in /build/docker by @dependabot[bot] in #853
- Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc by @dependabot[bot] in #854
- Bump github.com/testcontainers/testcontainers-go from 0.40.0 to 0.42.0 by @dependabot[bot] in #855
- Bump github.com/labstack/echo/v4 from 4.15.0 to 4.15.2 by @dependabot[bot] in #866
- Bump github.com/bytedance/sonic from 1.15.0 to 1.15.2 by @dependabot[bot] in #867
- Bump github.com/elastic/go-elasticsearch/v8 from 8.19.1 to 8.19.6 by @dependabot[bot] in #868
- Bump github.com/pgvector/pgvector-go from 0.3.0 to 0.4.0 by @dependabot[bot] in #869
- Bump go.opentelemetry.io/contrib/instrumentation/runtime from 0.64.0 to 0.69.0 by @dependabot[bot] in #870
- Bump pgregory.net/rapid from 1.2.0 to 1.3.0 by @dependabot[bot] in #874
- Bump github.com/rs/zerolog from 1.34.0 to 1.35.1 by @dependabot[bot] in #875
- Bump github.com/jackc/pgx/v5 from 5.9.2 to 5.10.0 by @dependabot[bot] in #876
- Bump golang.org/x/sync from 0.20.0 to 0.21.0 by @dependabot[bot] in #877
- Bump github.com/tidwall/gjson from 1.18.0 to 1.19.0 by @dependabot[bot] in #878
- Bump opensearchproject/opensearch-dashboards from 3.6.0 to 3.7.0 in /build/docker by @dependabot[bot] in #891
- Bump opensearchproject/opensearch from 3.6.0 to 3.7.0 in /build/docker by @dependabot[bot] in #892
- Bump github.com/lib/pq from 1.11.1 to 1.12.3 by @dependabot[bot] in #893
- Bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc from 1.40.0 to 1.44.0 by @dependabot[bot] in #894
- Bump github.com/nucleuscloud/neosync from 0.5.40 to 0.5.41 by @dependabot[bot] in #895
Full Changelog: v0.9.13...v0.9.14
v1.0.3
Changelog
- 889b9ae Add tsvector type support in DML adapter (#796)
- 86fcc6a Bump clickhouse/clickhouse-server in /build/docker (#816)
- b864f19 Bump docker/login-action from 3 to 4 (#768)
- fec59a2 Bump github.com/bytedance/sonic from 1.15.0 to 1.15.1 (#814)
- ddb617d Bump github.com/tidwall/gjson from 1.18.0 to 1.19.0 (#812)
- f823f77 Bump go.opentelemetry.io/otel from 1.41.0 to 1.43.0 (#761)
- b4c069d Bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc (#815)
- 415ae34 Bump goreleaser/goreleaser-action from 6 to 7 (#738)
- 106fa0c Bump opensearchproject/opensearch-dashboards in /build/docker (#793)
- ba7ad14 Fix schema restore failure with partitioned tables and add
--data-onlyflag (#775) - 9eaba98 Fix update failres on generated columns (#820)
- ada4af6 cli: improve docs/errors around snapshot/run commands (#807)
- 78f50b1 docs(xata): add cnpg_pooler_pgbouncer to Xata-managed roles (#819)
v0.9.13
v0.9.12
v1.0.2
⚡ Highlights
🔀 Batch WAL operations in stream mode
🆔 New settings for wal2json plugin: add_tables and filter_tables
🐛 Bug fixes
Dependency updates
What's Changed
- 1dc7cc1 Add
add_tablesandfilter_tableswal2jsonplugin options (#791) - 2e6db21 Added CLAUDE.md file (#734)
- 59a723d Batch WAL operations in stream mode (#774)
- a6ad9fe Exclude
pgstreamschema when cleaning up target database (#740) - 757f3fa Rename recorder schema workers -> snapshot workers
- 82b7a3f Set concurrency limit on snapshot recorder
- 9888095 Surface error even if we cannot parse ERROR lines in the output of restore (#771)
- 3b757ab Update configuration.md for new
wal2jsonplugin settings (#803) - 8b50ade Update Golang to 1.26.3 (#804)
- b7f3546 Use
IS NULLwhen building update queryWHEREclause