Skip to content

Releases: xataio/pgstream

v1.2.0

Choose a tag to compare

@github-actions github-actions released this 15 Jul 15:14
7d863ad

Highlights

🩺 New check command: epxerimental preflight validation before you run pgstream
🔐 New encrypted_aes_siv transformer
🗂 Support for "schema only" tables
🚀 Parallelized COPY workers for faster snapshots
🎯 Configurable transformer error policy
🛡 Strict mode: fail snapshots on dropped Postgres writes
🌙 Automated nightly & container image releases

What's Changed

New features

  • New command: check by @kvch in #901
  • Add schema compatibility validation for pgstream by @kv1sidisi in #896
  • Add source and target postgres connectivity checks by @kvch in #902
  • Add replication checks by @kvch in #904
  • Add replica identity check by @kvch in #911
  • Add preflight check for source SELECT privileges by @lghuy05 in #912
  • Add preflight check for source sequence SELECT privileges by @lghuy05 in #947
  • Move schema validation to preflight checks by @kvch in #951
  • [Preflight] Check source and target extensions compatibility by @kvch in #952
  • [Preflight] New category: resources by @kvch in #971
  • Quote schema/table/role identifiers in access check messages by @kvch in #944
  • Refactoring of check category connections by @kvch in #905
  • Add encrypted_aes_siv transformer by @geo-ocean in #998
  • Add support for "schema only" tables by @tsg in #999
  • Add new TableSelection for filtering tables by @kvch in #949
  • [Transformers] Configurable error policy by @kvch in #958
  • Add strict mode to WALWriter and counter for dropped messages by @kvch in #959
  • Fail snapshots on dropped Postgres writes if in strict mode by @danddanddand in #976
  • Parallelize COPY workers by @kvch in #995
  • Exclude PostgreSQL temp schemas from snapshot discovery by @danddanddand in #954
  • Skip legacy public PL/pgSQL handler functions on snapshot restore by @kvch in #906
  • Skip pgstream metadata DML for Postgres targets by @danddanddand in #977
  • Improve SQL parameter binding and escaping by @kvch in #965
  • validate: Exit code 1 if validation fails by @fiws in #948
  • Reload log level on SIGHUP by @pollychen-lab in #945
  • Resolve column types & timestamp once per result set by @kvch in #950
  • Check log level before logging in hot path by @kvch in #969
  • Call WalDataToDDLEvent once per event in the processor chain by @kvch in #970

Bug fixes

  • Correctly treat top level JSONB null value by @tsg in #991
  • Fix exclude_tables bug with multiple schemas by @tsg in #992
  • Fix snapshot CLI never auto-enabling bulk ingest with flags by @kvch in #993
  • Fix batch sender startup race between NewSender and Close by @kvch in #994
  • Fix data race in batch-bytes auto-tuner by @kvch in #957
  • Fix backoff MaxInterval wired to MaxElapsedTime by @kvch in #956
  • [Snapshot] Prevent GetTables shadowing outer result slice by @kvch in #960

CI / release

  • Automated container image release with attestation by @kvch in #955
  • Publish nightly releases by @kvch in #980

Dependency updates

  • Bump github.com/testcontainers/testcontainers-go/modules/elasticsearch from 0.42.0 to 0.43.0 by @dependabot[bot] in #913
  • Bump github.com/testcontainers/testcontainers-go/modules/postgres from 0.42.0 to 0.43.0 by @dependabot[bot] in #915
  • Bump github.com/testcontainers/testcontainers-go/modules/opensearch from 0.42.0 to 0.43.0 by @dependabot[bot] in #916
  • Bump github.com/testcontainers/testcontainers-go/modules/kafka from 0.42.0 to 0.43.0 by @dependabot[bot] in #917
  • Bump github.com/labstack/echo/v4 from 4.15.2 to 4.15.4 by @dependabot[bot] in #899
  • Bump actions/checkout from 6 to 7 by @dependabot[bot] in #898
  • Bump github.com/schollz/progressbar/v3 from 3.19.0 to 3.19.1 by @dependabot[bot] in #962
  • Bump clickhouse/clickhouse-server from 26.5.1-alpine to 26.6.1.1193-alpine in /build/docker by @dependabot[bot] in #961
  • Bump golang.org/x/term from 0.44.0 to 0.45.0 by @dependabot[bot] in #987
  • Bump golang.org/x/sync from 0.21.0 to 0.22.0 by @dependabot[bot] in #988
  • Bump actions/attest-sbom from 2 to 4 by @dependabot[bot] in #985
  • Bump actions/attest-build-provenance from 2 to 4 by @dependabot[bot] in #986

New Contributors

Full Changelog: v1.1.1...v1.2.0

Nightly build

Nightly build Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 16 Jul 04:12
7d863ad

Automated nightly build from main (7d863ad). These artifacts are unstable and intended for testing only.

v1.1.1

Choose a tag to compare

@github-actions github-actions released this 06 Jul 11:15
8d8585f

⚡ Highlights

🔒 SQL injection hardening in the replication path (identifier quoting, slot names, plugin options)
🔄 Materialized views refreshed after snapshot restore
🌳 Inherited / partition child rows excluded from CTID snapshots (ONLY)
🧩 Snapshot restore correctness: CLUSTER ON ordering, int4range/int8range typing, legacy handler functions

What's Changed

Security

  • Improve SQL parameter binding and escaping by @kvch in #965
    • WAL identifier quoting now validates a single complete quoted token instead of trusting leading/trailing ", so a crafted source object name can't break out onto the target.
    • Replication slot CREATE/DROP bind the slot name as a parameter.
    • START_REPLICATION escapes table-filter plugin options and validates the slot name before use.

Bug fixes

New Contributors

Full Changelog: v1.1.0...v1.1.1

v0.9.15

Choose a tag to compare

@github-actions github-actions released this 06 Jul 11:21
0ab8e05

What's Changed

Security

  • Improve SQL parameter binding and escaping by @kvch in #965
    • WAL identifier quoting now validates a single complete quoted token instead of trusting leading/trailing ", so a crafted source object name can't break out onto the target.
    • Replication slot CREATE/DROP bind the slot name as a parameter.
    • START_REPLICATION escapes table-filter plugin options and validates the slot name before use.

v1.1.0

Choose a tag to compare

@github-actions github-actions released this 16 Jun 09:25
048acef

Highlights

📜 Environment variables in YAML configuration
📏 Human-readable byte size settings
🪵 Configurable logging format (console / JSON)
🩺 New /health and /ready endpoints
🆕 New stdout target for debugging pipelines
🧹 Optional null-byte sanitizer processor
🌳 ltree and cube type support
🔢 Large integer / bigint precision fix
🔒 Custom TLS for search stores
🛠 --upgrade flag for v0.9.x → v1.x migration
🐛 Bug fixes (composite PKs, constraint ordering, transformer panics, …)
:dependabot: Dependency updates

What's Changed

New features

  • feat: support environment variables in YAML configuration by @mvanhorn in #790
  • Add support for human readable byte size settings by @kvch in #823
  • Add support for changing logging format by @kvch in #880
  • Add new healthcheck endpoints: /health and /ready by @kvch in #882
  • New target: stdout by @kvch in #883
  • Add optional processor to strip null bytes from string columns by @AasheeshLikePanner in #871
  • Add support for ltree by @kvch in #834
  • Add support for cube data type by @kvch in #831
  • Add support for custom TLS settings for the search stores by @kvch in #832
  • Add upgrade flag that cleanups 0.9 objects and reinits by @tsg in #788

Bug fixes

  • Fix precision loss on large integers and bigint by @kvch in #833
  • Support composite primary keys in WAL metadata by @blurskye in #805
  • Restore constraints before snapshot conflict updates by @blurskye in #806
  • Eliminate panics in transformers when multiple workers are configured by @kvch in #884
  • Register pgvector types by @rieg-ec in #809
  • Collect all error messages before stopping pgstream by @kvch in #841
  • Make includeGlobalDBObjects work for schema all by @tsg in #760
  • Update jackc/pgx to v5.9.2 to resolve CVEs by @fiws in #810

Dependency updates

  • Bump go.opentelemetry.io/otel/trace from 1.41.0 to 1.42.0 by @dependabot[bot] in #765
  • Bump docker/setup-buildx-action from 3 to 4 by @dependabot[bot] in #766
  • Bump opensearchproject/opensearch from 3.5.0 to 3.6.0 in /build/docker by @dependabot[bot] in #794
  • Bump github.com/testcontainers/testcontainers-go/modules/kafka from 0.40.0 to 0.42.0 by @dependabot[bot] in #813
  • Bump go.opentelemetry.io/contrib/instrumentation/runtime by @dependabot[bot] in #825
  • Bump github.com/testcontainers/testcontainers-go/modules/postgres by @dependabot[bot] in #826
  • Bump clickhouse/clickhouse-server in /build/docker by @dependabot[bot] in #828
  • Bump github.com/segmentio/kafka-go from 0.4.50 to 0.4.51 by @dependabot[bot] in #835
  • Bump github.com/elastic/go-elasticsearch/v8 from 8.19.3 to 8.19.6 by @dependabot[bot] in #836
  • Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc by @dependabot[bot] in #837
  • Bump github.com/labstack/echo/v4 from 4.15.1 to 4.15.2 by @dependabot[bot] in #838
  • Bump github.com/lib/pq from 1.11.2 to 1.12.3 by @dependabot[bot] in #839
  • Bump dependabot/fetch-metadata from 2 to 3 by @dependabot[bot] in #849
  • Bump pgregory.net/rapid from 1.2.0 to 1.3.0 by @dependabot[bot] in #856
  • Bump github.com/go-viper/mapstructure/v2 from 2.4.0 to 2.5.0 by @dependabot[bot] in #859
  • Bump github.com/jackc/pgx/v5 from 5.9.2 to 5.10.0 by @dependabot[bot] in #861
  • Bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc by @dependabot[bot] in #862
  • Bump github.com/rs/zerolog from 1.34.0 to 1.35.1 by @dependabot[bot] in #863
  • Bump github.com/pgvector/pgvector-go from 0.3.0 to 0.4.0 by @dependabot[bot] in #865
  • Update golang/x/crypto to v0.52.0 by @dependabot[bot] in #872
  • Bump golang.org/x/sync from 0.20.0 to 0.21.0 by @dependabot[bot] in #873
  • Bump opensearchproject/opensearch from 3.6.0 to 3.7.0 in /build/docker by @dependabot[bot] in #888
  • Bump opensearchproject/opensearch-dashboards from 3.6.0 to 3.7.0 in /build/docker by @dependabot[bot] in #889
  • Bump golang.org/x/term from 0.43.0 to 0.44.0 by @dependabot[bot] in #890

New Contributors

Full Changelog: v1.0.3...v1.1.0

v0.9.14

Choose a tag to compare

@github-actions github-actions released this 16 Jun 09:32
1b7c264

Highlights

🆕 tsvector type support
🛠 CLI docs/errors improvements for snapshot and run
🐛 Bug fixes: transformer panics with multiple workers, error collection on shutdown, pgvector type registration (all backported)
🔒 pgx upgraded to v5.9.2 to resolve CVEs
:dependabot: Dependency updates

What's Changed

Bug fixes

Dependency updates

  • Bump github.com/segmentio/kafka-go from 0.4.50 to 0.4.51 (#835) (#843) by @xata-bot in #843
  • Bump fgrosse/go-coverage-report from 1.2.0 to 1.3.0 by @dependabot[bot] in #850
  • Bump clickhouse/clickhouse-server in /build/docker by @dependabot[bot] in #851
  • Bump opensearchproject/opensearch from 3.4.0 to 3.6.0 in /build/docker by @dependabot[bot] in #852
  • Bump opensearchproject/opensearch-dashboards in /build/docker by @dependabot[bot] in #853
  • Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc by @dependabot[bot] in #854
  • Bump github.com/testcontainers/testcontainers-go from 0.40.0 to 0.42.0 by @dependabot[bot] in #855
  • Bump github.com/labstack/echo/v4 from 4.15.0 to 4.15.2 by @dependabot[bot] in #866
  • Bump github.com/bytedance/sonic from 1.15.0 to 1.15.2 by @dependabot[bot] in #867
  • Bump github.com/elastic/go-elasticsearch/v8 from 8.19.1 to 8.19.6 by @dependabot[bot] in #868
  • Bump github.com/pgvector/pgvector-go from 0.3.0 to 0.4.0 by @dependabot[bot] in #869
  • Bump go.opentelemetry.io/contrib/instrumentation/runtime from 0.64.0 to 0.69.0 by @dependabot[bot] in #870
  • Bump pgregory.net/rapid from 1.2.0 to 1.3.0 by @dependabot[bot] in #874
  • Bump github.com/rs/zerolog from 1.34.0 to 1.35.1 by @dependabot[bot] in #875
  • Bump github.com/jackc/pgx/v5 from 5.9.2 to 5.10.0 by @dependabot[bot] in #876
  • Bump golang.org/x/sync from 0.20.0 to 0.21.0 by @dependabot[bot] in #877
  • Bump github.com/tidwall/gjson from 1.18.0 to 1.19.0 by @dependabot[bot] in #878
  • Bump opensearchproject/opensearch-dashboards from 3.6.0 to 3.7.0 in /build/docker by @dependabot[bot] in #891
  • Bump opensearchproject/opensearch from 3.6.0 to 3.7.0 in /build/docker by @dependabot[bot] in #892
  • Bump github.com/lib/pq from 1.11.1 to 1.12.3 by @dependabot[bot] in #893
  • Bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc from 1.40.0 to 1.44.0 by @dependabot[bot] in #894
  • Bump github.com/nucleuscloud/neosync from 0.5.40 to 0.5.41 by @dependabot[bot] in #895

Full Changelog: v0.9.13...v0.9.14

v1.0.3

Choose a tag to compare

@github-actions github-actions released this 20 May 19:51
9eaba98

Changelog

  • 889b9ae Add tsvector type support in DML adapter (#796)
  • 86fcc6a Bump clickhouse/clickhouse-server in /build/docker (#816)
  • b864f19 Bump docker/login-action from 3 to 4 (#768)
  • fec59a2 Bump github.com/bytedance/sonic from 1.15.0 to 1.15.1 (#814)
  • ddb617d Bump github.com/tidwall/gjson from 1.18.0 to 1.19.0 (#812)
  • f823f77 Bump go.opentelemetry.io/otel from 1.41.0 to 1.43.0 (#761)
  • b4c069d Bump go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc (#815)
  • 415ae34 Bump goreleaser/goreleaser-action from 6 to 7 (#738)
  • 106fa0c Bump opensearchproject/opensearch-dashboards in /build/docker (#793)
  • ba7ad14 Fix schema restore failure with partitioned tables and add --data-only flag (#775)
  • 9eaba98 Fix update failres on generated columns (#820)
  • ada4af6 cli: improve docs/errors around snapshot/run commands (#807)
  • 78f50b1 docs(xata): add cnpg_pooler_pgbouncer to Xata-managed roles (#819)

v0.9.13

Choose a tag to compare

@github-actions github-actions released this 20 May 20:13
e87e37b

Changelog

  • e87e37b Backport: fix UPDATE failures on GENERATED ALWAYS AS IDENTITY columns (#820) (#822)

v0.9.12

Choose a tag to compare

@github-actions github-actions released this 11 May 14:40
6aba9d7

Changelog

v1.0.2

Choose a tag to compare

@github-actions github-actions released this 11 May 12:50
8b50ade

Highlights

🔀 Batch WAL operations in stream mode
🆔 New settings for wal2json plugin: add_tables and filter_tables
🐛 Bug fixes
:dependabot: Dependency updates

What's Changed

  • 1dc7cc1 Add add_tables and filter_tables wal2json plugin options (#791)
  • 2e6db21 Added CLAUDE.md file (#734)
  • 59a723d Batch WAL operations in stream mode (#774)
  • a6ad9fe Exclude pgstream schema when cleaning up target database (#740)
  • 757f3fa Rename recorder schema workers -> snapshot workers
  • 82b7a3f Set concurrency limit on snapshot recorder
  • 9888095 Surface error even if we cannot parse ERROR lines in the output of restore (#771)
  • 3b757ab Update configuration.md for new wal2json plugin settings (#803)
  • 8b50ade Update Golang to 1.26.3 (#804)
  • b7f3546 Use IS NULL when building update query WHERE clause