Skip to content

CHANGELOG.md

Latest commit

 

History

History
398 lines (306 loc) · 12.1 KB

CHANGELOG.md

File metadata and controls

398 lines (306 loc) · 12.1 KB
Raw

Changes for Datasplash

0.8.0 - 2026-07-09

Changed

  • bump beam to 2.74.0
  • set java 11 as target

0.7.36 - 2026-07-04

Changed

  • bump beam to 2.73.0 - final version to support Java 8.

0.7.35 - 2026-06-24

Added

  • bq: new key-fn option to read-bq.

Changed

  • bump clj to 1.12.5
  • bump slf4j-pi to 2.0.18
  • bump charred to 1.039
  • bump beam to 2.74.0

Fixed

  • ES unserializable function when parsing JSON.

0.7.34 - 2026-04-13

Changed

  • Bump charred to 1.038.
  • bump tools-io to 0.3.43, aircompressor to 2.0.3
  • bump beam to 2.72.0
  • bump nippy to 3.6.2

0.7.33 - 2026-02-04

Changed

  • bump beam to 2.71.0
  • bump tools.logging to 1.3.1, tools-io to 0.3.42
  • bump clj to 1.12.4
  • bump math.combinatorics to 0.3.2

Fixed

  • ES conf can be created without a document type

0.7.32 - 2025-10-24

Changed

  • bump beam to 2.67.0
  • bump clj to 1.12.2
  • bump nippy to 3.6.0

0.7.31 - 2025-07-03

Changed

  • bump beam to 2.66.0

Fixed

  • pubsub and ES ns fixes by @leathekd.

0.7.30 - 2025-06-10

Changed

  • bump to clojure 1.12.1.

Fixed

  • Fix generate-input not passing a coder when a collection was passed.

0.7.29 - 2025-05-14

Changed

  • Bump beam to 2.65.0.

0.7.28 - 2025-04-17

Changed

  • Bump beam to 2.64.0.
  • Bump slf4j to 2.x.
  • Bump charred to 1.037.
  • Bump nippy to 3.5.0.

Fixed

  • Fix bug on Elasticsearch write when using id, type & index functions.

0.7.27 - 2025-02-20

Changed

  • Bump beam to 2.63.0.

0.7.26 - 2025-02-03

Changed

  • Bump beam to 2.62.0.
  • Bump charred to 1.036.
  • Bump superstring to 3.2.1.

0.7.25 - 2024-12-06

Added

  • without-validation option in read-bw-raw.

Changed

  • Bump beam to 2.61.0.

0.7.24 - 2024-10-30

Added

  • BQ: temp-dataset and temp-project options for reading.

Changed

  • Bump beam to 2.60.0.

0.7.23 - 2024-09-16

Changed

  • Bump beam to 2.59.0. Downgrade kafka-clients to match beam's version.

0.7.22 - 2024-07-01

Changed

  • Build compression-type map from Compression enum.
  • Bump beam to 2.57.0.

0.7.21 - 2024-05-07

Changed

  • Bump beam to 2.56.0
  • update of BQ schema spec types to now allow :json and :range
  • add support to set collation for STRING type

0.7.20 - 2024-03-16

Changed

  • Bump beam to 2.54.0
  • Bump charred to 1.034
  • Bump math.combinatronics
  • Bump clojure
  • Downgrade slf4j to 1.7.x as v2 is not compatible with beam > 2.53.0
  • Replace :dev profile logback dep with slj4j-simple

0.7.19 - 2023-12-15

  • Bump beam to 2.52.0
  • Add taoensso/encore required dependency for nippy.

v0.7.18 - 2023-09-29

  • Bump beam to 2.50.0
  • Bump charred to 1.033
  • Bump superstring to 3.2.0

v0.7.17 - 2023-04-07

  • add ds/distinct-by, ds/keep
  • normalize BQ schema
  • support for maxLength, precision, scale fields in BQ tables
  • new PAssert testing utilities
  • clean up aot files to reduce size
  • BQ specs for schema-validation
  • new exported clj-kondo config
  • bump charred to 1.028, beam sdk to 2.46.0

v0.7.16 - 2023-01-27

  • Bump charred to 1.019, beam to 2.44.0

v0.7.15 - 2022-12-15

  • Bump charred to 1.018

v0.7.14 - 2022-12-02

  • adapt charred options to avoid throwing on empty lines
  • Bump charred to 1.016, beam to 2.43.0

v0.7.13 - 2022-10-26

  • Add BQ raw read :query-location option.
  • Add support for Pcoll set operations.
  • Bump beam to 2.42.0
  • Bump charred to 1.0.14

v0.7.12 - 2022-09-16

  • handle collection for generating pipeline args
  • Bump beam to 2.41.0
  • Bump charred to 1.0.12

v0.7.11 - 2022-08-23

  • Add BQ clustering & optimized-write options in write-bq-table.
  • Bump beam to 2.40.0.
  • Bump charred to 1.0.11.
  • Bump nippy to 3.2.0.

v0.7.10 - 2022-06-24

  • Bump nippy to fix serializing of boolean values in map collections.
  • Bump Beam to 2.39.0.

v0.7.9 - 2022-05-24

  • Bump charred to fix more white-space issues. Enable options to match previous output.
  • Add write-json-file-test.

v0.7.8 - 2022-05-16

  • More complete tests, again.
  • Add ability to specify a teardown function for a DoFn
  • Invoke start-bundle, finish-bundle and teardown-fn when specified
  • Reuse no-op function in dofn
  • switch from cheshire to com.cnuernber/charred.

v0.7.7 - 2022-05-13

  • Enforce target jdk as 1.8
  • Bump clojure to 1.11.1
  • more complete tests

v0.7.6 - 2022-04-23

  • don't require bq, pubsub, datastore in api.clj - Issue #94
  • Bump beam to 2.38.0

v0.7.5 - 2022-04-09

  • replace cheshire usage with jsonista by @domparry
  • Bump beam to 2.37.0; bump other deps

v0.7.4 - 2021-12-15

  • Datastore fix by @domparry for breaking change introduced by nippy bump in 0.7.3
  • Bump beam dep (2.34.0)
  • Bump tools.logging (1.2.2)
  • Clean up

v0.7.3 - 2021-08-14

  • Resolve various java compiler warnings
  • Bump dependencies including beam (2.31.0)
  • Bump nippy to 3.1.1, adding alter-var-root of nippy/*thaw-serializable-allowlist* to allow serializable org.apache.beam.sdk.values.KV
  • Fix probems with Id keys in Google Datastore storage

v0.7.2 - 2021-01-07

  • fix locking problem with requires in safe-exec => Issue #97
  • Update Apache Beam to 2.26.0

v0.7.1 - 2020-10-06

  • Update Apache Beam to 2.24.0

v0.7.0 - 2020-09-07

  • Add Kafka IO support
  • Add delete-datastore-raw to datastore features
  • add :schema-update-options for BQ write
  • Update Apache Beam to 2.22.0
  • Performance improvements
  • Fixed require bug in partition-by

v0.6.6 - 2019-09-11

Fixes

  • Fix cogroup-by for numbers of pcoll > 10 => Issue #83
  • Fix :many-files option in read-text-file => Issue #75

v0.6.5 - 2019-08-23

  • Add FileIO Read to read a PCollection of files
  • Add ElasticSearchIO
  • Upgrade to Apache Beam 2.15.0
  • Upgrade dependencies

v0.6.4 - 2019-01-16

  • Upgrade to Apache Beam 2.9.0
  • Upgrade dependencies

v0.6.3 - 2018-11-19

  • Add json-schema in bq write
  • Combine-fn accepts now map as arguments
  • Remove deprecated write-by

v0.6.2 - 2018-11-15

  • Upgrade to Apache Beam 2.8.0
  • Support dynamic file writes
  • Support :deflate and :zip compressions
  • bq/->time-partitioning now accepts :field and :require-partition-filter
  • Improve docs

v0.6.1 - 2018-03-20

Changes

  • Upgrade to Dataflow 2.3.0
  • Support time-partitioned Big Query tables (thanks to @neuromantik33)

Fixes

  • Fix broken input from BigQuery
  • Fix missing safe-exec from filename policy
  • Fix broken serializing after group-by in some cases
  • Fix rare (hopefully) bug introduced by being lazy. with the vals after a group-by. Now we are greedy.

v0.6.0 - 2018-02-13

Changes

  • Compatibility with Dataflow 2.2.0
  • Compatibility with Clojure 1.9 and specs (some code had to be ported to Java shims because proxy is not Serializable anymore in Clojure 1.9)

Fixes

  • Removed superfluous logging, safe-exec call and type hints

v0.5.3 - 2017-12-18

Fixes

  • Fix options on PipelineWithOptions to have it work again on Dataflow (oops, sorry!)

v0.5.2 - 2017-12-14

Changes

  • Fixed defoptions again and made it useful for interop with Beam
  • Added :checkpoint option, will be made more useful later
  • Switched to new PubSub API
  • Add support for custom filename-policy

Fixes

  • Fixed a possible bug causing StackOverflows on combine

v0.5.1 - 2017-10-05

Changes

  • Added wait-pipeline-result

Fixes

  • Fix a bug with incorrect call to seq in combine
  • Add a missing safe-exec in write-bq-table-raw

[v0.5.0] - 2017-09-18

Changes

  • Update to v2.x Dataflow version (Apache Beam)
  • Add new streaming features (custom timestamp in PubSub #30 (thanks to @RoIT), BigQuery streaming writes)
  • Add compressed TextIO output #31 (thanks to @RoIT)
  • Add FileNamePolicy compatibility for TextIO

[v0.4.1] - 2017-03-09

Changes

  • Handle recursion in RECORD schema generation for Big Query #9 (thanks to @torbjornvatn)
  • Add windows and trigger support #13 (thanks to @RoIT)
  • Bump Dataflow SDK to 1.9.0
  • Add support for standard SQL syntax for Big Query #11 (thanks to @torbjornvatn)
  • Read data from pubsub topics #10 (thanks to @torbjornvatn)

v0.4.0 - 2016-11-12

Breaking Changes

  • Update Datastore API to v1, add wrappers for creating Entities from and converting to Clojure maps. Some breakage in the datasplash.datastore namespace ensues.

Changes

  • Add pubsub namespace #6 (Thanks to @MartinSahlen)
  • Put slf4j deps in :dev profiles #8 (Thanks to @torbjornvatn)