All notable changes to this project will be documented in this file.
This project adheres to Semantic Versioning.
- Cache resolved avro schemas in the schema-registry formatter.
avroSchemaRegistryFormatter.unmarshalpreviously re-parsed the writer/reader schemas and re-ranSchemaCompatibility.Resolveon every message;Resolvefingerprints the whole schema (Schema.Stringover the entire tree), which is very expensive for large schemas. The resolved schema is now cached by the message's confluent schema ID and reused, so resolution runs once per schema ID instead of once per message. Expected to significantly improve consumer decode performance (CPU and allocations) for large schemas, with no behavior change.
- Added
CircuitBreakerStateChangedlifecycle hook invoked on every per-queue circuit breaker transition. The hook receives aLifecycleCircuitBreakerStateChangedpayload carrying the previous and nextCircuitBreakerState(open,halfopen,closed). Chained inChainLifecycleHooks.
- go1.26 upgrade
- Updated direct deps
- Updated golangci to use version 2 (updated github action)
- Use go fix ./... to modernize codebase (use 1.26 features)
- Updated dependencies (fix vulnerabilities)
- Clearer error message
client.Reader()when missing topic names
- Added
WithWorkTextMapPropagator - Updated module to go1.24
- Updated dependencies
- Fixed linter
- Forward/Backwards compatible schemas used to produce/consume messages were not being properly deserialized.
Test_SchemaRegistryReal_Avro_AutoRegisterSchemas_OldProducerCanBeConsumedByNewConsumerandTest_SchemaRegistryReal_Avro_AutoRegisterSchemas_NewProducerCanBeConsumedByOldConsumerdemonstrated bug. Schema registry aware avro formatter was updated to reconcile the bug.
- Include
DisableTracePropagationas a WriterOption
- Updated confluent-kafka-go-2.8.0
- Bugfix in
ChainLifecycleHookswhere incorrect nil checks could result in panic go get -uto upgrade deps
- Updated
go.uber.org/mockversion0.5.0 - Updated
golang.org/x/syncversion0.9.0 - Removed noisy log
Timed out on read
- Updated
confluent-kafka-goversion2.6.1along with other dependencies
- Updated
confluent-kafka-goversion2.6.0to address security vulnerability.
- Removed dependency on github.com/golang/mock (deprecated) in favor of github.com/uber-go/mock
- Bugfixes in
WithDisableCircuitBreakerandWithDisableBusyLoopBreakeroptions implementation
- Added
WithDisableCircuitBreakerandWithDisableBusyLoopBreakeroptions. These are variants of the now deprecatedDisableCircuitBreakerandDisableBusyLoopBreakeroptions. They provide a booling parameter which is more convenient for usage with code generation and for shimming with configuration.
- Update to allow subject name specification (not just TopicNameStrategy)
- Updated
avro_schema_registryformatter deserialization to require passed in schema (less susceptible to errors from inferred target schema)
- Added support for schema registry (avro, proto, json). Extended
zfmt.FormatterTypetypes to includeavro_schema_registry,proto_schema_registryandjson_schema_registry - Added lifecycle function
LifecyclePostReadImmediate - Added
workFactory.CreateWithFuncwhich is a convenience work factory method for creating work using a callback instead of an interface (can reduce boilerplate) in some scenarios. - During the creation of readers/writers an error is now returned if bootstrap servers is empty
- Updated golang version 1.23
- Updated otel versions
- Updated
WithDeadLetterTopicoption to borrow username and password from ConsumerTopicConfig when those issues aren't specified on DeadLetterTopicConfig
- Added dlt topic name in error logs on dlt write failure
Initial release to public github.com
- Updated otel to v1.27
- Updated semconv to v1.25
- Added consumer delay config
ProcessDelayMillisthis allows the consumption of messages, but delays processing until at least the configured delay has passed since the message was written. Useful for intermediate dead letter messages. - Added ability to add headers via writer option (WithHeaders).
- Updated so
linger.ms=0is the default. - Updated so
socket.nagle.disable=trueis the default. - Increased
SessionTimeoutMillisandMaxPollIntervalMillisdefaults to be greater thanProcessDelayMillisso that an inadvertent long running processor doesn't cause a rebalance. - Updated confluent.Config's usage of
config.AdditionalProps. They can now override any setting. Previously, some of the promoted configs couldn't be set via this map.
- Updated to be used with zworker.Work which requires a
work.Runinterface notwork.Do. The difference beenwork.Runis executed once, andzkafkais responsible for continuously looping, whereaswork.Dowould be continually executed in a loop. - Renamed
zkafka.WithOnDoneWithContexttozkafka.WithOnDoneand removed originalzkafka.WithOnDoneoption (which didn't provide a context.Context arg) - Updated
Writerinterface to includeWriteRawmethod. The concrete type has supported it for some time, but was waiting for a major version roll to update the interface. - Updated
zkafka.Messageheaders to be amap[string][]byteinstead ofmap[string]interface{}. This is closer to the transport representation and is more convenient and self documenting. The interface{} type was a holdover from the original implementation and was not necessary. - Removed
ExtractHeaderKeys(reduce surface area of package). Opinionated API (zillow specific) that now resides in zkafkacomproot - Added variadic arguments (
...zkafka.WriteOption) tokwriter.Write*methods . This allows future customization in a noninvasive way. - Removed
zcommondependency. Introduce hooks which can be used toward the same end - Changed interface{} -> any
- Added lifecycle methods
PostRead,PreWriteandPostFanout - Added
WithConsumerProviderandWithProducerProviderwhich is useful for full e2e testing. - Updated work to remove read messages that won't be worked, from the inwork count. Allows Faster shutdown
- Supports migration from Datadog statsd to Panoptes.
- Removes the
Metricsinterface and related options. RemovesNoopMetricsstruct. Rather than calling metrics classes directly, the user registers lifecycle hooks and calls the metric provider from the hooks. For example, zkafkacomproot registers hooks that call the zmetric provider. - Removes the
RequestContextExtractorinterface. Instead, use thePreProcessinglifecycle hook to extract information from the request and add it to the context. The context returned from thePreProcessinghook is used for the rest of the request lifecycle.
- Removes the dependency on opentracing-go. Opentracing-go was a stale dependency that was no longer receiving updates. The library is now instrumented with opentelemtry a stable tracing library, that was written with backwards compatability in mind with opentracing-go.
- Removed
WithTracer(opentracing.Tracer). UseWithTracerProviderandWithTextMapPropagatorinstead.
Updated to account for update in zfmt which changes the values of some of the formatter factory entry values.
To see further details on zmt update to V1. See migration guide here