View Source Changelog
4.1.0
- For v2 subscribers, pass offset
ack_fun
as a callback state init parameter. This makes it possible for callback implementation to acknowledge offsets easier. PR#591 - Type spec fix. PR#590
- Fix
brod:fold/8
to avoid exception when a stable offset points to en empty batch. PR#589
- For v2 subscribers, pass offset
4.0.0
- Remove
snappyer
from default dependency PR#547. Starting fromkafka_protocol-4.0
, it no longer requires compression libraries as rebar dependencies. Nowbrod-4.0
has it removed too. For more compression options, see kafka_protocol/README
- Remove
3.19.1
- Made brod-cli to work on OTP 26. PR#582
--ssl
option is now mandatory if TLS is to be used (previously it can be derived from--cacertfile
option)- TLS version defaults to 1.2, added
--ssl-versions
to support explictly setting TLS 1.3
- Support building on OTP 27 PR#585
Also fixed rebar.lock for dependency
kafka_protocol-4.1.5
- Guard against crashes in
brod_client:get_consumer/3
PR#587 (originally PR#581)
- Made brod-cli to work on OTP 26. PR#582
3.19.0
- Forward unhandled messages in topic/group consumer processes to handle_info/2 callbacks in order to support arbitrary message passing PR#580
3.18.0
3.17.1
- Upgrade
kafka_protocol
from 4.1.3 to 4.1.5- Allow space after
,
in comma-separated bootstrapping host:port list - Avoid
badmatch
exception when parsing SASL password file
- Allow space after
- Upgrade
3.17.0
- Deleted
supervisor3
as dependency, module is now moved to brod.git namedbrod_supervisor3
. - Upgrade
snappyer
version from 1.2.8 to 1.2.9. - Upgrade
kafka_protocol
from 4.1.2 to 4.1.3. - In rebar3.config move
coveralls
fromplugins
toproject_plugins
.
- Deleted
3.16.8
- Upgrade
supervisor3
from 1.1.11 to 1.1.12 to log supervisor status atdebug
level instead ofnotice
level. - Add
brod_group_subscriber_v2:get_workers
function to help monitor and check the health of a consumer group.
- Upgrade
3.16.7
- Upgrade
kafka_protocol
from 4.1.1 to 4.1.2 to allow handling multiply nested wrapped secrets.
- Upgrade
3.16.6
- Upgrade
kafka_protocol
from 4.1.0 to 4.1.1 to support defining SNI for bootstrap hosts without the need to set theverify_peer
config.
- Upgrade
3.16.5
- Improve the documentation
- Add
brod:get_partitions_count_safe/2
. It is ensured not to auto-create topics in Kafka even when Kafka has topic auto-creation configured. - Treat
offset = 0
commit record asearliest
. This is an issue introduced in 3.16.4 which supported subscriber managed offset commits to be represented as{begin_offset, Offset}
. The issue is:0
is altered by v2 group worker to-1
. Handled as offsets greater than 0, v2 group worker assumsbrod_topic_subscriber
would apply+1
to it, however-1
is actually interpreted aslatest
. #531 fixes it by replacing0
withearliest
.
3.16.4
- Allow special begin_offset from consumer managed commits.
- Fix specs for
brod_group_subscriber_v2.get_committed_offset
- Update kafka_protocol from 4.0.3 to 4.1.0. kafka_protocol 4.1.0 support a different version of the auth plugin interface that also pass the handshake version to the plugin. This is used by the SASL GSSAPI Kerberos plugin brod_gssapi (https://github.com/kafka4beam/brod_gssapi) so that the correct handshake protocol is used. The change is backwards compatible as the old interface will be used if the plugin module does not have a function with the name and arity that is required by the new interface. See https://github.com/kafka4beam/kafka_protocol/pull/102 for details.
3.16.3
- Fix specs for
delete_topics
. - Ensure that partition worker process is alive before returning it.
- Update kafka_protocol from 4.0.1 to 4.0.3 in rebar.lock file.
- Make consumer 'isolation_level' configurable.
- Fix specs for
3.16.2
- Update kafka_protocol from 4.0.1 to 4.0.3. Prior to this change the actual time spent in establishing a Kafka connection might be longer than desired due to the timeout being used in SSL upgrade (if enabled), then API version query. This has been fixed by turning the given timeout config into a deadline, and the sub-steps will try to meet the deadline. see more details here: https://github.com/kafka4beam/kafka_protocol/pull/92
- Catch
timeout
and otherDOWN
reasons when makinggen_server
call tobrod_client
,brod_consumer
and producer/consumer supervisor, and return asReason
in{error, Reason}
. Previously onlynoproc
reason is caught. (#492) - Propagate
connect_timeout
config tokpro
API functions astimeout
arg affected APIs: connect_group_coordinator, create_topics, delete_topics, resolve_offset, fetch, fold, fetch_committed_offsets (#458) - Fix bad field name in group describe request (#486)
3.16.1
- Fix
brod
script inbrod-cli
in release. - Support
rebalance_timeout
consumer group option
- Fix
3.16.0
- Update to kafka-protocol v4.0.1
3.15.6
- Eliminate long-lived anonymous function closures
3.15.5
- Fix exponential growth of
brod_producer
buffer
- Fix exponential growth of
3.15.4
- Avoid start_link for temp connection usages affected APIs: fetch_committed_offsets, fetch, resolve_offset, create_topics, delete_topics
3.15.3
- Try to commit acknowledged offsets when brod_group_subscriber_v2 terminates
- Fix process leak, close connection after offset is fetched in brod_utils:fetch_committed_offsets/3
3.15.2
- Producer: Do not format producer buffer in
gen_server
state - Consumer: Do not commit offsets when
unknown_member_id
error is received - Logging: Changed from
error_logger
tologger
macros - Don't shut down brod_group_subscriber_v2 on previous generation worker termination
- Fix brod_group_subscriber_v2 crash on shutdown
- Use GitHub Actions instead of Travis CI
- Added Elixir example
- Rename DEFAULT_TIMEOUT macro in public brod.hrl to BROD_DEFAULT_TIMEOUT.
- Producer: Do not format producer buffer in
3.15.0 (and 3.15.1, for some reason tagged on the same commit)
- Fix unknown types
- Build on OTP-23
3.14.0 NOTE: This release changes internal states of brod worker processes in a way that cannot by applied on a running system. brod application and all brod workers should be shot down for the time of the upgrade.
Introduced a new optional terminate/2 callback to brod_topic_subscriber and brod_group_subscriber_v2 behaviors
Introduced new API functions for starting topic subscribers: brod_topic_subscriber:start_link/1 and brod:start_link_topic_subscriber/1 Old APIs
- brod:start_link_topic_subscriber/5,
- brod:start_link_topic_subscriber/6
- brod:start_link_topic_subscriber/7
- brod_topic_subscriber:start_link/6
- brod_topic_subscriber:start_link/7
- brod_topic_subscriber:start_link/8 are deprecated and will be removed in the next major release
3.13.0
- Update supervisor3 dependency to 1.1.11
- brod_group_subscriber_v2 behavior handles worker crashes
- Makefile fix for hex publishing
- Reverse Changelog order
- Fix typos and dialyzer warnings
3.12.0
- Enable passing custom timeout to resolve_offset
3.11.0
- Improve compatibility with EventHub
3.10.0
- Stop supporting erlang.mk
- Stop supporting rebar
- Update kafka_protocol dependency to 2.3.6
- Add new brod_group_coordinator:update_topics API
3.9.5
- Bump kafka_protocol dependency to 2.3.3
3.9.4
- Handle undefined fetcher in fold loop exception
3.9.3
- Remove vsn-check dependency from default Makefile target This enables using brod with erlang.mk + hex
- Bump kafka_protocol dependency to 2.3.2
3.9.2
- Fix corrupted package published to hex
3.9.1
- Receive pending acks after assignments_revoked is invoked
3.9.0
- Updated kafka_protocol dependency to 2.3.1
- (Experimental) Added group_subscriber_v2 behavior
- Added API for topic deletion and creation
3.8.1
- Handle the case when high_watermark < last_stable_offset in fetch resp
3.8.0
- Bump to kafka_protocol 2.2.9 (allow
atom()
hostname) - Add
brod:fold/8
. This API spawns a process to fetch-ahead while folding the previously fetched batch.brod-cli
'sfetch
command is updated to call thisfold
API for better performance. - Add callbacks to allow
brod_client:stop_producer
andbrod_client:stop_consumer
to remove the stopped child references from the supervisor and clean up the client ets table to allow later restart. - Support scram SASL authentication in brod-cli
- Made possible to start-link or supervise
brod_consumer
in user apps, instead of always underbrod_client
'sbrod_consumers_sup
- Bump to kafka_protocol 2.2.9 (allow
3.7.11
- Fix a bug when dropping aborted transactions for compacted topics
3.7.10
- Compare begin_offset with last stable offset before advancing to next offset in case empty
batch is received. Prior to this version, fetch attempts on unstable messages (messages
belong to open transactions (transactions which are neither committed nor aborted),
may result in an empty message set, then
brod_consumer
orbrod_utils:fetch
jumps to the next offset (if it is less than high-watermark offset).
- Compare begin_offset with last stable offset before advancing to next offset in case empty
batch is received. Prior to this version, fetch attempts on unstable messages (messages
belong to open transactions (transactions which are neither committed nor aborted),
may result in an empty message set, then
3.7.9
- Fix brod-cli escript include apps
- Fix brod-cli sub-record formatting crash
- Upgrade to kafka_protocol 2.2.8 to discard replica_not_available error code in metadata response
- Fix empty responses field in fetch response #323
3.7.8
- Drop batches in aborted transactions (and all control batches) also improve offset fast-forwarding when empty batches are received
3.7.7
- Fix
badrecord
race: message-set is delivered tobrod_group_subscriber
after unsubscribed frombrod_consumer
.
- Fix
3.7.6
- Fix produce message input type spec to make it backward compatible (bug introduced in 3.7.3)
3.7.5
- Bump kafka_protocol version to 2.2.7
- Fix empty assignment handling. In case a group member has no partition assigned,
member_assignment
data field in group sync response can either benull
(kafka 0.10) or a struct having emptytopic_partitions
(kafka 0.11 or later). The later case was not handled properly inbrod
before this fix.
3.7.4
- Add callback to make user_data in group join request
3.7.3
- Bump kafka_protocol version to 2.2.3
- Discard stale async-ack messages to group subscriber
3.7.2
- Pr #298: Subscriber now automatically reconnects to Kafka on topic rebalances where the previous partition leader no longer holds the partition at all.
- Pr #299: Fix topic subscriber returning partition offsets from callback module's init.
3.7.1
- Fix brod_topic_subscriber and brod_group_subscriber re-subscribe behaviour to avoid fetching duplicated messages.
- Add 'random' and 'hash' partitioner for produce APIs
- Allow
brod_group_subscrber:assign_partitions/3
to return an updated cb_state brod_client:get_connection
is back- Make it possible to run tests on both mac and linux
- Position group leader member at the head of members list when assigning partitions
3.7.0
- Add
brod_group_subscriber:ack/5
andbrod_group_subscriber:commit/4
to let group subscribers commit offsets asynchronously - Pr #284: In compacted topics, Kafka may return a MessageSet which contains only messages before the desired offset. Just keep reading forward in this case.
- Issue #285
brod_consumer
no longer restart onleader_not_availble
andnot_leader_for_partition
error codes received in fetch response. It resets connection and rediscover leader after delay.
- Add
3.6.2
- Allow
brod_topic_subscriber
to explicitly start consuming from partition offset 0 (by passing in a committed offset of -1).
- Allow
3.6.1
- Make produce request version configurable as
produce_req_vsn
inbrod:producer_config()
- Upgrade
kafka_protocol
to2.1.2
to support alpine/busybox build
- Make produce request version configurable as
3.6.0
- Moved 3 modules to kafka_protocol:
brod_sock
->kpro_connection
brod_auth_backed
->kpro_auth_backend
brod_kafka_requests
->kpro_sent_reqs
#kafka_message.key
and#kafka_message.value
are now alwaysbinary()
(they were of specundefined | binary()
prior to this version). i.e. empty bytes are now decoded as<<>>
instead ofundefined
. This may cause dialyzer check failures.brod_client
no longer logs about metadata socket down, it had been confusing rather than being helpful- There is no more cool-down delay for metadata socket re-establishment
brod_group_coordinator
default session timeout changed from 10 seconds to 30, and heartbeat interval changed from 2 seconds to 5.- Add
brod:produce_cb/4
andbrod:produce_cb/6
to support user defined callback as produce ack handler. - Add
brod:produce_no_ack/3
andbrod:produce_no_ack/5
. min_compression_batch_size
is removed from producer config.- Support magic v2 batch/message format (message headers).
- Use rebar3 as primary build tool for travis-ci, erlang.mk is still supported.
- Support
SCRAM-SHA-256
andSCRAM-SHA-512
SASL authentication mechanisms.
- Moved 3 modules to kafka_protocol:
3.5.2
- Fix issue #263: Kafka 0.11 may send empty batch in fetch response when messages are deleted in compacted topics.
3.5.1
- Add
extra_sock_opts
client socket options. It would be helpful for tuning the performance of tcp socket.
- Add
3.5.0
- Add
*_offset
variants toproduce
APIs, returning the base offsets that were assigned by Kafka. Producers need to be restarted when upgrading to this version.
- Add
3.4.0
- Add
prefetch_bytes
consumer config.brod_consumer
should stop fetch-ahead only when bothprefetch_count
andprefetch_bytes
limits are exceeded
- Add
3.3.5
- Fix issue #252 -- Kafka 0.11 and 1.0 have more strict validations on compressed batch wrapper message. Changed kafka_protocol 1.1.2 has the wrapper timestamp and offsets fixed.
3.3.4
- Fix issue #247 -- revert the handling of offset = -1 in offset_fetch_response, bug introduced in 3.3.0 offset = -1 in offset_fetch_response is an indicator of 'no commit' should be ignored (not taken as 'latest')
3.3.3
- Add a --no-api-vsn-query option for brod-cli to support kafka 0.9
- Bump kafka_protocol to 1.1.1 to fix relative offsets issue so brod-cli can fetch compressed batches as expected, also brod_consumer can start picking fetch request version
- Upgrade roundrobin group protocol to roundrobin_v2 to fix offset commit incompatiblility with kafka spec and monitoring tools etc. see https://github.com/klarna/brod/issues/241 for details
3.3.2
- Enhancements
- Add sh script to wrap brod-cli escript for erts dir auto discover
- Detailed log for connection estabilishment failures
- Bug Fixes
- Demonitor producer pid after sync_produce_request timeout
- Enhancements
3.3.1
- Fix brod-cli commits command redandunt socket usage.
3.3.0
- New Features
- Support offset_fetch_request version 1 - 2
- Provide APIs to reset committed offsets
- Support offset commit in brod-cli
- Improved group coordinator logging: 1) stop showing member ID, 2) show callback process.
- Cache queried version ranges per kafka host
- Brod-cli 'offset' command by default resolves offsets for all partitions if '--partition' option is 'all' or missing
- Enhancements
- Brod rock with elvis with travis
- Travis run dialyzer check
- Bug Fixes
- Fixed brod-cli offset_fetch_request version when working with kafka 0.10.1.x or earlier
- Make group coordinator restart on heartbeat timeout
- New Features
3.2.0
- New Features
- Support produce_request version 0 - 2
- Support fetch_request version 0 - 3
- Support offsets_request version 0 - 1
- Support metadata_request version 0 - 2
- Support message create time by allowing
Value
to be[{Ts, Key, Value}]
inbrod:produce
andbrod:produce_sync
APIs - Bind all
#kafka_message{}
fields to variables so they can be used in brod-clifetch
command's--fmt
expression
- Bug Fixes
- Remove sasl application from dependent app list
- New Features
3.1.0
- New Features
- New message type option for topic and group subscribers that specifies whether to
handle a Kafka
message
ormessage_set
.
- New message type option for topic and group subscribers that specifies whether to
handle a Kafka
- New Features
3.0.0
- New Features
- New API
brod:connect_group_cordinator
to establish a sockt towards group coordinator broker. - New API
brod:fetch_committed_offsets
to fetch consumer group committed offsets. - New API
brod:list_groups
,brod:list_all_groups
andbrod:describe_groups/3
. - Brod-cli new command
groups
to list / describe consumer groups. - Brod-cli new command
commits
to list comsuer group committed offsets.
- New API
- Backward-incompatible changes
brod:get_offsets
API replaced withbrod:resolve_offset
. Reason:brod:get_offsets
andbrod_utils:fetch_offsets
are very confusing, because they look like fetching consumer group committed offsets. Also, the return value has been changed from a list of offsets to a single offset.brod:get_metadata
return value changed from#kpro_MetadataResponse{}
record tokpro:struct()
.brod_utils:fetch/4
is removed, usemake_fetch_fun/8
instead.--count
option removed frombrod-cli
offset
command, The command now resolves only one (or maybe none) offset at a time.- All type specs defined in
brod.hrl
are moved to exported types frombrod.erl
- Backward-compatible changes
#kafka_message{}
record is extended with new fieldsts_type
andts
.#kafka_message.crc
changed from signed to unsigned integer.
- New Features
2.5.1
- Fix ignored commit history when committed offset is 0 (bug)
2.5.0
- Pluggable SASL authentication backend (contributor: ElMaxo)
- Brod-cli support extra ebin to code path
- Fix group subscriber duplicated loopback messages (bug)
- SASL-PLAIN username password in text file
- Hide SASL-PLAIN password in an anonymous function in
brod_client
state
2.4.1 Fixed brod-cli typo fix
2.4.0 Added brod-cli