Skip to content

Releases: confluentinc/librdkafka

v2.14.2

03 Jun 10:28
6516057

Choose a tag to compare

librdkafka v2.14.2 is a maintenance release:

  • Fix duplicate groups in ListConsumerGroups when multiple brokers
    return the same group (#5417).
  • Fix data race in timers (#5089).
  • Update bundled OpenSSL, libcurl, zstd, zlib and cJSON
    dependencies (#5346).

Security considerations

Bundled dependencies were upgraded as follows (see #5346):
OpenSSL 3.0.15 → 3.5.6 (LTS) for source/autoconf builds, and to 3.6.2 in
vcpkg-based packages (no LTS available in vcpkg); libcurl 8.10.1 → 8.20.0
for source/autoconf builds and to 8.19.0 in vcpkg; zlib 1.3.1 → 1.3.2;
zstd 1.5.6 → 1.5.7; cJSON 1.7.14 → 1.7.19.

Fixes

General fixes

  • Issues: #5082.
    Fix data race in timers. The callback and its argument could have been modified after the lock is released.
    Happening since 1.x (#5089).

Consumer fixes

  • Fix crash (SIGSEGV) in rd_kafka_cgrp_handle_LeaveGroup() when coordinator
    is unavailable during consumer close. The error logging path dereferenced
    a potentially NULL broker pointer. Happening since 1.x.

Admin client fixes

  • Issues: #5417.
    Fix duplicate groups in ListConsumerGroups when multiple brokers return the same group.
    Happening since 1.x (#5417).

Checksums

Release asset checksums:

  • v2.14.2.zip SHA256 2c0a563a39d5c1bc2e7b3ae81bbad9aca23c586ddab9f659b51983d4dc67cffb
  • v2.14.2.tar.gz SHA256 d7eec9c31c817fa44402f679c252dfbf97e4c338a849a25c3579a31fd127beb8

v2.14.1

15 Apr 05:22
e0da09c

Choose a tag to compare

librdkafka v2.14.1 is a maintenance release:

  • Bundle prebuilt binaries for linux-s390x (#5365).

Checksums

Release asset checksums:

  • v2.14.1.zip SHA256 b6f52ae7a743e504e416e620c94e1192906daf15eafe661e4e2e6a7b793efc06
  • v2.14.1.tar.gz SHA256 bb246e754dee3560e9b42bf4e844dc05de4b146a3cae937e36301ffacdc456e7

v2.14.0

01 Apr 13:57
f1b831e

Choose a tag to compare

librdkafka v2.14.0 is a feature release:

  • KIP-768 Extend SASL/OAUTHBEARER to support OIDC claim mapping beyond the default sub claim (#5336).

Checksums

Release asset checksums:

  • v2.14.0.zip SHA256 372589ac63b06f9cac5d9b50d4ed1998f46b1a6cca991b691527ccc62b7cb7dd
  • v2.14.0.tar.gz SHA256 c05c03ef00a13a8463fac3e8918c04843c416f11ced58c889d806a88ca92cf99

v2.13.2

02 Mar 18:32
901af7c

Choose a tag to compare

librdkafka v2.13.2 is a maintenance release:

  • The librdkafka.redist NuGet package now includes binary for alpine-arm64 (#5237, @mclayton7)
  • Remove CPU usage regression when a subscription matches no topics (#5324).
  • Fix rd_kafka_consume_batch_queue incorrectly updating the application position on EOF or error messages (#5213).
  • Fix compilation without getentropy (Oleg Babin (@olegrok), Leo Singer (@lpsinger), #5288).
  • Use a truly random seed for pseudo-random number generation whenever available (#5288).
  • Fix rd_list destroy callback type mismatch by changing rd_kafka_assignor_destroy to take a void * argument, as expected by rd_list_init() destroy callbacks, and casting internally to rd_kafka_assignor_t * (#5195) (#5278).

Fixes

General fixes

Consumer fixes

  • Issues: #5324. Remove CPU usage regression when a subscription matches no topics. The increased CPU usage (~30%) was seen in particular when there are many topics in the clusters and the given subscription regex doesn't match any. Happening since 2.10.0 (#5324).
  • Issues: #4844. Fix rd_kafka_consume_batch_queue incorrectly updating the application position when receiving EOF or error messages, causing the position to move forward and likely be stored and committed. When storing the application offset the leader epoch is also considered for correct offset ordering in case of log truncation. Happening since 2.2.0 (#5213).

Checksums

Release asset checksums:

  • v2.13.2.zip SHA256 1b71b01a33f54c5d3e359935f73445fde4010215efdb899e3a746475d6aa178a
  • v2.13.2.tar.gz SHA256 14972092e4115f6e99f798a7cb420cbf6daa0c73502b3c52ae42fb5b418eea8f

v2.13.0

05 Jan 12:59
59b2f66

Choose a tag to compare

librdkafka v2.13.0 is a feature release:

  • KIP-482 Upgrade CreateAcls, DescribeAcls, DeleteAcls to the first version supporting this KIP (#5081).
  • KIP-482 Upgrade DescribeGroups, DeleteTopics, DeleteRecords, CreatePartitions, DeleteGroups to the first version supporting this KIP (#5083).
  • Strip trailing dot of hostname to fix SSL certificate verification issue (#5253).
  • Fix memory management for interceptors in rd_kafka_conf to prevent
    double-free errors (#5240).
  • Fix for the pseudo-random generator seed on Windows involving as well
    the uniqueness of the new consumer group protocol member id (#5265).
  • Add secure random generation functionality used for UUID uniqueness
    and secure salt generation in rd_kafka_UserScramCredentialUpsertion
    using OpenSSL or the POSIX or WIN32 equivalent calls when it
    isn't available (#5265).

Fixes

General fixes

  • Issues: #4348.
    Strip trailing dot of hostname to fix SSL certificate verification issue.
    Happening since 1.x (#5253).
  • Issues: #4142.
    Fix memory management for interceptors in rd_kafka_conf to prevent double-free errors.
    In case the client instance fails the users needs to destroy the configuration
    data structure, it was causing a double-free because the interceptors were
    already freed in the constructor.
    Happening since 1.x (#5240).
  • Issues: #5263, #3929.
    Fix for the pseudo-random seed on Windows. The function rand_r isn't present
    on Windows and the global seed wasn't based on the current microseconds and thread
    id. Also it wasn't called on every thread as required on this platform but
    only once per process. The fix allows on this platform the uniqueness of client side
    member id generation in next-generation consumer group protocol.
    Happening since 1.x (#5265).

Checksums

Release asset checksums:

  • v2.13.0.zip SHA256 73d731322b34c59fb5245d27172c71824e9323acd934e214d70a77954749e79d
  • v2.13.0.tar.gz SHA256 3bd351601d8ebcbc99b9a1316cae1b83b00edbcf9411c34287edf1791c507600

New Contributors

Full Changelog: v2.12.1...v2.13.0

v2.12.1

21 Oct 15:59
e1db7ea

Choose a tag to compare

librdkafka v2.12.1 is a maintenance release:

  • Restored macOS binaries compatibility with macOS 13 and 14 (#5219).

Fixes

General fixes

  • Fix to restore macOS 13 and 14 compatibility in prebuilt binaries present in librdkafka.redist.
    Happening since 2.12.0 (#5219).

Checksums

Release asset checksums:

  • v2.12.1.zip SHA256 da7571a0c1dc374aabb18af6ca01411d4bc597d321977980c8d3211ec5adf696
  • v2.12.1.tar.gz SHA256 ec103fa05cb0f251e375f6ea0b6112cfc9d0acd977dc5b69fdc54242ba38a16f

v2.12.0

08 Oct 18:36
2f22088

Choose a tag to compare

librdkafka v2.12.0 is a feature release:

KIP-848 – General Availability

Starting with librdkafka 2.12.0, the next generation consumer group rebalance protocol defined in KIP-848 is production-ready. Please refer the following migration guide for moving from classic to consumer protocol.

Note: The new consumer group protocol defined in KIP-848 is not enabled by default. There are few contract change associated with the new protocol and might cause breaking changes. group.protocol configuration property dictates whether to use the new consumer protocol or older classic protocol. It defaults to classic if not provided.

Enhancements and Fixes

  • Support for OAUTHBEARER metadata based authentication types,
    starting with Azure IMDS. Introduction available (#5155).
  • Fix compression types read issue in GetTelemetrySubscriptions response
    for big-endian architectures (#5183, Faidon Liambotis (@paravoid)).
  • Fix for KIP-1102 time based re-bootstrap condition (#5177).
  • Fix for discarding the member epoch in a consumer group heartbeat response when leaving with an inflight HB (#4672).
  • Fix for an error being raised after a commit due to an existing error in the topic partition (#4672).
  • Fix double free of headers in rd_kafka_produceva method (blindspot (@blindspotbounty), #4628).
  • Fix to ensure rd_kafka_query_watermark_offsets enforces the specified timeout and does not continue beyond timeout expiry (#5201).
  • New walkthrough in the Wiki about configuring Kafka cross-realm authentication between Windows SSPI and MIT Kerberos.

Fixes

General fixes

  • Issues: #5178.
    Fix for KIP-1102 time based re-bootstrap condition.
    Re-bootstrap is now triggered only after metadata.recovery.rebootstrap.trigger.ms
    have passed since first metadata refresh request after last successful
    metadata response. The calculation was since last successful metadata response
    so it's possible it did overlap with the periodic topic.metadata.refresh.interval.ms
    and cause a re-bootstrap even if not needed.
    Happening since 2.11.0 (#5177).
  • Issues: #4878.
    Fix to ensure rd_kafka_query_watermark_offsets enforces the specified timeout and does not continue beyond timeout expiry.
    Happening since 2.3.0 (#5201).

Telemetry fixes

  • Issues: #5179 .
    Fix issue in GetTelemetrySubscriptions with big-endian
    architectures where wrong values are read as
    accepted compression types causing the metrics to be sent uncompressed.
    Happening since 2.5.0. Since 2.10.1 unit tests are failing when run on
    big-endian architectures (#5183, Faidon Liambotis (@paravoid)).

Consumer fixes

  • Issues: #5199
    Fixed an issue where topic partition errors were not cleared after a successful
    commit. Previously, a partition could retain a stale error state even though the
    most recent commit succeeded, causing misleading error reporting. Now, successful
    commits correctly clear the error state for the affected partitions
    Happening since 2.4.0 (#4672).

Producer fixes

Checksums

Release asset checksums:

  • v2.12.0.zip SHA256 9b2f373e03f3d5d87c2075b3ce07ee9ea3802eea00cea41b99d8351a68d8a062
  • v2.12.0.tar.gz SHA256 1355d81091d13643aed140ba0fe62437c02d9434b44e90975aaefab84c2bf237

v2.11.1

18 Aug 20:51
69b1865

Choose a tag to compare

librdkafka v2.11.1 is a maintenance release:

  • Made the conditions for enabling the features future proof (#5130).
  • Avoid returning an all brokers down error on planned disconnections (#5126).
  • An "all brokers down" error isn't returned when we haven't tried to connect
    to all brokers since last successful connection (#5126).

Fixes

General fixes

  • Issues: #4948, #4956.
    Made the conditions for enabling the features future proof, allowing to
    remove RPC versions in a subsequent Apache Kafka version without disabling
    features. The existing checks were matching a single version instead of
    a range and were failing if the older version was removed.
    Happening since 1.x (#5130).

  • Issues: #5142.
    Avoid returning an all brokers down error on planned disconnections.
    This is done by avoiding to count planned disconnections, such as idle
    disconnections, broker host change and similar as events that can cause
    the client to reach the "all brokers down" state, returning an error and
    since 2.10.0 possibly starting a re-bootstrap sequence.
    Happening since 1.x (#5126).

  • Issues: #5142.
    An "all brokers down" error isn't returned when we haven't tried to connect
    to all brokers since last successful connection. It happened because the down
    state is cached and can be stale when a connection isn't needed to that
    particular broker. Solved by resetting the cached broker down state when any
    broker successfully connects, so that broker needs to be tried again.
    Happening since 1.x (#5126).

Checksums

Release asset checksums:

  • v2.11.1.zip SHA256 4a63e4422e5f5bbbb47f0ac1200e2ebd1f91b7b23f0de1bc625810c943fb870e
  • v2.11.1.tar.gz SHA256 a2c87186b081e2705bb7d5338d5a01bc88d43273619b372ccb7bb0d264d0ca9f

v2.11.0

03 Jul 16:26
c56a3e6

Choose a tag to compare

librdkafka v2.11.0 is a feature release:

  • KIP-1102 Enable clients to rebootstrap based on timeout or error code (#4981).
  • KIP-1139 Add support for OAuth jwt-bearer grant type (#4978).
  • Fix for poll ratio calculation in case the queues are forwarded (#5017).
  • Fix data race when buffer queues are being reset instead of being
    initialized (#4718).
  • Features BROKER_BALANCED_CONSUMER and SASL_GSSAPI don't depend on
    JoinGroup v0 anymore, missing in AK 4.0 and CP 8.0 (#5131).
  • Improve HTTPS CA certificates configuration by probing several paths
    when OpenSSL is statically linked and providing a way to customize their location
    or value (#5133).

Fixes

General fixes

  • Issues: #4522.
    A data race happened when emptying buffers of a failing broker, in its thread,
    with the statistics callback in main thread gathering the buffer counts.
    Solved by resetting the atomic counters instead of initializing them.
    Happening since 1.x (#4718).
  • Issues: #4948
    Features BROKER_BALANCED_CONSUMER and SASL_GSSAPI don't depend on
    JoinGroup v0 anymore, missing in AK 4.0 and CP 8.0. This PR partially
    fixes the linked issue, a complete fix for all features will follow.
    Rest of fixes are necessary only for a subsequent Apache Kafka major
    version (e.g. AK 5.x).
    Happening since 1.x (#5131).

Telemetry fixes

  • Issues: #5109
    Fix for poll ratio calculation in case the queues are forwarded.
    Poll ratio is now calculated per-queue instead of per-instance and
    it allows to avoid calculation problems linked to using the same
    field.
    Happens since 2.6.0 (#5017).

Checksums

Release asset checksums:

  • v2.11.0.zip SHA256 9e76a408f0ed346f21be5e2df58b672d07ff9c561a5027f16780d1b26ef24683
  • v2.11.0.tar.gz SHA256 592a823dc7c09ad4ded1bc8f700da6d4e0c88ffaf267815c6f25e7450b9395ca

v2.10.1

11 Jun 11:39
2af6749

Choose a tag to compare

librdkafka v2.10.1 is a maintenance release:

  • Fix to add locks when updating the metadata cache for the consumer
    after no broker connection is available (Marcin Krystianc (@marcin-krystianc), #5066).
  • Fix to the re-bootstrap case when bootstrap.servers is NULL and
    brokers were added manually through rd_kafka_brokers_add (#5067).
  • Fix an issue where the first message to any topic produced via producev or
    produceva was delivered late (by up to 1 second) (#5032).
  • Fix for a loop of re-bootstrap sequences in case the client reaches the
    all brokers down state (#5086).
  • Fix for frequent disconnections on push telemetry requests
    with particular metric configurations (#4912).
  • Avoid copy outside boundaries when reading metric names in telemetry
    subscription (#5105)
  • Metrics aren't duplicated when multiple prefixes match them (#5104)

Fixes

General fixes

  • Issues: #5088.
    Fix for a loop of re-bootstrap sequences in case the client reaches the
    all brokers down state. The client continues to select the
    bootstrap brokers given they have no connection attempt and doesn't
    re-connect to the learned ones. In case it happens a broker restart
    can break the loop for the clients using the affected version.
    Fixed by giving a higher chance to connect to the learned brokers
    even if there are new ones that never tried to connect.
    Happens since 2.10.0 (#5086).
  • Issues: #5057.
    Fix to the re-bootstrap case when bootstrap.servers is NULL and
    brokers were added manually through rd_kafka_brokers_add.
    Avoids a segmentation fault in this case.
    Happens since 2.10.0 (#5067).

Producer fixes

  • In case of producev or produceva, the producer did not enqueue a leader
    query metadata request immediately, and rather, waited for the 1 second
    timer to kick in. This could cause delays in the sending of the first message
    by up to 1 second.
    Happens since 1.x (#5032).

Consumer fixes

  • Issues: #5051.
    Fix to add locks when updating the metadata cache for the consumer.
    It can cause memory corruption or use-after-free in case
    there's no broker connection and the consumer
    group metadata needs to be updated.
    Happens since 2.10.0 (#5066).

Telemetry fixes

  • Issues: #5106.
    Fix for frequent disconnections on push telemetry requests
    with particular metric configurations.
    A NULL payload is sent in a push telemetry request when
    an empty one is needed. This causes disconnections every time the
    push is sent, only when metrics are requested and
    some metrics are matching the producer but none the consumer
    or the other way around.
    Happens since 2.5.0 (#4912).
  • Issues: #5102.
    Avoid copy outside boundaries when reading metric names in telemetry
    subscription. It can cause that some metrics aren't matched.
    Happens since 2.5.0 (#5105).
  • Issues: #5103.
    Telemetry metrics aren't duplicated when multiple prefixes match them.
    Fixed by keeping track of the metrics that already matched.
    Happens since 2.5.0 (#5104).

Checksums

Release asset checksums:

  • v2.10.1.zip SHA256 7cb72c4f3d162f50d30d81fd7f7ba0f3d9e8ecd09d9b4c5af7933314e24dd0ba
  • v2.10.1.tar.gz SHA256 75f59a2d948276504afb25bcb5713a943785a413b84f9099d324d26b2021f758