View Source KlifeProtocol.Messages.Produce (Klife Protocol v0.5.0)

Kafka protocol Produce message

Request versions summary:

  • Version 1 and 2 are the same as version 0.
  • Version 3 adds the transactional ID, which is used for authorization when attempting to write transactional data. Version 3 also adds support for Kafka Message Format v2.
  • Version 4 is the same as version 3, but the requester must be prepared to handle a KAFKA_STORAGE_ERROR.
  • Version 5 and 6 are the same as version 3.
  • Starting in version 7, records can be produced using ZStandard compression. See KIP-110.
  • Starting in Version 8, response has RecordErrors and ErrorMessage. See KIP-467.
  • Version 9 enables flexible versions.
  • Version 10 is the same as version 9 (KIP-951).
  • Version 11 adds support for new error code TRANSACTION_ABORTABLE (KIP-890).

Response versions summary:

  • Version 1 added the throttle time.
  • Version 2 added the log append time.
  • Version 3 is the same as version 2.
  • Version 4 added KAFKA_STORAGE_ERROR as a possible error code.
  • Version 5 added LogStartOffset to filter out spurious OutOfOrderSequenceExceptions on the client.
  • Version 8 added RecordErrors and ErrorMessage to include information about records that cause the whole batch to be dropped. See KIP-467 for details.
  • Version 9 enables flexible versions.
  • Version 10 adds 'CurrentLeader' and 'NodeEndpoints' as tagged fields (KIP-951)
  • Version 11 adds support for new error code TRANSACTION_ABORTABLE (KIP-890).

Link to this section Summary

Functions

Returns the message api key number.

Receive a binary in the kafka wire format and deserialize it into a map.

Returns the current max supported version of this message.

Returns the current min supported version of this message.

Receives a map and serialize it to kafka wire format of the given version.

Link to this section Functions

Returns the message api key number.

Link to this function

deserialize_response(data, version, with_header? \\ true)

View Source

Receive a binary in the kafka wire format and deserialize it into a map.

Response content fields:

  • responses: Each produce response ([]TopicProduceResponse | versions 0+)

    • name: The topic name (string | versions 0+)

    • partition_responses: Each partition that we produced to within the topic. ([]PartitionProduceResponse | versions 0+)

      • index: The partition index. (int32 | versions 0+)

      • error_code: The error code, or 0 if there was no error. (int16 | versions 0+)

      • base_offset: The base offset. (int64 | versions 0+)

      • log_append_time_ms: The timestamp returned by broker after appending the messages. If CreateTime is used for the topic, the timestamp will be -1. If LogAppendTime is used for the topic, the timestamp will be the broker local time when the messages are appended. (int64 | versions 2+)

      • log_start_offset: The log start offset. (int64 | versions 5+)

      • record_errors: The batch indices of records that caused the batch to be dropped ([]BatchIndexAndErrorMessage | versions 8+)

        • batch_index: The batch index of the record that cause the batch to be dropped (int32 | versions 8+)

        • batch_index_error_message: The error message of the record that caused the batch to be dropped (string | versions 8+)

      • error_message: The global error message summarizing the common root cause of the records that caused the batch to be dropped (string | versions 8+)

      • current_leader: (LeaderIdAndEpoch | versions 10+)

        • leader_id: The ID of the current leader or -1 if the leader is unknown. (int32 | versions 10+)

        • leader_epoch: The latest known leader epoch (int32 | versions 10+)

  • throttle_time_ms: The duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota. (int32 | versions 1+)

  • node_endpoints: Endpoints for all current-leaders enumerated in PartitionProduceResponses, with errors NOT_LEADER_OR_FOLLOWER. ([]NodeEndpoint | versions 10+)

    • node_id: The ID of the associated node. (int32 | versions 10+)

    • host: The node's hostname. (string | versions 10+)

    • port: The node's port. (int32 | versions 10+)

    • rack: The rack of the node, or null if it has not been assigned to a rack. (string | versions 10+)

Returns the current max supported version of this message.

Returns the current min supported version of this message.

Link to this function

serialize_request(map, version)

View Source

Receives a map and serialize it to kafka wire format of the given version.

Input content fields:

  • transactional_id: The transactional ID, or null if the producer is not transactional. (string | versions 3+)

  • acks: The number of acknowledgments the producer requires the leader to have received before considering a request complete. Allowed values: 0 for no acknowledgments, 1 for only the leader and -1 for the full ISR. (int16 | versions 0+)

  • timeout_ms: The timeout to await a response in milliseconds. (int32 | versions 0+)

  • topic_data: Each topic to produce to. ([]TopicProduceData | versions 0+)

    • name: The topic name. (string | versions 0+)

    • partition_data: Each partition to produce to. ([]PartitionProduceData | versions 0+)

      • index: The partition index. (int32 | versions 0+)

      • records: The record data to be produced. (records | versions 0+)