Kafka Offsets: Separate Log Progress from Returned AI Records

Identify the partition and distinguish the two consumer positions

This explanation uses KafkaConsumer, clients 4.1.2 API, under Apache Kafka's versioned 4.1 documentation, checked October 20, 2026. It evaluates an AI input handoff rather than prescribing a Kafka configuration.

An offset supplies a numerical address within a partition. That address lets a consumer describe where reading continues. Offset 103 in one partition does not identify offset 103 in another partition or supply a global sequence across a topic.

The live consumer position concerns the next read location and advances as consumption proceeds, including movement past filtered entries. The committed consumer position is the separately stored recovery reference. Kafka's design discussion distinguishes reading, processing and saving progress because a failure can occur between those events.

Also distinguish a producer transaction's commitment from a consumer's offset commitment. The first determines the transaction's visibility to this reader; the second saves where that consumer recovers. Neither term alone describes a completed external application action.

References: [K41][KD41]

An open transaction bounds the readable prefix

A read_committed consumer admits committed transactional data and nontransactional data within its readable prefix. It excludes aborted transactional data. Commit and abort markers occupy log offsets but are hidden from the application's returned records.

For the numerical boundary here, use the clients 4.1.2 endOffsets contract: its last stable offset, or LSO, is the minimum of the high watermark and the smallest offset belonging to an open transaction. This is an exclusive boundary; eligible records are below it. The high watermark is the last successfully replicated offset plus one.

Later nontransactional data can therefore exist beyond an open transaction without being available to this reader yet. Eligibility also differs from an actual poll result: returned batches can be partial. The case explicitly supplies eventual complete reading, without assigning a delivery time or requiring one poll to return the whole eligible set.

References: [K41][KC41]

KF-38 starts with six entries at offsets 100 through 105

KF-38 is an original fictional log, not an executed Kafka test. Fix one cluster, topic EX-AI and partition 0. Consider only the interval starting at offset 100. The reader starts there with read_committed isolation. There is no compaction, retention removal, application filtering, rewind or duplicate delivery in this case. Authorized reading and sufficient fetch opportunity are stipulated.

At the first observation, offset 100 contains nontransactional data record R1. Offset 101 contains data record A1 from a transaction that has already aborted; offset 102 contains its abort marker. Offset 103 contains nontransactional data record R2.

Offset 104 contains data record O1 from a different, still-open transaction. Offset 105 contains nontransactional data record R3. These labels identify separate fictional data records, without specifying their payloads or business meanings.

All six entries are stipulated successfully replicated, so the high watermark is 106. The transaction containing O1 is the only open transaction affecting the boundary. No further entry arrives during this first observation.

First observation: two records return while R3 waits

The exclusive LSO is the smaller of 106 and 104: 104. R1 at 100 and R2 at 103 are eligible. A1 at 101 is excluded because its transaction aborted, and the abort marker at 102 is hidden.

O1 at 104 remains outside the readable prefix while its transaction is open. R3 at 105 is nontransactional, but it is also beyond the boundary. Its existence in the replicated log does not let this read_committed reader pass O1 to return it.

During this observation, stipulate that the reader eventually receives the entire eligible data population: R1, then R2. That is two returned data records, potentially across multiple polls. R3's absence follows from transaction visibility rather than an absent log entry or an application rejection.

The boundary value 104 does not measure this consumer's live position or its saved recovery position. Neither position is assigned an exact value at this observation. A transaction boundary, a returned record list and a consumer-position measurement answer different questions.

References: [K41][KC41]

Second observation: four cumulative data records across seven positions

Now stipulate that the transaction containing O1 commits, with its commit marker at offset 106. All entries through 106 are successfully replicated. There are no remaining open transactions, no other new records and no further writes during the second observation. The high watermark and exclusive read_committed end boundary are now 107.

The same reader continues without replay and eventually reads the complete newly eligible population: O1 at 104, followed by R3 at 105. The commit marker at 106 is hidden. This observation adds two returned data records. Across both observations, the complete returned population is R1, R2, O1 and R3: four records at offsets 100, 103, 104 and 105.

The interval from 100 up to the exclusive end 107 spans seven occupied positions in this case. Its accounting is four returned data records, one aborted data record at 101 and two markers at 102 and 106: 4 + 1 + 2 = 7. The gaps in the returned offsets are consistent with the stipulated log. Offset subtraction has counted an address extent, not seven business events.

Separately stipulate that the consumer has saved committed recovery position 107 after this complete read. It names where recovery resumes under the case's retained, valid-position assumptions. The saved value is an additional premise, not something inferred from seeing the last returned data record at 105. Reading past the trailing marker and persisting recovery progress are distinct facts.

References: [K41][KD41]

Four returned records leave the application outcome unresolved

The four-record result identifies input to the consumer. It supplies no one-to-one mapping to database rows, model requests or completed business actions. An application might combine several records into one feature row, reject a record during validation or create several actions from one input. KF-38 specifies none of those downstream behaviors.

Kafka's design discussion shows why reading and saved progress need separate treatment: processing can remain undone after progress is saved, or already completed work can be encountered again when progress was not saved before a failure. This article selects no recovery implementation. Its narrower conclusion is that recovery position 107 alone cannot certify the external effects of the four inputs.

Any promise about a database or model endpoint needs evidence for that application's processing and coordination contract. A committed Kafka transaction and read_committed visibility do not, by themselves, supply that evidence. The related API idempotency guide examines the separate scope within which repeated requests suppress additional effects.

References: [KD41]

Compare the returned population behind an end-offset or lag figure

An offer showing only an end-offset or lag figure provides an incomplete description of an AI dataset handoff. Establish what its number measures: which cluster, topic and partition, which reader isolation, which observation, and which end boundary and consumer position. In KF-38's first observation, the replicated high watermark is 106 while the read_committed end boundary is 104. Kafka's client reference also makes its read_committed fetch-lag metrics relative to the LSO.

An offer that additionally identifies the starting address, applicable visibility rule and actual returned offsets can explain the population being handed over. For KF-38, that evidence distinguishes the initial 100-and-103 input from the eventual 100, 103, 104 and 105 input. It accounts for both the withheld R3 and the excluded A1, rather than treating every offset gap as a missing application record.

Compare that population with the buyer's requested result. A progress-monitoring requirement may be answered by a properly defined progress metric. A requirement to hand over the eventual four-record input also needs evidence of that actual return. A declared read mode establishes a selection rule; a returned record list establishes what reached the reader during the stated observation. Neither establishes downstream completion.

The partition sequence also leaves source observation coverage and event-time correctness to establish separately. It creates no global sort across partitions and cannot recover a physical change never captured upstream. The related OPC UA guide addresses source sampling and retention; the Iceberg guide addresses selection of a committed table state. Those are different boundaries in an AI input chain.

References: [K41]

Connect a separate equipment-cabling requirement to its physical scope

If the platform choice also requires physical connections between installed equipment, Leviathan's Structured Cabling & Fiber Installation service provides a route to discuss agreed fiber and copper work, approved pathways, endpoint schedules, labels and connection records. Use the related project inquiry with a nonconfidential equipment and site outline, required connections and material-supply responsibilities.

That inquiry concerns the physical installation. Kafka administration, consumer integration and application-outcome guarantees belong to separately appointed software and platform providers.

References: [L1]