AI Worker Pressure: Partial Stalls and Whole-Group Lost Progress

Ask how long work waited, alongside what it consumed

An AI data worker can depend on an I/O operation before its next computation can proceed. A CPU-usage figure describes consumption; it does not supply the time relationships between that worker's waits and those of its companions. Likewise, a current memory charge or a reclaim-event count leaves the duration of overlapping waits unanswered.

Linux 6.18's Pressure Stall Information documentation provides a useful distinction for that question. Its some measure covers time with resource-stalled tasks. Its full measure covers the subset when all non-idle tasks in scope are stalled on that resource together. The difference matters when several tasks could otherwise progress independently.

The related CPU request/limit and host-memory guides explain their own quantities. This guide adds the missing time relationship, using I/O pressure in one selected task group. It does not infer why a real worker waited from a low usage graph.

References: [P1]

Define the group before interpreting full

The documentation describes both system and cgroup2 pressure reporting. A cgroup report concerns its grouped tasks. Full therefore needs the selected population attached: full for a worker group is a different claim from full for the whole host. Other host tasks may continue running while the selected group waits.

The resource also remains part of the identity. This example concerns I/O. System-level CPU full is undefined and reported as zero for compatibility; that zero must not be promoted into evidence that the host has no CPU contention.

A report for an entire service can conceal a smaller group's experience, while a narrowly selected group cannot establish what happened to omitted workers. Ask which application tasks the supplied scope actually represents before comparing its percentages.

References: [P1]

PSI-2: eight task-milliseconds contain two milliseconds of shared waiting

PSI-2 is an entirely fictional comparison over ten milliseconds, from time 0 up to time 10. Exactly two tasks, A and B, remain non-idle throughout this stable selected group. The only relevant I/O stalls are the supplied intervals below. There are no other tasks, idle periods or omitted I/O waits. These assumptions define the teaching example, not a host measurement.

In profile P, A waits from 0 up to 4 milliseconds. B waits from 2 up to 6. Each task waits four milliseconds, giving four plus four, or eight task-milliseconds. That sum counts time separately for each task.

At least one task waits continuously from 0 up to 6: six milliseconds of some time. Both wait from 2 up to 4: two milliseconds of full time. The shared two-millisecond interval is already inside the six-millisecond union. Six plus two would not be the elapsed time with any waiting.

For completeness, the six milliseconds divide into four with only one task stalled and two with both stalled. The final four milliseconds have neither task in the stipulated I/O stall state. The intervals include their start and exclude their end, so a boundary instant is not an extra duration.

Move one wait and the full time becomes zero

Keep A's interval at 0 up to 4 milliseconds, but move B's interval to 4 up to 8. This is profile Q. Each task still waits four milliseconds, so the sum remains eight task-milliseconds.

Now at least one waits from 0 up to 8: eight milliseconds of some time. The two intervals meet at time 4 without overlapping. No positive-duration interval has both tasks stalled, so full time is zero.

Q's zero full time does not mean no I/O waiting. It means these two non-idle tasks never share an I/O-stalled interval under the stated assumptions. At each point during the first eight milliseconds, one task is outside that stall state. That is an opportunity for other progress, not evidence that the task actually executed or completed useful work.

A summary retaining only the common eight task-milliseconds cannot recover whether the waits overlapped. Comparing only some would also miss the reversal: P has less some time than Q, yet more full time. Neither one-number ranking supplies the application's required outcome.

Use a common time basis for totals and percentages

The named kernel documentation expresses the absolute total in microseconds and separately describes recent trends over ten, sixty and three hundred seconds. Those trend fields answer a different time-basis question from this ten-millisecond construction.

Under PSI-2's idealized complete intervals, P's some/full accumulated durations correspond to 6,000 and 2,000 microseconds. Q's correspond to 8,000 and zero. The common elapsed window is 10,000 microseconds. Dividing each duration by that window gives P at 60% some and 20% full, and Q at 80% some and 0% full.

These are custom-interval arithmetic results. They are not recorded kernel counters, a prediction of avg10, or a short-window reconstruction of the other trend fields. For an actual supplied total-counter comparison, the identities and observations must support the stated interval. The related cumulative-counter guide explains discontinuities and why the cumulative level itself is not the interval result.

A missing pressure observation is also not a supplied zero. Preserve that absence instead of manufacturing the Q result. The related gauge guide develops the separate problem of incomplete observation coverage.

References: [P1]

Choose the evidence that answers the worker-progress question

Consider two fictional returns for PSI-2. Return One supplies only eight task-milliseconds and calls the two profiles equivalent. Return Two supplies the declared group, resource, observation window and individual stall intervals, then explains both some and full.

One correctly adds each task's waiting, but leaves simultaneous group waiting unresolved. Two supports the narrower, useful finding: P contains a two-millisecond interval in which neither member can move beyond its I/O wait; Q does not. If the buyer needs to understand whether companion work is also blocked, that difference belongs in the comparison.

It still does not establish that Q completes more inference requests or responds faster. The example contains no work sizes, dependencies between tasks, execution schedule, queue lengths or accepted outputs. A task outside the I/O stall state need not be running, and another dependency can govern the service. Keep pressure evidence alongside the actual useful-work and response requirements.

For a real offer, ask the responsible platform provider to connect its supplied resource-pressure view to the intended worker population and operating phase. A whole-host report and an application-group report need that relationship explained. If another task joins the group, or the workload or window changes, the two-task comparison no longer describes that scope.

The completed evidence can identify which kind of waiting was observed without selecting a cause or remedy. These numbers do not identify a faulty SSD, establish that more memory or CPU will help, or supply an operating threshold. Those conclusions require the appointed team's application and system evidence.

Take a separate equipment decision into its physical work package

Resource-pressure interpretation belongs with the appointed platform and application teams. It does not itself establish a need to add or replace equipment.

If their supported equipment plan also needs physical installation, Leviathan's Rack-and-Stack / Colocation service can discuss agreed mounting, IT cabling, labels and physical handoff records. Use the related project inquiry with a nonconfidential equipment list, rack positions, site and requested physical work. Monitoring design, Linux tuning and software acceptance remain separate responsibilities.

References: [L1]