--- title: "Prometheus Alerts: Firing, Grouping and Received Notifications" description: "Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20." canonical: "https://www.leviathansystems.co/articles/prometheus-alert-state-versus-notification-delivery" last-updated: "2026-10-20" --- # Prometheus Alerts: Firing, Grouping and Received Notifications Leviathan Systems — Technical content Published 2026-10-20; updated 2026-10-20. A firing alert is a rule-engine result. Alertmanager can group several alert identities into one notification and apply notification gates before dispatch. A separately evidenced receiver receipt establishes a later result; human acknowledgment, response action and actual repair each need further evidence. Compare an alerting offer by where its promised output ends and which identities remain visible. ## Follow the instance behind the rule name An alerting rule evaluates an expression. Prometheus v3.5.0 describes returned vector elements as active for their label sets. One named rule can therefore have several alert instances. Rules can add or replace labels; retain the effective alert labels when comparing instances. Labels identify an instance; annotations carry explanatory information. The v3.5.0 implementation updates an existing active instance's value and annotations without creating a new identity. A changed identifying label set must be treated separately. Seeing the same rule name twice is insufficient evidence of uninterrupted activity by the same instance. For a buyer, the distinction is practical: a single rule badge can conceal several affected instances. Preserve the identities through the later notification account. Their existence establishes an evaluated condition, without supplying a hardware diagnosis or the application's complete outcome. This explanation uses Prometheus v3.5.0 and Alertmanager v0.28.1, checked October 20, 2026. These references establish neither the releases in an offered platform nor a verified pairing. The current Alertmanager concepts page supplements the tagged references for grouping and silences. References: [P1](https://github.com/prometheus/prometheus/blob/v3.5.0/docs/configuration/alerting_rules.md), [P2](https://github.com/prometheus/prometheus/blob/v3.5.0/rules/alerting.go#L393-L480), [A1](https://github.com/prometheus/alertmanager/blob/v0.28.1/docs/configuration.md#route), [A2](https://prometheus.io/docs/alerting/latest/alertmanager/) ## Persistence delays firing; retention can extend it The optional for duration requires an instance to remain active at successive evaluations before firing. While waiting, it is pending. With no persistence duration, it can fire on its first successful evaluation. The mechanism concerns elapsed persistence for that identity, rather than a universal required number of samples. The separate optional keep_firing_for behavior can retain firing after the condition ceases to match. Consequently, a newer reading that looks different can coexist with a firing state. That explanation applies only when the actual rule and history support it; a screenshot alone identifies no cause. This also separates two buyer questions: when did the rule become firing, and what does its current state mean under its retention behavior? Neither answer establishes when a notification reached a receiver. References: [P1](https://github.com/prometheus/prometheus/blob/v3.5.0/docs/configuration/alerting_rules.md), [P2](https://github.com/prometheus/prometheus/blob/v3.5.0/rules/alerting.go#L393-L480) ## Routing selects handling; grouping changes the notification unit In Alertmanager v0.28.1, incoming alerts enter the root route, then follow label-matching child routes. If no child matches, handling stays at the current node. Continuation behavior can allow matching subsequent siblings. Grouping can collect distinct identities into one notification. Sharing the grouping labels does not make their full label sets identical. A group can retain the distinction between instances while presenting their alerts together. The buyer therefore needs both the notification and its included identities to understand what the message represents. Reducing the message count through grouping need not reduce the number of firing conditions represented. Conversely, two firing identities supply no evidence that two messages were delivered. A notification count and an alert-instance count use different units. References: [A1](https://github.com/prometheus/alertmanager/blob/v0.28.1/docs/configuration.md#route), [A2](https://prometheus.io/docs/alerting/latest/alertmanager/) ## A firing instance can still await or miss notification The tagged route reference distinguishes three waits. group_wait precedes an initial group notification; group_interval governs notification about new alerts joining an already notified group; repeat_interval governs resending after a successful send. Repeat timing is also bounded by Alertmanager's data-retention period. None is the rule's persistence duration or a receiver-delivery guarantee. Notification gates matter separately. A matching active silence suppresses notifications for an alert. Route time intervals can also restrict sending. These controls concern notification handling; they do not change the evidence that Prometheus supplied a firing instance. Inhibition supplies another conditional gate. A target must match the inhibition rule, and a qualifying source alert must exist with matching values for the required equal labels. The tagged implementation excludes resolved sources and handles the self-inhibition case separately. A higher-severity label by itself is insufficient evidence that another alert was inhibited. Absent mail leaves the cause unresolved. It cannot distinguish permitted waiting, suppression, a delivery problem or an earlier missing observation. Establish the applicable evidence before assigning any of those explanations. This discussion selects no production timing, silence or inhibition policy. References: [A1](https://github.com/prometheus/alertmanager/blob/v0.28.1/docs/configuration.md#route), [A2](https://prometheus.io/docs/alerting/latest/alertmanager/), [I1](https://github.com/prometheus/alertmanager/blob/v0.28.1/inhibit/inhibit.go#L118-L235) ## OPS-ALERT-02: two firing identities, one received notification OPS-ALERT-02 is wholly fictional. Alert identities A and B share a rule name and cluster label but have different instance-label values. Both conditions are true at every stipulated successful evaluation at 0, 60 and 120 seconds. The fictional persistence requirement is two minutes. There are no evaluation gaps, identity changes or intervening resets of pending state. At 0 seconds, each instance begins pending. At 60 seconds, each has persisted for one minute and remains pending. At 120 seconds, two minutes have elapsed since first activation, and both become firing. Three evaluation instants appear in this example because of its stipulated clock; three samples are not a general firing requirement. The duration and interval are illustrative, rather than proposed settings. Now supply separate premises for the notification layer. Alertmanager receives both firing instances. Its applicable route groups by their shared rule and cluster labels, excluding the distinguishing instance label from the grouping key. All notification gates permit sending, and the applicable wait has elapsed. At the one stipulated destination, independent receiver evidence supplies exactly one received grouped notification containing both A and B. No elapsed delivery time is supplied. The account contains two firing identities and one received notification. Reporting the message as one detected condition discards the retained distinction between A and B. Reporting two firing identities as two delivered messages invents a receipt the example does not supply. Grouping explains how one message can represent both identities; the independently supplied receipt establishes that it actually did so in this invented account. Remove that receipt and the rule states plus grouping policy establish no received-message count. These are stipulated facts and reasoning, without a pipeline measurement, real incident or operational test. References: [P1](https://github.com/prometheus/prometheus/blob/v3.5.0/docs/configuration/alerting_rules.md), [P2](https://github.com/prometheus/prometheus/blob/v3.5.0/rules/alerting.go#L393-L480), [A1](https://github.com/prometheus/alertmanager/blob/v0.28.1/docs/configuration.md#route), [A2](https://prometheus.io/docs/alerting/latest/alertmanager/) ## Compare the promised endpoint beyond dispatch Use receiver receipt to mean independent evidence from the identified destination that identifies the notification received. An outgoing attempt supplies a different observation. Receipt by an automated integration also leaves unanswered whether a person read or acknowledged it. Human acknowledgment identifies a further event. A subsequent response action identifies what someone or an application did. Evidence of actual repair must address the affected system or application's agreed return condition. Acknowledging a message supplies no action record, and taking an action supplies no result by itself. Compare two fictional offers against OPS-ALERT-02. Offer One supplies rule-state history and the intended routing policy. Offer Two also supplies the independent grouped receiver receipt identifying A and B. Two supports receipt of that notification; One leaves receipt unresolved. Neither offer, with only those materials, establishes acknowledgment or repair. The distinction concerns supported outcomes, without implying a difference in hardware reliability. A buyer seeking detection needs the rule evidence and its instance coverage. A buyer seeking delivered notifications additionally needs the destination and receipt evidence. A buyer seeking an operating response needs the separately defined acknowledgment, action and outcome responsibilities. Ask the proposed platform or observability provider which endpoint it supplies and who owns what follows. A lower message count alone cannot rank those offers. Related guides retain the neighboring questions: Redfish covers metric time and resource health; sampled traces cover observed request populations; cumulative counters cover reset-aware aggregation. The GPU Operator guide addresses component management, and the recovery-objectives guide addresses evidence of usable service return. Those decisions complement this rule-to-receiver account. ## Scope a separate physical equipment project If the selected infrastructure also needs physical equipment work, Leviathan's Complete Physical Deployment service brings receiving, rack integration, agreed hardware QA, onsite installation and cabling into a defined package. Use the service and project-inquiry links with a nonconfidential equipment summary, site, requested physical stages and receiving team. Alerting is an educational subject here. Monitoring integration, software operation and incident response remain with the separately appointed platform and operating teams. The project inquiry concerns the physical equipment package. References: [L1](https://www.leviathansystems.co/services/turnkey-gpu-deployment) ## Primary sources - [P1 — Prometheus v3.5.0: Alerting rules](https://github.com/prometheus/prometheus/blob/v3.5.0/docs/configuration/alerting_rules.md). Checked 2026-10-20. Defines expression activity, optional persistence and firing retention, labels, annotations and the separate notification layer. - [P2 — Prometheus v3.5.0: Alert-instance identity and state transitions](https://github.com/prometheus/prometheus/blob/v3.5.0/rules/alerting.go#L393-L480). Checked 2026-10-20. Confirms identity from effective labels, updates to annotations and values, and elapsed-time transition from pending to firing. - [A1 — Alertmanager v0.28.1: Routing and notification timing](https://github.com/prometheus/alertmanager/blob/v0.28.1/docs/configuration.md#route). Checked 2026-10-20. Tagged route reference for label matching, grouping, notification waits and time gates; no installed configuration is assumed. - [A2 — Prometheus: Alertmanager grouping, inhibition and silences](https://prometheus.io/docs/alerting/latest/alertmanager/). Checked 2026-10-20. Current concepts overview for grouped notifications and active silences; used alongside the tagged behavior references. - [I1 — Alertmanager v0.28.1: Inhibition matching](https://github.com/prometheus/alertmanager/blob/v0.28.1/inhibit/inhibit.go#L118-L235). Checked 2026-10-20. Confirms target matching, qualifying sources, equal-label comparisons and resolved-source and self-inhibition handling. - [L1 — Leviathan Systems: Complete Physical Deployment](https://www.leviathansystems.co/services/turnkey-gpu-deployment). Checked 2026-10-20. Defines the separately selectable physical delivery package and project inquiry, without supplying an alerting or operating service. ## Related Leviathan resources - [Redfish metric time and resource health](/articles/redfish-metric-report-time-versus-resource-health) - [Sampled traces and the request error population](/articles/ai-observability-trace-sampling-versus-request-metrics) - [Cumulative counter resets before aggregation](/articles/inference-cumulative-counter-resets-before-aggregation) - [GPU Operator component-management scope](/articles/nvidia-gpu-operator-versus-device-plugin-scope) - [Recovery objectives and observed service return](/articles/ai-workload-recovery-objectives-versus-restore-observations) - [Complete Physical Deployment service](/services/turnkey-gpu-deployment) - [Discuss a separate physical equipment project](/contact?service=turnkey-gpu-deployment&from=%2Farticles%2Fprometheus-alert-state-versus-notification-delivery) ## Have a separate physical equipment project? Share the equipment, site, requested physical stages and receiving team to discuss installation scope, requirements and availability. [Discuss the scope](/contact?service=turnkey-gpu-deployment&from=/articles/prometheus-alert-state-versus-notification-delivery)