Create alert rules

An alert rule watches every instance of a project that carries a set of tags. At each evaluation, it checks a condition for each of these instances, and it notifies its channels when an instance's alert fires and when it resolves.

Before you beginCopy Anchor Link

• To create, change, silence, or delete rules, you need the Owner or Contributor role on the project or its subscription, or a custom role with Manage access to Observability. With Read access, as with the Reader role, you can view rules but not change them.
• Tag the instances that you want to watch. See Organize resources with tags.
• For a metric rule, turn on monitoring on those instances. See View virtual machine metrics.
• To be notified, add a notification channel to the project first. A rule without channels still tracks its state, but it notifies no one. See Send alerts by email and webhook.

Create a ruleCopy Anchor Link

While you type the selector, the form shows how many instances match right now, and for a metric rule, how many of them have monitoring off. Below the condition, it repeats the condition as a sentence.

Choose a signalCopy Anchor Link

Instance in ERROR stateCopy Anchor Link

Instance in ERROR state fires while an instance is in the Error status, and resolves when it leaves it. An instance whose cloud status the platform doesn't recognize also shows Error in the console, but it doesn't fire the rule. The condition is fixed: you only set Fire after and Evaluation window. Each evaluation reads the instance's current status, so the window doesn't change the result; use Fire after to wait before the rule fires.

VM metricCopy Anchor Link

VM metric compares a metric of each instance with a threshold. You can alert on these metrics:

Disk latency, page cache, and swap are charted on an instance's Monitoring tab, but you can't alert on them. The metrics are the ones on the charts; see Read the charts.

A metric rule skips the matched instances whose monitoring is off: it doesn't evaluate or bill them. The rule's page shows how many it skips.

Choose instances with a tag selectorCopy Anchor Link

The rule watches every instance of the project that carries all the key–value pairs of the selector. For example, the selector env: prod and tier: web matches an instance tagged with both, but not one tagged only env: prod.

• Exact match. Keys and values match exactly, including case. There are no wildcards.
• Resolved at every evaluation. An instance that gets the tags later joins the rule at the next evaluation. An instance that loses a tag, or is deleted, leaves it, and an alert that it still holds open resolves once.
• No match is allowed. A rule whose selector matches nothing stays enabled, shows No data, and starts watching as soon as an instance carries the tags.

When you create a rule from a chart, the form offers that instance's tags as pairs that you can add with one click.

Set the conditionCopy Anchor Link

For a metric rule, set these fields:

• Aggregation over the window: average (the default) or maximum of the metric over the window. Average smooths short spikes; maximum reacts to any peak. The window is read as at most 60 samples, one per minute up to an hour and sparser beyond, so over long windows the maximum can miss shorter peaks.
• Comparison: >, ≥, <, or ≤. The default is >.
• Threshold: the value to compare with, in the metric's unit.

For both signals, set these fields:

• Evaluation window: the period that the metric is aggregated over. Rates such as CPU, disk, and network figures are computed over at least the last 120 seconds, so a 60-second window reflects the last two minutes.
• Fire after: how long the condition must hold before the rule fires. With 0, it fires at the next evaluation that meets the condition.

For disks and network interfaces, the rule compares the worst device: the highest value for > and ≥, and the lowest for < and ≤. So one rule covers every disk or interface of an instance.

How rules are evaluatedCopy Anchor Link

The platform evaluates every enabled rule on a regular cycle, separately for each matched instance. How often the cycle runs is set by the platform operator. Each instance has its own state and its own Fire after clock.

A new rule shows No data until its first evaluation.

For a metric rule, an instance has no data when it isn't running on the cloud yet, when the window holds no samples, or when the metrics store can't be reached. A missing reading restarts the instance's Fire after clock.

Notifications. The rule notifies its channels when:

• an instance's alert fires; an instance that goes from firing to no data and back to firing isn't notified twice;
• an instance's alert resolves, including through a period of no data;
• an open alert closes because monitoring was switched off for the instance, the instance left the selector, or the rule was silenced or deleted.

No data alone notifies no one. Every fire and resolve also appears in the rule's State history, and in the notification feed of the users who can read the project's alerts.

Manage a ruleCopy Anchor Link

To open a rule, select it in Observability › Alerts. The list shows each rule's state, selector, condition, number of channels, and when it was last evaluated. The rule's page shows its Properties, the Matched instances with each instance's state and last value, its Notification channels, and its State history, newest first.

Edit a ruleCopy Anchor Link

• A change to the selector or the condition makes the rule due at once and restarts each instance's clock. An instance that's firing and still breaches the new condition stays firing, without a new notification.
• A change to the name or the channels doesn't affect the evaluation.
• Widening the selector counts against the project's rule limit, like a new rule. See Limits.

Silence or resume a ruleCopy Anchor Link

On the rule's page, select Silence, and then Silence rule. Silencing a rule:

• resolves every alert that it holds open, and notifies the channels once for each firing instance;
• stops its evaluation, its notifications, and its billing.

A silence has no end date: the rule stays silent until someone selects Resume. Resuming starts the evaluation again, with every instance's clock at zero. Turning off Enabled in the rule's form does the same as silencing it.

Delete a ruleCopy Anchor Link

On the rule's page, select Delete, and confirm. The rule's open alerts resolve first, with a notification, and then its evaluation and billing stop. You can't restore a deleted rule.

ResultCopy Anchor Link

After you create a rule, the console opens its page. The rule shows No data until its first evaluation, in the next evaluation cycle. Then each matched instance shows its own state and last value. The page refreshes by itself while the rule is enabled.

LimitsCopy Anchor Link

Instance tags accept longer keys and values than a selector: a tag key can have up to 128 characters, and a value up to 256 characters or none. A selector can only match a tag whose key has at most 64 characters and whose value has 1–255 characters.

TroubleshootingCopy Anchor Link

A metric rule shows No data when every instance that it matches has monitoring off. Turn on monitoring for them. See View virtual machine metrics.

RelatedCopy Anchor Link

• Send alerts by email and webhook
• View virtual machine metrics
• Organize resources with tags
• Monitoring & Alerts FAQ
• To manage alert rules through the REST API, see POST/scopes/{scopeId}/observability/alert-rules.