Send alerts by email and webhook

Add email and webhook notification channels to a project, confirm email addresses, verify webhook signatures, and test or rotate channels.

A notification channel is an email address or a webhook URL that alert rules send their notifications to. Channels belong to a project, and every rule of the project can use them.

Before you begin

  • To add, test, rotate, or delete channels, you need the Owner or Contributor role on the project or its subscription, or a custom role with Manage access to Observability. With Read access, as with the Reader role, you can view the channels, and webhook URLs are shortened to their scheme and host.
  • In the console, select the project whose alerts you want to send. The channels page shows the channels of that project.

Open the channels page

In the console, go to SettingsNotifications.

Select Delivery channels.

The page, Notification channels, lists each channel with its destination and its status.

Add an email channel

On the channels page, select Add channel.

Select Email.

Enter a Name and the Email address.

Select Add channel.

Verify an email address

An address that belongs to an active user of your organization is verified when you add it. Any other address receives a confirmation email, and the channel receives no alerts until the recipient confirms it.

The recipient opens the link in the confirmation email. They don't need to sign in.

On the Confirm alert notifications page, they select Confirm address.

The channel's status shows where it stands:

StatusMeaning
VerifiedThe channel receives notifications.
Awaiting confirmationThe confirmation link was sent and hasn't been used. The link is valid for 7 days and works once.
Link expiredThe link wasn't used within 7 days.

To send a new link, open the channel's Row actions menu (⋯), and select Resend confirmation. A new link replaces the previous one.

Notifications for an unconfirmed address are skipped, not held back: the address doesn't receive them after it's confirmed.

Add a webhook channel

On the channels page, select Add channel.

Select Webhook.

Enter a Name and the Webhook URL.

Select Add channel.

Copy the signing secret with Copy, and store it with your receiver.

The secret is shown once

The console shows a webhook's signing secret only after you add the channel or rotate its secret. If you lose it, rotate the secret. See Rotate a signing secret.

A webhook channel is verified when you add it. Its URL must meet these rules:

  • It uses https://, and it has at most 2,048 characters.
  • It doesn't contain a user name or password.
  • Its host resolves only to public IP addresses. Private, loopback, link-local, and other non-public ranges are refused. The platform checks the host at every send, and refuses a non-public IP address written in the URL when you add the channel.

The platform doesn't follow redirects.

Verify webhook requests

The platform sends each notification as an HTTPS POST with a JSON body and these headers:

HeaderValue
Content-Typeapplication/json
User-Agentpubliccloud-alerting/1
X-Publiccloud-Eventalerting.state_changed, or alerting.test for a test
X-Publiccloud-DeliveryThe notification's eventId. A test has no such header.
X-Publiccloud-Signaturet=<timestamp>,v1=<signature>

In the signature header, <timestamp> is the send time in Unix seconds, and <signature> is the hex-encoded HMAC-SHA256 of <timestamp>.<body>, keyed with the channel's signing secret. To verify a request, compute the same HMAC over the raw request body, as received, and compare it with v1. To reject replayed requests, also refuse requests whose timestamp is older than a tolerance that you choose.

Verify a webhook signature
import hashlib
import hmac
import time

def verify(secret: str, header: str, body: bytes, tolerance: int = 300) -> bool:
    fields = dict(item.split("=", 1) for item in header.split(","))
    timestamp = int(fields["t"])
    expected = hmac.new(
        secret.encode(), f"{timestamp}.".encode() + body, hashlib.sha256
    ).hexdigest()
    fresh = abs(time.time() - timestamp) <= tolerance
    return fresh and hmac.compare_digest(expected, fields["v1"])

secret is the whole signing secret, including its whsec_ prefix. header is the value of X-Publiccloud-Signature, and body is the raw request body.

The body has these fields:

FieldMeaning
typealerting.state_changed, or alerting.test for a test.
eventIdThe ID of the state change. A retry carries the same ID, so use it to drop duplicates.
projectIdThe project of the rule.
subscriptionIdSet instead of projectId for the alerts of a budget on a whole subscription.
ruleId, ruleNameThe rule that changed state.
instanceId, instanceNameThe instance that the notification is about. null for a budget alert.
fromStateThe state before the change: OK, FIRING, or INSUFFICIENT_DATA.
toStateFIRING when an alert fires, and OK for every notification that resolves an alert.
ruleStateThe state the instance moved to. It differs from toState only when an alert closes without a reading, for example INSUFFICIENT_DATA after monitoring was switched off.
resolvedtrue when the notification resolves an alert.
valueThe measured value, in the metric's unit. For an ERROR state rule, 1 in ERROR and 0 otherwise. null without a reading.
occurredAtThe time of the change, in UTC, in ISO 8601 format.
reasonWhy an alert closed other than by a reading: rule_disabled, rule_deleted, target_deleted, monitoring_disabled, or period_reset when a budget's new period starts. Otherwise null.

How notifications are delivered

Respond with a 2xx status code within 10 seconds. The platform gives up on a request after 15 seconds in total.

OutcomeWhat happens
2xx responseDelivered.
5xx, 408, or 429 response, a timeout, a connection error, or a host that doesn't resolveRetried after 1 minute, 5 minutes, 15 minutes, and 1 hour. After the fifth failed attempt, the notification is dropped.
Any other response, such as a redirect or a 4xx status, or a URL that the rules above refuseNot retried.

While a notification waits for a retry, later notifications of the same rule to the same channel wait behind it, so they arrive in order. Emails follow the same retries when the mail relay fails.

Email notifications. By default, the subject is the new state in brackets, [FIRING] or [RESOLVED], followed by the rule name and the instance. A test has [TEST]. These tags aren't translated, so you can filter your inbox on them. The body names the change, the value, the instance, the project, and the time. The email's language is set when the channel is added, from the language preference of the browser that added it, with English as the fallback.

Send a test notification

On the channels page, open the channel's Row actions menu (⋯).

Select Send test.

The platform sends the test right away, without retries:

  • Test notification delivered. The destination accepted it.
  • Not sent: the address is not confirmed yet. Confirm the address first.
  • For a failure, the message names the reason. See Troubleshooting.

You can send one test per channel every 60 seconds.

Rotate a signing secret

On the channels page, open the webhook channel's Row actions menu (⋯), and select Rotate signing secret.

In the confirmation, select Rotate signing secret.

Copy the new secret, and update your receiver.

The old secret stops signing at once, so requests fail your receiver's check until it has the new secret. To get the secret of a webhook channel that a budget created, rotate it in the budget's Edit dialog. See Edit or delete a budget.

Delete a channel

On the channels page, open the channel's Row actions menu (⋯), and select Delete.

Confirm with Delete.

Every rule that listed the channel stops notifying it, and the channel's address or URL is erased. You can't restore a deleted channel.

A channel marked Managed by a budget follows a budget's alert recipient, and you can't delete it here. Change the budget instead. See Create budgets and threshold alerts.

Result

The channel appears in the list. Once its status is Verified, it receives the notifications of every rule that lists it. To add a channel to a rule, see Create alert rules.

Limits

ResourceLimitAdjustable?Notes
Channels per projectSet by the platform operatorYes, by the operatorChannels that budgets manage don't count.
Webhook channels per projectSet by the platform operatorYes, by the operatorCounted within the channels per project.
Channel name1–255 charactersNoUnique in the project.
Email addressUp to 254 charactersNo
Webhook URLUp to 2,048 charactersNoHTTPS only.
Channels per rule10No
Test notifications1 per channel every 60 secondsNo
Confirmation emails1 per channel every 60 secondsNo
Confirmation linkValid for 7 days, onceNo

Troubleshooting

MessageCauseResolution
The channel was added, but the confirmation email could not be sent. Resend it later.The confirmation email couldn't be sent.Select Resend confirmation later.
This link is invalid, already used or expired. Ask the project owner to resend it.The link is wrong, was already used, or is older than 7 days.Ask a project owner or contributor to resend the confirmation.
The channel could not be added., or a message from the platformThe name is already used in the project, the address or URL is refused, or the project has reached its channel limit.Use another name or destination, delete a channel, or ask the platform operator to raise the limit.

When a test fails, the message names one of these reasons:

ReasonCause
webhook_http_<status>The receiver answered with this HTTP status code.
webhook_timeoutThe receiver didn't answer in time.
webhook_connection_errorThe connection or the TLS handshake failed.
webhook_host_unresolvableThe URL's host doesn't resolve.
webhook_target_blockedThe host resolves to a non-public IP address.
webhook_scheme_not_allowedThe URL isn't a valid HTTPS URL.
smtp_not_configuredThe platform has no mail relay. Contact the platform operator.
smtp_delivery_errorThe mail relay refused or failed the message.

On this page