Send alerts by email and webhook
Add email and webhook notification channels to a project, confirm email addresses, verify webhook signatures, and test or rotate channels.
A notification channel is an email address or a webhook URL that alert rules send their notifications to. Channels belong to a project, and every rule of the project can use them.
Before you begin
- To add, test, rotate, or delete channels, you need the Owner or Contributor role on the project or its subscription, or a custom role with Manage access to Observability. With Read access, as with the Reader role, you can view the channels, and webhook URLs are shortened to their scheme and host.
- In the console, select the project whose alerts you want to send. The channels page shows the channels of that project.
Open the channels page
In the console, go to SettingsNotifications.
Select Delivery channels.
The page, Notification channels, lists each channel with its destination and its status.
Add an email channel
On the channels page, select Add channel.
Select Email.
Enter a Name and the Email address.
Select Add channel.
Verify an email address
An address that belongs to an active user of your organization is verified when you add it. Any other address receives a confirmation email, and the channel receives no alerts until the recipient confirms it.
The recipient opens the link in the confirmation email. They don't need to sign in.
On the Confirm alert notifications page, they select Confirm address.
The channel's status shows where it stands:
| Status | Meaning |
|---|---|
| Verified | The channel receives notifications. |
| Awaiting confirmation | The confirmation link was sent and hasn't been used. The link is valid for 7 days and works once. |
| Link expired | The link wasn't used within 7 days. |
To send a new link, open the channel's Row actions menu (⋯), and select Resend confirmation. A new link replaces the previous one.
Notifications for an unconfirmed address are skipped, not held back: the address doesn't receive them after it's confirmed.
Add a webhook channel
On the channels page, select Add channel.
Select Webhook.
Enter a Name and the Webhook URL.
Select Add channel.
Copy the signing secret with Copy, and store it with your receiver.
The secret is shown once
The console shows a webhook's signing secret only after you add the channel or rotate its secret. If you lose it, rotate the secret. See Rotate a signing secret.
A webhook channel is verified when you add it. Its URL must meet these rules:
- It uses
https://, and it has at most 2,048 characters. - It doesn't contain a user name or password.
- Its host resolves only to public IP addresses. Private, loopback, link-local, and other non-public ranges are refused. The platform checks the host at every send, and refuses a non-public IP address written in the URL when you add the channel.
The platform doesn't follow redirects.
Verify webhook requests
The platform sends each notification as an HTTPS POST with a JSON body and these headers:
| Header | Value |
|---|---|
Content-Type | application/json |
User-Agent | publiccloud-alerting/1 |
X-Publiccloud-Event | alerting.state_changed, or alerting.test for a test |
X-Publiccloud-Delivery | The notification's eventId. A test has no such header. |
X-Publiccloud-Signature | t=<timestamp>,v1=<signature> |
In the signature header, <timestamp> is the send time in Unix seconds, and <signature> is the hex-encoded HMAC-SHA256 of <timestamp>.<body>, keyed with the channel's signing secret. To verify a request, compute the same HMAC over the raw request body, as received, and compare it with v1. To reject replayed requests, also refuse requests whose timestamp is older than a tolerance that you choose.
import hashlib
import hmac
import time
def verify(secret: str, header: str, body: bytes, tolerance: int = 300) -> bool:
fields = dict(item.split("=", 1) for item in header.split(","))
timestamp = int(fields["t"])
expected = hmac.new(
secret.encode(), f"{timestamp}.".encode() + body, hashlib.sha256
).hexdigest()
fresh = abs(time.time() - timestamp) <= tolerance
return fresh and hmac.compare_digest(expected, fields["v1"])secret is the whole signing secret, including its whsec_ prefix. header is the value of X-Publiccloud-Signature, and body is the raw request body.
The body has these fields:
| Field | Meaning |
|---|---|
type | alerting.state_changed, or alerting.test for a test. |
eventId | The ID of the state change. A retry carries the same ID, so use it to drop duplicates. |
projectId | The project of the rule. |
subscriptionId | Set instead of projectId for the alerts of a budget on a whole subscription. |
ruleId, ruleName | The rule that changed state. |
instanceId, instanceName | The instance that the notification is about. null for a budget alert. |
fromState | The state before the change: OK, FIRING, or INSUFFICIENT_DATA. |
toState | FIRING when an alert fires, and OK for every notification that resolves an alert. |
ruleState | The state the instance moved to. It differs from toState only when an alert closes without a reading, for example INSUFFICIENT_DATA after monitoring was switched off. |
resolved | true when the notification resolves an alert. |
value | The measured value, in the metric's unit. For an ERROR state rule, 1 in ERROR and 0 otherwise. null without a reading. |
occurredAt | The time of the change, in UTC, in ISO 8601 format. |
reason | Why an alert closed other than by a reading: rule_disabled, rule_deleted, target_deleted, monitoring_disabled, or period_reset when a budget's new period starts. Otherwise null. |
How notifications are delivered
Respond with a 2xx status code within 10 seconds. The platform gives up on a request after 15 seconds in total.
| Outcome | What happens |
|---|---|
2xx response | Delivered. |
5xx, 408, or 429 response, a timeout, a connection error, or a host that doesn't resolve | Retried after 1 minute, 5 minutes, 15 minutes, and 1 hour. After the fifth failed attempt, the notification is dropped. |
Any other response, such as a redirect or a 4xx status, or a URL that the rules above refuse | Not retried. |
While a notification waits for a retry, later notifications of the same rule to the same channel wait behind it, so they arrive in order. Emails follow the same retries when the mail relay fails.
Email notifications. By default, the subject is the new state in brackets, [FIRING] or [RESOLVED], followed by the rule name and the instance. A test has [TEST]. These tags aren't translated, so you can filter your inbox on them. The body names the change, the value, the instance, the project, and the time. The email's language is set when the channel is added, from the language preference of the browser that added it, with English as the fallback.
Send a test notification
On the channels page, open the channel's Row actions menu (⋯).
Select Send test.
The platform sends the test right away, without retries:
- Test notification delivered. The destination accepted it.
- Not sent: the address is not confirmed yet. Confirm the address first.
- For a failure, the message names the reason. See Troubleshooting.
You can send one test per channel every 60 seconds.
Rotate a signing secret
On the channels page, open the webhook channel's Row actions menu (⋯), and select Rotate signing secret.
In the confirmation, select Rotate signing secret.
Copy the new secret, and update your receiver.
The old secret stops signing at once, so requests fail your receiver's check until it has the new secret. To get the secret of a webhook channel that a budget created, rotate it in the budget's Edit dialog. See Edit or delete a budget.
Delete a channel
On the channels page, open the channel's Row actions menu (⋯), and select Delete.
Confirm with Delete.
Every rule that listed the channel stops notifying it, and the channel's address or URL is erased. You can't restore a deleted channel.
A channel marked Managed by a budget follows a budget's alert recipient, and you can't delete it here. Change the budget instead. See Create budgets and threshold alerts.
Result
The channel appears in the list. Once its status is Verified, it receives the notifications of every rule that lists it. To add a channel to a rule, see Create alert rules.
Limits
| Resource | Limit | Adjustable? | Notes |
|---|---|---|---|
| Channels per project | Set by the platform operator | Yes, by the operator | Channels that budgets manage don't count. |
| Webhook channels per project | Set by the platform operator | Yes, by the operator | Counted within the channels per project. |
| Channel name | 1–255 characters | No | Unique in the project. |
| Email address | Up to 254 characters | No | |
| Webhook URL | Up to 2,048 characters | No | HTTPS only. |
| Channels per rule | 10 | No | |
| Test notifications | 1 per channel every 60 seconds | No | |
| Confirmation emails | 1 per channel every 60 seconds | No | |
| Confirmation link | Valid for 7 days, once | No |
Troubleshooting
| Message | Cause | Resolution |
|---|---|---|
| The channel was added, but the confirmation email could not be sent. Resend it later. | The confirmation email couldn't be sent. | Select Resend confirmation later. |
| This link is invalid, already used or expired. Ask the project owner to resend it. | The link is wrong, was already used, or is older than 7 days. | Ask a project owner or contributor to resend the confirmation. |
| The channel could not be added., or a message from the platform | The name is already used in the project, the address or URL is refused, or the project has reached its channel limit. | Use another name or destination, delete a channel, or ask the platform operator to raise the limit. |
When a test fails, the message names one of these reasons:
| Reason | Cause |
|---|---|
webhook_http_<status> | The receiver answered with this HTTP status code. |
webhook_timeout | The receiver didn't answer in time. |
webhook_connection_error | The connection or the TLS handshake failed. |
webhook_host_unresolvable | The URL's host doesn't resolve. |
webhook_target_blocked | The host resolves to a non-public IP address. |
webhook_scheme_not_allowed | The URL isn't a valid HTTPS URL. |
smtp_not_configured | The platform has no mail relay. Contact the platform operator. |
smtp_delivery_error | The mail relay refused or failed the message. |
Related
- Create alert rules
- What is Monitoring & Alerts?
- Create budgets and threshold alerts
- To manage channels through the REST API, see POST
/scopes/{scopeId}/observability/notification-channels.