How To: Process a System Alert WebHook

After deploying the RealTheory collector to a Kubernetes cluster (See Installing RealTheory in a Cluster), the cluster and its associated objects become available in the RealTheory API within minutes.

To receive a webhook callback, you must register and configure a webhook (See Configuring a Webhook). Once configured, you can confirm that alerts are reaching your service by reviewing the delivery history (See Viewing Webhook Delivery Status).

Process the System Alert Webhook

When a system alert is raised that you have configured for a callback, RealTheory sends an HTTP POST request to the URL registered for your webhook. The body of the request is a SystemAlertPayload object, serialized as JSON, that wraps the SystemAlert along with the domain and delivery metadata.

POST <your-webhook-url>
Content-Type: application/json
Authorization: Basic <base64(username:password)>
HTTP-X-HUB-SIGNATURE-256: sha256=<signature>

<SystemAlertPayload>

where

• <your-webhook-url> - is the endpoint URL you registered when configuring the webhook.
• Content-Type - is always application/json.
• Authorization - is an HTTP Basic authentication header, included only when a username and password were supplied when configuring the webhook. It allows your service to authenticate the request.
• HTTP-X-HUB-SIGNATURE-256 - is an HMAC-SHA256 signature of the raw request body, included only when a secret was supplied when configuring the webhook. The value is formatted as sha256= followed by the lower-case hexadecimal HMAC, and allows your service to verify the authenticity and integrity of the payload. See Verify the Payload Signature below.
• <SystemAlertPayload> - is the JSON body of the request. See SystemAlertPayload for a full description of each field.

Note: Your service should return an HTTP status code in the 2xx range to acknowledge successful receipt of the payload. Any other status code is treated as a delivery failure and is recorded in the webhook delivery status.

Example

POST https://myservice.example.com/realtheory/alerts
Content-Type: application/json
HTTP-X-HUB-SIGNATURE-256: sha256=7d38cdd689735b008b3c702edd92eea23791c5f6f853a0d2b9b6e1f8a3c9d4e2

{
    "domainId": "8ca6aa4c-8c7d-9fb9-c751-3e038f90d283",
    "domainName": "RealTheory, Inc.",
    "version": "1.0",
    "type": "alert",
    "action": "create",
    "url": "",
    "consoleUrl": "https://myendpoint.realtheory.io/alerts/12a3ae02-5b90-3701-6534-d4d9ab751e5d",
    "content": {
        "type": "System",
        "metadata": {
            "objectType": "Pod",
            "objectId": "14aecb16-ed6a-41f4-9be5-6e29b4e5cdce",
            "objectName": "logger-67b576f56b-krvqs",
            "clusterId": "f99e055f-b35d-208b-63b1-f4222b80151a",
            "clusterName": "realtheory-cluster-01",
            "nodeId": "c9e25d58-29a2-4d5f-b158-244f052f7d6f",
            "nodeName": "aks-userpool-15373879-vmss00002w",
            "namespaceId": "0b3c50fe-5116-44bd-b229-543251be4e74",
            "namespaceName": "web-app",
            "deploymentId": "b3039e89-64f9-44de-8164-e97a95e17880",
            "deploymentName": "logger",
            "podId": "14aecb16-ed6a-41f4-9be5-6e29b4e5cdce",
            "podName": "logger-67b576f56b-krvqs"
        },
        "id": "12a3ae02-5b90-3701-6534-d4d9ab751e5d",
        "key": "PodOOMKilledHealthCheck::14aecb16ed6a41f49be56e29b4e5cdce",
        "title": "The pod has been recently terminated with the OOMKilled status.",
        "message": "The 'Pod OOM Killed Check' has identified a high impact health concern on pod 'logger-67b576f56b-krvqs'.",
        "details": "Last Pod Status: OOMKilled, Last Exit Code: 137, Last Terminated: 2024-09-05 16:14:36Z",
        "level": "High",
        "category": "Health",
        "source": "Pod OOM Killed Check (health check)",
        "sourceId": "ddc94c84-4e77-40d9-a76c-cb8bac863f28",
        "state": "New",
        "advisoryId": "00000000-0000-0000-0000-000000000000",
        "created": "2024-08-07T10:16:35.0176406Z",
        "updated": "2024-09-05T16:20:09.6787782Z",
        "postponedUntil": null,
        "expires": "2024-09-05T16:25:08.1769592Z"
    },
    "timestamp": "2024-09-05T16:20:09.8312009Z"
}

Note: Content has been omitted from the JSON body above for brevity.

Verify the Payload Signature

If a secret was supplied when configuring the webhook, every request includes an HTTP-X-HUB-SIGNATURE-256 header. To verify that a request genuinely originated from RealTheory and was not altered in transit, recompute the signature on your side and compare it to the header value.

• Read the raw request body exactly as received, before any deserialization or re-encoding.
• Compute the HMAC-SHA256 of the raw body using the webhook secret as the key.
• Encode the result as a lower-case hexadecimal string and prefix it with sha256=.
• Compare the computed value to the value in the HTTP-X-HUB-SIGNATURE-256 header using a constant-time (fixed-time) string comparison. If the values do not match, reject the request.

Note: The signature is computed over the raw, unmodified request body. Recomputing it from a re-serialized object may produce a different result and cause verification to fail.

Process the Payload

After the request has been authenticated and, where applicable, its signature verified, your service can process the payload.

• Inspect the type and action fields to determine why the webhook was invoked. For a system alert, type is alert and action is create.
• Use the domainId and domainName fields to associate the alert with the correct RealTheory account.
• Read the content field, which is a SystemAlert object, to obtain the details of the alert, such as its level, category, title, message, and the metadata describing the Kubernetes object the alert is linked to.
• Use the consoleUrl field to provide a direct link back to the alert in the RealTheory portal, for example within a notification or ticket created by your service.