video-iac/k8s/monitoring/README.md

Monitoring and Alerting

Slack Webhook Configuration

The Slack notifications use a specific Incoming Webhook URL structure: https://hooks.slack.com/services/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

These ID components represent:

• T0L5RA3E0: Slack Workspace ID (e.g., JamKazam)
• B01SM8RC346: Bot/App Configuration ID (unique to the specific "Incoming Webhook" integration created in the Slack app management)
• XDDOrcPE7eAXJPMCvc5FxIva: The Secret Token for authentication

Updating the Webhook

If you need to change the channel or regenerate the URL:

1. Go to Slack App Management.
2. Select the relevant App (e.g., "Monitoring" or "Incoming Webhooks").
3. Navigate to Incoming Webhooks.

Generating New Webhooks for Specific Channels

If you need to send alerts to a new channel (like #video-cluster-prd-alerts), the existing URL might not work if it's locked to a single channel. To create a new one:

1. Navigate to Slack App Management.
2. Select your App (likely named "Prometheus Alerts", "Incoming Webhooks", or similar).
3. In the left sidebar, click Incoming Webhooks.
4. Scroll to the bottom and click the button "Add New Webhook to Workspace".
5. Select the specific channel you want to post to (e.g., #video-cluster-prd-alerts).
6. Click Allow.
7. Copy the new Webhook URL generated in the table.
8. Update the api_url in the corresponding values.yaml file.