History

Seth Call 7d2ff00fbb Fix prod creds		2025-12-14 10:58:46 -06:00
..
charts	custom/patched chart	2025-12-12 22:34:44 -06:00
templates	remove default config	2025-12-13 23:20:13 -06:00
1	more	2025-04-06 17:58:43 -05:00
Chart.lock	Try to sync crds always with pre-commit	2025-12-12 15:31:23 -06:00
Chart.yaml	Revert "Attempt to fix enableOLTPReceiver bug in monioring bringup"	2025-12-05 19:02:06 -06:00
README.md	Update all webhooks	2025-12-13 22:57:18 -06:00
slack_webhooks.yaml	more	2025-04-06 17:58:43 -05:00
values-production.yaml	Fix prod creds	2025-12-14 10:58:46 -06:00
values-staging.yaml	Try new email settings	2025-12-14 08:51:59 -06:00

README.md

Monitoring and Alerting

Slack Webhook Configuration

The Slack notifications use a specific Incoming Webhook URL structure: https://hooks.slack.com/services/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

These ID components represent:

T0L5RA3E0: Slack Workspace ID (e.g., JamKazam)
B01SM8RC346: Bot/App Configuration ID (unique to the specific "Incoming Webhook" integration created in the Slack app management)
XDDOrcPE7eAXJPMCvc5FxIva: The Secret Token for authentication

Updating the Webhook

If you need to change the channel or regenerate the URL:

Go to Slack App Management.
Select the relevant App (e.g., "Monitoring" or "Incoming Webhooks").
Navigate to Incoming Webhooks.

Generating New Webhooks for Specific Channels

If you need to send alerts to a new channel (like #video-cluster-prd-alerts), the existing URL might not work if it's locked to a single channel. To create a new one:

Navigate to Slack App Management.
Select your App (likely named "Prometheus Alerts", "Incoming Webhooks", or similar).
In the left sidebar, click Incoming Webhooks.
Scroll to the bottom and click the button "Add New Webhook to Workspace".
Select the specific channel you want to post to (e.g., #video-cluster-prd-alerts).
Click Allow.
Copy the new Webhook URL generated in the table.
Update the api_url in the corresponding values.yaml file.