Jul 6, 2025·10 min read·

Setup Grafana Alerts to PagerDuty for Website Down Monitoring

Get notified instantly when your website goes down — configure Grafana alerting rules with PagerDuty integration for reliable incident response.

GrafanaPagerDutyMonitoringAlerts

Grafana → PagerDuty Alert Flow

Website Down Detection & Notification

WebsiteHealth Check

Probe

▼

BlackboxExporter

Scrape

▼

PrometheusMetrics

Query

▼

GrafanaAlert Rule

Alert

▼

PagerDutyIncident

Call

▼

On-CallPhone/SMS

WebsiteHealth Check

Probe

BlackboxExporter

Scrape

PrometheusMetrics

Query

GrafanaAlert Rule

Alert

PagerDutyIncident

Call

On-CallPhone/SMS

Why Grafana + PagerDuty?

When your website goes down at 3 AM, you need to know immediately — not when a customer tweets about it. Grafana monitors your site health via Prometheus metrics, and PagerDuty ensures the right person gets woken up with proper escalation policies.

Grafana

Flexible alerting rules, multi-condition evaluation, silence periods, and dashboards for visual monitoring.

PagerDuty

On-call scheduling, escalation policies, phone/SMS/push notifications, and incident timeline tracking.

Prerequisites

▹Grafana installed (v9+ recommended)
▹Prometheus scraping your website (or Blackbox Exporter)
▹PagerDuty account (free tier works)
▹Website running with a health endpoint

Step 1: Install Grafana on Ubuntu

# Install prerequisites

sudo apt-get install -y apt-transport-https software-properties-common wget

# Add Grafana GPG key

sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

# Add Grafana repository

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

# Install Grafana

sudo apt-get update
sudo apt-get install -y grafana

# Start and enable Grafana

sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server
sudo systemctl status grafana-server

Access: Open http://your-server-ip:3000 — Default login: admin / admin (change immediately).

Step 2: Install Prometheus

# Create prometheus user

sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus

# Download Prometheus

cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.53.0/prometheus-2.53.0.linux-amd64.tar.gz
tar xvfz prometheus-2.53.0.linux-amd64.tar.gz

# Install binaries

sudo cp prometheus-2.53.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.53.0.linux-amd64/promtool /usr/local/bin/
sudo cp -r prometheus-2.53.0.linux-amd64/consoles /etc/prometheus/
sudo cp -r prometheus-2.53.0.linux-amd64/console_libraries /etc/prometheus/
sudo chown -R prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool /etc/prometheus

# /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

# Create systemd service

sudo tee /etc/systemd/system/prometheus.service <<EOF
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file /etc/prometheus/prometheus.yml \
  --storage.tsdb.path /var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
EOF

# Start Prometheus

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
sudo systemctl status prometheus

Verify: Open http://your-server-ip:9090 — Prometheus UI should load.

Step 3: Add Prometheus Data Source in Grafana

Open Data Sources

Grafana → Connections → Data sources → Add data source

Select Prometheus

Search "Prometheus" → Click it

Configure URL

URL: http://localhost:9090 → Click Save & test→ Should show "Data source is working"

Step 4: Setup Blackbox Exporter

Blackbox Exporter probes your website and exposes metrics to Prometheus — it checks if your site is up, response time, SSL expiry, etc.

# Install Blackbox Exporter

sudo apt-get install prometheus-blackbox-exporter

# /etc/prometheus/blackbox.yml

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      follow_redirects: true
      preferred_ip_protocol: "ip4"

# Add to prometheus.yml

- job_name: 'website-monitor'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
        - https://aiwithaman.com
        - https://aiwithaman.com/blog
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: localhost:9115

# Restart services

sudo systemctl restart prometheus-blackbox-exporter
sudo systemctl restart prometheus

Step 5: Create PagerDuty Integration Key

Create a Service

Go to PagerDuty → Services → New Service→ Name it "Website Monitoring"

Add Integration

Under Integrations → Add Integration → Search "Events API V2" → Click Add

Copy Integration Key

Copy the Integration Key(routing key) — you'll need this in Grafana. It looks like: a1b2c3d4e5f6g7h8i9j0...

Step 6: Configure PagerDuty in Grafana

Open Contact Points

Grafana → Alerting → Contact points → Add contact point

Configure PagerDuty

• Name: PagerDuty - Website
• Type: PagerDuty
• Integration Key: paste your key from Step 2
• Severity: critical

Test

Click Test → You should receive a test alert in PagerDuty. Then Save.

Step 7: Create Alert Rule for Website Down

Navigate to Alert Rules

Grafana → Alerting → Alert rules → New alert rule

# PromQL query for website down

probe_success{job="website-monitor"} == 0

Alert Configuration

• Rule name: Website Down - aiwithaman.com
• Query: probe_success == 0
• Condition: IS BELOW 1
• Evaluate every: 1 minute
• For: 2 minutes (avoids flapping)
• Contact point: PagerDuty - Website
• Summary: Website is DOWN

Step 8: Add Response Time Alert (Optional)

Alert when response time exceeds a threshold — catch slow performance before it becomes downtime.

# Alert if response time exceeds 3 seconds

probe_http_duration_seconds{phase="transfer", job="website-monitor"} > 3

# Alert if SSL certificate expires within 30 days

probe_ssl_earliest_cert_expiry - time() < 86400 * 30

Step 9: Setup Notification Policy

Configure Routing

Grafana → Alerting → Notification policies

• Default contact point: PagerDuty - Website
• Group by: alertname, instance
• Group wait: 30s
• Group interval: 5m
• Repeat interval: 4h

Step 10: PagerDuty Escalation Policy

Recommended Setup

• Level 1: Primary on-call → immediate push + SMS
• Level 2: After 5 min no ACK → phone call to primary
• Level 3: After 10 min → escalate to secondary on-call
• Level 4: After 15 min → notify team lead

Step 11: Setup Phone Call Alerts

Configure PagerDuty to actually call your phone when a critical alert fires — push notifications can be missed, but a phone call at 3 AM won't be.

Add Phone Number

PagerDuty → My Profile → Contact Information → Add your phone number with country code. Verify via SMS code.

Configure Notification Rules

My Profile → Notification Rules → Set up:

• Immediately: Push notification + SMS
• After 2 min: Phone call
• After 5 min: Phone call again (if not acknowledged)

Set High-Urgency for Website Down

Go to Services → Website Monitoring → Settings → Set urgency to High (triggers phone calls). Low urgency only sends push/email.

Setup On-Call Schedule

People → On-Call Schedules → New Schedule

• Name: "Website On-Call"
• Add team members with rotation (daily/weekly)
• Set handoff time (e.g., 9 AM)
• Link this schedule to your escalation policy

Test Phone Call

Go to your service → New Incident → Create a test incident manually. Your phone should ring within 2 minutes. Acknowledge to stop the calls.

Tip: Save PagerDuty's number in your contacts so you recognize the call. Whitelist it from Do Not Disturb so it rings even at night.

Step 12: Test the Full Flow

# Temporarily break your health check to trigger alert

docker stop aiwithaman

Expected: After 2 minutes, Grafana fires → PagerDuty creates incident → you get notified. Start the container back: docker start aiwithaman → alert auto-resolves.

Pro Tips

Add Multiple Endpoints

Monitor your homepage, blog, API, and health check endpoint separately. Each gets its own alert rule.

Silence During Maintenance

Before deployments, create a silence in Grafana: Alerting → Silences → New silence — set for your deployment window.

Dashboard + Alert Together

Create a Grafana dashboard showing uptime %, response time graph, and current alert status — single pane of glass for your site health.

Summary

Install Blackbox Exporter to probe your website
Configure Prometheus to scrape the probes
Create PagerDuty service + integration key
Add PagerDuty contact point in Grafana
Create alert rule: probe_success == 0
Set evaluation: every 1min, for 2min
Configure notification policy and escalation
Test by stopping your service

Comments

No comments yet. Be the first to comment!

All Posts Get in Touch