> ## Documentation Index
> Fetch the complete documentation index at: https://docs.servers.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring

> Metrics, logs, dashboards and alerts for your install.

With `monitoring.enabled` the helmfile installs kube-prometheus-stack (Prometheus, Alertmanager, Grafana,
node-exporter, kube-state-metrics) and, with `monitoring.logs`, Loki and Alloy. Grafana is at
`https://<domains.grafana>`; the admin password is in `monitoring/grafana-admin`.

## What is scraped

| Source | How | Used for |
| - | - | - |
| Nodes, pods, kube-state-metrics | kube-prometheus-stack defaults | Capacity, restarts, OOM kills |
| metrics-server | Kubernetes metrics API (not Prometheus) | HPAs of user apps; billing CPU/memory |
| Traefik | ServiceMonitor, router and service labels | Requests, latency, **bytes per app** |
| CloudNativePG | PodMonitor `atoll-db` | Replication lag, WAL archiving, connections |
| cert-manager | ServiceMonitor | Certificate expiry |
| Prometheus picks up every ServiceMonitor and PodMonitor in the cluster | | |
| (`serviceMonitorSelectorNilUsesHelmValues: false`). | | |

## Who can reach it

NetworkPolicies in `monitoring` (chart `charts/network-policies`) admit only Grafana, Prometheus,
Alertmanager and Alloy to the monitoring pods, the ingress controller to Grafana, and atoll-server to
Prometheus (billing). Loki runs without multi-tenancy (`auth_enabled: false`), which is fine only because
nothing outside the monitoring components can reach it; app and build pods are also stopped by their own
egress policies. Grafana is the one entry point, behind its admin login.

## Billing data

Billing counts the CPU and memory running app pods request (their size, see
[Scaling](https://docs.servers.dev/self-hosting/kubernetes.md#scaling)) and reads bandwidth from Prometheus at `ATOLL_BILLING_PROMETHEUS_URL`. Apps that
scale to zero are served through their `w--<app>` Service, which billing maps to the same project. Traefik labels its
metrics with the Kubernetes service of each app Ingress, for example:

```promql theme={null}
sum by (service) (increase(traefik_service_responses_bytes_total{service=~"atoll-apps-.*"}[1h]))
sum by (service) (increase(traefik_service_requests_bytes_total{service=~"atoll-apps-.*"}[1h]))
```

`service` looks like `atoll-apps-<project>--<module>-80@kubernetes`. Web apps are served by atoll-server, which
reports their bytes itself.

## App metrics

The panel's Metrics tab (`GET /api/projects/<project>/metrics?module=&preview=&range=1h|24h|7d|30d`, members)
reads Prometheus at `ATOLL_METRICS_PROMETHEUS_URL` (unset: `ATOLL_BILLING_PROMETHEUS_URL`; neither: the tab says
metrics are unavailable). For module `shop--api` in `atoll-apps`, with a window `W` of the range's step but at least 2m:

| Metric | Source | PromQL |
| - | - | - |
| CPU (cores) | cAdvisor | `sum(max by (pod) (rate(container_cpu_usage_seconds_total{namespace="atoll-apps",pod=~"shop--api-[a-z0-9]+-[a-z0-9]+",container="app"}[W])))` |
| Memory (bytes) | cAdvisor | the same with `avg_over_time(container_memory_working_set_bytes{…}[W])` |
| Replicas | kube-state-metrics | `max(max_over_time(kube_deployment_status_replicas_available{namespace="atoll-apps",deployment="shop--api"}[W]))` |
| Requests/s | Traefik | `sum(rate(traefik_service_requests_total{service=~S}[W]))` |
| p50 / p95 latency | Traefik | `histogram_quantile(0.95, sum by (le) (rate(traefik_service_request_duration_seconds_bucket{service=~S}[W])))` |
| Error rate | Traefik | `(sum(rate(…{service=~S,code=~"5.."}[W])) or vector(0)) / sum(rate(…{service=~S}[W]))` |

`S` matches the module's Service, its wake Service (`w--shop--api`, while it scales to zero) and the custom-domain
Services leading to it, e.g. `atoll-apps-(w--)?shop--api-[a-z0-9]+@kubernetes|atoll-domains-shop--domain-…`.
Traefik's latency buckets are set finer than its defaults in `values/traefik.yaml.gotmpl` for useful percentiles.
Answers are cached on the server (15 seconds for the last hour, longer for longer ranges).

## Logs

Alloy tails every pod through the Kubernetes API and ships to Loki with labels `namespace`, `pod`,
`container` and `atoll_project` (from the `atoll.app/project` label of app and build pods). In Grafana → Explore
→ Loki:

```logql theme={null}
{namespace="atoll"} |= "ERROR"
{namespace="atoll-apps", atoll_project="hello"}
{namespace="atoll-builds"} |= "error:"
```

## Alerts worth adding

kube-prometheus-stack ships general Kubernetes alerts. Add PrometheusRules for the platform:

| Alert | Expression (sketch) |
| - | - |
| Server down | `kube_deployment_status_replicas_available{namespace="atoll",deployment="atoll-server"} < 1` |
| Postgres not archiving | `cnpg_pg_stat_archiver_seconds_since_last_archival > 600` |
| Postgres replica lag | `cnpg_pg_replication_lag > 60` |
| Certificate expiring | `certmanager_certificate_expiration_timestamp_seconds - time() < 7*24*3600` |
| Builds waiting for capacity | `sum(kube_pod_status_phase{namespace="atoll-builds",phase="Pending"}) > 0` for 10m |
| Storage filling | `kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.15` |

Route Alertmanager to email or chat through `kube-prometheus-stack` values (`alertmanager.config`) in your site
file's release overrides.

## Without the stack

Turning `monitoring.enabled` off skips Prometheus, Grafana and Loki. metrics-server stays (HPAs need it), but
billing has no bandwidth source then.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.