Monitoring

With monitoring.enabled the helmfile installs kube-prometheus-stack (Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics) and, with monitoring.logs, Loki and Alloy. Grafana is at https://<domains.grafana>; the admin password is in monitoring/grafana-admin.

What is scraped

Source	How	Used for
Nodes, pods, kube-state-metrics	kube-prometheus-stack defaults	Capacity, restarts, OOM kills
metrics-server	Kubernetes metrics API (not Prometheus)	HPAs of user apps; billing CPU/memory
Traefik	ServiceMonitor, router and service labels	Requests, latency, bytes per app
CloudNativePG	PodMonitor `atoll-db`	Replication lag, WAL archiving, connections
cert-manager	ServiceMonitor	Certificate expiry
Prometheus picks up every ServiceMonitor and PodMonitor in the cluster
(`serviceMonitorSelectorNilUsesHelmValues: false`).

Who can reach it

NetworkPolicies in monitoring (chart charts/network-policies) admit only Grafana, Prometheus, Alertmanager and Alloy to the monitoring pods, the ingress controller to Grafana, and atoll-server to Prometheus (billing). Loki runs without multi-tenancy (auth_enabled: false), which is fine only because nothing outside the monitoring components can reach it; app and build pods are also stopped by their own egress policies. Grafana is the one entry point, behind its admin login.

Billing data

Billing counts the CPU and memory running app pods request (their size, see Scaling) and reads bandwidth from Prometheus at ATOLL_BILLING_PROMETHEUS_URL. Apps that scale to zero are served through their w--<app> Service, which billing maps to the same project. Traefik labels its metrics with the Kubernetes service of each app Ingress, for example:

sum by (service) (increase(traefik_service_responses_bytes_total{service=~"atoll-apps-.*"}[1h]))
sum by (service) (increase(traefik_service_requests_bytes_total{service=~"atoll-apps-.*"}[1h]))

service looks like atoll-apps-<project>--<module>-80@kubernetes. Web apps are served by atoll-server, which reports their bytes itself.

App metrics

The panel’s Metrics tab (GET /api/projects/<project>/metrics?module=&preview=&range=1h|24h|7d|30d, members) reads Prometheus at ATOLL_METRICS_PROMETHEUS_URL (unset: ATOLL_BILLING_PROMETHEUS_URL; neither: the tab says metrics are unavailable). For module shop--api in atoll-apps, with a window W of the range’s step but at least 2m:

Metric	Source	PromQL
CPU (cores)	cAdvisor	`sum(max by (pod) (rate(container_cpu_usage_seconds_total{namespace="atoll-apps",pod=~"shop--api-[a-z0-9]+-[a-z0-9]+",container="app"}[W])))`
Memory (bytes)	cAdvisor	the same with `avg_over_time(container_memory_working_set_bytes{…}[W])`
Replicas	kube-state-metrics	`max(max_over_time(kube_deployment_status_replicas_available{namespace="atoll-apps",deployment="shop--api"}[W]))`
Requests/s	Traefik	`sum(rate(traefik_service_requests_total{service=~S}[W]))`
p50 / p95 latency	Traefik	`histogram_quantile(0.95, sum by (le) (rate(traefik_service_request_duration_seconds_bucket{service=~S}[W])))`
Error rate	Traefik	`(sum(rate(…{service=~S,code=~"5.."}[W])) or vector(0)) / sum(rate(…{service=~S}[W]))`

S matches the module’s Service, its wake Service (w--shop--api, while it scales to zero) and the custom-domain Services leading to it, e.g. atoll-apps-(w--)?shop--api-[a-z0-9]+@kubernetes|atoll-domains-shop--domain-…. Traefik’s latency buckets are set finer than its defaults in values/traefik.yaml.gotmpl for useful percentiles. Answers are cached on the server (15 seconds for the last hour, longer for longer ranges).

Logs

Alloy tails every pod through the Kubernetes API and ships to Loki with labels namespace, pod, container and atoll_project (from the atoll.app/project label of app and build pods). In Grafana → Explore → Loki:

{namespace="atoll"} |= "ERROR"
{namespace="atoll-apps", atoll_project="hello"}
{namespace="atoll-builds"} |= "error:"

Alerts worth adding

kube-prometheus-stack ships general Kubernetes alerts. Add PrometheusRules for the platform:

Alert	Expression (sketch)
Server down	`kube_deployment_status_replicas_available{namespace="atoll",deployment="atoll-server"} < 1`
Postgres not archiving	`cnpg_pg_stat_archiver_seconds_since_last_archival > 600`
Postgres replica lag	`cnpg_pg_replication_lag > 60`
Certificate expiring	`certmanager_certificate_expiration_timestamp_seconds - time() < 7243600`
Builds waiting for capacity	`sum(kube_pod_status_phase{namespace="atoll-builds",phase="Pending"}) > 0` for 10m
Storage filling	`kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.15`

Route Alertmanager to email or chat through kube-prometheus-stack values (alertmanager.config) in your site file’s release overrides.

Without the stack

Turning monitoring.enabled off skips Prometheus, Grafana and Loki. metrics-server stays (HPAs need it), but billing has no bandwidth source then.

Install

Components

Operate

What is scraped

Who can reach it

Billing data

App metrics

Logs

Alerts worth adding

Without the stack

​What is scraped

​Who can reach it

​Billing data

​App metrics

​Logs

​Alerts worth adding

​Without the stack

What is scraped

Who can reach it

Billing data

App metrics

Logs

Alerts worth adding

Without the stack