Operations

High Availability

For production deployments, configure Limitador with Redis backend for metric persistence across pod restarts.

Why HA Matters

Default in-memory storage means:

All hit counts lost on pod restart
Metrics reset on reschedule or scale down
No persistence across cluster maintenance

Configure Redis Persistence

See Configuring Redis storage for rate limiting.

For local development: Limitador Persistence.

Production considerations:

HA: Use Redis Sentinel or Cluster
Persistence: Configure RDB snapshots or AOF logs
Monitoring: Monitor memory and connection pool
Backup: Implement regular backups
Scaling: Size for expected metric volume

Verify connection:

# Check Limitador logs
kubectl logs -n kuadrant-system deployment/limitador | grep -i redis

# Test persistence across restart
# WARNING: Only run in non-production or during a maintenance window.
# This will disrupt in-flight requests while pods restart.
kubectl delete pod -n kuadrant-system -l app=limitador
kubectl logs -n kuadrant-system deployment/limitador | grep -i redis
# Counters should reload from Redis, not reset

Maintenance

Monitor ServiceMonitor Health

The ODH/RHOAI monitoring stack scrapes ServiceMonitor / PodMonitor targets with the OpenTelemetry Collector (Target Allocator discovers jobs; the collector scrapes; metrics are remote-written to Prometheus). Thanos Query serves PromQL only — it has no scrape pool, so /api/v1/targets on data-science-thanos-querier-route is empty or unimplemented. Prometheus Status → Targets does not list these jobs either.

The collector's prometheus receiver sets job to its own scrape job (data-science-collector-prometheus). The original ServiceMonitor/PodMonitor job is preserved as exported_job (for example limitador-limitador or redhat-ods-applications/maas-controller-metrics). Filter scrape health on exported_job, not job.

Do not use the usage-logs collector (usage-logs-collector) here. That collector ingests gateway access logs into Loki. Metrics scraping uses the monitoring-stack OpenTelemetryCollector in the DSCI monitoring namespace.

# Replace <monitoring-namespace> with DSCI `spec.monitoring.namespace`
# (e.g. opendatahub or redhat-ods-monitoring).
# Replace <cluster> with your cluster's apps domain (e.g. apps.mycluster.example.com).

# 1. Confirm monitors exist (MaaS labels ServiceMonitors with monitoring.opendatahub.io/scrape=true)
kubectl get servicemonitor,podmonitor -A -l monitoring.opendatahub.io/scrape='true'

# 2. Discovery: Target Allocator job list (what should be scraped)
# The monitoring OpenTelemetryCollector is named data-science-collector, so its
# Target Allocator Service is data-science-collector-targetallocator.
kubectl -n <monitoring-namespace> port-forward svc/data-science-collector-targetallocator 8080:80
# In another terminal:
curl -s localhost:8080/jobs | jq
# Job IDs look like serviceMonitor/<namespace>/<name>/0 — inspect endpoints:
# curl -s localhost:8080/jobs/serviceMonitor%2F<namespace>%2F<name>%2F0/targets | jq
# Look for maas, limitador, authorino, kserve jobs. Missing jobs mean the monitor
# was not discovered (selector, namespace label, or NetworkPolicy).

# 3. Scrape health: `up` is emitted by the prometheus receiver and stored in Thanos.
# Match exported_job (original target). job is always data-science-collector-prometheus.
curl -s -H "Authorization: Bearer $(oc whoami -t)" --get \
  --data-urlencode 'query=up{exported_job=~".*(maas|limitador|authorino|kserve).*"}' \
  "https://data-science-thanos-querier-route-<monitoring-namespace>.<cluster>/api/v1/query" | \
  jq '.data.result[] | {exported_job: .metric.exported_job, instance: .metric.instance, up: .value[1]}'
# up==1 is the analogue of a Prometheus target UP. No series usually means the
# Target Allocator never allocated the job, not that Thanos lost the target.

# 4. If up is 0 or missing, check the monitoring-stack collector logs (exclude usage-logs-collector)
kubectl logs -n <monitoring-namespace> \
  -l 'app.kubernetes.io/component=opentelemetry-collector,app.kubernetes.io/name!=usage-logs-collector' \
  --tail=200 | grep -iE 'scrape|error|limitador|authorino|maas'

Cleanup

How you remove resources depends on how they were applied. ./scripts/observability/install-observability.sh applies TelemetryPolicy and Istio Telemetry from deployment/base/observability/, plus these resources when their conditions match:

Resource	When
Limitador `ServiceMonitor` (`limitador-metrics`)	Kuadrant is not already scraping Limitador `/metrics`
Authorino `ServiceMonitor` (`authorino-server-metrics`)	Authorino Service exists and nothing else scrapes `/server-metrics`
Gateway `Service` + `ServiceMonitor` (`istio-gateway-metrics`)	Gateway Deployment exists in `openshift-ingress`
PrometheusRules (`authorino-maas-metadata-evaluator-high-failure-rate`, `authorino-maas-authentication-alerts`)	Authorino Deployment exists (`kuadrant-system` or `rh-connectivity-link`)

PersesDashboard ownership

Deployment mode	How applied	Ownership	Cleanup
Operator (recommended)	Metrics Usage dashboard via `usageMetricsDashboard` on `Config`. Usage-log dashboards are always reconciled while `Config/default` exists (`usageLogging` does not gate them).	Controller `ownerReference` on `Config` (`configs.maas.opendatahub.io/default`); field manager `maas-controller`.	Metrics Usage: set `usageMetricsDashboard=false`. Usage-log dashboards are recreated while the controller runs. See Operator-managed dashboards below.
Kustomize (development)	`kustomize build` of `deployment/components/observability/observability/dashboards/` (and optionally `usage-logs/dsci-managed/`).	Label `app.kubernetes.io/managed-by: maas-observability` on the metrics Usage dashboard. No `Config` controller reference.	Delete `dashboard-3-maas-usage-admin` by name. For usage-logs, `kubectl delete -k` only sticks when the operator is not also reconciling those objects.
`install-observability.sh`	TelemetryPolicy, Istio Telemetry, and the conditional monitors/rules in the table above.	`app.kubernetes.io/managed-by: maas-observability` on the kustomize base and gateway Service/ServiceMonitor. Limitador monitor and PrometheusRules do not all carry that label.	Delete the applied objects by name (see Telemetry and ServiceMonitors). Dashboards are unaffected.

Identify operator-owned dashboards:

kubectl get persesdashboard -A -o json | jq -r '
  .items[] | select(.metadata.ownerReferences[]? | .kind=="Config" and .controller==true)
  | "\(.metadata.namespace)/\(.metadata.name)"'

Operator-managed dashboards

During upgrade, the MaaS controller removes the stale 3.4 metrics dashboard dashboard-3-maas-usage-admin, datasource kuadrant-prometheus-datasource, and ConfigMap prometheus-web-tls-ca from its applications namespace (for example, redhat-ods-applications). The dashboard is preserved when spec.display.name is Usage (legacy), identifying the current opt-in metrics dashboard even when the applications and monitoring namespaces are the same. The obsolete datasource and ConfigMap are removed by name; resources with other names or in other namespaces are unaffected.

The lifecycle reconciler runs general stale-resource cleanup before module setup, independently of monitoring configuration. Cleanup runs once successfully per controller lifetime. Concurrent reconciliations are serialized, failures are retried on subsequent reconciliation, and missing Perses APIs leave cleanup pending until they are available.

Requires a running maas-controller (LifecycleReconciler), a Config instance, a monitoring namespace, and Perses CRDs (PersesAvailable). See Setup.

Usage-log dashboards (dashboard-4, dashboard-5) are reconciled whenever Config/default exists. Setting usageLogging does not remove them. Deleting Config/default garbage-collects controller-owned copies.

# Metrics Usage dashboard (dashboard-3): operator deletes the CR it owns
kubectl patch configs.maas.opendatahub.io default --type=merge \
  -p '{"spec":{"usageMetricsDashboard":false}}'

Kustomize dashboards

Delete only the Kustomize-applied metrics Usage dashboard, by name and namespace; if Config is the controller owner, do not delete the CR by hand — see Operator-managed dashboards.

kubectl delete persesdashboard dashboard-3-maas-usage-admin -n opendatahub

Usage-log resources

The controller applies deployment/components/observability/usage-logs/dsci-managed/ while Config/default exists. When spec.usageLogging is true, it applies usage-logs/ instead. Objects rendered by both overlays, including the tenancy proxy and usage-log dashboards, are updated in place. Controller-owned objects rendered only by the other overlay are deleted. usage-logs/ includes dsci-managed/ and adds the usage-logs collector, so turning usageLogging on creates that collector and its RBAC and network policies, and turning it off or omitting it removes those controller-owned objects. The per-tenant gateway EnvoyFilter is applied separately. Controller-owned objects from the selected overlay are recreated if deleted by hand, and are garbage-collected when Config/default is deleted.

kubectl delete -k deployment/components/observability/usage-logs/dsci-managed removes a development-only apply. Skip it while maas-controller is reconciling the same objects.

Telemetry and ServiceMonitors

Kustomize/development cleanup only. Do not run this in operator-managed mode. For operator-managed telemetry, update the tenant configuration and let the Tenant reconciler manage those resources.

# Always applied by the script (from deployment/base/observability/)
kubectl delete telemetrypolicy -n openshift-ingress maas-telemetry
kubectl delete telemetry -n openshift-ingress latency-per-subscription

# Conditional: Limitador ServiceMonitor (only if the script deployed it)
kubectl delete servicemonitor -n kuadrant-system limitador-metrics --ignore-not-found

# Conditional: Authorino /server-metrics ServiceMonitor
kubectl delete servicemonitor -n kuadrant-system authorino-server-metrics --ignore-not-found

# Conditional: Istio Gateway metrics
kubectl delete servicemonitor,service -n openshift-ingress istio-gateway-metrics --ignore-not-found

# Conditional: PrometheusRules (namespace is kuadrant-system or rh-connectivity-link)
kubectl delete prometheusrule -n kuadrant-system \
  authorino-maas-metadata-evaluator-high-failure-rate \
  authorino-maas-authentication-alerts --ignore-not-found
kubectl delete prometheusrule -n rh-connectivity-link \
  authorino-maas-metadata-evaluator-high-failure-rate \
  authorino-maas-authentication-alerts --ignore-not-found

Troubleshooting Missing Metrics

# 1. Verify service exposes metrics
kubectl exec -n <namespace> <pod> -- curl localhost:<port>/metrics

# 2. Verify ServiceMonitor exists and labeled with: "monitoring.opendatahub.io/scrape: 'true'"
kubectl get servicemonitor -n <namespace> \
  -l 'monitoring.opendatahub.io/scrape=true'

# 3. Verify ODH monitoring stack enabled.
#    MaasTenantConfig is Degraded for a missing stack only when Config.spec.usageMetricsDashboard is true and tenant telemetry is enabled.
kubectl get maastenantconfig default-tenant -n models-as-a-service  -o json | jq '.status.conditions[] | select(.type=="Degraded" or .type=="MaaSPrerequisitesAvailable") | {type, status, message}'

kubectl get dscinitialization default-dsci -o json | jq '.status.conditions[] | select(.type=="MonitoringReady" or .type=="MonitoringStackAvailable") | {type, status}'

# 4. Confirm the Target Allocator discovered the monitor and `up` is 1
# (see Monitor ServiceMonitor Health above). Do not use Thanos /api/v1/targets.

# 5. Query stored samples via Thanos (PromQL only — this is not a scrape-target API)
# Replace <monitoring-namespace> with DSCI `spec.monitoring.namespace` (e.g., opendatahub)
# Replace <cluster> with your cluster's apps domain (e.g., apps.mycluster.example.com)
# For example: https://data-science-thanos-querier-route-redhat-ods-monitoring.apps.mycluster.example.com/api/v1/query?query=authorized_hits_total
curl -s -H "Authorization: Bearer $(oc whoami -t)" --get \
  --data-urlencode 'query=<metric_name>' \
  "https://data-science-thanos-querier-route-<monitoring-namespace>.<cluster>/api/v1/query"

Troubleshooting Dashboard Issues

# 1. Confirm Perses is available (ODH/RHOAI observability console).
#    When tenant telemetry is enabled, MaasTenantConfig is Degraded unless PersesAvailable is True.
#    When usageLogging is false, it is also Degraded unless LokiStackAvailable and
#    UsageLogsCollectorAvailable are True. See Setup.
kubectl get dscinitialization default-dsci -o json | \
  jq '.status.conditions[] | select(.type=="PersesAvailable" or .type=="LokiStackAvailable" or .type=="UsageLogsCollectorAvailable") | {type, status}'

# 2. Confirm PersesDashboard CRs exist
kubectl get persesdashboard -A | grep maas

# 3. Run the panel query in Prometheus or Loki directly (see Setup)

# 4. Verify the time range includes when metrics or logs were generated

# 5. Check for lazily-registered metrics
# Some metrics appear only after first event (e.g., queue_time after first queued request)

Capacity Planning

Prometheus storage:

# Check storage size
kubectl exec prometheus-data-science-monitoringstack-0 -n <monitoring-namespace> -- \
  df -h /prometheus

# View retention
kubectl get monitoringstack data-science-monitoringstack -n <monitoring-namespace> -o yaml | \
  grep -A 5 retention

Metric cardinality:

# Thanos Query has no local TSDB, so /api/v1/status/tsdb is not available.
# Count active series per MaaS metric via PromQL instead.
# Replace <monitoring-namespace> with DSCI `spec.monitoring.namespace` (e.g., opendatahub)
# Replace <cluster> with your cluster's apps domain (e.g., apps.mycluster.example.com)
curl -s -H "Authorization: Bearer $(oc whoami -t)" --get \
  --data-urlencode 'query=count by (__name__) ({__name__=~"authorized_hits_total|authorized_calls_total|limited_calls_total"})' \
  "https://data-science-thanos-querier-route-<monitoring-namespace>.<cluster>/api/v1/query" | \
  jq '.data.result[] | {metric: .metric.__name__, series: .value[1]}'

Watch: authorized_hits_total{user!=""}, authorized_calls_total{user!=""}, istio_request_duration_milliseconds_bucket{subscription!=""}.

Regular Maintenance Tasks

Task	Frequency	Action
Storage Check	Weekly	Monitor Prometheus storage usage
ServiceMonitor Health	Daily	Check Target Allocator jobs and `up` series (not Thanos `/api/v1/targets`)
Cardinality Review	Monthly	Review high-cardinality metrics
Dashboard Testing	After deployment	Verify Perses dashboards load in the ODH console
Backup Redis (HA)	Daily	Backup Redis data

Known Limitations

Blocked Features

Feature	Blocker	Workaround
`model` label on `authorized_calls_total` / `limited_calls_total`	Kuadrant wasm-shim doesn't pass `responseBodyJSON` context	Use `authorized_hits_total` for per-model breakdown
Input/output token split	TokenRateLimitPolicy sends single `hits_addend`	Total tokens via `authorized_hits_total`; response body has `usage.prompt_tokens` and `usage.completion_tokens` but wasm-shim doesn't split
Input/output per user	vLLM doesn't label with `user`	Total tokens per user via `authorized_hits_total{user!=""}`; vLLM prompt/gen metrics are per-model only
Rate-limited in Istio metrics	WASM plugin `sendLocalReply()` short-circuits filter chain	Use `limited_calls_total` from Limitador (has correct labels)
Policy health metrics	`kuadrant_policies_enforced`, `kuadrant_policies_total` not in RHCL 1.x	`limitador_up` and `datastore_partitioned` available now
maas-api metrics	Requires HTTPS scrape + `/metrics` get RBAC	Use ServiceMonitor `maas-api-metrics` with bearer token; grant scrapers `nonResourceURLs: ["/metrics"]` get
PromQL `_total` suffix	OTel prometheus receiver stores Limitador counters as `authorized_hits_total` (and the same for `authorized_calls` / `limited_calls`)	Query the `_total` names; Grafana panels that omit `_total` return no data

Total vs Split

Total token consumption per user is available via authorized_hits_total{user!=""}. Input/output split at gateway requires wasm-shim to send two counter updates.

Available Metrics

Feature	Metric	Label
Latency per subscription	`istio_request_duration_milliseconds_bucket`	`subscription`
Tokens per user	`authorized_hits_total`	`user`
Tokens per subscription	`authorized_hits_total`	`subscription`
Tokens per model	`authorized_hits_total`	`model`
Requests per user	`authorized_calls_total`	`user`
Requests per subscription	`authorized_calls_total`	`subscription`
Rate limited per user	`limited_calls_total`	`user`
Rate limited per subscription	`limited_calls_total`	`subscription`

Reporting Issues

Check Setup prerequisites
Review troubleshooting procedures above
Search GitHub Issues
Report with: MaaS version, failing query/panel, expected vs actual, relevant logs