Setup

Prerequisites

User Workload Monitoring

User Workload Monitoring must be enabled for Prometheus to scrape metrics.

Required

Without User Workload Monitoring, ServiceMonitors will not be processed and no metrics will be collected.

kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true
EOF

# Verify prometheus-user-workload pods are running
kubectl get pods -n openshift-user-workload-monitoring

Kuadrant Observability

Enable observability on the Kuadrant CR for rate-limiting metrics (authorized_hits, authorized_calls, limited_calls):

kubectl patch kuadrant kuadrant -n kuadrant-system --type=merge \
  -p '{"spec":{"observability":{"enable":true}}}'

# Verify PodMonitor was created
kubectl get podmonitor -n kuadrant-system kuadrant-limitador-monitor

RHOAI Dashboard Observability Tab (Optional)

For Perses-based dashboards in the RHOAI Dashboard:

Install Cluster Observability Operator and OpenTelemetry Operator from OperatorHub
Configure DSCI monitoring.metrics - see Platform Setup
Enable observabilityDashboard: true on OdhDashboardConfig - see Feature Flags

# Verify
kubectl get csv -A | grep -E 'cluster-observability|opentelemetry'
kubectl get pods -n redhat-ods-monitoring | grep perses

See Managing observability (RHOAI 3.4).

Installation

Option 1: Operator-Managed (Recommended)

Enable via Tenant CR:

apiVersion: maas.opendatahub.io/v1alpha1
kind: Tenant
metadata:
  name: default-tenant
  namespace: models-as-a-service
spec:
  telemetry:
    enabled: true
    metrics:
      captureOrganization: true
      captureUser: false      # GDPR
      captureGroup: false     # High cardinality
      captureModelUsage: true

Or patch:

kubectl patch tenant default-tenant -n models-as-a-service --type=merge \
  -p '{"spec":{"telemetry":{"enabled":true}}}'

This creates:

TelemetryPolicy (maas-telemetry) - Adds subscription, model, organization_id labels to Limitador metrics (user and group labels disabled by default)
Istio Telemetry (latency-per-subscription) - Adds subscription label to gateway latency

Verify:

kubectl get telemetry -n openshift-ingress latency-per-subscription

Prerequisites

Requires OpenShift Service Mesh 2.4+, Kuadrant/RHCL, and deployed Gateway.

AuthPolicy Dependency

Istio Telemetry reads X-MaaS-Subscription header injected by AuthPolicy. Without header injection, subscription label will be empty.

Option 2: Kustomize (Development)

Development Only

Production deployments should use operator-managed telemetry (Option 1).

# Deploy base telemetry + conditional ServiceMonitors
./scripts/observability/install-observability.sh [--namespace NAMESPACE]

# Deploy Grafana dashboards
./scripts/observability/install-grafana-dashboards.sh

Manual deployment:

# Base telemetry (requires Gateway + AuthPolicy)
kustomize build deployment/base/observability | kubectl apply -f -

# Conditional ServiceMonitors (auto-detects Kuadrant monitors)
./scripts/observability/install-observability.sh

# Grafana dashboards (discovers Grafana instance)
./scripts/observability/install-grafana-dashboards.sh

Kustomize entrypoints:

Path	Contents
`deployment/base/observability/`	TelemetryPolicy, Istio Telemetry, metadata-evaluator PrometheusRule
`deployment/components/observability/grafana/`	GrafanaDashboard CRs
`deployment/components/observability/prometheus/`	Standalone Prometheus (dev/test)

Operator vs Kustomize drift:

Resource	Kustomize	Operator
TelemetryPolicy	`base/observability/`	Yes (Tenant reconciler)
Istio Telemetry	`base/observability/`	Yes (Tenant reconciler)
Limitador ServiceMonitor	Conditional	Kuadrant PodMonitor when `observability.enable: true`
Authorino /server-metrics	`authorino-server-metrics-servicemonitor.yaml`	No (Kuadrant only scrapes `/metrics`)
Grafana Dashboards	`components/observability/grafana/`	No (same CRs used)