Troubleshooting

This guide helps you diagnose and resolve common issues with MaaS Platform deployments.

Common Issues

Getting 501 Not Implemented errors: Traffic is not making it to the Gateway.
- Verify Gateway status and HTTPRoute configuration

Getting 401 Unauthorized errors when trying to create an API key: Authentication to maas-api is not working.

Verify maas-api-auth-policy AuthPolicy is applied
Check if your cluster uses a custom token review audience:

# Detect your cluster's audience
AUD="$(kubectl create token default --duration=10m 2>/dev/null | \
  cut -d. -f2 | jq -Rr '@base64d | fromjson | .aud[0]' 2>/dev/null)"
echo "Cluster audience: ${AUD}"

If the audience is NOT https://kubernetes.default.svc, patch the AuthPolicy:

kubectl patch authpolicy maas-api-auth-policy -n opendatahub \
  --type=merge --patch "
spec:
  rules:
    authentication:
      openshift-identities:
        kubernetesTokenReview:
          audiences:
            - ${AUD}
            - maas-default-gateway-sa"

Getting 401 errors when trying to get models: Authentication is not working for the models endpoint.
- Create a new API key and use it in the Authorization header
- Verify gateway-auth-policy AuthPolicy is applied
- Validate that the service account has post access to the llminferenceservices resource per MaaSAuthPolicy
  - Note: this should be automated by the ODH Controller
Getting 404 errors when trying to get models: The models endpoint is not working.
- Verify model-route HTTPRoute exist and is applied
- Verify the model is deployed and the LLMInferenceService has the maas-default-gateway gateway specified
- Verify that the model is recognized by maas-api by checking the maas-api/v1/models endpoint (see Validation Guide - List Available Models)
Rate limiting not working: Verify AuthPolicy and TokenRateLimitPolicy are applied
- Verify gateway-rate-limits RateLimitPolicy is applied
- Verify TokenRateLimitPolicy is applied (e.g. gateway-default-deny or per-route policies)
- If multiple TokenRateLimitPolicies target the same HTTPRoute, see Quota and Access Configuration
- Verify the model is deployed and the LLMInferenceService has the maas-default-gateway gateway specified
- Verify that the model is rate limited for request bursts (request-rate limiting) — see Validation Guide - Test Rate Limiting
- Verify that the model returns 429 for token-heavy prompts (token-rate limiting) — see Validation Guide - Test Rate Limiting
Routes not accessible (503 errors): Check MaaS Default Gateway status and HTTPRoute configuration
- Verify Gateway is in Programmed state: kubectl get gateway -n openshift-ingress maas-default-gateway
- Check HTTPRoute configuration and status
High-concurrency inference returns intermittent 500 or 503 errors: The RHCL/Kuadrant WASM authentication path may be timing out under burst load.
- Confirm the errors occur during concurrent request scenarios and not for low-volume requests
- Increase AUTH_SERVICE_TIMEOUT from the default 200ms to 2s through the RHCL operator Subscription configuration. See High-concurrency authentication timeout.
Metrics not appearing in dashboards: Prometheus is not scraping MaaS components.
- Verify User Workload Monitoring is enabled — see Observability Setup
- Verify Kuadrant observability is enabled — see Observability Setup
- Check prometheus-user-workload pods are running:
```
kubectl get pods -n openshift-user-workload-monitoring
```
- Verify ServiceMonitors/PodMonitors exist:
```
kubectl get servicemonitor,podmonitor -A | grep -E "(maas|kuadrant|limitador)"
```

Rate limiting metrics missing (authorized_calls, limited_calls): Kuadrant observability is not enabled.

Enable observability on Kuadrant CR:

kubectl patch kuadrant kuadrant -n kuadrant-system --type=merge \
  -p '{"spec":{"observability":{"enable":true}}}'

Verify the PodMonitor was created:

kubectl get podmonitor -n kuadrant-system

RHOAI Dashboard Observability tab returns 503 Service Unavailable: The Dashboard cannot reach the Perses backend.

The error typically appears as {"statusCode": 503, "code": "FST_REPLY_FROM_SERVICE_UNAVAILABLE", ...}. This is a Fastify/Dashboard-level error (not a gateway 503) indicating the monitoring stack is not deployed or Perses is not running. The most common causes are missing operators (COO, OpenTelemetry) or DSCI monitoring.metrics not being configured.

See RHOAI Dashboard Observability Tab for the full prerequisites and verification checklist.
GenAI Studio tab not visible in Dashboard: Requires llamastackoperator set to Managed in the DSC and the genAiStudio feature flag enabled on OdhDashboardConfig.

See OdhDashboardConfig Feature Flags for setup.
TLS certificate errors (curl: (60) SSL certificate problem): Your cluster uses self-signed or internal CA certificates that are not in your system trust store. See TLS Certificate Validation below.
Cannot create MaaSSubscription or MaaSAuthPolicy (no endpoints available for service "maas-controller-webhook-service"): The maas-controller pods are not running or not ready.

MaaS uses admission webhooks to validate resource creation. When the controller is unavailable (pod crash, upgrade, or scaled to 0), the webhook endpoint becomes unreachable and creates are rejected.
- Check controller pod status:
```
kubectl get pods -n models-as-a-service -l control-plane=maas-controller
```
- Check webhook service endpoints:
```
kubectl get endpoints maas-controller-webhook-service -n models-as-a-service
```
- Check controller logs for errors:
```
kubectl logs -n models-as-a-service deployment/maas-controller
```
Creates succeed once controller pods are healthy. Model inference requests are unaffected during controller downtime (data plane continues operating normally).
Cannot create AITenant (must be created in the configured AITenant infrastructure namespace): The object is being created outside the namespace configured by --aitenant-namespace (default ai-tenants).
- Check which namespace the controller is configured to accept:
```
kubectl get deployment maas-controller -n opendatahub -o jsonpath='{.spec.template.spec.containers[0].args}'
```
- Create the AITenant in that namespace instead of the target tenant namespace:
```
kubectl get namespace ai-tenants
```
- If the error is no endpoints available for service "maas-controller-webhook-service", follow the same webhook health checks as issue 12 above.

Conflicting AuthPolicy Detection

MaaS automatically detects non-MaaS AuthPolicies (e.g., from KServe or other controllers) that target the same HTTPRoutes used by MaaS-governed models. When a conflict is detected, MaaS sets a ConflictingAuthPolicy condition on the affected MaaSAuthPolicy resource and emits a Kubernetes warning event.

Symptoms

MaaSAuthPolicy has condition ConflictingAuthPolicy=True
Warning events on MaaSAuthPolicy resources referencing non-MaaS AuthPolicies
Unexpected authentication behavior (wrong policy may win when multiple AuthPolicies target the same HTTPRoute)

Diagnosis

Check for conflicting AuthPolicy conditions:

# Check MaaSAuthPolicy conditions
kubectl get maasauthpolicy -n models-as-a-service -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .status.conditions[?(@.type=="ConflictingAuthPolicy")]}{.status}{"\t"}{.message}{end}{"\n"}{end}'

# List all AuthPolicies targeting a specific model's HTTPRoute
MODEL_NS="<model-namespace>"
MODEL_NAME="<model-name>"
kubectl get authpolicy -n "$MODEL_NS" -o json | \
  jq -r ".items[] | select(.spec.targetRef.name == \"$MODEL_NAME\" and .spec.targetRef.kind == \"HTTPRoute\") | .metadata.name + \" (managed-by: \" + (.metadata.labels[\"app.kubernetes.io/managed-by\"] // \"unknown\") + \")\""

# Check for warning events
kubectl get events -n models-as-a-service --field-selector reason=ConflictingAuthPolicy

Remediation

KServe anonymous auth policies (*-kserve-route-authn): These are typically created when security.opendatahub.io/enable-auth is misconfigured on the InferenceService. Set the annotation to "true" on MaaS-governed InferenceServices to prevent KServe from creating conflicting anonymous AuthPolicies:
```
kubectl annotate inferenceservice <name> -n <namespace> \
  security.opendatahub.io/enable-auth="true" --overwrite
```
Custom AuthPolicies: If another team deployed a custom AuthPolicy targeting the same HTTPRoute, coordinate to determine which controller should own authentication for that route. Either:
- Remove the conflicting AuthPolicy if MaaS is the intended auth authority
- Or remove the model from the MaaSAuthPolicy if MaaS should not govern that route

Verify resolution: After applying the fix, the ConflictingAuthPolicy condition should transition to False on the next reconciliation:

kubectl get maasauthpolicy -n models-as-a-service -o jsonpath='{range .items[*]}{.metadata.name}{": "}{range .status.conditions[?(@.type=="ConflictingAuthPolicy")]}{.status}{end}{"\n"}{end}'

TLS Certificate Validation

By default, curl validates TLS certificates against your system CA bundle. If you encounter certificate verification errors (e.g., curl: (60) SSL certificate problem: self-signed certificate), use one of the approaches below.

Recommended: Use the Ingress CA/Certificate Chain

For OpenShift clusters with self-signed or internal certificates, use the ingress trust chain (or your platform-provided CA bundle) with curl --cacert:

# Get ingress certificate chain/trust bundle (platform-specific source)
oc get secret -n openshift-ingress router-certs-default \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/ingress-cert-chain.crt

# Use --cacert in curl commands
curl -sS --cacert /tmp/ingress-cert-chain.crt \
  -H "Authorization: Bearer $API_KEY" \
  "https://maas.${CLUSTER_DOMAIN}/maas-api/v1/models" | jq .

Alternatively, if your cluster uses the OpenShift service CA:

# Extract the service CA bundle
oc get configmap -n openshift-config-managed service-ca-bundle \
  -o jsonpath='{.data.service-ca\.crt}' > /tmp/service-ca.crt

# Use --cacert in curl commands
curl -sS --cacert /tmp/service-ca.crt \
  -H "Authorization: Bearer $API_KEY" \
  "https://maas.${CLUSTER_DOMAIN}/maas-api/v1/models" | jq .

Development/Testing Only: Disable Verification

Security Warning

The -k flag disables all TLS certificate validation. An attacker on the network path can present a forged certificate and intercept your API key, token, or other credentials. Never use -k in production or when sending credentials over untrusted networks.

For isolated development or test environments only, you can add the -k flag:

# INSECURE: Only for isolated dev/test environments
curl -sS -k -H "Authorization: Bearer $API_KEY" \
  "https://maas.${CLUSTER_DOMAIN}/maas-api/v1/models" | jq .

Adding the CA to Your System Trust Store

For a permanent solution, add the ingress certificate chain to your system trust store so that all tools (curl, Python, browsers) trust it automatically:

# Linux (Fedora/RHEL)
sudo cp /tmp/ingress-cert-chain.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

# Linux (Debian/Ubuntu)
sudo cp /tmp/ingress-cert-chain.crt /usr/local/share/ca-certificates/ingress-cert-chain.crt
sudo update-ca-certificates

# macOS
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain /tmp/ingress-cert-chain.crt

For detailed TLS configuration options, see TLS Configuration.

Additional Resources

Validation Guide — Manual validation steps
Observability Guide — Metrics, monitoring, and dashboards
scripts/README.md — Deployment scripts documentation