Troubleshooting

This guide helps you diagnose and resolve common issues with MaaS Platform deployments.

Common Issues

1. Getting 501 Not Implemented errors: Traffic is not making it to the Gateway.
   • Verify Gateway status and HTTPRoute configuration

2. Getting 401 Unauthorized errors when trying to create an API key: Authentication to maas-api is not working.

   • Verify maas-api-auth-policy AuthPolicy is applied
   • Check if your cluster uses a custom token review audience:

   # Detect your cluster's audience
   AUD="$(kubectl create token default --duration=10m 2>/dev/null | \
     cut -d. -f2 | jq -Rr '@base64d | fromjson | .aud[0]' 2>/dev/null)"
   echo "Cluster audience: ${AUD}"
   

   If the audience is NOT https://kubernetes.default.svc, patch the AuthPolicy:

   kubectl patch authpolicy maas-api-auth-policy -n opendatahub \
     --type=merge --patch "
   spec:
     rules:
       authentication:
         openshift-identities:
           kubernetesTokenReview:
             audiences:
               - ${AUD}
               - maas-default-gateway-sa"
   

3. Getting 401 errors when trying to get models: Authentication is not working for the models endpoint.

   • Create a new API key and use it in the Authorization header
   • Verify gateway-auth-policy AuthPolicy is applied
   • Validate that the service account has post access to the llminferenceservices resource per MaaSAuthPolicy
     • Note: this should be automated by the ODH Controller
4. Getting 404 errors when trying to get models: The models endpoint is not working.
   • Verify model-route HTTPRoute exist and is applied
   • Verify the model is deployed and the LLMInferenceService has the maas-default-gateway gateway specified
   • Verify that the model is recognized by maas-api by checking the maas-api/v1/models endpoint (see Validation Guide - List Available Models)
5. Rate limiting not working: Verify AuthPolicy and TokenRateLimitPolicy are applied
   • Verify gateway-rate-limits RateLimitPolicy is applied
   • Verify TokenRateLimitPolicy is applied (e.g. gateway-default-deny or per-route policies)
   • If multiple TokenRateLimitPolicies target the same HTTPRoute, see Quota and Access Configuration
   • Verify the model is deployed and the LLMInferenceService has the maas-default-gateway gateway specified
   • Verify that the model is rate limited for request bursts (request-rate limiting) — see Validation Guide - Test Rate Limiting
   • Verify that the model returns 429 for token-heavy prompts (token-rate limiting) — see Validation Guide - Test Rate Limiting

6. Routes not accessible (503 errors): Check MaaS Default Gateway status and HTTPRoute configuration

   • Verify Gateway is in Programmed state: kubectl get gateway -n openshift-ingress maas-default-gateway
   • Check HTTPRoute configuration and status

7. High-concurrency inference returns intermittent 500 or 503 errors: The RHCL/Kuadrant WASM authentication path may be timing out under burst load.

   • Confirm the errors occur during concurrent request scenarios and not for low-volume requests
   • Increase AUTH_SERVICE_TIMEOUT from the default 200ms to 2s through the RHCL operator Subscription configuration. See High-concurrency authentication timeout.

8. GenAI Studio tab not visible in Dashboard: Requires llamastackoperator set to Managed in the DSC and the genAiStudio feature flag enabled on OdhDashboardConfig.

   See OdhDashboardConfig Feature Flags for setup.

9. GatewayClass stuck in Accepted: Unknown ("Waiting for controller"): A conflicting OSSM subscription prevents the openshift-ingress operator from managing Gateway API on OCP versions where the ingress operator uses OLM for OSSM management (4.19, 4.20, 4.21 before 4.21.22). On OCP 4.21.22+, 4.22+, the ingress operator uses the Sail Library directly and manual OSSM subscriptions do not cause this conflict.

   On affected versions, the openshift-ingress ClusterOperator manages OSSM 3 via OLM and pins it to a version compatible with the cluster. Two scenarios cause this failure:

   • An OSSM v2.x subscription (servicemeshoperator) blocks OSSM v3 installation — v2 and v3 cannot coexist, and the ingress operator reports GatewayAPIOSSMConflict.

   • A manually installed OSSM v3 subscription (servicemeshoperator3, e.g. from the stable channel via OperatorHub) pins a version the ingress operator cannot manage.

   • Check for a conflicting OSSM subscription (v2 or v3):

   kubectl get subscription -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' | grep -i servicemesh
   

   • Delete the conflicting subscription and its CSV (replace <namespace> and <operator-name> with the values from the previous step, e.g. openshift-operators and servicemeshoperator3 or servicemeshoperator):

   kubectl delete subscription <operator-name> -n <namespace>
   kubectl delete csv -n <namespace> -l operators.coreos.com/<operator-name>.<namespace>
   

   • Wait for the openshift-ingress operator to reinstall OSSM at the pinned version:

   kubectl wait --for=condition=Available clusteroperator/ingress --timeout=300s
   

   • Do not approve the OSSM upgrade InstallPlan that may appear in openshift-operators — approving it re-breaks the gateway

   • Verify the GatewayClass is now accepted (Accepted must be True):

   kubectl get gatewayclass openshift-default -o jsonpath='{.metadata.name}{"\t"}{range .status.conditions[?(@.type=="Accepted")]}{.status}{"\t"}{.message}{end}{"\n"}'
   

   See Install Gateway API Controller for the full warning and context.

10. TLS certificate errors (curl: (60) SSL certificate problem): Your cluster uses self-signed or internal CA certificates that are not in your system trust store. See TLS Certificate Validation below.

11. Cannot create MaaSSubscription or MaaSAuthPolicy (no endpoints available for service "maas-controller-webhook-service"): The maas-controller pods are not running or not ready.

    MaaS uses admission webhooks to validate resource creation. When the controller is unavailable (pod crash, upgrade, or scaled to 0), the webhook endpoint becomes unreachable and creates are rejected.

    • Check controller pod status:

    kubectl get pods -n models-as-a-service -l control-plane=maas-controller
    

    • Check webhook service endpoints:

    kubectl get endpoints maas-controller-webhook-service -n models-as-a-service
    

    • Check controller logs for errors:

    kubectl logs -n models-as-a-service deployment/maas-controller
    

    Creates succeed once controller pods are healthy. Model inference requests are unaffected during controller downtime (data plane continues operating normally).

12. Cannot create AITenant (must be created in the configured AITenant infrastructure namespace): The object is being created outside the namespace configured by --aitenant-namespace (default ai-tenants).

    • Check which namespace the controller is configured to accept:

    kubectl get deployment maas-controller -n opendatahub -o jsonpath='{.spec.template.spec.containers[0].args}'
    

    • Create the AITenant in that namespace instead of the target tenant namespace:

    kubectl get namespace ai-tenants
    

    • If the error is no endpoints available for service "maas-controller-webhook-service", follow the same webhook health checks as issue 11 above.

13. Management endpoints return AUTH_FAILURE on RHCL v1.4.0: All management endpoints (/v1/models, /v1/subscriptions, /v1/api-keys) return AUTH_FAILURE while inference endpoints work. RHCL v1.4.0 contains a Wasm shim bug that prevents auth calls from reaching Authorino. Upstream Kuadrant (ODH) is not affected.

    • Confirm you are on RHCL v1.4.0:

    kubectl get subscription -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,VERSION:.status.currentCSV' | grep -i rhcl
    

    • Upgrade RHCL to v1.4.1 or later

Conflicting AuthPolicy Detection

MaaS automatically detects non-MaaS AuthPolicies (e.g., from KServe or other controllers) that target the same HTTPRoutes used by MaaS-governed models. When a conflict is detected, MaaS sets a ConflictingAuthPolicy condition on the affected MaaSAuthPolicy resource and emits a Kubernetes warning event.

Symptoms

• MaaSAuthPolicy has condition ConflictingAuthPolicy=True
• Warning events on MaaSAuthPolicy resources referencing non-MaaS AuthPolicies
• Unexpected authentication behavior (wrong policy may win when multiple AuthPolicies target the same HTTPRoute)

Diagnosis

Check for conflicting AuthPolicy conditions:

# Check MaaSAuthPolicy conditions
kubectl get maasauthpolicy -n models-as-a-service -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .status.conditions[?(@.type=="ConflictingAuthPolicy")]}{.status}{"\t"}{.message}{end}{"\n"}{end}'

# List all AuthPolicies targeting a specific model's HTTPRoute
MODEL_NS="<model-namespace>"
MODEL_NAME="<model-name>"
kubectl get authpolicy -n "$MODEL_NS" -o json | \
  jq -r ".items[] | select(.spec.targetRef.name == \"$MODEL_NAME\" and .spec.targetRef.kind == \"HTTPRoute\") | .metadata.name + \" (managed-by: \" + (.metadata.labels[\"app.kubernetes.io/managed-by\"] // \"unknown\") + \")\""

# Check for warning events
kubectl get events -n models-as-a-service --field-selector reason=ConflictingAuthPolicy

Remediation

1. KServe anonymous auth policies (*-kserve-route-authn): These are typically created when security.opendatahub.io/enable-auth is misconfigured on the InferenceService. Set the annotation to "true" on MaaS-governed InferenceServices to prevent KServe from creating conflicting anonymous AuthPolicies:

   kubectl annotate inferenceservice <name> -n <namespace> \
     security.opendatahub.io/enable-auth="true" --overwrite
   

2. Custom AuthPolicies: If another team deployed a custom AuthPolicy targeting the same HTTPRoute, coordinate to determine which controller should own authentication for that route. Either:

   • Remove the conflicting AuthPolicy if MaaS is the intended auth authority
   • Or remove the model from the MaaSAuthPolicy if MaaS should not govern that route

3. Verify resolution: After applying the fix, the ConflictingAuthPolicy condition should transition to False on the next reconciliation:

   kubectl get maasauthpolicy -n models-as-a-service -o jsonpath='{range .items[*]}{.metadata.name}{": "}{range .status.conditions[?(@.type=="ConflictingAuthPolicy")]}{.status}{end}{"\n"}{end}'
   

TLS Certificate Validation

By default, curl validates TLS certificates against your system CA bundle. If you encounter certificate verification errors (e.g., curl: (60) SSL certificate problem: self-signed certificate), use one of the approaches below.

Recommended: Use the Ingress CA/Certificate Chain

For OpenShift clusters with self-signed or internal certificates, use the ingress trust chain (or your platform-provided CA bundle) with curl --cacert:

# Get ingress certificate chain/trust bundle (platform-specific source)
oc get secret -n openshift-ingress router-certs-default \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/ingress-cert-chain.crt

# Use --cacert in curl commands
curl -sS --cacert /tmp/ingress-cert-chain.crt \
  -H "Authorization: Bearer $API_KEY" \
  "https://maas.${CLUSTER_DOMAIN}/maas-api/v1/models" | jq .

Alternatively, if your cluster uses the OpenShift service CA:

# Extract the service CA bundle
oc get configmap -n openshift-config-managed service-ca-bundle \
  -o jsonpath='{.data.service-ca\.crt}' > /tmp/service-ca.crt

# Use --cacert in curl commands
curl -sS --cacert /tmp/service-ca.crt \
  -H "Authorization: Bearer $API_KEY" \
  "https://maas.${CLUSTER_DOMAIN}/maas-api/v1/models" | jq .

Development/Testing Only: Disable Verification

Security Warning

The -k flag disables all TLS certificate validation. An attacker on the network path can present a forged certificate and intercept your API key, token, or other credentials. Never use -k in production or when sending credentials over untrusted networks.

For isolated development or test environments only, you can add the -k flag:

# INSECURE: Only for isolated dev/test environments
curl -sS -k -H "Authorization: Bearer $API_KEY" \
  "https://maas.${CLUSTER_DOMAIN}/maas-api/v1/models" | jq .

Adding the CA to Your System Trust Store

For a permanent solution, add the ingress certificate chain to your system trust store so that all tools (curl, Python, browsers) trust it automatically:

# Linux (Fedora/RHEL)
sudo cp /tmp/ingress-cert-chain.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

# Linux (Debian/Ubuntu)
sudo cp /tmp/ingress-cert-chain.crt /usr/local/share/ca-certificates/ingress-cert-chain.crt
sudo update-ca-certificates

# macOS
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain /tmp/ingress-cert-chain.crt

For detailed TLS configuration options, see TLS Configuration.

Additional Resources

• Validation Guide — Manual validation steps
• Observability Guide — Metrics, monitoring, and dashboards
• scripts/README.md — Deployment scripts documentation