Skip to content

Reconciliation Flow

This document describes how the MaaS Controller reconciles resources, ownership, and lifecycle.

MaasTenantConfig reconciler

The MaasTenantConfig reconciler watches MaasTenantConfig CRs named default-tenant (maas.opendatahub.io/v1alpha1) in tenant namespaces. It:

  • Resolves platform context from the owning AITenant (Gateway and OIDC) and validates cluster dependencies (for example AuthConfig / Kuadrant CRDs).
  • Runs prerequisite checks in the application namespace before applying manifests.
  • Renders and applies embedded kustomize manifests with Server-Side Apply, then waits until the maas-api Deployment in the app namespace is available.
  • Manages finalizers and teardown for Removed / Unmanaged management-state annotations.

Cross-namespace and cluster-scoped platform pieces (gateway AuthPolicy, telemetry, cluster RBAC) are tracked with labels and cleaned up when needed; in-namespace workloads use ownerReference where appropriate (see diagram below).

MaasTenantConfig status

MaasTenantConfig.status exposes a high-level phase: Pending, Active, Degraded, or Failed, plus conditions aligned with platform aggregation (for example Ready, DependenciesAvailable, MaaSPrerequisitesAvailable, DeploymentsAvailable, Degraded). Typical outcomes:

  • Failed — blocking prerequisites missing, gateway invalid, or platform apply failed.
  • Pending — manifests applied but maas-api not ready yet.
  • Active — platform applied and maas-api deployment available; may still be Degraded if non-blocking warnings exist.

For spec fields (API keys and telemetry), see MaasTenantConfig CR. Gateway and OIDC platform context belongs to AITenant.

MaasTenantConfig resource layout

The MaasTenantConfig CR is namespace-scoped and lives in the tenant namespace for MaaS platform configuration. The default AITenant uses models-as-a-service as its tenant namespace; non-default AITenants use their derived tenant namespaces. It owns resources across three scopes — same-namespace children use standard ownerReference, while cluster-scoped and cross-namespace children use tracking labels.

graph TB
    subgraph "tenant namespace (for example models-as-a-service)"
        Tenant["MaasTenantConfig CR<br/>default-tenant"]
        API["maas-api Deployment"]
        CM["ConfigMaps"]
        SVC["Services"]
        SA["ServiceAccounts"]
        NP["NetworkPolicies"]
        HR["HTTPRoutes"]
        AP2["maas-api AuthPolicy"]
    end

    subgraph "openshift-ingress namespace"
        AP["gateway AuthPolicy"]
        DR["DestinationRule"]
        TP["TelemetryPolicy"]
        IT["Istio Telemetry"]
    end

    subgraph "Cluster-scoped"
        CR["ClusterRoles"]
        CRB["ClusterRoleBindings"]
    end

    Tenant -->|ownerRef| API
    Tenant -->|ownerRef| CM
    Tenant -->|ownerRef| SVC
    Tenant -->|ownerRef| SA
    Tenant -->|ownerRef| NP
    Tenant -->|ownerRef| HR
    Tenant -->|ownerRef| AP2
    Tenant -.->|tracking labels| CR
    Tenant -.->|tracking labels| CRB
    Tenant -.->|tracking labels| AP
    Tenant -.->|tracking labels| DR
    Tenant -.->|tracking labels| TP
    Tenant -.->|tracking labels| IT

    style Tenant fill:#4a90d9,color:#fff
    style AP fill:#f5a623,color:#fff
    style DR fill:#f5a623,color:#fff
    style TP fill:#f5a623,color:#fff
    style IT fill:#f5a623,color:#fff

Solid arrows = standard ownerReference (automatic GC). Dashed arrows = tracking labels (finalizer-based cleanup). Orange resources = cross-namespace children that require tracking labels.

Reconciler Behavior

MaaSModelRef Reconciler

What it does: - Validates that the referenced model resource exists (e.g., LLMInferenceService) - Validates that HTTPRoute exists for the model (or creates it for certain kinds) - Updates status with endpoint URL and readiness phase

Watch triggers: - MaaSModelRef changes - HTTPRoute changes (fixes startup race when KServe creates route after MaaSModelRef) - LLMInferenceService changes (for backend spec/status updates)

Status transitions: - Pending → HTTPRoute doesn't exist yet (KServe still deploying) - Ready → HTTPRoute exists and model is accessible - Failed → Referenced model not found or validation failed

Finalizer behavior: - Adds finalizer to MaaSModelRef - On deletion, triggers cascade deletion of all AuthPolicies and TokenRateLimitPolicies for this model - Removes finalizer after cleanup completes

MaaSAuthPolicy Reconciler

What it does: - Creates one Kuadrant AuthPolicy per referenced model - Aggregates multiple MaaSAuthPolicies targeting the same model into a single AuthPolicy - Attaches AuthPolicy to the model's HTTPRoute via targetRef

Watch triggers: - MaaSAuthPolicy changes - MaaSModelRef changes (re-reconcile when model created/deleted) - HTTPRoute changes (re-reconcile when route appears) - Generated AuthPolicy changes (overwrite manual edits unless opted out)

Opt-out annotation: 

apiVersion: kuadrant.io/v1
kind: AuthPolicy
metadata:
  annotations:
    opendatahub.io/managed: "false"  # Controller won't overwrite or delete

MaaSSubscription Reconciler

What it does:

  • Creates one Kuadrant TokenRateLimitPolicy per model (aggregating every MaaSSubscription that references that model)
  • Adds one limit entry per distinct set of tokenRateLimits among those subscriptions, so subscriptions with identical rates share an entry:
    • The limit is named after its rates: tokens-{limit}-per-{window}, with one -{limit}-per-{window} segment per rate, deduplicated and sorted as strings (for example tokens-1000-per-1h-99999-per-1s). Windows are compared as written, so 1m and 60s are different rate sets.
    • New subscriptions (status.rateLimitIdentity=short): when matches auth.identity.selected_subscription_id (first 16 hex chars of SHA-256 of the model-scoped key); counters are selected_subscription_id and userid.
    • Pre-upgrade subscriptions (status.rateLimitIdentity=legacy): when and counters still use auth.identity.selected_subscription_key, so live Limitador quotas are not reset.
    • If the same rate set has both generations, the controller emits two limits (tokens-… for legacy members, tokens-…-id for short-ID members) because one limit can have only one counter expression. That extra limit is the size trade-off for keeping old quotas; see Upgrade to 3.6.
    • A group of one gets the single equality without parentheses. Unlimited subscriptions share tokens-unlimited with mixed predicates and no counters.
  • AuthPolicy is responsible for resolving subscription selection (via maas-api) and setting the matching identity before TRLP runs; see Authentication Internals.
  • Exempts /v1/models from token consumption limits where configured so discovery still works when quotas are exhausted

Kuadrant copies every limit into every route-match ActionSet of the gateway's wasm config (kuadrant-{gateway-name}, an EnvoyFilter on Kuadrant 1.5+ or a WasmPlugin on 1.4.x), so the number of limits drives that object's size (RHOAIENG-95277). Grouping by rate means a subscription at an existing rate adds one clause to a predicate instead of a whole limit. See Subscription Cardinality.

Watch triggers: - MaaSSubscription changes - MaaSModelRef changes (re-reconcile when model created/deleted) - HTTPRoute changes (re-reconcile when route appears) - Generated TokenRateLimitPolicy changes (overwrite manual edits unless opted out)

Opt-out annotation: 

apiVersion: kuadrant.io/v1alpha1
kind: TokenRateLimitPolicy
metadata:
  annotations:
    opendatahub.io/managed: "false"  # Controller won't overwrite or delete

Lifecycle: Deletion Behavior

MaaSModelRef deleted: - Controller uses finalizer to cascade-delete all AuthPolicies and TokenRateLimitPolicies for that model - Parent MaaSAuthPolicy and MaaSSubscription CRs remain intact - Underlying LLMInferenceService is not affected

MaaSSubscription deleted: - Aggregated TokenRateLimitPolicy is deleted, then rebuilt from remaining subscriptions - If no subscriptions remain, model falls back to gateway defaults (401/403 from auth, or 429 from TRLP safety net)

MaaSAuthPolicy deleted: - Aggregated AuthPolicy is rebuilt from remaining auth policies - If no auth policies remain, model falls back to gateway default deny (401/403)

Orphaned policies warning: An opted-out policy (opendatahub.io/managed: "false") can become permanently orphaned (no longer reconciled and not deleted) when: - The last MaaSAuthPolicy/MaaSSubscription referencing a model is deleted - A model is removed from spec.modelRefs (edit rather than deletion) - A MaaSModelRef is deleted

Manually delete orphaned opted-out resources when no longer needed.