MaaS Testing and Contribution Guide

This guide covers what tests exist in the MaaS project, how to run them, and how to add new tests when contributing a feature or fix.

Test Overview

Type	Location	Language	Runner
Unit tests	`maas-api/internal/`, `maas-api/cmd/`	Go	`make test`
Unit tests	`maas-controller/pkg/`	Go	`make test`
E2E tests	`test/e2e/tests/`	Python (pytest)	`run-tests-quick.sh`, `smoke.sh`
CI smoke / E2E	`test/e2e/scripts/prow_run_smoke_test.sh`	Bash + pytest	Konflux integration
Integration tests	`test/integration/postgres.sh`	Bash	See Integration Tests (PostgreSQL)
Integration tests	opendatahub-tests	Python (pytest)	ODH CI / Nightly

Repository Structure (Testing)

maas-api/
├── internal/**/*_test.go        # Unit tests for API handlers, services, auth
├── cmd/*_test.go                # Unit tests for CLI/server setup
└── test/fixtures/               # Shared Go test helpers (fakes, test data)

maas-controller/
└── pkg/**/*_test.go             # Unit tests for CRD reconcilers

test/e2e/
├── tests/                       # Pytest E2E test modules
│   ├── conftest.py              # Shared session-scoped fixtures
│   └── test_*.py                # Test modules (see table below)
├── fixtures/                    # Kustomize overlays for test CRs
├── scripts/prow_run_smoke_test.sh  # Full deploy + E2E (CI entrypoint)
├── smoke.sh                     # Run all E2E modules locally
├── run-tests-quick.sh           # Quick pytest run (no deploy)
└── requirements.txt             # Python dependencies

E2E Test Modules and Groups

Tests are organized into xdist groups for parallel execution. All tests in a group run on the same pytest-xdist worker, and a worker takes the next queued group when it finishes one. See Parallel E2E for details on how groups work.

Group	Module	What it covers
`readonly`	`test_smoke.py`	Health checks, catalog shape, basic inference
`readonly`	`test_config_tenant.py`	Read-only Config CR validation
`readonly`	`test_tenant.py`	Read-only MaasTenantConfig validation
`readonly`	`test_tenant_discovery.py`	Read-only /v1/tenants endpoint checks
`readonly`	`test_networkpolicy.py`	NetworkPolicy + connectivity checks
`readonly`	`test_gateway_filter_chain.py`	Gateway filter order (ipp-pre -> auth -> ipp -> endpoint picker); its InferencePool traffic test is `serial`
`api_keys`	`test_api_keys.py`	API key CRUD, admin authorization, validation
`api_keys`	`test_subscription.py`	Subscription enforcement, rate limiting, auth flows
`api_keys`	`test_subscription_list_endpoints.py`	Subscription listing endpoints
`api_keys`	`test_x_api_key_auth.py`	X-API-Key header authentication (IPP ExternalModel)
`models`	`test_models_endpoint.py`	`/v1/models` subscription-aware filtering
`models`	`test_gateway_scoped_authpolicy.py`	Gateway AuthPolicy structure, lifecycle, enforcement gaps
`models`	`test_model_identity_conflict.py`	MaaSModelRef model-identity collision detection
`models`	`test_multi_tenant_integration.py`	Full lifecycle, two-tenant coexistence (placed here to even out pass 1)
`security`	`test_negative_security.py`	Header spoofing, expired keys, cross-model access
`security`	`test_namespace_scoping.py`	Namespace isolation behavior
`mt_lifecycle`	`test_aitenant_lifecycle.py`	AITenant create/migrate/delete
`mt_lifecycle`	`test_multi_tenant_maas_api.py`	Per-tenant Deployment, Service, HTTPRoute; dual-listener AuthConfig/wasm checks are `@pytest.mark.nightly`
`mt_lifecycle`	`test_tenant_namespace_discovery.py`	Namespace-label discovery, webhook validation
`mt_lifecycle`	`test_crd_watch_resilience.py`	Dynamic CRD watch registration (restarts controller)
`tenant_isolation`	`test_tenant_auth_isolation.py`	Cross-tenant key rejection, OIDC scoping
`tenant_isolation`	`test_tenant_rate_limit_isolation.py`	Per-tenant rate limit independence
`tenant_isolation`	`test_tenant_subscription_isolation.py`	Per-tenant subscription scoping
`tenant_isolation`	`test_tenant_discovery_isolation.py`	Per-tenant /v1/tenants isolation
`tenant_inference`	`test_tenant_model_inference.py`	Cross-gateway auth isolation and per-Gateway AuthPolicy EnvoyFilter selectors
`tenant_ipp`	`test_per_tenant_ipp_isolation.py`	Per-tenant IPP stacks, routing isolation
`tenant_auto_resolve`	`test_tenant_auto_resolve.py`	MaaSModelRef tenant auto-resolution from the gateway parentRef
`inference_priority`	`test_inference_priority.py`	Subscription `inferencePriority`: InferenceObjective lifecycle, `flowControlStatuses`, gateway flow-control headers (`serial`; skips without the `inferenceobjectives.llm-d.ai` CRD)
`external`	`test_external_oidc.py`	External OIDC token flows (skipped unless `EXTERNAL_OIDC=true`)
`external`	`test_external_models.py`	ExternalModel/ExternalProvider, egress, body routing

Running Tests

Unit Tests (Go)

maas-apimaas-controller

cd maas-api
make test    # runs go test with race detection + coverage
make lint    # golangci-lint

cd maas-controller
make test    # runs go test with race detection + coverage
make lint    # golangci-lint

Both generate a coverage.html report in their respective directories.

Integration Tests (PostgreSQL)

Integration tests for maas-api verify JSONB storage, GIN index queries, and NULL handling against a real PostgreSQL database.

Integration tests skip protection:

Note: TestPostgres is compiled with the other unit tests. To avoid the postgresql integration tests running during unit testing with make test the following must be true for the postgresql tests to be included in make test.

Build tag check (requires -tags=integration)
testing.Short() check (skipped in short mode)
Environment variable check (skips if TEST_DATABASE_URL not set)

./test/integration/postgresql.sh
if [ $? -eq 0 ]; then 
  echo "Integration tests passed" 
fi

E2E Tests (Python)

Prerequisites

OpenShift cluster with MaaS deployed, oc logged in as cluster-admin, Python 3.10+ (3.11 recommended; pytest 9).

Quick (local dev)Smoke (all modules)Full CI (deploy + test)Direct pytest

# Cluster must already have MaaS deployed
./test/e2e/run-tests-quick.sh

# Run specific file or filter
./test/e2e/run-tests-quick.sh tests/test_api_keys.py
./test/e2e/run-tests-quick.sh -k test_create_api_key

./test/e2e/smoke.sh

# Deploys the full platform, models, fixtures, then runs E2E
./test/e2e/scripts/prow_run_smoke_test.sh

# Skip deploy if cluster is already set up
SKIP_DEPLOYMENT=true ./test/e2e/scripts/prow_run_smoke_test.sh

cd test/e2e
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

export GATEWAY_HOST="maas.apps.your-cluster.example.com"
export E2E_SKIP_TLS_VERIFY=true

# Mixed-marker modules need separate passes (or use run_e2e_tests.sh for the full suite).
pytest tests/ -m "not serial" -v
pytest tests/ -m serial -v
pytest tests/test_subscription.py -m "not serial" -v
pytest tests/test_api_keys.py::TestAPIKeyCRUD -m "not serial" -v

Key Environment Variables

The E2E framework auto-discovers most values from the cluster. These are the most common overrides:

Variable	Description
`GATEWAY_HOST`	Gateway hostname (required unless `MAAS_API_BASE_URL` is set)
`MAAS_API_BASE_URL`	Full MaaS API URL (auto-derived if not set)
`TOKEN`	User bearer token (falls back to `oc whoami -t`)
`ADMIN_OC_TOKEN`	Admin token; admin tests skip if unset
`E2E_SKIP_TLS_VERIFY`	Set `true` to skip TLS verification
`MODEL_NAME`	Override model ID (defaults to first from catalog)
`E2E_MODEL_PATH`	Override the model route path used by inference tests (for example, `/llm/facebook-opt-125m-simulated`; omit `/v1`)
`EXTERNAL_OIDC`	Set `true` to enable external OIDC tests
`E2E_PARALLEL_WORKERS`	pytest-xdist worker count for pass 1 (default `7`). Set to `1` for single-worker pass 1 without xdist; pass 2 stays serial.
`LLMISVC_CONTROLLER_CPU_REQUEST` / `LLMISVC_CONTROLLER_CPU_LIMIT`	CPU that `deploy-platform.sh` gives `llmisvc-controller-manager` on the test cluster (defaults `1` / `2`; shipped is 100m). Set the limit empty to keep the shipped resources.

See test/e2e/tests/conftest.py and individual test module docstrings for the full set of supported variables.

The governed chat/completions smoke regressions are covered by tests/test_smoke.py::test_chat_completions_gateway_alive (non-streaming body content) and tests/test_smoke.py::test_chat_completions_streaming_sse (text/event-stream, JSON data: chunks, data: [DONE]). Both use the governed facebook-opt-125m-simulated LLMInferenceService and MaaSModelRef deployed by scripts/deploy-models.sh, plus the simulator-subscription API key created by the api_key_headers fixture. MODEL_NAME selects the route/catalog model, while E2E_MODEL_NAME sets the ID sent in the inference request body (default facebook/opt-125m). The full CI runner sets GATEWAY_HOST and MODEL_NAME; for a local run, set GATEWAY_HOST to the gateway hostname and MODEL_NAME to the catalog model ID. Set E2E_MODEL_PATH when the catalog URL does not identify the model route. SSE parsing is centralized in test_helper.assert_openai_sse_chat_stream.

Parallel E2E (pytest-xdist)

run_e2e_tests.sh and run-tests-quick.sh run two marker-filtered passes by default (default E2E_PARALLEL_WORKERS=7). Use --serial-only to run pass 2 only (-m serial); that mode does not execute non-serial tests. Direct pytest invocations must use the same split when a module mixes @serial and worker-tenant tests.

Pass 1: -m "not serial" — with --dist=loadgroup -n 7 when workers > 1, or single-worker execution when E2E_PARALLEL_WORKERS=1
Pass 2 (serial): -m serial — tests that mutate shared cluster state (subscription delete/restore, controller/Kuadrant scaling)

Set E2E_PARALLEL_WORKERS=1 to disable xdist on pass 1 while keeping the serial/non-serial split (useful for debugging on shared clusters). Pass ./run_e2e_tests.sh --serial-only only when intentionally limiting the run to serial mutators.

xdist Groups

Every test file must have a module-level xdist_group marker:

import pytest

pytestmark = pytest.mark.xdist_group("api_keys")

All tests in a file run on the same xdist worker, grouped with other files sharing the same group name. Tests without an xdist_group marker are scattered round-robin across workers, which can cause resource conflicts.

Group Assignment Rules

When adding a new test file, assign it to a group using these criteria (in priority order):

Mutation scope: Tests that create, modify, or delete the same CRs (MaaSAuthPolicy, MaaSSubscription, MaaSModelRef, etc.) must be in the same group. This prevents two workers from concurrently mutating the same resource.
Functional domain: Group tests by what they exercise. API key tests go with api_keys, gateway AuthPolicy tests go with models, multi-tenant lifecycle tests go with mt_lifecycle, etc.
Read-only tests go in readonly: Tests that only read cluster state (no creates, no deletes, no patches) belong in the readonly group. This group finishes fast and never conflicts with other groups.
Self-managing tenant tests go in mt_lifecycle or a tenant_* group: Tests that create their own AITenant CRs belong in one of these groups. mt_lifecycle for lifecycle operations (create/delete/migrate), tenant_isolation for cross-tenant isolation assertions using the shared_test_tenants fixture. A long isolation module that builds its own tenant pair gets its own group (tenant_inference, tenant_ipp), so its runtime doesn't add to tenant_isolation's.
Externally-gated tests go in external: Tests that require external infrastructure (Keycloak OIDC, external endpoints) and use pytest.mark.skipif to self-gate belong in the external group.
Prefer existing groups over new ones: Adding more groups increases cluster resource pressure. Only create a new group if the test's mutation scope genuinely conflicts with all existing groups. The exception is wall clock: with --dist=loadgroup pass 1 cannot finish before its longest group, so split a group whose runtime alone sets the pass length, as long as the parts share no tenants or CRs and make no negative assertions (quiet logs, counters) on components the others touch. Pass 1 has more groups than workers: xdist starts the groups with the most tests first and hands the rest to whichever worker drains first, so a long group with few tests starts late.

Serial Marker

Tests that mutate global cluster state must be marked @pytest.mark.serial:

@pytest.mark.serial
def test_delete_shared_subscription_then_restore(self):
    ...

Serial tests run in Pass 2 on a single worker. Use @serial when the test:

Deletes or modifies a shared fixture (e.g., simulator-subscription, simulator-access)
Scales or restarts a cluster operator (maas-controller, Kuadrant)
Modifies a CRD or cluster-scoped resource that other groups depend on

Do not use @serial for tests that only create/delete their own uniquely-named resources — those are safe to run in parallel within their group.

CI Pipeline

CI runs automatically on every PR and push to main. Here's how the different test types fit together:

graph TB
    A[PR opened / push to main] --> B{What changed?}
    B -->|maas-api/**| C[GitHub Actions: lint + unit tests]
    B -->|maas-controller/api/**| D[GitHub Actions: verify-codegen]
    B -->|Any non-docs change| E[Konflux: Build images]
    E --> F[Integration test on ephemeral cluster]
    F --> G[prow_run_smoke_test.sh → E2E pytest]
    B -->|docs/** or *.md only| H[Skip build + E2E]

System	Trigger	What runs
GitHub Actions	`maas-api/**` changes	golangci-lint, `make test` (Go unit tests), image build
GitHub Actions	`maas-controller/api/` or `deployment/` changes	`make verify-codegen`, kustomize manifest validation
Konflux	Any non-docs PR or push to `main`	Builds multi-arch images, then runs full E2E on an ephemeral OpenShift cluster

Konflux provisions a fresh cluster, deploys ODH + MaaS with the PR's built images, and runs test/e2e/scripts/prow_run_smoke_test.sh. Nightly builds use the same script against the latest main images — there is no separate nightly test suite.

Deploy phase timings and fail-fast

/group-test is sequential: cluster provision → deploy + validate → pytest. prow_run_smoke_test.sh writes UTC start/end stamps for deploy_platform, deploy_models, validate, pytest_pass1, and pytest_pass2 to phase-timings.txt.

Where	Path
Local	`$PROJECT_ROOT/test/e2e/reports/phase-timings.txt` (or `$ARTIFACT_DIR` / `$ARTIFACTS` / `$LOG_DIR` if set)
Konflux / Prow	`artifacts/<job>/<step>/phase-timings.txt` (OpenShift CI copies `ARTIFACT_DIR`)

Each line looks like 2026-08-19T20:01:02Z deploy_platform start. A start without a matching end means that phase failed or the job was killed.

Readiness gates fail the job before pytest so a bad DSC, AuthPolicy, or ODH install does not burn another ~40 minutes of pytest:

DataScienceCluster wait in prow_run_smoke_test.sh exits 1 and dumps DSC conditions.
After models are applied, AuthPolicy Enforced=True is required (wait_for_auth_policies_enforced returns 1 on timeout). SKIP_AUTH_CHECK still defaults to true before models exist (RHOAIENG-48760 chicken-egg).
.github/hack/install-odh.sh exits 1 if the operator webhook, DSCInitialization, or DataScienceCluster never become Ready.
validate-deployment.sh retries after polling gateway/pods, not after a fixed 30s sleep.

SKIP_DEPLOYMENT=true still skips platform and model install; timings are then recorded only for validate and pytest.

Docs-only changes

PRs that only touch docs/** or *.md files skip Konflux builds and E2E entirely (controlled via CEL expressions in .tekton/ pipeline definitions).

Adding New Tests

Adding a Go Unit Test

Create a *_test.go file next to the code you're testing (standard Go convention)
Use testify/assert and testify/require for assertions
For maas-api handlers: use net/http/httptest and the fake listers in maas-api/test/fixtures/
For maas-controller reconcilers: use controller-runtime/pkg/client/fake with scheme setup
Run make test in the relevant directory to verify

Quick reference — a minimal handler test in maas-api:

func TestMyHandler_Returns200(t *testing.T) {
    req := httptest.NewRequest(http.MethodGet, "/v1/my-endpoint", nil)
    rec := httptest.NewRecorder()
    handler := NewMyHandler(/* fake dependencies */)

    handler.ServeHTTP(rec, req)

    require.Equal(t, http.StatusOK, rec.Code)
}

Quick reference — a minimal reconciler test in maas-controller:

func TestMyReconciler_Succeeds(t *testing.T) {
    client := fake.NewClientBuilder().WithScheme(scheme).WithObjects(/* seed */).Build()
    r := &MyReconciler{Client: client, Scheme: scheme}

    result, err := r.Reconcile(ctx, reconcile.Request{NamespacedName: key})

    require.NoError(t, err)
    require.False(t, result.Requeue)
}

CRD codegen

If you modify API types under maas-controller/api/, run make -C maas-controller generate manifests and commit the generated files. CI will fail if they're stale.

Adding an E2E Test

Pick the right module — add your test to an existing test_*.py file if it fits the scope (see the E2E test modules table). Create a new module only if your feature doesn't fit any existing one.
Assign an xdist group — every test file must have a module-level xdist_group marker. Follow the group assignment rules to pick the right group. If adding to an existing file, it already has a group.
```
import pytest

pytestmark = pytest.mark.xdist_group("api_keys")
```
Mark serial tests — if your test mutates shared cluster state (deletes shared fixtures, scales operators), add @pytest.mark.serial. See Serial Marker for criteria.
Use shared fixtures — conftest.py provides session-scoped fixtures like maas_api_base_url, headers, token, admin_headers, api_key, etc. Import TLS_VERIFY from conftest for HTTP calls.
Add test resources if needed — if your feature requires new MaaS CRs (models, subscriptions, auth policies), add a kustomize overlay under test/e2e/fixtures/ and include it in the base kustomization.yaml.
Use pytest discovery for CI — run_e2e_tests.sh collects every test_*.py module under test/e2e/tests/. New modules do not need a separate CI registration step. Give the module an appropriate xdist_group marker and mark cluster-wide mutations as serial.
Use skip markers for optional features — if your test depends on optional infrastructure (e.g., external OIDC, IPP ExternalModel CRD), gate it with pytest.mark.skipif so the same module runs cleanly in all environments.

Integration Testing with ODH Operator

MaaS is a component of Open Data Hub (ODH). Beyond the in-repo E2E tests, MaaS is also validated as part of the broader ODH integration test suite in opendatahub-tests.

How It Works

The opendatahub-tests repo contains cross-component integration tests for the entire ODH platform, including model serving and MaaS functionality.
These tests run in ODH CI and nightly pipelines against full ODH deployments with all components installed.
MaaS-related integration tests validate that MaaS works correctly when deployed through the ODH operator alongside KServe, Authorino, and other platform components.

Contributing Integration Tests

If your change affects how MaaS integrates with the ODH operator or other ODH components, you may need to add or update tests in the opendatahub-tests repo:

Follow the opendatahub-tests contributing guide and developer guide
Tests are organized by component — look for MaaS-related tests under the relevant directory
The project uses openshift-python-wrapper for Kubernetes/OpenShift API interactions
Follow the project's style guide and run pre-commit checks before submitting

When to Add In-Repo vs Integration Tests

Scenario	Where to add tests
New MaaS API endpoint or controller behavior	In-repo: Go unit tests + E2E in `test/e2e/`
MaaS CRD changes or subscription logic	In-repo: controller unit tests + E2E
Interaction with ODH operator, KServe, or Authorino	`opendatahub-tests` repo
End-to-end model serving through the full ODH stack	`opendatahub-tests` repo
Bug fix with regression test	In-repo (unit or E2E depending on scope)

Test Ownership Across the AI Gateway Stack (AIGO / AIGC / MaaS)

AI Gateway now spans three repos: ai-gateway-operator (AIGO), ai-gateway-controller (AIGC), and this repo (MaaS). Where a test belongs depends on what it needs to observe, not which repo you happen to be changing:

Repo	Owns	What it tests	Test suite
AIGO	Component setup & dependency management	`AIGateway` CR reconciles to `Ready`; sibling controller Deployments (`maas-controller`, `ai-gateway-controller`, `batch-gateway-operator`) become `Available`; aggregate status (`ModelsAsAServiceReady`, etc.) rolls up correctly. Never the request path.	`test/e2e/*_test.go` (Go — deploy prereqs, create CR, assert status)
AIGC	Deploying the Praxis-backed stack + AIGC's own control-plane logic	Its own reconciliation logic (`pkg/tenant`, `pkg/render`, `pkg/controller` for ExternalModel/ExternalProvider) via real fixtures. For MaaS-level/request-path behavior, AIGC does not write new test content — it fetches and re-runs this repo's own pytest suite against its Praxis-backed deployment, pinned via `test/maas-e2e.lock`, proving the same behavior holds under Praxis as under IPP.	`test/kind-env`, `test/openshift-env` (real fixtures) + a runtime checkout of `test/e2e/tests/` from this repo in `test/maas-e2e/`, fetched at the commit pinned in `test/maas-e2e.lock`
MaaS (this repo)	The single source of truth for MaaS-level resource/request behavior	Subscription enforcement, auth policy, rate limiting, API-key lifecycle, and anything visible at the MaaS API/Gateway boundary — including OpenAI resource-API routing and identity-header propagation. Written once here, run twice: in-repo against IPP, and via AIGC's runtime checkout in `test/maas-e2e/` against Praxis.	`test/e2e/tests/*.py` (pytest)

The upstream-first rule: if MaaS-level behavior has a coverage gap that's exposed while working in AIGC or AIGO, add the test to this repo's test/e2e/tests/ first (skip-gated with pytest.mark.skipif if the behavior is genuinely Praxis-only, mirroring the existing EXTERNAL_OIDC skip pattern — see Group Assignment Rules), then have the downstream repo bump its pin to pick it up. Don't accept a parallel Go/mock test in AIGO, and don't let a bespoke copy accumulate in AIGC. ai-gateway-controller#37 is a worked example of this lifecycle: a gap was fixed upstream in MaaS (#1508); once it landed, AIGC removed its own side-workarounds and re-pinned test/maas-e2e.lock.

Checklist: Adding a New Test

Unit test: Add *_test.go alongside your source in maas-api/ or maas-controller/; run make test
E2E test: Add to an existing test_*.py or create a new module in test/e2e/tests/
xdist group: Ensure the test file has a pytestmark = pytest.mark.xdist_group("group_name") marker (see group assignment rules)
Serial marker: Add @pytest.mark.serial if the test mutates shared fixtures or scales operators
Test fixtures: If new CRs are needed, add a kustomize overlay in test/e2e/fixtures/
Pytest discovery: Confirm the new module follows the test_*.py naming convention and is collected with pytest tests/ --collect-only -q
Skip markers: Use pytest.mark.skipif for tests requiring optional infrastructure
Local validation: Run make test and/or ./test/e2e/run-tests-quick.sh before pushing
Integration tests: If your change affects ODH operator integration, update tests in opendatahub-tests