Telemetry Defaults and Cardinality Guidance

This document provides recommended defaults for MaaS telemetry dimensions, explains cardinality risks, and gives guidance for safely scaling observability.

Estimates, Not Load-Tested

The projections in this document are analytical estimates based on the Prometheus time-series model, not validated with load testing. Real-world cardinality depends on usage patterns, model adoption breadth, and label value counts that vary by deployment. These estimates should be validated with customer scale data before being used for capacity planning.

How MaaS Telemetry Works

Per-user metrics are generated by Limitador (Kuadrant's rate-limiting engine) at the Istio gateway, not by maas-api. The TelemetryPolicy defines which labels are added to Limitador counters:

Label	Source	Default	Cardinality
user	auth.identity.userid	Disabled (captureUser: false)	Unbounded — grows with user base
model	responseBodyJSON("/model")	Enabled (captureModelUsage: true)	Unbounded — grows with deployed models (see Model Cardinality)
subscription	auth.identity.selected_subscription	Always enabled	Unbounded — grows with MaaSSubscription CRs
organization_id	auth.identity.subscription_info.organizationId	Enabled (captureOrganization: true)	Unbounded — grows with tenant count
cost_center	auth.identity.subscription_info.costCenter	Always enabled	Unbounded — grows with org structure

All labels are effectively unbounded

Unless MaaS enforces hard limits on unique values per label, every dimension can grow without constraint. The organization_id and cost_center labels are attributes of a subscription (each subscription belongs to exactly one org and cost center), so they do not independently multiply cardinality. The primary cardinality drivers are user × model × subscription. See Cardinality Analysis for details.

Metrics Affected

Metric	Type	Observed Labels	Purpose
authorized_hits	Counter	subscription, model, limitador_namespace	Token consumption (from usage.total_tokens in response body)
authorized_calls	Counter	subscription, limitador_namespace	Requests allowed through rate limiting
limited_calls	Counter	limitador_namespace	Requests denied by rate limits

Label availability varies by metric

The model label is available only on authorized_hits. The limited_calls metric carries only limitador_namespace — it does not include subscription or other TelemetryPolicy labels. When captureUser is enabled, the user label appears on authorized_hits and authorized_calls. The organization_id and cost_center labels appear only when the corresponding fields are set on the MaaSSubscription CR.

Metrics that intentionally avoid high-cardinality labels:

• istio_request_duration_milliseconds_bucket — uses subscription only (histograms have high base cardinality from bucket boundaries)
• vllm:* — uses model_name only (vLLM has no visibility into user identity)

Model Cardinality

The model label is not bounded by MaaS. Real deployments can have significantly more models than a typical dev cluster:

Deployment Size	Typical Model Count	Source
Dev/test	5-10	Small set for development
Production (small)	10-50	Focused enterprise use case
Production (medium-large)	~200	RHAISTRAT-730 sizing guidance
Aggregator platforms	500+	e.g., OpenRouter serves 653 models

A single API call to any model creates time series in the retention period. Users who experiment across many models amplify the effective cardinality multiplier.

Cardinality Analysis

Cardinality Dimensions

The theoretical maximum series count for a single metric is the cross-product of all independent label values:

worst_case = users × models × subscriptions × limitador_namespaces

organization_id and cost_center are not independent multipliers — each subscription belongs to exactly one org and one cost center, so they are determined by the subscription value.

The primary cardinality drivers are:

• user — grows with the active user base; this is the largest dimension
• model — grows with deployed models (see Model Cardinality)
• subscription — grows with MaaSSubscription CRs

At small to moderate scale (tens of models, tens of subscriptions), the non-user dimensions alone are manageable. Cardinality becomes a concern primarily when per-user labels are enabled and the active user base is large.

These projections are not load-tested

The cardinality impact on Prometheus ingestion, Limitador memory, and scrape duration has not been validated with load testing. Actual thresholds depend on hardware, retention settings, and usage patterns. Monitor cardinality growth in your deployment before enabling high-cardinality labels like user.

Key Consideration: Active Users

Only users who made at least one request during the retention period create series. Inactive users do not contribute to cardinality. The number of unique user × model × subscription combinations that actually occur in practice depends on usage patterns and is typically much smaller than the theoretical maximum.

Impact on Limitador

Cardinality does not only affect Prometheus. Each unique label combination creates a counter inside Limitador itself:

Impact Area	Description
Memory	Limitador holds all active counters in memory (or Redis). High cardinality increases memory pressure on the Limitador pod.
/metrics endpoint	Prometheus scrapes Limitador's /metrics endpoint every 30s. Each series adds ~100 bytes to the response. At 500k series, scrape responses reach ~50 MB, potentially exceeding the 30s scrape interval and causing missed scrapes.
Counter lookup	Each incoming request requires a counter lookup by label combination. Larger counter sets may increase lookup latency, though this is typically sub-millisecond with hash-based storage.
Redis (if used)	With Redis-backed persistence, each counter is a Redis key. High cardinality increases Redis memory usage and network traffic for counter sync.

Limitador impact is not well-characterized

The effect of high label cardinality on Limitador performance (memory, lookup latency, scrape duration) has not been load-tested. This is a gap that should be addressed before recommending per-user labels at scale.

Dangerous Combinations

High-Risk Configurations

Enabling all dimensions at scale can overwhelm both Prometheus and Limitador:

• All labels enabled + user at scale: Enabling user alongside model and subscription creates a series for each unique combination a user touches. At large user counts this can grow rapidly.
• Per-user histograms: Never add user to Istio latency histograms. Each histogram has 23 buckets by default — with 10k users this creates 230k series from one metric alone.
• Short scrape intervals: Reducing scrape interval below 30s doubles ingestion load without proportional observability benefit. At high cardinality, scrape duration may already exceed the interval.

Realistic Mitigation Options

Option 1: Drop High-Cardinality Labels by Default (Recommended)

Disable per-user and other high-cardinality labels by default. Operators opt in to specific labels based on their scale and monitoring capacity.

The current default already disables user (captureUser: false). To further reduce cardinality, operators can also disable organization_id and model via the MaasTenantConfig CR:

spec:
  telemetry:
    metrics:
      captureUser: false          # default — disabled
      captureOrganization: false  # default is true; disable to reduce cardinality
      captureModelUsage: false    # default is true; disable to reduce cardinality
      # subscription and cost_center are always enabled

Operators who need per-user billing data enable the user label explicitly, accepting the cardinality cost.

Option 2: Move High-Cardinality Data to Logs

Instead of encoding user, organization_id, and cost_center as metric labels (which create time series), emit them as structured log entries. Logs can be aggregated and queried via log analytics (e.g., OpenShift Logging, Loki) without creating Prometheus cardinality pressure.

Metrics remain for aggregated operational visibility (per-model, per-tier). Logs handle per-user attribution and billing queries.

Option 3: Enforce Hard Limits on Label Values

Add controller-side validation that caps the number of unique values for each label dimension. For example, reject TelemetryPolicy configurations that would produce more than a configurable cardinality budget. This makes the worst-case series count predictable.

Recording Rules (Query Optimization Only)

Pre-aggregate metrics for dashboards so they don't scan all series on every panel load:

groups:
- name: maas-aggregated
  interval: 1m
  rules:
  - record: maas:authorized_hits:rate5m_by_model
    expr: sum by (model) (rate(authorized_hits[5m]))
  - record: maas:authorized_calls:rate5m_by_subscription
    expr: sum by (subscription) (rate(authorized_calls[5m]))

Recording rules are not a cardinality fix

Recording rules improve query performance but do not reduce storage, ingestion, or Limitador memory pressure. They are a useful complement to other mitigations, not a standalone solution.

Cardinality Alert

Add this alert to detect cardinality growth before it causes problems:

- alert: MaaSHighMetricCardinality
  expr: count({__name__=~"authorized_hits|authorized_calls|limited_calls"}) > 100000
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "MaaS Limitador metrics exceed 100k active series"
    description: "Review telemetry label cardinality. See docs/advanced-administration/telemetry-defaults-and-cardinality.md"

Open Questions

The following questions should be answered with real customer data before finalizing telemetry defaults:

1. What is the realistic effective multiplier per user? How many unique model/subscription combinations does a typical user touch during a retention period?
2. What is the Limitador performance impact? At what counter count does Limitador's /metrics scrape exceed the scrape interval? Does counter lookup latency degrade?
3. What model counts are realistic? Do production customers deploy 10 models or 200? This is the biggest variable after user.
4. Which labels are needed for billing vs operational monitoring? Can we separate the billing pipeline (logs or dedicated store) from operational metrics (Prometheus)?

Prometheus Capacity Reference

These are rough reference points, not validated limits. Actual capacity depends on retention, query load, and hardware.

Configuration	Approximate Active Series
OpenShift User Workload Monitoring (default)	~1M
OpenShift User Workload Monitoring (tuned, 16-32GB)	~5M
Standalone Prometheus (64GB, NVMe)	~10M
Mimir / Thanos / VictoriaMetrics	100M+

See Also

• Observability Guide — full metrics reference, dashboards, and installation