Skip to content
CVE scanning

CVE scanning

We scan for known vulnerabilities (CVEs) with Trivy at three moments:

WhenToolScopeBlocks?
Pull requestTrivy CLI in security-scan.ymlThe image built from the PRNo, informational table
ReleaseTrivy in release-create.ymlThe final image, then the published digestYes, for hardened releases
RunningTrivy Operator in the clusterEvery workload image in every namespaceNo, reports and metrics

The CI scans are covered in Hardened releases and signing. This page is about the third one: a release that was clean on the day it shipped can become vulnerable when a new CVE is published, so we keep scanning what actually runs.

GitOps promotion pull requests also carry a Container image security (Trivy) table for the images being promoted; see Promotion.

Trivy Operator in the cluster

Trivy Operator runs in the trivy namespace of the production cluster, installed by the Argo CD Application trivy-production:

SettingValue
Chartaquasecurity/trivy-operator 0.27.0
Valuesuug-ai/gitops: environments/production/trivy/values.yaml
Applicationenvironments/production-bijlokehof/argocd-application-trivy.yaml (overrides scanJobsConcurrentLimit: 5, scanJobTTL: 1h, scanner memory request 512Mi)
ScannerTrivy 0.60.0 in standalone mode, databases from mirror.gcr.io
NamespacesAll (targetNamespaces: "")
SeveritiesUNKNOWN,LOW,MEDIUM,HIGH,CRITICAL, unfixed included
Report lifetimescannerReportTTL: 24h: each image is rescanned at least daily against a fresh database
RevisionsCurrent ReplicaSets only, so old rollouts don’t add noise

The operator watches workloads. When a Pod starts with an image it hasn’t scanned, it starts a scan job (at most five at a time); scan jobs run without privileges, with all capabilities dropped and a read-only root filesystem.

Besides vulnerabilities, the same operator produces:

ReportWhat it finds
VulnerabilityReportCVEs in OS packages and application dependencies, per container image
ExposedSecretReportCredentials baked into an image
ConfigAuditReportRisky workload settings: root user, missing limits, privileged containers
RbacAssessmentReportOverly broad Roles and ClusterRoles
ClusterComplianceReportCIS Kubernetes 1.23, NSA hardening, Pod Security Standards baseline and restricted, every six hours

Read the reports

Reports are Kubernetes resources next to the workload they describe:

# Vulnerability summary for the Hub images
kubectl -n kerberos-hub get vulnerabilityreports \
  -o custom-columns='IMAGE:.report.artifact.repository,TAG:.report.artifact.tag,CRITICAL:.report.summary.criticalCount,HIGH:.report.summary.highCount,SCANNED:.report.updateTimestamp'

# Every CVE in one report, with its fix version
kubectl -n kerberos-hub get vulnerabilityreport <name> -o json \
  | jq -r '.report.vulnerabilities[] | [.severity, .vulnerabilityID, .resource, .installedVersion, .fixedVersion] | @tsv'

Metrics and dashboards

The operator exposes Prometheus metrics, scraped by a ServiceMonitor every 30 seconds into the cluster’s kube-prometheus-stack (Application grafana-prometheus-production). Per-CVE and per-image metrics are on (metricsVulnIdEnabled, metricsImageInfo), as are metrics for secrets, config audits, RBAC, infrastructure and compliance.

The main metric is trivy_image_vulnerabilities: one series per container image and severity, labelled with the workload’s namespace, resource_kind, resource_name, image_repository and image_tag.

Grafana provisions the Trivy Operator - Vulnerabilities dashboard (grafana.com dashboard 16337) from the grafana.dashboards values in GitOps. It shows counts by severity and namespace and a table of CVEs per image.

Useful queries:

# Critical and high findings per namespace
sum by (namespace, severity) (trivy_image_vulnerabilities{severity=~"Critical|High"})

# Our own images with any critical finding
sum by (namespace, image_repository, image_tag) (
  trivy_image_vulnerabilities{severity="Critical", image_repository=~"uug-ai/.*"}
) > 0

# Images with an exposed secret
sum by (namespace, image_repository) (trivy_image_exposedsecrets) > 0

Alerting

Alerts are Grafana-managed, provisioned from grafana.alerting in environments/production-bijlokehof/grafana-prometheus-production/values.yaml:

FileContent
contactpoints.yamlslack, email and slack-and-email contact points
policies.yamlDefault route to slack, grouped by folder and alert name, repeated every 4 hours
rules.yamlAlert rules, each with its own receiver

The Prometheus Alertmanager that ships with kube-prometheus-stack routes every alert to a null receiver and notifies nobody. Add rules to Grafana, not as PrometheusRule resources.

As of 18 November 2026 the provisioned rules cover RabbitMQ queue backlog and blackbox endpoint probes only. No rule alerts on vulnerabilities or Kyverno policy results yet. New CVEs show up on the dashboard and in the reports, but nobody is notified.

To add one, put a rule group in rules.yaml that follows the existing rabbitmq and blackbox groups (a Prometheus query A, a threshold expression C, and notification_settings.receiver). Scope it to our own images, or third-party charts will keep it firing; at the time of writing the cluster reports several hundred critical findings, none of them in UUG.AI images.

Responding to a finding

  1. Find the image and CVE in the dashboard or the VulnerabilityReport.
  2. Check fixedVersion. If there is a fix, upgrade the Go module or bump the digest-pinned base image in the component’s repository.
  3. Cut a hardened release. Its gate fails while the CVE is still present, so a green release confirms the fix.
  4. Promote through GitOps. After the rollout, the operator scans the new image and the finding drops off the dashboard.

For third-party images (ingress, storage, monitoring), upgrade the chart in GitOps rather than patching the image.