CVE scanning
We scan for known vulnerabilities (CVEs) with Trivy at three moments:
| When | Tool | Scope | Blocks? |
|---|---|---|---|
| Pull request | Trivy CLI in security-scan.yml | The image built from the PR | No, informational table |
| Release | Trivy in release-create.yml | The final image, then the published digest | Yes, for hardened releases |
| Running | Trivy Operator in the cluster | Every workload image in every namespace | No, reports and metrics |
The CI scans are covered in Hardened releases and signing. This page is about the third one: a release that was clean on the day it shipped can become vulnerable when a new CVE is published, so we keep scanning what actually runs.
GitOps promotion pull requests also carry a Container image security (Trivy) table for the images being promoted; see Promotion.
Trivy Operator in the cluster
Trivy Operator runs in the trivy namespace of the production cluster,
installed by the Argo CD Application trivy-production:
| Setting | Value |
|---|---|
| Chart | aquasecurity/trivy-operator 0.27.0 |
| Values | uug-ai/gitops: environments/production/trivy/values.yaml |
| Application | environments/production-bijlokehof/argocd-application-trivy.yaml (overrides scanJobsConcurrentLimit: 5, scanJobTTL: 1h, scanner memory request 512Mi) |
| Scanner | Trivy 0.60.0 in standalone mode, databases from mirror.gcr.io |
| Namespaces | All (targetNamespaces: "") |
| Severities | UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL, unfixed included |
| Report lifetime | scannerReportTTL: 24h: each image is rescanned at least daily against a fresh database |
| Revisions | Current ReplicaSets only, so old rollouts don’t add noise |
The operator watches workloads. When a Pod starts with an image it hasn’t scanned, it starts a scan job (at most five at a time); scan jobs run without privileges, with all capabilities dropped and a read-only root filesystem.
Besides vulnerabilities, the same operator produces:
| Report | What it finds |
|---|---|
VulnerabilityReport | CVEs in OS packages and application dependencies, per container image |
ExposedSecretReport | Credentials baked into an image |
ConfigAuditReport | Risky workload settings: root user, missing limits, privileged containers |
RbacAssessmentReport | Overly broad Roles and ClusterRoles |
ClusterComplianceReport | CIS Kubernetes 1.23, NSA hardening, Pod Security Standards baseline and restricted, every six hours |
Read the reports
Reports are Kubernetes resources next to the workload they describe:
# Vulnerability summary for the Hub images
kubectl -n kerberos-hub get vulnerabilityreports \
-o custom-columns='IMAGE:.report.artifact.repository,TAG:.report.artifact.tag,CRITICAL:.report.summary.criticalCount,HIGH:.report.summary.highCount,SCANNED:.report.updateTimestamp'
# Every CVE in one report, with its fix version
kubectl -n kerberos-hub get vulnerabilityreport <name> -o json \
| jq -r '.report.vulnerabilities[] | [.severity, .vulnerabilityID, .resource, .installedVersion, .fixedVersion] | @tsv'Metrics and dashboards
The operator exposes Prometheus metrics, scraped by a ServiceMonitor every
30 seconds into the cluster’s kube-prometheus-stack (Application
grafana-prometheus-production). Per-CVE and per-image metrics are on
(metricsVulnIdEnabled, metricsImageInfo), as are metrics for secrets,
config audits, RBAC, infrastructure and compliance.
The main metric is trivy_image_vulnerabilities: one series per container
image and severity, labelled with the workload’s namespace,
resource_kind, resource_name, image_repository and image_tag.
Grafana provisions the Trivy Operator - Vulnerabilities dashboard
(grafana.com dashboard 16337) from the grafana.dashboards values in GitOps.
It shows counts by severity and namespace and a table of CVEs per image.
Useful queries:
# Critical and high findings per namespace
sum by (namespace, severity) (trivy_image_vulnerabilities{severity=~"Critical|High"})
# Our own images with any critical finding
sum by (namespace, image_repository, image_tag) (
trivy_image_vulnerabilities{severity="Critical", image_repository=~"uug-ai/.*"}
) > 0
# Images with an exposed secret
sum by (namespace, image_repository) (trivy_image_exposedsecrets) > 0Alerting
Alerts are Grafana-managed, provisioned from grafana.alerting in
environments/production-bijlokehof/grafana-prometheus-production/values.yaml:
| File | Content |
|---|---|
contactpoints.yaml | slack, email and slack-and-email contact points |
policies.yaml | Default route to slack, grouped by folder and alert name, repeated every 4 hours |
rules.yaml | Alert rules, each with its own receiver |
The Prometheus Alertmanager that ships with kube-prometheus-stack routes every
alert to a null receiver and notifies nobody. Add rules to Grafana, not as
PrometheusRule resources.
To add one, put a rule group in rules.yaml that follows the existing
rabbitmq and blackbox groups (a Prometheus query A, a threshold
expression C, and notification_settings.receiver). Scope it to our own
images, or third-party charts will keep it firing; at the time of writing the
cluster reports several hundred critical findings, none of them in UUG.AI
images.
Responding to a finding
- Find the image and CVE in the dashboard or the
VulnerabilityReport. - Check
fixedVersion. If there is a fix, upgrade the Go module or bump the digest-pinned base image in the component’s repository. - Cut a hardened release. Its gate fails while the CVE is still present, so a green release confirms the fix.
- Promote through GitOps. After the rollout, the operator scans the new image and the finding drops off the dashboard.
For third-party images (ingress, storage, monitoring), upgrade the chart in GitOps rather than patching the image.