Leaving prometheus-operator without breaking anything: a calm migration to Victoria Metrics
Cloud native observability with the VictoriaMetrics stack, part 1 of 3
Most teams i meet do not wake up one morning wanting to migrate their monitoring. They wake up to a Prometheus pod that was OOMKilled during the night or to a storage bill that grew faster than the cluster or to a capacity review where someone asks why the monitoring stack needs more memory than the application it monitors.
At that point VictoriaMetrics usually comes up in the conversation. And the very next question is always the same:
“OK, but we have two hundred ServiceMonitors, forty Prometheus Rules and dashboards everywhere. Do we have to rewrite all of that?”
Short answer: no. Longer answer: this article.
I will walk you through how the VictoriaMetrics operator deals with your existing prometheus-operator objects, how ilike to run the migration in phases and the traps I have seen in my previous experiences catch people. No big bang, no weekend of heroics.
Versions used while writing: VictoriaMetrics operator v0.75.2 and VictoriaMetrics v1.155.0 (October 2026). Flags and defaults can change between releases, so check the changelog of the versions you run.
What you already have
If you run kube-prometheus-stack or a similar setup, your monitoring configuration lives in Kubernetes objects defined by prometheus-operator:
ServiceMonitorandPodMonitordescribe what to scrape.Probedescribes blackbox checks.ScrapeConfigcovers everything else (static targets, other service discovery mechanisms).PrometheusRuleholds recording and alerting rules.AlertmanagerConfigholds routing and receivers.
These objects are the real value of your setup. They were written by many teams over time and they encode a lot of knowledge. The Prometheus server itself is just the engine that reads them.
So the goal of a good migration is simple: keep the objects, change the engine.
How the VictoriaMetrics operator reads them
The VictoriaMetrics operator was designed with this in mind. It has its own CRDs, but it also watches the prometheus-operator ones and converts them.
By default the operator creates the matching VictoriaMetrics object in the same namespace as the original and keeps it in sync, labels included, when the original changes. Your teams keep writing ServiceMonitor objects in their Helm charts and nothing changes for them.
Each conversion can be switched off individually with the operator flag -controller.disableReconcileFor (values ServiceMonitor, PodMonitor, PrometheusRule, Probe, ScrapeConfig, AlertmanagerConfig), or entirely with operator.disable_prometheus_converter: true in the Helm chart.
Three details are worth knowing before you start, because they explain most of the surprises i have seen.
1. The Prometheus CRDs are not shipped with the VictoriaMetrics operator. The converter needs the monitoring.coreos.com CRDs to be installed. If you plan to remove kube-prometheus-stack at the end, check how your version of the chart installed those CRDs and make sure they survive the uninstall. Otherwise the source objects disappear, and with them the input of the converter.
2. Deletion is not synchronized by default. If you delete a ServiceMonitor, the generated VMServiceScrape stays. The documentation presents this as a safety feature: you can remove prometheus-operator objects, or run both operators side by side, without losing anything. The price is orphans. If you want deletions to follow, enable owner references on converted objects:
# values.yaml of the victoria-metrics-operator Helm chart
operator:
# converted VM objects are deleted by Kubernetes when the Prometheus object is deleted
enable_converter_ownership: true(The equivalent operator environment variable is VM_ENABLEDPROMETHEUSCONVERTEROWNERREFERENCES=true.)
3. Metadata sync has a strategy. By default the prefer-prometheus strategy applies: labels and annotations come from the Prometheus object and values you set manually on the VM object are dropped at the next update. You can change this per object with the operator.victoriametrics.com/merge-meta-strategy annotation (prefer-victoriametrics, merge-victoriametrics-priority, merge-prometheus-priority). If a specific VM object should stop following its source altogether, set operator.victoriametrics.com/ignore-prometheus-updates: enabled on it. You can also exclude label or annotation prefixes from the sync with VM_FILTERPROMETHEUSCONVERTERLABELPREFIXES and VM_FILTERPROMETHEUSCONVERTERANNOTATIONPREFIXES (for example helm.sh,argoproj.io).
If you use Argo CD, there is one more switch you will want. The operator documentation explicitly lists out-of-sync issues with converted objects in Argo CD deployments, and provides an option that adds Argo CD compare and sync options to them:
# values.yaml of the victoria-metrics-operator Helm chart
operator:
prometheus_converter_add_argocd_ignore_annotations: true(Environment variable: VM_PROMETHEUSCONVERTERADDARGOCDIGNOREANNOTATIONS=true.)
The migration, in five calm phases
Here is the sequence I recommend. It is boring on purpose. Boring migrations are the ones that work.
Phase 0: inventory
Before installing anything, count what you have:
for kind in servicemonitors podmonitors probes scrapeconfigs prometheusrules alertmanagerconfigs; do
printf "%-22s %s\n" "$kind" "$(kubectl get $kind -A --no-headers 2>/dev/null | wc -l)"
doneAlso note the current number of active series in Prometheus (prometheus_tsdb_head_series) and your retention. These two numbers will help you size VictoriaMetrics and, later, compare both systems.
Phase 1: shadow run
Install the VictoriaMetrics operator and a VMSingle (single node is enough for most clusters to start with) next to Prometheus. Do not touch Prometheus.
Get samuel desseaux’s stories in your inbox
Join Medium for free to get updates from this writer.
A minimal VMAgent that picks up every converted object looks like this:
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
name: shadow
namespace: monitoring
spec:
# with empty selectors, select every scrape object in every namespace
selectAllByDefault: true
remoteWrite:
# the operator names the Service of a VMSingle "vmsingle-<name>"
- url: "http://vmsingle-main.monitoring.svc:8428/api/v1/write"Now both systems scrape the same targets. Yes, this doubles the scrape load on your exporters for a while. On most clusters it is not a problem, but check it on large clusters or on fragile exporters.
Phase 2: compare
This is the phase most people skip and the one i care about most. For a week or two, compare:
- Series count. Should be very close. A large gap usually means a relabeling difference, a target that one side cannot reach or a selector difference (see the traps below).
- A handful of key queries. Pick your ten most important dashboard panels and run the same query on both sides. The VictoriaMetrics documentation describes MetricsQL as backwards-compatible with PromQL, so the queries run as they are. Expect some differences on
rate()andincrease(), and they are intentional: MetricsQL does not extrapolate these functions and it takes into account the last sample before the lookbehind window. On slow counters,increase()in VictoriaMetrics returns integers where Prometheus returns fractional values. Know this before someone opens a ticket saying "the numbers are wrong". - Alerts. Run a
VMAlertwith your convertedVMRuleobjects, but send notifications to a test receiver. Compare what fires on both sides.
A minimal VMAlert for this phase:
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlert
metadata:
name: shadow
namespace: monitoring
spec:
selectAllByDefault: true
evaluationInterval: 30s
datasource:
url: "http://vmsingle-main.monitoring.svc:8428"
notifier:
# a VMAlertmanager named "test" is exposed as "vmalertmanager-test" on port 9093
url: "http://vmalertmanager-test.monitoring.svc:9093"If you want history in VictoriaMetrics from day one, vmctl can import data from a Prometheus snapshot (vmctl prometheus --prom-snapshot=...). I only do it when the team really needs long history right away. Very often a few weeks of parallel run is enough.
Phase 3: switch the readers
Point Grafana to VictoriaMetrics. A Prometheus datasource works, only the URL changes. Then switch real alert routing to VMAlert and VMAlertmanager. Keep Prometheus running for a few more days as a safety net. If something looks wrong, you switch the URL back.
Phase 4: decommission, then decide about CRDs
Remove the Prometheus server and Alertmanager, but keep the monitoring.coreos.com CRDs as long as you rely on the converter.
Then comes a question you do not have to answer right away: do you keep writing ServiceMonitor objects forever, or do you move to native VMServiceScrape? My usual advice:
- Keep the Prometheus CRDs for everything that comes from third-party Helm charts. Many charts ship a
ServiceMonitorand you do not want to patch them. - Use native VM CRDs for new internal work, especially when you need options that only exist on the VM side.
Both can live side by side. There is no deadline.
Traps i have seen
Doubling everything by accident. If you also install the victoria-metrics-k8s-stack chart with its own default scrape objects, you can end up scraping the kubelet or kube-state-metrics twice: once through the converted ServiceMonitors and once through the native objects of the chart. Check for duplicate jobs in the VMAgent targets page.
Selectors that do not match. kube-prometheus-stack configures Prometheus, by default, to select only the objects labelled with its Helm release name. With selectAllByDefault: true and no selectors, VMAgent picks up everything, including objects your old Prometheus ignored on purpose. Sometimes that is a nice surprise, sometimes it is a cardinality explosion. Look at the series count after phase 1, and reproduce the old selectors on VMAgent if you need the same perimeter.
Forgotten consumers. Grafana is the obvious reader. But there may be others: an autoscaler using Prometheus metrics, a cost tool, a script in CI. Search for the Prometheus service URL in your repositories before phase 4.
Argo CD fighting the operator. Already mentioned, but enable the option early rather than after the first unexplained diff.
What you get at the end
The point of all this is not to add a new logo to your stack. Teams usually move for lower memory use, better compression on disk and a clear path to a cluster topology when one node is not enough anymore. But measure those gains on your own workload: phase 2 gives you real numbers for your capacity review, instead of benchmarks run by someone else.
And your teams did not have to change a single ServiceMonitor.
In part 2 we will look at the other door into the VictoriaMetrics stack: OpenTelemetry. We will build an OpenTelemetry Collector gateway that sends metrics, logs and traces to VictoriaMetrics, VictoriaLogs and VictoriaTraces, and we will talk about the topic that always comes back: cardinality.
I am Samuel Desseaux, I am Lead Architect observability and my job is to design observability platforms for regulated and industrial environments . If you ran this kind of migration and hit a trap i did not list, tell me in the comments: I will add it.
References
- VictoriaMetrics operator, Prometheus integration and conversion: https://docs.victoriametrics.com/operator/integrations/prometheus/
- MetricsQL, differences with PromQL: https://docs.victoriametrics.com/victoriametrics/metricsql/
- vmctl, migrating data from Prometheus: https://docs.victoriametrics.com/victoriametrics/vmctl/
