A Kubernetes cluster is flat by default. Any pod can open a connection to any other pod, in any namespace, on any port. The database behind one application is one TCP connect away from every other workload on the cluster; so is the admin port of the cache sitting next to it. That works when everything is trustworthy, and it is exactly the property an attacker leans on when something isn’t: one compromised web frontend becomes a foothold on every database, every internal API, and every metrics endpoint in the cluster.
Namespace isolation turns the default around. Each namespace becomes its own trust domain: its pods talk to each other, to the platform services they genuinely need (DNS, the API server, the ingress proxy, the metrics scraper), and to nothing else unless a rule says so. A compromised pod then sees the namespace it lives in — not the cluster. The cost is that every legitimate cross-namespace connection has to be stated. Most of them are easy to state. The ones that aren’t are the subject of this post.
Every operator has the same networking problem. CloudNativePG has to reach its Postgres pods for failover, switchover and backups. The Dragonfly operator dials its instances to issue REPLICAOF. Strimzi talks to its brokers, the OpenSearch operator to its nodes. The workloads an operator manages are its operands, and the traffic between the two is infrastructure, not an exception — the moment you put namespaces under default-deny, you have to say so somewhere.
The obvious ways to say so are both wrong, in opposite directions: one rule with far too much reach, or a rule per namespace that somebody has to remember. We wanted the precision of the second without maintaining it by hand. It turned out the operator had already told us everything we needed: in its custom resource (CR).
How we isolate a namespace
Our clusters run Cilium, and isolation is a label. A namespace with network-policy.tgbyte.io/isolated: "true" is picked up by a handful of cluster-wide base rules (DNS, the API server, health checks, metrics scraping, ingress from the proxy) and by one generate rule in Kyverno, a policy engine that can create and modify resources in reaction to others. That rule creates the policy a cluster-wide rule cannot express:
generate:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
name: allow-intra-namespace
namespace: "{{request.object.metadata.name}}"
synchronize: true
data:
spec:
endpointSelector: {}
ingress:
- fromEndpoints: [{}]
egress:
- toEndpoints: [{}]Empty selectors in a namespaced Cilium policy mean “this namespace” — exactly the bit a CiliumClusterwideNetworkPolicy can’t say, because its selectors span every namespace. So the shape has to exist once per namespace, but nobody has to write it: Kyverno stamps it out from the label and removes it again when the label goes. Protecting a namespace is one line in the cluster’s Argo CD values (the Git-managed configuration Argo CD deploys the cluster from), and the namespace is default-deny in both directions with its own pods still talking to each other.
That works beautifully for applications. It is the operators that don’t fit, because they live in one namespace and need to reach pods in many.
The rule we had, and why it was too wide
Our first answer was a single cluster-wide rule: any isolated namespace admits cnpg-system and dragonfly-operator-system. Honest and simple. It was also, when we counted, every operator admitted into every isolated namespace — while the operands lived in a small fraction of them. Most of that reach was granted to namespaces that had never seen a database.
The precise alternative is a rule per operand namespace. We have that pattern for genuinely app-specific clients, and it is fine at a handful. For operators it means every new Postgres cluster ships with a network policy someone has to remember, and the failure mode is silent: the operator can’t reach the new instance, and you find out when the first failover doesn’t happen.
The insight: the CR is the fact
What makes a namespace “one that CNPG needs to reach”? The presence of a postgresql.cnpg.io/v1/Cluster in it — the operator’s own custom resource, not a label somebody set or a naming convention. The same holds for every operator: a Dragonfly, a ClickHouseInstallation, a Kafka, an OpenSearchCluster. The CR is created before the pods it produces, so it is the earliest possible signal.
So we let Kyverno record that fact where Cilium can see it: as a label on the namespace. We call such a label a fact throughout: something true about the namespace that Kyverno derived, rather than something a human declared.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: label-namespaces-with-cnpg
spec:
failurePolicy: Ignore
background: true
mutateExistingOnPolicyUpdate: true
rules:
- name: mark-namespace
match:
any:
- resources:
kinds:
- postgresql.cnpg.io/v1/Cluster
mutate:
targets:
- apiVersion: v1
kind: Namespace
name: "{{ request.namespace }}"
patchStrategicMerge:
metadata:
labels:
network-policy.tgbyte.io/cnpg: "true"Three lines matter. background: true and mutateExistingOnPolicyUpdate: true make the policy apply to clusters that already exist, not only to ones created after it (Kyverno re-checks existing resources in a background sweep) — so rolling it out labels every database namespace on day one. failurePolicy: Ignore keeps a Kyverno outage from blocking anyone’s database. And mutate.targets is Kyverno’s “mutate existing” mode: the trigger is the Cluster, but the object that changes is the Namespace.
Note what the policy does not contain: any list of namespaces. It has no idea where databases run. It doesn’t need to.
Note also what it does not do: remove the label when the last Cluster leaves the namespace. That is a reconciliation of its own, and we come back to it below.
The Cilium side: one rule per operator
With the fact on the namespace, the cluster-wide rule can require it. Cilium exposes namespace labels to policy selectors under k8s:io.cilium.k8s.namespace.labels.<label>, so the ingress rule becomes:
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: base-operator-cnpg
spec:
description: cnpg-system may reach the isolated namespaces that hold its operands.
endpointSelector:
matchLabels:
k8s:io.cilium.k8s.namespace.labels.network-policy.tgbyte.io/isolated: "true"
k8s:io.cilium.k8s.namespace.labels.network-policy.tgbyte.io/cnpg: "true"
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: cnpg-systemOne rule per operator, admitting each exactly into the namespaces that hold its CR — today, and after the next database is created, without anyone touching a policy. On a cluster where an operator isn’t installed, the selector simply matches nothing.
Be precise about what this grants, though: it is namespace-level access. Every pod in cnpg-system may reach every pod in a labelled namespace, and as written, on every port — a Cilium rule without toPorts allows all L4 ports, and the selectors above name namespaces, not pods. Tightening it is the same rule with more selectors, and the two kinds of selector cost very different amounts.
Ports are the cheap half. Every operator documents what it needs: CNPG’s networking guide names 8000 and 5432, Elastic’s ECK guide names 9200, Strimzi’s generated broker policy admits the operator on 9090, 9091, and the Kafka Agent port 8443. So our production rules carry a toPorts list per operator, taken from those documents and checked against measured traffic.
Pods are the expensive half. The operator’s own pod labels on fromEndpoints would admit only the controller, not everything else that runs in its namespace, and the operand’s pod labels on endpointSelector would narrow the other side the same way. Those we have left at namespace level, because each label is one more thing that has to still match after the operator’s next upgrade.
The operator namespace itself gets the mirror image — an egress rule selecting cnpg-system and allowing toEndpoints with the same namespace label — so the operators can be isolated too; one of those per operator we run.
Four details that make it hold
Labels are set, never unset. A mutate-existing rule fires on the admission events it matches — here the creation and update of a Cluster — and sets the label. Nothing in it reacts to the last Cluster leaving a namespace, and the background sweep only re-applies the mutation, it never reverts one.
Cleaning up properly is a separate reconciliation: on a delete, decide whether any matching CR is still present in that namespace, and only then drop the label — a rule that removed it on every CR deletion would break the moment two databases share a namespace.
We have not automated that step. A stale label admits the operator into a namespace that no longer holds an operand, which is the reach it had before this design, and it goes away when the namespace is retired.
Allow rules must stay allow rules. In Cilium, a policy that selects a pod and lists egress destinations does two things: it allows those destinations, and it switches egress default-deny on for that pod. For a cluster-wide rule that is meant purely as an allowance — the operator egress twin above, which also reaches clusters where the operator namespace is not isolated — the second effect is unwanted. Cilium has a flag for exactly this, and every rule of that kind carries it:
spec:
enableDefaultDeny:
egress: falseDefault-deny for an operator namespace comes from its isolation label and the generated intra-namespace policy, never from a cluster-wide allowance.
Facts follow the CRDs. Kyverno rejects a policy that matches a kind the cluster has never heard of. Our policies live in a base shared by every cluster plus a per-cluster overlay, and the facts are split accordingly: facts for operators that run on every cluster live in the shared base, a fact for a cluster-specific operator lives in that cluster’s overlay. A server-side dry run against each cluster — the API server validates the policy without storing it — is part of the review.
Kyverno needs two kinds of permission. Kyverno’s controllers assemble their RBAC from every ClusterRole that carries an rbac.kyverno.io/aggregate-to-<controller> label, and a mutate-existing rule needs two such grants. They fail differently when missing:
- Read access on the trigger, the operator’s CR:
get,listandwatchfor the admission, background and reports controllers. Without it the policy has, in our experience, been accepted and left inert: the webhook warns that a controller “requires permissions get,list,watch” for the kind, and the background sweep never sees a trigger. - Write access on the target, the
Namespace:updateorpatchfor the background controller that performs the mutation. Kyverno checks this one when the policy is installed, so a missing grant fails the policy’s own admission — loud, at least.
Either way the role has to exist before the policy that needs it (in Argo CD terms: one sync wave earlier — Argo CD applies resources in numbered waves, one after the other), and the server-side dry run surfaces both.
Measure, then write
None of the rules were written from documentation alone. Before each one we listed the actual flows with Hubble, Cilium’s flow observability tool — which namespaces the operator connected to, on which ports, over days — and wrote the rule to match. That is how we know the Dragonfly operator only ever uses its instances’ admin port, and that CNPG polls the instance manager on 8000.
Measurement has a blind spot, though: a rare path — Strimzi’s rolling update through the Kafka Agent port, a CNPG switchover — never shows up in a quiet window, and a rule built from traffic alone would deny it on the day it matters. So the port lists come from the operator’s documentation first, and the measurement confirms the documentation rather than replacing it.
After a rule lands, the same tooling closes the loop: no drops involving the operator, and the rare paths exercised once on purpose.
The same trick, elsewhere
Once you see a namespace label as a derived fact, operators are just the clearest case. We derive the same way:
scrapedfrom aServiceMonitororPodMonitorin the namespace, so Prometheus is admitted only where it has something to scrape (with one assumption, below);ingressfrom anIngressorIngressRoute, so the proxy reaches only namespaces that publish something;backupsfrom a volume claim annotated for backup,load-balancerfrom aLoadBalancerservice,host-networkfrom a host-networked pod.
The scraped fact assumes that monitors select targets in their own namespace. Both kinds can reach into other namespaces through spec.namespaceSelector; a monitor that does would need the derivation to label the selected namespaces instead.
Each one removes a list somebody would otherwise maintain. Each one also shares the cleanup caveat from above: the label is set when the object appears and stays when it goes, until a reconciliation that checks the namespace for remaining objects takes it away.
Takeaways
- The operator already declared its intent. Its CR says where it will work. Derive network facts from that instead of restating them by hand.
- Kyverno “mutate existing” turns a CR into a namespace label; Cilium turns the label into namespace-level policy. Neither side keeps a list. Pod- and port-level precision is more selectors on the same rule; label removal is a reconciliation of its own.
- A cluster-wide allow rule without
enableDefaultDeny: falseis also a deny rule. Pure allowances carry the flag; default-deny comes from the isolation label alone. - Documentation for the ports, measurement for the proof. The operator’s docs name the rare paths a quiet window never shows; Hubble confirms the list and catches what the docs left out.
Every database, cache and Kafka namespace now admits its operator without a single per-namespace policy in git, and the next one will too — the day someone creates its CR.