Part 1 of a series.

Where does GitOps go if AI manages all our configuration? Can we finally update a fleet of Helm charts without at least one platform break? When we run hundreds of GitOps reconciliations at once, does AI tell us what on earth is going on?

Don’t know, maybe, can you repeat the question?

This piece is my contribution to that question. My answer is: GitOps never went away. Being AI-enabled does change some things.

In this piece, I shall propose that we extend the GitOps principles to accommodate configuration that is generated and processed at machine speed. For a number of reasons, I think this could be called Generative GitOps. More on that later.

Where are we today?

At ConfigHub we are helping customers who are using Kubernetes and GitOps and adding AI.

Many people who grew up with cloud native are turning to AI. Can automation remove bottlenecks and make platform engineering profitable? How will AI impact our CVE strategy?

In this context “adding AI” can mean several things

Adapt my application platform to include AI in all stages of the software delivery lifecycle.

Make changes faster eg. to Helm or YAML. I call this the ‘sandblasting’ approach.

Deploy new GPU-based stacks for training and inference.

Add an ‘on demand cloud’ of dev-test sandboxes (single-tenant agents)

Everyone wants to build more productively, test and deploy faster, and secure and operate apps automatically. But above all they want to be SAFER. In my last blog post I wrote about how we are all now living on Mythos time and how much this could cost if we get it wrong (yowch!).

When it comes to safety, GitOps automation is a trusted operating model for Kubernetes applications and fleets. This is partly because it converges deterministically. Now AI is here and with it a non-deterministic IT model. Could one small change do real damage? Let’s look at what happens with an example. To increase the “surprise factor” let’s add Helm to the story.

A story that shows the problem when AI meets GitOps

A service keeps running out of memory. An AI agent is told to fix it. It makes the obvious one-line change: in the Helm chart, it raises one value, memory from 512Mi to 1Gi. The agent commits the change, and GitOps does exactly what it should. The change is reconciled and rolled out across the fleet, which we shall assume today is fifty environments in all. Every GitOps principle holds. Nothing broke in the way GitOps measures broken: declarative, versioned, pulled, reconciled; we are all green.

And yet the fleet has done three different things, and the agent could not have foreseen any of them from the one line it changed.

  • Most environments inherit that value and correctly move to 1Gi. Good.

  • One production region had a deliberate override of 768Mi, set months ago because it runs on smaller machines. Whether the one-line edit respects that override or quietly wipes it depends on how the Helm values files are layered. That is nowhere in the diff.

  • One region has a hard memory quota. At 512Mi the fleet just fit. At 1Gi, multiplied across every replica, it goes over the limit. The new pods are refused, the rollout stalls in that region, and nobody notices for an hour.

One edit led to many rendered results and then three different outcomes. Kaboom! Now what?

Where the rendering does the damage. The agent edited the chart, but the chart is not what runs. Helm renders that one value out into multiple live results, and in doing so it flattens away the things that matter: that one region overrides this value, and that the values fan out across all fifty at once. The diff shows the single line a human wrote; but it cannot show the fifty rendered results that follow. That gap, between the line you edit and the results that actually run, is where the change becomes unsafe.

Where the AI can make it worse. A person making this change might make one a week, and might just remember the smaller-machines exception. An agent makes hundreds of changes at machine speed, remembers nothing from one to the next, and every single edit looks correct on its own. GitOps faithfully applies all of them. The reconciler is doing its job perfectly. It simply has no way to know that “raise the memory” has collided with a deliberate exception and a hard limit. So the system stays correct by its own rules and walks straight into an outage. That is what unsafe looks like here: nothing misbehaved, and it still broke.

Can this be fixed?

With exceptional architecture care, the best teams avoid some of these problems. But for most of us humans, it would be better to not cause problems in the first place. We have to be SAFE.

Look at what the agent actually needed to know:

Some of it was knowable in advance. The chart renders the same way wherever it runs. Which region overrides this value, and how far the value fans out, are fixed facts. They only look invisible because Helm threw them away at render time. Recover those relationships and hold them as data, and you can see the override and the fan-out before anything ships. Call this the deterministic part. It was always there; files just hid it.

Some of it was not knowable in advance. Whether 1Gi actually breaches the quota depends on what that cluster is doing at that moment: how many replicas are live, how much memory is already in use. That answer does not live in the chart at all. It lives in the running system. No amount of reading the config tells you. Call this the non-deterministic part.

This is the whole point. Safe configuration at machine speed means telling these two apart and handling each one properly. The deterministic part, the structure and the fan-out the generator forgot, belongs in a data store that remembers it. The non-deterministic part, the live facts the config cannot contain, has to come in as context from the running system, either as a fact that gates the change or as a live truth reconciled back into intent.

Traditional GitOps did not have to reason this way. There was one loop, not zillions of concurrent loops. There was no complex hierarchy of platform charts and custom configs. Human operators could just about hold all the information in their heads, or in merge comments, so they remembered exceptions, and they knew to check the cluster. AI removes the human from that loop. So now we need to track config and context.

Winning back trust

In the last few weeks I wrote about how we need a new approach to software operations, which may take us beyond what happens with Git. But we can still do GitOps if we use enhanced systems of record designed for both AI context as well as modern configuration.

The problem is that using tools like AI can be non-deterministic, and this can lead to surprises when combined with highly context-dependent deterministic config generation tools like Helm. We get unpredictable blast radius and this is ultimately very expensive.

So for safety we need a data store for the deterministic structure and live context for the non-deterministic facts. A graph system of both config and context. It must work with Argo, Flux, Helm, Kustomize, OPA, Kyverno, Kubernetes Server-Side Apply, etc, as building blocks. The missing piece is the system that connects them into a queryable, proven, bounded, and auditable model for generated and human-written configuration → and this might go beyond Git.

Also the open GitOps principles and model need an AI update. The core principles are sound: we can evolve them to accommodate the deterministic and non-deterministic generation of config by machines in a way that is safe and trusted. People say: “never trust, always verify”. So to trust AI and GitOps, we must be able to verify that configuration data produced is correct and know how it is generated — which is its provenance and context. Generative GitOps.

Ultimately we want one AI+GitOps system that can answer questions like:

Who changed this exact value?

Was this actor allowed to change it?

Which environments will be affected?

What is the blast radius?

Can we apply safety checks to the change before it is applied?

Can we safely accept a live fix back into desired intent?

Will it be more secure than config-as-code?

And what kinds of apps can I write?

This is EXCITING.

So I might pause here and come back tomorrow.

— alexis