I gave a 45 minute presentation entitled “ Your Game Is A Harness” as part of Indie Rodeo in Calgary on 9/12/26. After speaking with a lot of people after my talk, I thought it would be good to more broadly share the ideas I presented there in a form that could be enjoyed by people beyond just those who attended the conference.
Also, because I talk about AI a lot on this blog, people assume I use AI to write. This is not true! Everything below is 100% human authored, with me writing the post with basically only the slides I presented as a reference (this is not a transcription). As much as I enjoy using AI to write code and tools, I get a similar enjoyment out of writing “ the old way”, especially as I have no reason to try and make any part of this process more efficient.
So with that all out of the way, I present to you my first ever blogification of a talk I gave. Enjoy!
Introduction
[I give a small introduction to myself and my own work, that I’m omitting here, as you’re on my website]
I want to talk about using LLMs in game development. Not one-shot type demos that look cool on Twitter, but like real development. Where you’re working eight hours a day, 5 days a week, etc.
And before I get into it, I think it’s useful to level set about LLMs. Or at least get into some basics so the rest of what I talk about here makes sense. I recognize that for a lot of this people this will be review, so if you just want to get into the dev-loop stuff, scroll down to the next section.
My journey into using LLMs for dev work started with a moodboarding site I made called mood.site . This was pre “ agentic” anything, but I wanted to test and see if ChatGPT at the time could actually code anything. The process was a lot of me just blindly copy/pasting code into ChatGPT’s web interface. The process sucked, but I was surprised at the results. I built the site! It worked! I still needed to verify a fair amount of stuff myself, but it was enough to convert me. If I didn’t have access to an LLM I wouldn’t have been able to make that site happen in near the amount of time it ended up taking me (~2-3 weeks of nights, probably like 20 hours total). I’d probably still be working on building the first version today.
Since then, mood has become something of a testbed for me moving through LLM stuff (and a mildly successful project on its own!), and has inspired me to kick up lots of other projects that use similar stacks. Additionally, LLMs have gotten better, especially with things like Claude Code and later Claude models. From my own experience, I feel like now there is very little, if anything, I think an LLM can’t do when it comes to a web-based stack.
It could be easy to then say LLMs can do everything , as so much of software is web software. But whenever I tried to use an LLM on building a non-web game thing , it was this resounding takeaway:
“ LLMs are Bad At Games”
Any time I tried to use an LLM for a game-y thing, it would fall over. Maybe this is my own bias. I’ve been a game programmer for nearly 15 years, I think I know how things work. I can sniff out “ bad” game code a mile away. But when it comes to Svelte web-apps, I’m decidedly less well versed and am inclined to think that whatever code the LLM produces is “ good”. Maybe it’s actually garbage. But there is reason to believe the Svelte and web-stack stuff is quality, in part due to the fact that there are LOTS of high quality web projects with their source available online. Comparatively for games, I imagine there are orders of magnitude less data to train on.
Not only this, but games have a lot of stuff in them that isn’t code. Games aren’t just about putting static images and text in a layout. There are tons of things in games that aren’t easily representable by code. AND then on top of that there is all the same text layout stuff of web, but only as a subset of the 100 other systems a game needs to handle.
This is part of the reason why a lot of people have been making games in Three.js — they can easily represent assets as plain boxy primitives and skip a lot of the asset generation/conforming work and processing normally required for games. As a result, a lot of that stuff is slop because it lacks any real authorial voice outside of whatever the LLM median is.
A key idea of the one-shot Three.JS workflow (as well as LLMs proficiency at building web apps) is that everything is code. By having stuff as code, it’s very easy for an LLM to be able to reason about it, make changes, etc.
However for traditional games, this is not the case. Or at least the full story. Games obviously have code, but there is a lot of other stuff that is not code. This means that there are key components of games that are effectively opaque to an LLM . If your game then relies on the effective puppeting and transforming of that opaque stuff, then an LLM is going to fall over when trying to work on your game.
I think a lot of the gamedev community maybe gave LLMs a go a bit ago and ran directly into this. The “ LLMs are bad/not useful/etc.” sentiment (massive caveat here, just talking about capability specifically and not potential society-ending implications) came from running headfirst into these limitations.
This is to say that, structurally, LLMs are bad at reading games. At least in their default state.
LLM Basics
Let’s talk more about why, and what we can do about it.
Any call/prompt to any LLM service breaks down into three basic components. A system prompt, an assistant prompt, and a user prompt:
System prompt: Generic text shared across all users of a service. This usually describes the basics of the LLM itself, with something like “ You are Opus 5, an intelligent Large Language Model created by Anthropic…”. This isn’t to say these aren’t important (they are), but more that they aren’t super-relevant to our use case. You can see some examples of posted (and exfiltrated) system prompts for models here .
Assistant prompt: This is kind of a sidecar prompt section meant to help guide the AI in terms of what it is responding to. You can think of it basically as the model’s basic “ interaction” vector with your prompt, and represents something like the model’s “ response” to the user based on some input. You usually do not have to think about this.
User Prompt : This is really The Thing. Most any input you put into any AI tool goes here. Any submitted chat, code, etc. gets dumped into this section. When people talk about “ context”, they are largely talking about “ how many tokens are stored in the user (and assistant) portion of a prompt”.
It may be easy to think that each chat is a “ fresh” version of the above, but what actually happens is that, as you work with a model in a given session (the bounds of “ session” vary based on what service you are using — this can be a single Claude Code session before closing it, a “ chat” accessible in the side panel of a web hosted model, etc.), every prior message you sent to the model is re-submitted every time you send a message. It’s kind of like playing the old playground game where, on your turn, you have to say everyone else’s thing again before then adding your own thing for the next person to have to add on.
Context and Context Pollution
As you do this over and over, you build up the “ context” of the session you’re working in.
This is all well and good, but a key thing to notice here is that you only really are able to address the “ latest” thing in your context (i.e., you submit a new message). Things from previous messages get submitted over and over again to the model. Hence, you want to make sure what’s in your context is “ good”, otherwise you can find yourself in degenerate scenarios where your context becomes polluted due to “ bad” things earlier continually informing new work.
Managing context then is a key thing in working with LLMs, and a new engineering practice called “ context engineering” has emerged in light of these constraints. Context engineering asks “ how can we get the best information into the context, with as little pollution as possible”.
There is a great blog post here on “ ways long contexts fail” that goes a bit more into this, but I’ll highlight the key part here:
Context Engineering itself is a really big topic that is outside the scope of this talk/blog, but we will be using some of the tenets of it here to explore how to help LLMs do a better job working with games.
Said differently, getting LLMs to work well with games is a context problem, not something that is intrinsically linked to how games or LLMs “ are” (or “ aren’t”!).
The complicating factor here though is, games are weird . Or at least: we have no real template to build a game. We have game engines, but there is effectively 0 standardized tooling for anything, in part because every game manifests its own needs differently from one another.
This is acutely unique to games (imo), but it manifests enough in other technological domains that a new concept has emerged in order to get agents to effectively “ work” with whatever your work product is: Harnesses.
Some Description of the Idea of a Harness
The general idea of a Harness is that it’s some set patterns of actions an agent is meant to take in response to some well (or loosely) defined expected input. You’re probably already using a harness like Claude Code, but these can be augmented, extended, distilled, abstracted, etc. to better support the specific work you are trying to do. There’s also been a LOT of action in this space recently, with many harnesses being built for developing all sorts of things:
The key things these are all pointing to is: “ how can I better understand what I’m building, and guide that process as it is developed”. Harnesses can tie in things like iterating with a browser, telemetry, profiling, and even full SDLC work like making tickets, documents, etc.
However, the thing none of these do well is actually talk to your game. Your game is a bit of a black box, and most of what’s visible to an LLM is only the code-as-written, which as we talked about before, isn’t really enough. We know LLMs are already good at working with code, so it’s not like some harness itself solves that “ games aren’t just code” issue we ran into in the first place.
What really needs to happen here is that we can take all the stuff in games, and try to turn it into something the LLM is able to work with. We need to crack open the black box of your game and make more of it addressable directly by an LLM .
During the talk version of this I had this Very Funny Slide that said that the arrow here was the actual content of this talk. The joke hits nowhere near as hard in blog form but I’m leaving it here for posterity:
Let’s Make a Game Legible To AI Agents
Here’s some very tangible, actionable things you can do for your game to make it more legible and workable by an LLM .
As a brief aside before getting started, it’s also worth noting here that a lot of these tips are good for game development in general . They get supercharged a bit with AI , but doing any of this will likely make working on your own games generally just better .
Fixed Randomness
A running theme through this whole talk/post is the concept of determinism . Basically, how accurately can you reproduce a very specific state in your game. The general goal is that it should be very easy, and very exact. Doing this at the very end of your production will be VERY hard to do, and would likely require fundamental changes to your game in the 11th hour.
Instead, if you assume this constraint before you write your first line of code, and stick to it as a goal, you’ll be paving the path to a much more productive end game and not find yourself fighting off entropy and spaghetti code.
Starting off here is pretty easy. At the start of your game, create a single source of seeded randomness. Make sure any call to anything non-deterministic uses this source of randomness. This means that your game will be able to deterministically recreate almost any state in the game. If your code calls Random.Float or whatever in 30 places to determine some gameplay behavior, you know that the 30th call will be the same every time. Easy.
Screenshot-ability
Why have reproducible states? Many reasons, but for an agent, imagine you’re iterating many times on trying to correct something. It’s important for the agent to be able to get to the same state from scratch over and over again without needing to churn through state replication failures.
And once you get there, how do you know? How can you evaluate those changes?
Well for one, screenshots! Most engines have some built in screenshot API or functionality:
I also recommended adding in the ability to easily toggle UI layers, gameplay layers, etc. on/off that way you can reduce visual clutter that distracts the LLM from the layer you’re trying to address (ie, dont have gameplay under a UI if you’re trying to fix some UI style thing).
Taking a before and after picture of your game between agent changes is a nice way for an agent to be able to understand what changed.
When you take two screenshots (Before/After) a change, the LLM , even if it’s multi-modal (ie, has vision capabilities), will likely have a VERY hard time not only understanding what the change is, but understanding if the change it made moved it closer to its own end goal.
This noise from image data is its own form of context rot: naiively throwing screenshots at models can muddy their own sense of progress, leaving both them and you confused.
A visual example can help here: Imagine you take a screenshot of the home page of this conference:
You ask the model to “ Make the skeleton a pirate”. Assuming if (and with text LLMs, unlikely) this can be applied as an annotation to the image ( Google Stitch does this), your image will be passed to the model provider with your tag, but then also all the other stuff you aren’t trying to address (in green below).
This means that everything in green is something akin to “ context pollution”. It is data the LLM is processing as well as your requested change, but it’s not just thinking about what you requested, but is thinking about the whole image.
You need to get the LLM to think about only what needs to change. How do you judge what has changed?
If you take two images and “ diff” them, like you do with code changes, you can easily see what has changed. Combined with clever uses of cropping, you can have an LLM understand, with a quantitative measure, how closely some output post-change is close to your provided input/target image.
Luckily, you don’t have to do this yourself. There is a very good free, command-line image-diffing tool called odiff that does exactly this .
Given two different images, odiff finds the “ difference” between the two, and can output that difference as a “ diff” image:
odiff also can give you a quantitative difference value like “ 85% similar”, that an LLM is very good at reading, and can then use that as a stable baseline of eval across each agent turn as it strives to reduce the difference/increase the similarity.
Said differently, as you are working, these agent screenshot -> progress diff check become the basis for continual agent evaluation towards whatever your task is:
However, this obviously isn’t all. Games are much more than just their visual elements. “ Progress” here breaks down into it’s own little inner loop:
This inner-loop is likely the place where an agent will spend most of its time, doing multiple turns on its own before going back to the screenshot comp/diff (assuming that your change has some visual implication).
The goal here is to be able to make every change be the minimum amount of change required for a desired outcome, and have those changes able to be quickly verified. You want an incredibly stable, “ boring” baseline that can be evaluated in near-isolation to ensure that anything built on top of it doesn’t produce wide ranging technical debt (aka, spaghetti). So let’s talk more about deterministic state.
Deterministic State
The rest of the items in the above graph also deal in some part with deterministic state, but I want to talk about specific strategies and techniques for achieving it, as they will be essential for everything else.
There’s a simple question to ask to understand/know how close you are to deterministic state: how easily can you save and load an exact state in your game? Or said differently, the most important part of state reproduction is having a strategy for efficiently saving and loading your game from the start .
Saving/Loading is usually something amateur developers will put off to the end of a game, but this is a grave mistake, as many more seasoned developers will tell you. It’s easy to think of saving/loading as a “ feature” that you can add at some point during a game’s development, but saving and loading, when leveraged at the VERY start of a game and programmed in conjunction with , can be a development superpower that has a near innumerable list of benefits.
Let’s talk about one strategy for doing this, and why it makes ALL of development easier.
We’re going to take a small jaunt here into CS-style programming theory. Most people that spend extensive time with object-oriented type architectures for games will find that, near the end of their project, something like the above statement “ A Tree is a Dog” is true. They look at their code, and what things actually do in a game, and realize that, these two seemingly very different objects with separate OOP hierarchies are, at the end of the day, actually more or less the same kind of “ thing” from the perspective of the code.
A tree is an object that has a mesh and a collider. A dog is an object that has a mesh, collider, and can move. “ Movement” is not an intrinsically “ dog” thing (other things in the game may move), and hence a tree and a dog are basically just compositions of components that have some game-representation. From the code’s perspective, they are effectively the same.
This can be counter-intuitive at first, as it feels wrong, but there is a chakra-unlocking insight here that, when you hold on to, starts to make game programming much easier and not a constant uphill battle against a wall of abstraction that occurs when you try to maintain a strong OOP-style object model in your game.
Thing-Driven Development
Instead, if you take the idea I’m proposing and run with it, you can assume from the very start of your game that everything is actually the same thing (maybe even use the type Thing ). If you were to define Thing in code, you’d have something that looks like this:
You can see a Thing has some stat-type values on it as well as a shared idea of “ liveness” from the isAlive flag. Otherwise, the “ types” of thing is just an enum value, and based on that type the implementing code can take some action.
And as It turns out, AI REALLY likes this.
With this type of architecture, there is a single source of truth for your object model. If instead you have 100 classes of 100 different types of objects, an AI will need to likely understand all of those (and their relationships) to take action. This fills up your context, and makes reasoning about how to fit into that pattern messy — should I (the AI ) extend some class? Morph the type of a class to better capture this new thing? Make a new class? It’s hard to say!