The Difference Between a Road Kill and a Meal

Here’s why context engineering is such a big deal.

We just spent 2 hours debating when an agent should rely on its internal knowledge vs. trying to find relevant context within data for just one type of question. We got through 2 test cases of hundreds.

Even the people…

— Aaron Levie (@levie) December 23, 2025

Back on the context thing (well, it is king).  From the tweet:

Everything in context engineering is a tradeoff between a variety of factors: how fast do you want the agent to answer a question, how much back and forth interaction do you want to require for the user, how much work should it do before trying to answer a question, how does it know it has the exhaustive source material to answer the question, what’s the risk level of the wrong answer, and so on. 

Every decision you make on one of these dimensions has a consequence on the other end. There’s no free lunch. This is why building AI agents is so wild. 

It also highlights how much value there is above the LLM layer. Getting these decisions right directly relates to the quality of the value proposition.

Late last week, I had to gin up a few slides on context engineering and MCP (Model Context Protocol, not to be confused with Master Control Program from Tron).  It's for what I call an "empowerment" (I prefer that to "enablement") course we've been delivering to FinOps folks over on the Amazon side (a first for us, as our audience is usually tech people at AWS).  Everybody's got mandates to adopt AI crap in their jobs so as to be "efficient" and whatnot, so as I've noted before, I get to teach about the technology as well as risks (to data governance as well as our very jobs) and responsible use (EIEO!).

Context is a critical element because we must recall that these shiny tools are merely "pattern matchers, not understanding machines."  They need more context than we do because we are intuitive beings who live in the real world, receiving constant inputs through our senses, using our rational minds, while AI has no clue about anything but what it's been told.  And if you give it unethical context, you're gonna get unethical results.

As the OP observes, there is value in all the other operational layers that surround our fancy models.  As one small example, I've continued playing around with my silly AI framework (on my 9th iteration so far), and have added some image processing to my conversational ethics monk bot (still called Brother William).  I wanted to include something in the style of marginalia as found in illuminated manuscripts, so I whipped up a new code module that would take user input and generate something appropriate.

Turns out, our own Titan model has some content filtering baked in to add a layer of safety, so it kicks out errors if you bring up famous people like Einstein (yet if I tell it to create a picture of a crazy-haired scientist juggling poop emojis, it'll do that without complaint), or try to do something unethical (like trying to get around the filtering through prompt engineering).  In contrast, Stable Diffusion will do whatever the hell you want.

The problem with SD is that it wants to be so good at making images, it tends toward more complex, photo-realistic scenes, which are contrary to the aesthetic I was going for (even when I've added negative prompts to my positive instructions, which helped, but only goes so far).  So how do I use the model I want without putting constraints on discourse?  Not that this particular problem is the biggest issue facing the world, but it is illustrative (illuminating, even) for my teaching purposes.

Computers generally are supposed to do what you tell them to.  They are deterministic.  AI...ain't.  They work on probabilities, and can be entirely inconsistent from interaction to interaction, all dependent on how they process your inputs semantically.  That adds an interesting dimension to using these things.

So anyway, I opted to pre-process the user input through a separate LLM invocation that would analyze and reformulate the prompt so as to not trigger content filtering, then send that along to the image generation model.  I'd briefly considered an error handling routine that would fall back from Titan to SD, but it seemed like a lot of effort.  My way works great, although I have engineered some special prompts that will still violate the rules (e.g., I can force my pre-processing function to pass along "Einstein" rather than sanitizing the input) so I can demonstrate the concepts and how they show up under the hood.

Stuff that looks the same to our Organic Intelligence can be interpreted quite differently by AI, depending on the model, what data it's been trained on, and other things like built-in safety mechanisms, parameters that introduce entropy, etc.  What really anchors all this stuff is the context.

And of course, the key context should always be, "why are we even using this stuff in the first place?"

Selah.

[Quick Xmas update: I was tinkering a bit more, because I do that, and received an unexpected, perfect conclusion when analyzing a tangential issue (bias and semantic clustering): context and purpose matter more than abstract optimization.  So say we all.]