How my grocery agent works

I built an agent that takes the recurring thinking out of our weekly groceries: what we're eating, what's running low, and what goes in the cart. Why I wanted that is a story of its own: We kept fighting about groceries. So I built an AI to end it

This post is about how it works, and what I learned building it.

What are we actually eating?

My first goal was pretty simple: I wanted to say how many days we need food for, and get a proposal for both the menu and the groceries.

That immediately comes with a few conditions. Dinners are vegetarian by default, and they have to be something everyone will actually eat. I don't want the same dishes every week, but I also don't want four experimental recipes nobody is waiting for.

So the agent currently lands on about four dinners a week. Usually one of them is new, and the rest come from recipes that went down well before. A dish doesn't come back within three weeks.

I also try to stay around a weekly budget of €110. That's not a hard limit, mostly a way to keep the agent from going wild on expensive ingredients.

Up to this point, you could just as well ask ChatGPT for four vegetarian recipes every week.

The more interesting part starts with the groceries themselves.

We already had years of training data

We've been ordering from Picnic for quite a while. Without realising it, we had built up a pretty interesting dataset: our order history.

The agent looks at the last eight weeks. Not to simply reorder whatever we bought back then, but to recognise patterns.

Which milk do we normally buy? How often do we buy butter? Which products come back in almost every order, and which ones only once every few weeks?

I use that as a sort of primitive stock model. Because I absolutely don't want to start keeping track of how many grams of rice are left in the cupboard. Then I'd just have swapped one household chore for another.

Instead, I work with order rhythms. Milk is restocked after about six days. Spread after fourteen. Ontbijtkoek (Dutch spiced breakfast cake) turns out to be more of a once-every-six-weeks thing for us.

The logic is roughly:

When was this product last ordered?
                 +
How often do we normally need it?
                 +
Have I said there's still enough, or that it's almost gone?
                 =
Add it again, or not

That's obviously not perfect stock management. But it doesn't need to be. If the system usually decides correctly that we probably still have enough spread, that's already a lot smarter than blindly ordering the same standard list every week.

I didn't really build a grocery app

Technically, this might be my favourite part: I've hardly built a traditional application at all.

There's no elaborate frontend. No backend of my own with a database full of products and recipes.

I use Cursor as the agent. Underneath it is a skill that describes the workflow and the house rules. A few YAML files act as memory. And through MCP (Model Context Protocol), the agent can use the Picnic tools directly: browsing recipes, searching products, reading past deliveries and filling the cart.

Cursor agent
     ↓
groceries skill
     ↓
YAML with our preferences and rules
     ↓
Picnic MCP
     ↓
Picnic

That makes most of the "code" simply declarative. Preferences, substitutions and restock rhythms are plain text I can read and edit, instead of something the agent has to remember from last week's chat.

The flow, step by step

The weekly run deliberately works in steps:

1. Look at the order history and work out which basics probably need restocking
2. Put together a menu: a few favourites plus one new recipe
3. Show me the menu and the list, and wait
4. After approval: add each recipe to the cart, fine-tune the ingredients, add the basics
5. Show the total, and save the menu for next time

The agent may fill the cart. It may not place the order or pick a delivery slot. I do that myself in the Picnic app.

The first order wasn't right, of course

An agent can neatly read a recipe and order exactly the ingredients it lists. But correct according to the recipe doesn't mean it makes sense for our household.

A recipe calls for chopped onion. The agent searches for chopped onion and finds a bag of pre-chopped onions. Technically perfectly fine. Except I would never buy that myself. There are loose onions right there in the shop, and I'm perfectly capable of chopping them.

The same thing happened with pesto: the agent picked a jar, while we prefer fresh pesto from the chiller. And during a summer week, it suggested a dish with winter vegetables. Not wrong, but not how I would put a weekly menu together.

I don't correct those mistakes by hand every week. I turn them into rules. Pre-chopped onions become whole onions or shallots. Pesto is fresh. Recipes and vegetables have to fit the season.

That, for me, is the real feedback loop:

agent makes the menu
      ↓
agent fills the cart
      ↓
I check and adjust
      ↓
agent sees what should have been different
      ↓
rule or preference gets updated
      ↓
better next week

I find that much more interesting than trying to come up with hundreds of rules in advance. The agent is allowed to make mistakes, as long as I only have to correct each mistake once.

Some lessons were more technical:

• Recipes have to go in as recipes. In one of the first weeks, the agent added all the ingredients as separate products. The cart was fine, but Picnic only emails the cooking instructions when you add a recipe as a recipe. So now every recipe goes in first, and the fine-tuning happens afterwards.
• One item at a time. Adding products to the cart in parallel caused lock errors. Sequential is slower, but it works every time.

A dashboard in the kitchen

The feedback loop doesn't stop at the cart anymore. There's now a Home Assistant dashboard on a tablet in the kitchen.

It shows this week's dinners with the full recipe, so nobody has to dig through an email while cooking. After dinner, a dish gets a rating from 1 to 5. The dashboard also shows six candidates for next week, and we pick the four we want.

When the agent plans the next week, it first pulls those ratings and picks from Home Assistant. Five stars and a recipe becomes a favourite. One or two stars and it won't come back. So the agent no longer just learns from what I change in the cart, but from what we actually thought of the food. More about the dashboard: A kitchen tablet that rates dinner.

Except we don't buy everything at Picnic

There's still an obvious gap. We also shop in between: a quick trip to Jumbo because we forgot something, some extra things for lunch, an ingredient that ran out unexpectedly. Those purchases never show up in the agent's history.

So I've started taking photos of receipts. For now that's all they are: photos. The plan is for a receipt to become just another input source, with the products pulled out and added to the purchase history.

I find that a funny development. A receipt used to be something that sat in my jacket pocket for a few days before ending up in the bin. Now it's about to become data for my own grocery agent. And it would make the history a lot more realistic: not just what we order online, but what actually comes into the house.

What's next

Stock is still an estimate, and I expect quite a few more household exceptions to surface. But the original problem is starting to look solvable, and that's more than I expected when I started.