I have been trying to get a better picture of where our money goes. Not just whether we are spending too much, but which parts of our budget are actually growing over time.
Online marketplaces made that surprisingly difficult.
A transaction would arrive from my bank with a date, an amount and a marketplace such as Amazon, eBay or Etsy as the payee. That was enough to know where the money went, but not what I bought. Was it something for the house? Groceries? Electronics? A birthday present?
Without that context, everything ended up in Shopping. Technically categorised, but not especially useful.
Starting with Actual#
I already use Hermes Agent as a personal agent, so I asked it to help improve my finance tracking. It helped set up Actual Budget and import my transaction history across my different banks.
That gave me a much better place to inspect my spending, but also made the limitations of the data more obvious. A bank transaction only knows about the payment, not what was behind the order. With online marketplaces covering so many different parts of life, putting every purchase into one category loses most of the information that makes a budget useful.
Finding the missing context#
Hermes came up with a simple idea: the bank knew how much I paid, the order confirmation emails knew what I ordered. If those two sources could be matched, the single opaque bank transaction could be split into the items that actually made it up.
I had Hermes turn that idea into a skill and supporting scripts, which it runs as a recurring sync. For the email side, it uses gws, a command-line interface for Google Workspace, to find the relevant order messages in my inbox. It then compares them with the corresponding bank charge, using the amount and timing as evidence.
Amazon makes this harder because its emails are intentionally opaque. In my case, it often sends one email per item but a single bank charge for the total. The scripts group messages with the same order ID, sum their item totals, then use the calculated amount and date to find the corresponding charge. This is fuzzy matching rather than an exact reconciliation, but it correctly matches roughly 80% of my orders.
When a purchase contains several items, Hermes splits the transaction between their categories. When it contains a single item, it simply recategorises it. If the evidence is unclear, it leaves the transaction in ‘Shopping’ rather than guessing. The same process works across Amazon, eBay, Etsy and other retailers.
Instead of this:
| Payee | Category | Amount |
|---|---|---|
| Online marketplace | Shopping | £75.00 |
I get something closer to this:
| Item | Category | Amount |
|---|---|---|
| Laundry detergent | Household | £10.00 |
| USB-C cable | Electronics | £5.00 |
| Blender | Kitchen | £60.00 |
The items and amounts are illustrative, but the shape is what matters: one bank transaction becomes several useful pieces of budget data.
Letting the agent make the first pass#
Hermes also chooses a category for each item by inferring what it is. That part is autonomous: I do not approve every item before it appears in Actual.
Normally, I would be wary of letting an agent make decisions about financial data. Here, the cost of a mistake is small: the split is easy to inspect and a category is easy to change. Hermes handles the repetitive first pass, while I correct anything that looks wrong. The result does not need to be perfect to be substantially more useful than putting everything into ‘Shopping’.
Importantly, Hermes is not deciding what I should spend or moving money around. It is restoring context that was lost between placing an order and seeing the payment arrive in my budget.
Hermes can infer what an item is, but not why I bought it. If it was a gift, only I know that.
Seeing where the money went#
The biggest improvement has been much clearer spending patterns. Previously, an increase in Shopping told me almost nothing. I would have needed to open each marketplace, find every order, compare it with my bank statement and manually recategorise every item. Unsurprisingly, I rarely did that.
Now those purchases contribute to the categories they actually belong to. If household spending rises, I can see it. Appliance purchases no longer look like an unexplained shopping spike. Small electronics purchases stop disappearing inside the same bucket as toiletries and groceries.
The total amount of money has not changed, but my understanding of it has.
Connecting the pieces#
The interesting part of this project is not really online marketplaces or even budgeting. It is the way the agent found useful context spread across two systems. My bank feed was accurate but incomplete; my email archive had plenty of detail, but it was disconnected from the place where I manage money.
Hermes became useful when it joined them together and acted on the result. That is where I think personal agents have the most potential: not as another place to ask questions, or as an all-knowing system making decisions on my behalf, but as a way to connect the fragments of information I already have.
Sometimes the useful insight is simply that ‘Shopping’ is not a category.