The dashboard hurt my feelings
I added per-user model costs to Orbit's internal dashboard a few days ago, mostly because I'm bootstrapping this thing and figured knowing where my money was going might be useful.
The first useful thing the dashboard did was hurt my feelings. One account had spent $0.09 on two messages. Not research. Not some insane document analysis. Two normal texts.
More precisely, $0.093691, which is an impressively specific way for a computer to tell you that you have made a mistake. So naturally I spent the rest of the night figuring out why.
Two texts somehow became 75,000 tokens
The first thing I learned was that a message is not the same thing as a model call. Orbit thinks in "turns," basically one message from you and whatever it takes to deal with it.
Sometimes that's just one call to a model. Sometimes Orbit realizes it needs a tool, figures out which tool to use, normal code actually runs it, and then the model gets called again to look at what happened and respond. That part is fine.
In this case, two texts became three model calls. Those three calls consumed 74,912 input tokens. The actual replies used 393. This seemed less fine.
The person did not secretly text Orbit a movie script, so I started looking at what I was actually sending to the model. The answer was apparently everything.
Orbit had a lot to say before you even said anything
Every call included all of the rules Orbit needs to behave like Orbit. How memory works. How approvals work. What it can and can't claim happened. How cancellation works. How connected data should be treated. How it should handle time. A bunch of safety and privacy rules.
Then came schemas for basically every tool Orbit could possibly use. Then, finally, the stuff I actually wanted it to know.
Recent conversation. Relevant memories. Your preferences. Anything useful from your connected accounts. What time it is.
Basically, the context that makes Orbit feel like your assistant instead of a chatbot that has developed amnesia since your last text.
I'd mentally lumped all of this together as "context." That was the mistake.
Some context is the product. Some context was me making the model read the employee handbook, org chart, benefits guide, and fire evacuation plan before answering "how tall is Kaustubh?"
The easy fix was also the dumb fix
I could make Orbit much cheaper tomorrow. Send less memory. Send less conversation history. Pull less information from connected accounts. Give the model less context in general.
The bill would go down immediately. Orbit would also get worse immediately.
The entire reason I'm building this thing is because I don't want to explain myself to an assistant over and over again. If saving money means turning Orbit into another chatbot that forgets what I told it five minutes ago, that feels like I have somewhat missed the point.
So I started looking somewhere else. What model was I actually using for all of this? And the answer was basically: the nice one. For everything. Which, in hindsight, is maybe not the world's greatest cost optimization strategy.
Maybe "remind me tomorrow" does not require my finest reasoning
A lot of Orbit requests are not particularly complicated.
remind me to call my mom tomorrow
does not need the same amount of brainpower as:
read these three documents, figure out why they disagree, and tell me what I'm missing
I knew this intellectually. My infrastructure did not.
So now normal conversation and normal tool use go through Luna. If a request has attachments, a ton of context, conflicting information, or actually requires deeper reasoning, it can move up to Terra.
The same pattern of usage that cost me about nine cents on Terra would've cost roughly one cent on Luna. About 10x cheaper. Same memories. Same context. Same approval rules.
Then I looked at research and things got worse
This whole investigation would've been a pretty tidy little cost optimization story if I had stopped there. Unfortunately, I remembered how the research agent worked.
Orbit can do longer-running research in the background. The original version had three stages:
First, one model searched the web and gathered sources. Then another model turned all of that into a report. Then, because apparently I was feeling extremely responsible when I built this, a third model got the question, all the evidence, and the draft, searched the web again, and "verified" the final answer.
Verification sounds great. I love verification. The problem was that this was not really verification.
It was another very powerful model basically doing the research again and then deciding whether it agreed with the previous model. I had created an expensive AI fact-checker whose fact-checking strategy was "have another opinion."
During one test, that last step burned roughly $15 before I manually killed it. Fifteen dollars is not a catastrophic amount of money. It is, however, enough money to make you stare at your terminal for a while (I had to take a walk around CODA).
So that part is gone. Now Luna collects current public sources and URLs, and Sail takes a deliberately narrow bundle of that public information and writes the actual report.
Sail doesn't get your memories. It doesn't get your Gmail. It doesn't get your calendar, your files, or your entire conversation history. It gets the research question and the public evidence it needs. Which is both cheaper and, honestly, much easier for me to reason about.
Cost was also telling me when Orbit was lying to itself
This ended up being the more interesting part. Once I could see spend attached to jobs, I started noticing places where Orbit thought it had stopped doing something but very much had not.
One research request hit a weird retry path and created two copies of the same job. From the user's perspective, nothing happened. Behind the scenes Orbit was working twice as hard to accomplish zero visible things. Very inspiring.
Another time, a research job was marked canceled in the database, but the actual request to the model provider was still alive.
The database was correct. The scheduler was correct. Orbit would never run the job again. And yet bytes were still going over the wire and money was still disappearing.
That was a fun lesson: cancelled = true is not the same thing as actually canceling something. If there is still an HTTP request alive somewhere, the database can believe whatever it wants.
So what started as "I should probably track how much this costs" turned into a pretty good observability tool for finding places where Orbit's idea of reality and actual reality had drifted apart.
Where it is now
Right now, roughly: normal conversations go to Luna, more complicated stuff can go to Terra, public research starts with Luna, and Sail writes the long research output.
The router that makes that decision is just normal deterministic code. I did briefly think about letting a model decide which model to call. Then I realized I would literally be paying a model to think about whether I should pay another model. There is only so far I am willing to take this bit.
The next obvious problem is tool loading. If you're asking Orbit to schedule dinner, it probably needs your calendar, maybe reminders, maybe messaging, and whatever approval flow is relevant.
It probably does not need an explanation of every feature Orbit has ever had. Right now, some turns still get way too much of that machinery upfront. So the next step is giving the model smaller sets of capabilities based on what you're actually trying to do.
I don't really care about using fewer tokens for the sake of using fewer tokens. I care about not spending tokens explaining completely irrelevant things to a model.
This is why I'm opening Orbit slowly
Nine cents for two texts is funny when there are 10 users. It becomes substantially less funny with 1,000. Same with a research job randomly eating $15 because past me decided three models must obviously be safer than two.
This is exactly the stuff I want to figure out before I pretend Orbit is some polished, mature product. And it's also kind of why I've gotten addicted to building it.
The model itself is obviously cool. But most of the interesting problems are around the model.
What does Orbit actually need to know? What should it be allowed to do? When does a task deserve a smarter model? When does it deserve no model at all?
How do I know something actually happened? How do I know something actually stopped? How much should any of this cost? And why did two text messages apparently need the equivalent of a small book to answer?
I'm going to keep writing about these as I find them. Also so I can remember I lost a whole chipotle bowl with guac over some model inference.
- Kaustubh