Thank you for sharing! Based on your experience, do you think a two-model system might fare better? For example, two models in serial where the second model is trained to "sniff out" potential hallucinations and fact check them (and possibly iterate with the first model)?
You are however likely to observe better results in smaller models since they're usually more strapped for "cognitive capacity", so two separate calls reduce the load in each request, and hallucination in my experience is a common side effect of overloading an LLM cognitively.
Does anyone know why there hasn’t been more widespread adoption of OpenAI’s Harmony format? Or will it just take another model generation to see adoption?
It's a good question, opinionated* answer: it's the whackiest one by far. I'm not sure it's actually good in the long run. It's very much more intense than the other formats, and idk how to describe this, but I think it puts the model in a weird place where it has to think in this odd framework of channels, and the channel names also shade how it thinks about what it's doing.
It's less of a problem than I'm making it sound, obviously the GPTs are doing just fine. But the counterexample of not having such a complex and unique format and still having things like parallel tool calls has also played out just fine.
When I think on it, the incremental step that made the more classical formats work might have been them shifting towards the model having tokens like <parameter=oldText>...</parameter><parameter=newText>...</parameter> helped a ton, because you could shift to json-ifying stuff inside the parameters instead of having LLM do it.
Also fwiw, the lore on harmony was Microsoft pushed it on them to avoid issues with 2023 Bing and prompt injection and such. MS VP for Bing claimed this so not sure how true it is - not that he's unreliable, he's an awesome guy, just, language is loose. Maybe he meant "concept of channels" and not Harmony in toto. Pointing it out because it may be an indicator it was rushed and over-designed, which would explain it's relative complexity compared to ~anyone else.
* I hate talking about myself, but hate it less than being verbose and free-associating without some justification of relevant knowledge: quit Google in late 2022 to build a Flutter all-platform LLM client, based on llama.cpp / any 3rd party provider you can think of. Had to write Harmony parsing twice, as well as any other important local model format you can think of.
Yeah but this seems like a bolt-on and not something they train their model to understand at the token level like how they do tool calls. Maybe Anthropic has a token-level skills support (e.g. <SKILL_START>skill prompt<SKILL_END>).
Long-running tasks are an open topic of discussion, and I think MCP intends to address it in the future.
There are a few proposals floating around, but one issue is that you don't always know whether a task will be long-running, so having separate APIs for long-running tasks vs "regular" tool calls doesn't fully address the problem.
It'd be nice to more closely integrate MCP into something like Airflow, with hints as to expected completion time.
Real world LLM is going to be built on non-negligible (and increasingly complicated) deterministic tools, so might as well integrate with the 'for all possible workflows' use case.
RSC is indeed very cool. It also serves as a superior serialization format compared to JSON. For example, it can roundtrip basic types such as `Date` and `Map` with no extra effort.
One thing I would like to see more focus on in React is returning components from server functions. Right now, using server functions for data fetching is discouraged, but I think it has some compelling use cases. It is especially useful when you have components that need to fetch data dynamically, but you don't want the fetch / data tied to the URL, as it would be with a typical server component. For example, when fetching suggestions for a typeahead text input.
It's already in use in several places (e.g., https://www.mintlify.com/docs/ai/skillmd#skills-discovery-en...) and is supported by `npx skills add`.