The A/B test infrastructure is. It’s easy to wire up experiments because there is always a template/recipe to do so. Multiple subsystems can be strung together (eg: Experiment arm XX in subsystem A triggers Experiment arm YY in subsystem B). Analysis is simplified too since all the core metrics are already in place and ready for analysis.
Personally, I don’t run multiple sessions / agents (apart from the one claude code spawns on it’s own).
But it seems you need an agent workspace manager - perhaps something like gastown? https://github.com/gastownhall/gastown
True. This was written with the context of Google which was, to a large extent and up until very recently, a good example of the resilience and magic a slime mold can be.
It is not true today, and as you mentioned "fundamentally incompatible with a next-quarter profit driven.."
To add to this, MCPs usually focus on capabilities. This is especially useful in multi-hop situations.
As an illustration, lets say I ask the agent to “Read the issue Bob posted today from Linear (issue tracker), pull relevant logs from DataDog (log tool) and suggest a fix”
In the MCP world, my agent doesn’t have to spend tokens / time understanding APIs or the protocol (as the parent comment suggests). Instead, it relies on the tool provider for the best way to “read the issue” or “pull logs”.
Same thing, in the API world, would usually involve multiple API hits (and possibly using the wrong APIs or params for them), interpreting the results and hitting few more APIs again.
My biggest problem is that msps can't be chained together and that the response schema is not know models are very good at filtering with jq to get only the relevant fields
Thanks for the link. Very succinct. Coincidentally, just a week back, I ordered the book it mentions.
I completely agree with everything you mentioned (And thanks for giving in numbers too!).
Since you’ve clearly thought this through, if I can push you to the hypothetical - at what point would you want to run multiple tests in parallel? Will it be largely a function of your dev team (more productive everyone) or active users (more frequent feature build outs / cross cutting rollouts) ?
1) running multiple tests in parallel becomes important where the product/dev frequency exceeds waiting for a single test to finish, which is roughly a week for us. the business is currently relatively stable and doesn't change a lot, and I don't have big ideas that would need measurement so frequently, but if I did, I'd probably look for ways of running things in parallel. So I don't think it's a function of the dev team as much as what you reasonably want to measure. Having said that, I can imagine a larger dev team implying multiple product people, and those might want to have their own experiments/ideas measured in parallel, so there is something there definitely about the size of the dev team as well
2) if we wanted to be very scientific about the whole thing, we'd need multiple tests in parallel, particularly for novelty/primacy holdouts and to reconfirm that certain things actually hold across time. e.g. if I decided that variant A is better than variant B but our traffic changes (which it kind of it did over the last 3-4 months with AI overviews killing clicks from Google), maybe the reduced set of people we get behaves differently than the larger set we measured against a year ago. so I'd probably want to re-run critical experiments and just confirm them.
In addition, same set of questions can also be asked to people who've recently left the organisation.
I'd use the following criteria to filter ex employees:
- Reliable 1st/2nd level connections
- Folks who have shown stability in the past (generally stick to jobs for > 2 years)
Obviously, they'd come with biases. But, talking 2-3 people should also help reliably gather broad culture patterns.
Should avoid listening to evidently disgruntled folks.
> DON'T BE AFRAID TO DISMANTLE THE WRONG ABSTRACTION
Couldn't agree more with the statement, though I don't completely agree with the author's suggestion to copy paste. Duplicating code _is debt_. It may help us go faster now, but it'll almost inevitably come back to bite. It is manageable if 1/2 people do it 1/2 times - definitely not manageable if 5/6 people do it 5/6 times.
I believe the general hesitation of not touching a piece of code (or, getting by with that optional param) is due to the fear of fucking things up. Having your code test covered gives an amazing amount of confidence to rip apart old abstractions to yield newer ones that serve the purpose of the _current code_. To me, this route is more preferable to duplicating code.
Even with the best of intentions, Hacking an abstraction with that one optional parameter is inevitable. Tests help in our ability to repay that debt faster - on time & in full.
Basically they make all abstractions a lot cheaper - easier to write and easier to throw away. Thereby solving the problem of having a 'wrong abstraction' too early.
It is almost always better to copy/paste a function to accommodate "that one optional parameter" that breaks the original abstraction, than to add the parameter to the function signature. The "cost" of a broken/leaky abstraction is at least an order of magnitude higher than that of duplicated code.
While I completely agree with your _sentiments_ I think there is a conflation between choosing _the right tool_ for the job vs using 'too many things'.
I shun the unnecessary - completely with you on that.
But I'd prefer gaining relative mastery over _1 decent tool_ in each of the areas that you pointed out so that I have enough tools in my toolkit which prevent me from using the wrong tool for the wrong job. For example, I'd hate to use shell scripts to do something that ansible does really well.
A modern solution, fortunately or unfortunately, is built up of multiple smaller tool sets as you pointed out - and, if used correctly, each enhance productivity tremendously. Writing a frontend app (something that I've only recently started doing since I'm on my own) is immensely more productive if working with something like react rather than with a relatively old framework based on the jvm.
What I'm trying to say is - tech we use is ultimately a tool - we should optimise for productivity. In that case, Boring Technology helps being more productive since we know a lot more about it which makes it easier for us to bend it to our will as well as debug/diagnose the unknowns.
But it also doesn't mean we continue to use `grep` when silversearcher/ripgrep is out there in the world :)
That's the lens that I look with when I come across new tools/technologies (regardless of how long they've been around) - do they have the potential to be net-productivity enhancers over a long-ish period of time && by how much (0.5x? 5x?).
Thank you for starting such an interesting thread.
Going through all the comments a few days after the question was asked is very rewarding ! So many books marked as 'to read' :)
My contribution:
If you're not familiar with Quantum Physics, do check out 'Through Two doors at once'. There were numerous instances while reading the book that I had to just put it down and think deeply - mostly philosophical thoughts around what we are and how magical nature is. The subject matter is very very approachable - even to someone like me who hasn't read a physics book in like a decade.
Thank you for this - and the references to how you came across the concept.
I was personally only recently introduced to anki [1] & [2]
However, I tried it out but I couldn't end up using it to 'remember books' or broader concepts that the books convey indirectly. I've started summarising books and using a manual form of spaced repetition to remember them better.
Do you have any advice on organising such knowledge better?
I just take handwritten notes, then convert them into carefully designed and tagged cards in Anki. Even if I forget something, searching my Anki deck will usually not just tell me the information, but what book it came from, and often which page. I'm pretty diligent about careful note collation. I've definitely gotten better since I read "The Organized Mind", a book which was so enlightening for my persona that I built more than 1000 cards to remember as many of its concepts as I could.
Regardless, I'm reading my previous comment and should admit that I'm quite intense about efficiency in learning. Less so about money. I've spent a lot of time tutoring, making this an important subject to me and I get.... emotional. I apologize if my original comment seems rude. It certainly feels that way to me.
reply