I love the assumption that an ubiquitous feature used by the most scaled e-mail app in the world uses the same expensive state of the art model that the author of the blog uses.
My money would be that the gmail model is heavily distilled to reduce cost, reducing its flexibility for user-level detailed system prompts.
The problem the author tackles with is a well known one in machine learning - and nothing really new. I do agree that a world in which we allow per-user system fine-tuning of models that have a scaled utility through a large number of tasks for a single user, but that only works for apps that have a high frequency of usage. It doesn’t make sense to system prompt an app you use rarely.
And you can’t ignore costs, especially as all the commercially available API’s right now operate at cost, skewing the perception to the end-user (end-developer?) of how much it costs to run ai in a scaled setting.
I do agree with the horseless carriage thing do, it’s a neat mental model for what is likely happening.
My money would be that the gmail model is heavily distilled to reduce cost, reducing its flexibility for user-level detailed system prompts.
The problem the author tackles with is a well known one in machine learning - and nothing really new. I do agree that a world in which we allow per-user system fine-tuning of models that have a scaled utility through a large number of tasks for a single user, but that only works for apps that have a high frequency of usage. It doesn’t make sense to system prompt an app you use rarely.
And you can’t ignore costs, especially as all the commercially available API’s right now operate at cost, skewing the perception to the end-user (end-developer?) of how much it costs to run ai in a scaled setting.
I do agree with the horseless carriage thing do, it’s a neat mental model for what is likely happening.