This is cool! I don’t see many people doing write ups on their tech stack as much any more. It’s nice to see the inside of a production-grade app like this.
I’m curious, why command+r for the model? What benefits does it have over other SOTA models?
It’s unlikely an individual would need this much capacity. Folks who need tokens at this level are apps with lots of users that don’t have their own GPUs. Think character.ai type apps.
Kuzco, Inc. (https://kuzco.xyz) | Full-stack SWEs and MLE | Full-time | San Francisco, CA
We're building a serverless LLM inference network that makes use of underutilized capacity from GPU data centers. Our product is a scheduler for running LLM inference workloads on GPUs located all over the world. We currently have over 6,000 GPUs on our network, and are growing quickly.
Things we need help with:
* Improve core scheduling algorithms
* Optimize vLLM inference runtime
* Improve logging and observability stack
* Building our user-facing dashboard and APIs
* New products
We're well-funded and have a clear path to profitability. We're currently a four person team of staff-level engineers and looking to add two more engineers to our team. Our office is near the Ferry Building in downtown San Francisco. We pay well and offer significant equity.
Pontificating on what actual profit will be generated by LLM companies is fair game, but reaching a conclusion that LLMs have no mass market utility is blisteringly insane. I could easily measure the increase in value I alone have had from using them in the tens of thousands of dollars.
It's more plausible for me that we will see a notable productivity increase in a lot of sectors of the economy over the next decade. Part of me wonders if this is an additional reason why the Russell 2000 has been spiking lately (investors concluding that there is more money to be made from the general productivity increases in the wider economy than the tech companies providing the LLMs that don't seem to possess any monopolies on the technology), but this is just my speculation.
I think you are misunderstanding my point, and perhaps I should've worded it more-precisely"
"Mass market utility" here refers to its ability to sell at the scale it would need to substantiate its costs. As it stands, LLMs do not have mass-market utility at the scale that they need to substantiate their costs. It is really that simple. If they did, these companies would be profitable, and they would be having a meaningful effect on productivity, which they are not.
And he admitted he is “neither an engineer nor an economist, nor do I have privileged information”.
So why should we listen to him? ChatGPT has saved me a lot of time. Luckily it’s well trained on AWS API’s, tge SDK, CDK, Terraform and Kubernetes. Anything it isn’t trained on, I just give it links to the documentation
If I read you correctly, you are implying that the first statement is obviously false. Can you fill me in on the mass-market utility? Because I keep coming to the same conclusion as the author.
The "killer app", as far as I can tell, is essentially natural language search. However, the core function (in my opinion) has existed since DuckDuckGo added contextual infoboxes to the right of search results ("knowledge panels"), and the benefit of using natural language has existed since Siri and has never seemed to add much to the experience for me. AI image generators seems to be used mostly by youtube creators and spammers. The main users of AI language generation seem to be spammers and crappy content farms on TikTok.
Commercials for AI products ALWAYS lie by speeding up the time it takes for results to arrive, and the most impressive demos always seem to end up as some version of the same useless feature: "what am I looking at right now?" Who needs that? AI-assisted coding also seems to have a similar issue. Demos that supposedly show off the technology never actually use it to create the kind of code that is actually worth money.
I'd be happy to be proven wrong here, but I keep looking and I never find that killer app.
Translation is another. It's uncertain to what extent experienced professional translators are losing work—I work in the field, and I've heard anecdotal evidence on both sides—but it's clear that LLM-driven machine translation is already producing significant value for millions of people around the world. This might not be apparent to monolingual people in English-speaking countries, but it's a major factor driving the rapid adoption of LLMs here in Japan.
If OpenAI's new voice mode turns out to be as versatile and context-aware as promised, it should be equally valuable for interactive spoken interpretation.
There's definitely some value in undercutting minimum-wage outsourced call centres. But enough to justify these valuations? This is stuff that was already being partially automated back in the '80s; I can see the argument that OpenAI is the next Autonomy, but when you're this heavily leveraged an incremental improvement is a failure.
OpenAI is the platform everyone else is building on top of.
Think Microsoft in the late 80s through the 90s.
If you are charging for the platform, you can let thousands of other businesses try the risky hard to scale stuff and you'll get a cut of everything no matter who wins.
Microsoft wasn't taking any risks there though, they were a profitable business selling positive margin projects the whole time - that's more where NVidia is than where OpenAI is.
Or could we have instead done some simple web service that would do most things... The sad reality is that companies don't want easy and efficient customer service... As that would allow customers to cancel their continued payments...
Credit card companies and large banks each have call centers with multiple tens of thousands of employees.
> The sad reality is that companies don't want easy and efficient customer service... As that would allow customers to cancel their continued payments...
This is not true at all. The true reason is that a well trained call center employee can easily cost a company $20 per customer issue resolved (total cost inclusive of training, office space, equipment, etc).
For any low margin business (e.g. hardware under $500 USD!) that basically destroys the entire profit from that customer.
Uh... you realize they were already algorithms, right? Meaning, they have a flow chart they follow. They don't make any decisions, they take input and respond to output.
The only reason they're not computer programs is because they NEED to be human. So the human on the other end trusts them. Even though everyone understands they have no free will or reasoning abilities (or if they use them they get canned).
AI doesn't fit that use case. Number 1 is because AI IS NOT algorithmic. So it's a liability to use. Number 2 is it's not human. Again, if you're going the no human route there's infinite cheaper, more reliable, faster, and overall better in every way programs you can use.
> what am I looking at right now?" Who needs that?
Anyone who is sight impaired, or doesn’t have their glasses on while reading a menu, or looking at a sign in another language.
It’s clearly not in final form. In 1978, people couldn’t see the use of home computers. In 1988, most people were saying the same thing about email. In 1998, most people were saying the same thing about the internet.
It might not prove out, but evaluating something super early isn’t all that interesting. Let’s see where we are in 2030 at least.
> It’s clearly not in final form. In 1978, people couldn’t see the use of home computers. In 1988, most people were saying the same thing about email. In 1998, most people were saying the same thing about the internet.
Give it a few more years for more people to forget about how hard things like NFTs got pushed with the same sort of arguments- then this'll come off better.
That one technology worked does not automatically mean any other technology will. There is no argument that begins "A worked and so B will too". Also, email is pretty much dead as a technology - it's just spam and password reset emails at this point.
I think parent was not saying new tech will succeed, merely there is an extended hangover/negativity due to recent busts.
Regarding email - it's probably one of the most used bits of technology in existence, and I bet it has a similar OOM economic impact to something like Excel. Calling it a dead technology is not accurate - so much business is done via email, especially internally at SMB.
I realize that these are commonly-held tropes, but where is the actual article that says this? There's the famous "the internet isn't a big deal" thing in Newsweek (and funnily enough that piece is extremely prescient in many other ways!), but I don't know if I've seen the kind of hype-busting then.
But also...the reason they might have is that email kind of sucked back then. Of course you wouldn't see the promise in something that was clunky and slow and nobody used.
This isn't a comparable to LLMs, though, because even someone who found them clunky could see why you'd want to send an email versus sending a letter.
Or 5%, right? What's the minimum amount that's "killer", on a societal level and on an industry-specific level? Human transcription might be a good case study.
Actually, I would expect the robots to get better at blue collar work faster than LLMs at white collar work. The reason is that there are a lot of highly repetitive physical skills for which you can easily build a dataset for and then fine-tune a model.
People have built dedicated plastering robots, strawberry picking robots etc. Each robot only really needs to be good at one thing. Almost nobody is making the foolish mistake of building humanoid robots that then tackle the problem exactly the way humans do. The plastering robots spray the plaster, which is much faster.
Don’t worry you’re not alone. I’ll believe it when I see it. So far I haven’t seen it. And I work in AI (albeit at a low level).
If I can compare it to Google search back in the 90s. There weren’t all these evangelists saying in 5-10 years blah blah blah. We just used it, in our daily lives. I don’t use AI at all except at work. And I couldn’t even tell you what product it’s going toward because I don’t think 99% of employees know. Why we don’t know, who knows!
There absolutely were hordes of evangelists touting this or that tech company (including google, redhat, amazon etc as well as pets.com and a bunch of others who didn't survive) saying in 5-10years these were companies would be the biggest thing since sliced bread. In some cases they were right.
The VC money is betting on a similar sort of monopolistic dynamic occurring in the AI space. They're not saying that 100s of billions of dollars of value is going to accrue to openAI because of chatGPT they're saying 100s of billions of dollars of value is going to accrue in the space and the most likely outcome is that the vast majority of that is going to be siphoned up by a couple of behemoths. Other than the incumbents (Microsoft, Google) OpenAI seems best positioned at the moment and whoever else creates breakthroughts will probably be acquired anyway just like instagram got bought out by facebook as soon as they got large enough to matter.
> saying in 5-10years these were companies would be the biggest thing since sliced bread
It's not about growth, it's that nobody had to make future claims about web search or online ordering being useful to the average person, they already were useful when someone tried them.
Ironically enough, if AI powered code generation actually worked, OpenAI would have a business model right there. But it doesn't, and so they don't. What they hope is that other people will somehow make their business model for them - and then they'll sell their technology into that.
But AI code generation is garbage, just like AI text generation is. AI generated books, and songs, and poetry etc is just appalling rubbish that nobody wants to read or hear. AI generated art is ugly and generic, and stands out immediately as zero-effort and near-zero cost. Except of course, it cost alot of money to make, but nobody is willing to pay for it and it's so far been bankrolled by VC capital.
"create an express server that uses CORS to only accept requests from <mydomain> that has 3 async endpoints defined, 2 GET endpoints titled <a, b> and 1 POST enndpoint titled <c>. The POST endpoint should accept JSON, and the GET endpoints will return JSON. Stub out the route handlers, they will be defined in another file. For the POST endpoint parse out the userId from <header field> and also pass it in along with the JSON"
"Here is a typescript definition of the JSON being posted to this endpoint, validate all string fields are less than 100 characters long, and that none of the values are null, then write the data to a redis DB at <___location> using auth values already loaded up in the environment. Data should be written to <userid/___location>"
"Now add exception and error handling around this code, cover the following case: Data is invalid - Return an error message to the user with HTTP error code 400, Redis is not accessible - HTTP 500. Also log errors using the <logging library> that has already been imported and initialized up above, and increment an appropriately named counter using the <analytics library> that is also already imported and initialized."
If you are getting garbage AI code, you are not using proper tools. If you just go to ChatGPT and ask it for stuff, you won't get very good code (and IMHO GPT4o is much worse at coding).
Use a proper editor that prompt engineers code generation (such as Cursor), and LLMs can write some really good code.
Can it end to end generate an entire service with just a vague description? No. But it can easily 2x-4x development speed. Heck I've thrown code at GPT4 and asked it to "please look for common concurrency bugs that may have been overlooked in this code" and watched as GPT4 saved me literal hours of debugging.
What OpenAI has is a problem monetizing the value their product brings, since the value to each customer is so different. When I'm at home, I'm happy to pay $15 a month in API usage fees to accelerate my coding, but a company should be happy to pay hundreds of dollars per month to accelerate the speed code is written. Figure $100 an hour average cost for a dev on the east or west coast of the USA, if GPT4 saves 10 hours a month (easy!) then OpenAI should be charging at least $200-$500 a seat licenses per month.
Agent for X is valuable and lots of startups are trying to build it in some vertical. As these agents evolve, they have the potential threaten incumbents in their own respective vertical. It’s also unclear how we’ll orchestrate all the agents or discover them. None of this is very obvious and it’s very possible to hit a major tech roadblock. But nobody knows.
No serious company is claiming AGI is around the corner. There's going to be scams like Rabbit, Friend and similar, but there are scams for everything.
Agents are evolving though. The coding agents a few months ago were a joke (Devin) and now they're actually usable for basics.
That's what I meant - Devin was a joke, I've seen the promo and didn't use it. But I experimented with Plandex recently and it's usable for various small tasks. Not great, but usable if you make the right choices and getting better.
The killer app is the chatbot itself? Fastest app adoption / scaling ever? Like I think people are looking for an 'and then' to be sold, but it's unclear why that has to be the case.
Awesome project! I’m sure someone would be willing to buy this from you if the traffic is big enough. If you want to move on to other engineering projects this may be best path. If you’re interested in learning sales and marketing yourself I would look into affiliate marketing with hair loss product companies. Happy to chat more about this if you want help.
Completely agree. Explaining how systems work in plain english is much more valuable than just giving inputs and outputs of individual functions. We want to understand how a system and it's subsystems work independently and interdependently.
We're not there yet with Autodoc; there is still tons of work to do.
If you have't tried the demo, give it a shot. You might be surprised.
Have you considered finding a way to instead write a text editor plugin that allows me to talk to a GPT and ask questions about a codebase? This would be a serious technology that moves beyond the in-code documentation paradigm.
I’m not prepared to install this at the moment, but if you were to give an example of costs for what I suggested above, it might convince me to try it out
I’m curious, why command+r for the model? What benefits does it have over other SOTA models?