For people finding this thread via web search in the future:
screen.studio is macOS screen recording software that checks for updates every five minutes. Somehow, that alone is NOT the bug described in this post. The /other/ bug described in this blog is: their software also downloaded a 250MB update file every five minutes.
The software developers there consider all of this normal except the actual download, which cost them $8000 in bandwidth fees.
To re-cap:
Screen recording software.
Checks for updates every five (5) minutes. That's 12 times an hour.
I choose software based on how much I trust the judgement of the developers. Please consider if this feels like reasonable judgement to you.
Yea, it seems like the wrong lesson was learned here: It should have been "Don't abuse your users' computers," but instead it was, "When you abuse your users' computers, make sure it doesn't cost the company anything."
$8000 for 2 petabytes of traffic is pretty cheap for them also.
There are plenty of shitty ISPs out there who would charge $$ per gigabyte after you hit a relatively small monthly cap. Even worse if you're using a mobile hotspot.
I would be mortified if my bug cost someone a few hundred bucks in overages overnight.
It got one of their customers booted off of their ISP; they did cover that person's overage costs though (and hopefully that person could get their account back).
I work with developers in SCA/SBOM and there are countless devs that seem to work by #include 'everything'. You see crap where they include a misspelled package name and then they fix it by including the right package but not removing the wrong one!.
The lack of dependency awareness drives me insane. Someone imports a single method from the wrong package, which snowballs into the blind leading the blind and pinning transitive dependencies in order to deliver quick "fixes" for things we don't even use or need, which ultimately becomes 100 different kinds of nightmare that stifle any hope of agility.
In a code review a couple of years ago, I had to say "no" to a dev casually including pandas (and in turn numpy) for a one-liner convenience function in a Django web app that has no involvement with any number crunching whatsoever.
Coincidentally, Copilot has been incredibly liberal lately with its suggestions of including Pandas or Numpy in a tiny non-AI Flask app, even for simple things. I expect things to get worse.
There's a ton you can do with sqlite, which is in the Python standard library. You just have to think about it and write some SQL instead of having a nice Pythonic interface.
To push back on this, I consider pandas/numpy so crucial to Python as a whole they are effectively stdlib to me. I wouldn't blink at this because it would happen sooner or later.
Unless is was absolutely critical the server have as small as a footprint as humanly possible and it was absolutely guaranteed there would never need to be included in the future of course. However, that first constraint is the main one.
>> their software also downloaded a 250MB update file every five minutes
> How on earth is a screen recording app 250 megabytes
How on earth is a screen recording app on a OS where the API to record the screen is built directly into the OS 250 megabytes?
It is extremely irresponsible to assume that your customers have infinite cheap bandwidth. In a previous life I worked with customers with remote sites (think mines or oil rigs in the middle of nowhere) where something like this would have cost them thousands of dollars per hour per computer per site.
For a long time iOS did not have features to limit data usage on WiFi. They did introduce an option more recently for iPhone, but it seems such an option is not available to MacOS. Windows supported it as far as I could remember using it with tethering.
Or.. Why on earth you need to check for updates 288x per day. It sounds and seems more like 'usage monitoring' rather than being sure that all users have the most recent bug fixes installed. What's wrong with checking for updates upon start once (and cache per day). What critical bugs or fixes could have been issued that warrant 288 update checks.
8 of the 10 Gbits I meant :) sorry I see it reads a bit weird yes. So 8 gbit for single line is the max currently. But huge competition on the horizon, so I expect soon more :)
I read it as 1 x 8GB connection
But that’s only because I think 8GB is offered in my area.
I’ve limited my current service to 1.5GB / 1GB fibre, because well I only run gbit Ethernet … so more sounds totally unnecessary
It sounds right, and this is the kind of thing I'd expect if developers are baking configuration into their app distribution. Like, you'd want usage rules or tracking plugins to be timely, and they didn't figure out how to check and distribute configurations in that way without a new app build.
They probably just combined all phoning home information into one. Usage monitoring includes version used, which leads to automatic update when needed (or when bugged...).
Is it normal to include the Electron framework like that? Is it not also compiled with the binary? Might be a stupid question, I'm not a developer. Seems like a very, very heavy program to be doing such a straightforward function. On MacOS, I'm sure it also requires a lot of iffy permissions. I think I'd stick with the built-in screen recorder myself.
I'm not sure why there are both app.asar and app.asar.unpacked. I did just run `npx @electron/asar extract app.asar` and confirmed it's a superset of app.asar.unpacked. The unpacked files are mostly executables, so it may have something to do with code signing requirements.
Tauri is not as platform-agnostic as Electron is because it uses different web views depending on the platform. I ran into a few SVG-related problems myself when trying it out for a bit.
For example, on Linux, it uses WebKitGTK as the browser engine, which doesn't render the same way Chrome does (which is the web view used on Windows), so multi-platform support is not totally seamless.
Using something like Servo as a lightweight, platform-independent web view seems like the way forward, but it's not ready yet.
I suspect the real reason electron got used here is that ChatGPT/Copilot/whatever has almost no Tauri example code in the training set, so for some developers it effectively doesn't exist.
It's about time Linux desktops adopt some form of ${XDG_WEB_ENGINES:-/opt/web_engines} convention to have web-based programs to fetch their engines as needed and play nice with each other.
It's relevant in the broader context, cross-platform is a significant reason people choose Electron, and lighter alternatives like Tauri still have some issues there.
seconded -- tried to use tauri for a cross-platform app but the integrated webview on android is horrendous. had to rewrite basic things from scratch to work around those issues, at which point why am I even using a "cross-platform" framework?
I imagine if you stick to desktop the situation is less awful but still
Screen recording straight from a regular browser window, though it creates GIFs instead of video files. Links to a git repo so you can set it up locally.
The app itself is probably much bigger than 250mb. If it is using Electron and React/other JS library like a million other UIs just the dependencies will be almost that big.
For context, the latest iOS update is ~3.2GB, and the changelog highlights are basically 8 new emojis, some security updates, some bug fixes. It makes me want to cry.
I would be so embarrassed about this bug that I would be terrified to write it up like this. Also admitting that your users were forced to download 10s or 100s of gigabytes of bogus updates nearly continuously. This is the kind of thing that a lot of people would just quietly fix. So kudos (I guess) to blogging about it.
Not dialup. Just bad last-mile wiring, as far as I can tell.
Apparently such service is still somehow available; I found https://www.dialup4less.com with a web search. Sounds more like a novelty at this point. But "real" internet service still just doesn't work as well as it's supposed to in some places.
Canada. But yes, I've heard the stories about Germany, and Australia too.
In point of fact, I can fairly reliably download at that rate (for example I can usually watch streaming 1080p video with only occasional interruptions). The best case has been over 20Mbit/s. (This might also be partly due to my wifi; even with a "high gain" dongle I suspect the building construction, physical ___location of computer vs router etc. causes issues.)
Microsoft InTune WUDO has a similar bug costing my department 40000 € internal charging per month for firewall log traffic of blocked tcp 7680 requests. 86000 requests per day per client, 160 million per day total. MS confirmed the bug but did nothing to fix it.
They are building features right now. There are a lot of bugs which Microsoft will never fix, or it fixes them after years. (Double click registered on mouse single clicks, clicking "x" to close the window, closes also the window underneat, GUI elements rendered as black due to monitor not recognized etc).
Yes, all packets get logged (metadata only). Otherwise we wouldn’t know there is an issue.
Those packets consume bandwidth and device utilization, too but this is flat fee, whereas log traffic is measured per GB so we investigated where an unexpected growth came from.
Little Snitch catches these update request checks and I realize now that it should have an additional rule meta which is *how often* this endpoint request should be allowed (LS should allow throttling not just yes / no)
Obviously five minutes is unnecessarily frequent, but one network request every five minutes doesn't sound that bad to me. Even if every app running on my computer did that, I'm not sure I'd notice.
People complaining about 5 minute update checks hopefully don't use Windows 10/11.
A while ago I did some rough calculations with numbers Microsoft used to brag about their telemetry, and it came out to around 10+ datapoints collected per minute. But probably sent in a lower frequency.
I also remember them bragging about how many million seconds Windows 10 users used Edge and how many pictures they viewed in the Photo app. I regret not having saved that article back then as it seems they realized how bad that looks and deleted it.
Try installing Adobe's Creative Cloud and/or any of its related products. I ultimately setup an instance of AdGuard just to block Adobe's insane telemetry traffic.
When I built an app that “phones home” regularly, I added the ability for the backend to respond to the client with an override backoff that the client would respect over the default.
Seems like the proper fix would have been to remove the file from the server when they realized the increased traffic. Then clients would just fail to check the update each time and not tie up bandwidth.
Why not just use http retry-after? then you can use middleware/proxy to control this behavior. Downside here is that system operation becomes more opauqe and fragmented across systems.
The client might have a feature to retry certain failures, and it’s using a particular rate, probably not retrying n times one right after the other in rapid succession. This is called backoff.
The server can return an override backoff so the server can tell the client how often or how quickly to retry.
It’s nice to have in case some bug causes increased load somewhere, you can flip a value on the server and relieve pressure from the system.
Exactly. Without going too deep into the architecture, the clients are sending data to the backend in real time, but often that data is not actionable during certain periods, so the backend can tell the clients to bundle the data and try again after a certain amount of time, or just discard the data it's currently holding and try again later (i.e. in 5/10/n seconds)
Presumably the back end could tell the client not to check again for some amount of time. Sounds similar but different to cache TTLs, as those are passive.
Several months ago I was dealing with huge audio interruption issues - typical sign of some other, blocking, high-priority process taking too long.
Turns out Adobe's update service on Windows reads(and I guess also writes) about 130MB of data from disk every few seconds. My disk was 90%+ full, so the usual slowdown related to this was occurring, slowing disk I/O to around 80MB/s.
Disabled the service and the issues disappeared. I bought a new laptop since, but the whole thing struck me as such an unnecessary thing to do.
I mean, why was that service reading/writing so much?
Thats only half as bad as a certain company that had all thier users download an unwanted OS upgrade on the theory that one day they might click the install button by accident.
"We will stop filling your drives with unwanted windows 14 update files to you once you agree the windows 12 and 13 eulas and promise to never ever disconnect from the internet again."
I don’t know this software, but my sense is that this would be exactly the type of desired functionally in order to bypass rejected user metric sharing by parsing update request metrics, but maybe you are right and the Developers really do believe you can’t go more than 5 minutes on an out-of-date version…
No, those are abstraction over a TCP socket and introduce more complexity than you'd need for something like this. There's nothing wrong with occasionally polling for updates.
Web push, FCM, APNS, etc are free because they only have a single systemwide TCP channel open - and that channel is already open whether or not your app chooses to use it.
Your app can also be ready to receive notifications even when the app isn't running - using zero RAM. Inetd on Linux allows similar stuff (although no ability to handle ip changes or traverse NAT makes it fairly useless in the consumer world).
This stuff is important because polling dominates power use when idle - especially network polling which generally requires hundreds of milliseconds of system awakeness to handle tens of network packet arrivals simply for a basic http request.
Did you know, a typical android phone, if all polling is disabled, has a battery life of 45 days?
That's a lot of abstraction to implement a simple update check and I suspect it's very much not worth it to save a minuscule amount of battery life on a laptop. This is ignoring that you're incorrect about how Web Push works, so you'd need that extra TCP connection anyway and at that point, there's no point in bothering with Web Push. FCM is the same deal (and now you get to pay for extra stuff for a trivial problem, woo) and APN seems like the wrong solution to auto-updates.
> To re-cap: Screen recording software. Checks for updates every five (5) minutes. That's 12 times an hour.
The tone might be somewhat charged, but this seems like a fair criticism. I can’t imagine many pieces of software that would need to check for updates quite that often. Once a day seems more than enough, outside of the possibility of some critical, all consuming RCE. Or maybe once an hour, if you want to be on the safe side.
I think a lot of people are upset with software that they run on their machines doing things that aren’t sensible.
For example, if I wrote a program that allows you to pick files to process (maybe some front end for ffmpeg or something like that) and decided to keep an index of your entire file system and rebuild it frequently just to add faster search functionality, many people would find that to be wasteful both in regards to CPU, RAM and I/O, alongside privacy/security, although others might not care or even know why their system is suddenly slow.
For contrast: Chrome, a piece of software which has a huge amount of attackable surface area, and lives in a spot with insane stakes if a vulnerability is found, checks for updates every five hours, last I read.
Noone is commenting on the actual bug. The fact that it auto downloads 250mb updates is user-hostile. On top of that, checking every 5 minutes? What if I'm on a mobile connection?
Why not just follow every Mac app under the sun and prompt if there's an update when the app is launched and download only if the user accepts?
I think the critique here is not directed to 1 individual, the guy who actually wrotw the code. That would be ok, can happen. Here we are talking about the most valued company in the world, which hopefully has many architects, designers and literally an army of testers… and then make such a brutal error.
> Screen Studio is a screen recorder for macOS. It is desktop app. It means we need some auto-updater to allow users to install the latest app version easily.
No, it doesn't mean that.
Auto updater introduced series of bad outcomes.
- Downloading update without consent, causing traffic for client.
- Not only that, the download keeps repeating itself every 5 minutes? You did at least detect whether user is on metered connection, right... ?
- A bug where update popup interrupts flow
- A popup is a bad thing on itself you do to your users. I think it is OK when closing the app and let the rest be done in background.
- Some people actually pay attention to outgoing connections apps make and even a simple update check every 5 minutes is excessive. Why even do it while app is running? Do on startup and ask on close. Again some complexity: Assume you're not on network, do it in background and don't bother retrying much.
- Additional complexity for app that caused all of the above. And it came with a price tag to developer.
Wouldn't app store be perfect way to handle updates in this case to offload the complexity there?
App store updates are perfect: no unnecessary complications, no unnecessary work (assuming Screen Studio is published and properly updated in the app store), and the worst case scenario is notifications about a new Screen Studio version ending up in a Screen Studio recording in progress.
Thinking of it, the discussed do-it-yourself update checking is so stupid that malice and/or other serious bugs should be assumed.
Exactly. The AppStore already exists and does updates (either automatically or manually, configurable by the user). The developer didn't have to lift a finger to get this functionality. Imagine sitting down and spending time adding functionality to your application that is already provided for free by the operating system, and then after all that, doing it incorrectly!
Starting from the paid developer accounts, the Apple app store isn't "provided for free by the operating system" and it is a source of endless busywork, fear and suffering, but the argument stands: a professional Macintosh software vendor uses the app store because Macintosh users expect it, so it can be assumed that "properly" publishing new software version to the app store is a sunken cost that should be made as useful as possible.
By "provided for free" I mean the App Store comes with the OS, costs nothing (monetarily) to the developer over the existing annual Apple Developer Program fee, which pretty much all macOS developers pay anyway, and can be counted on to exist on all macOS installations.
> malice and/or other serious bugs should be assumed
Going back to the blog post and re-reading it with this possibility in mind is quite a trip.
> It turns out thousands of our users had the app running in the background, even though they were not using it or checking it for weeks (!). It meant thousands of users had auto-updater constantly running and downloading the new version file (250MB) over and over again every 5 minutes
This could easily have been data exfiltration from client computers instead, and few (besides the guy whose internet contract got cancelled for heavy traffic) would have noticed.
I find the misbehavior of indie/boutique MacOS apps always insisting on starting at login very irritating. Unless the app needs to run some heavy background preparation steps before becoming usable, there is literally no sense it starting at login. Also when dormant, check for update(once every 24h), and nag the user if they want to update, but please do not auto download! A lot of non-tech folks use 128/256GB versions of macbook with trillions of photos already clogging their device, an app downloading new updates to add to the pain unless the user asks to do so is outright malice.
Yeah no, publishing to the App Store is a nightmare in cost and time. I can 100% guarantee they still saved money on 30% fees even after this $8000 snafu.
Screen Studio has 32k followers, lets say 6% are end users, 2000 users at $229, that is $137k in App Store fees.
I am going to say writing your own app update script is a wash time wise, as getting your app published is not trivial, especially for an app that requires as many permissions as screen studio.
Some people don’t like using the AppStore. I like to keep backups of installers so I can control the version. And if it gets pulled from the AppStore, I’ll always have a copy.
If you’re a small shop or solo dev, it is real hard to justify going native on three platforms when electron gives it for (near) free. And outside of HN, no one seems to blink at a 250MB bundle.
There are alternatives like Tauri that use the system browser and allow substantially smaller bundles, but they’re not nearly as mature as Electron, and you will get cross platform UI bugs (some of which vary by user’s OS version!) from the lack of standardization.
I’d actually seen this project before because the author did a nice write up on using React portal to portal into electron windows[1], which is something I decided to do in my app.
I’d just assumed his was a cross platform project.
I’m not excusing this bug. There are several poor decisions that went into this issue, but my contention is that using electron (with the resulting 250mb bundle) is not one of them.
You should probably not roll your own auto-updater.
If you do, checking every 5 minutes for updates is waaaay too often (and likely hurts battery life by triggering the radio).
And triggering a download without a user-prompt also feels hostile to me.
The app size compounds the problem here, but the core issue is bad choices around auto-updating
Different versions of electron bundle different versions of chromium. There can/will be rendering differences between them.
Tauri is an alternative framework that uses whatever web view the OS provides, saving ~200mb bundle size. On Mac that’s a (likely outdated) version of Safari. On Windows it’ll be Edge. Not sure what Linux uses, I’d guess it varies by distro.
The promise of Electron (and it’s an amazing value prop) is that your HTML/JS UI will always look and work the same as in your dev environment, no matter what OS the host is running.
I don’t have the time or inclination to test my app on the most recent 3 releases of the most popular operating systems every time I change something in the view layer. With Electron, I trade bundle size for not having to do so.
I do think alternatives like Tauri are compelling for simple apps with limited UI, or where a few UI glitches are acceptable (e.g. an internal app). Or for teams that can support the QA burden.
I go into more detail in a sibling comment, but Tauri does not provide a standardized web runtime. The webview you get depends on your OS and OS version. They’re all “WebKit”, but definitely do not all render the same. I have built a Tauri app and switched to Electron after encountering multiple x-plat rendering bugs.
And even when nothing changed?!? Fucking lazy developers aka "I have an idle ≥1Gb/s pipe to the download server". What happened to rsync/zsync/zstd (with dictionary)? There are so many good tools freely available to reduce wasted bandwidth when you insist on reinventing the wheel. sigh
Per [1] That would be 39MB (uncompressed; probably about half that compressed) to include ffmpeg-darwin-arm64, since OS X doesn't ship with ffmpeg installed.
FWIW the transitive dependencies of the nixOS ffmpeg add up to 764MB, but dynamically linking is always much larger than statically linking, and that calculation will include more than just the shared-libraries.
Also note that he app includes an ffmpeg that is 39MB uncompressed.
That was a thing I thought was missing from this writeup. Ideally you only roll up the update to a small percent of users. You then check to see if anything broke (no idea how long to wait, 1 day?). Then you increase the percent a little more (say, 1% to 5%) and wait a day again and check. Finally you update everyone (who has updates on)
yes obviously something as mature as the App store supports phased rollout. I believe it is even the default setting once you reach certain audience sizes. Updates are always spread over 7 days slowly increasing the numbers
> Wouldn't app store be perfect way to handle updates
But then the HN crowd would complain "why use an app store? that's gate keeping, apple could remove your app any day, just give me a download link, and so on..."
You can? Don’t check for updates every 5 minutes. Daily or even weekly would be sufficient for an app like this (if auto-updater is even necessary at all.. just show a notification)
I find it ludicrous that the developers of an app as insignificant as a screen recorder would think it's necessary to check for updates every 5 minutes.
The big clouds love these people too. So much of the software industry is just an outrageous combination of inexperience and "YOLO". Every problem can be solved by just giving AWS another $100,000 this month because we don't have time (and don't know how) to make even basically optimized software. So just burn the gas and electricity and give more money to the YAML merchants at Amazon.
Data centers are big and scary, no body wanted to run their own. The hypothetical cost savings of firing half the IT department was too good to pass up.
AWS even offered some credits to get started, first hit's free.
Next thing you know your AWS spend is out if control. It just keeps growing and growing and growing. Instead of writing better software, which might slow down development, just spend more money.
Ultimately in most cases it's cheaper in the short term to give AWS more money.
Apart of me wants to do a 5$ VPS challenge. How many users can you serve with 5$ per month. Maybe you actually need to understand what your server is doing ?
I work for a big org. We’re about to deploy a small grafana setup for monitoring some test environments. Double digit spend per month, maximum. It’s pretty close to impossible to get IT, infosec, purchasing and finance to agree to this in a period of time that I’ll still be employed (and I’m not planning on leaving).
But, on the AWS marketplace I can click a button, a line item is added to our bill, and infosec are happy because it’s got the AWS checkmark beside it. Doesn’t matter what it costs, as long it goes through the catalog.
That’s why big companies use AWS.
At my last job, I worked for a vc backed startup. I reached out to our fund, and they put us in touch with AWS, who gave us $100k in credits after a courtesy phone call.
We are throwing everything under the bus, including the user's battery, CPU, memory, bandwidth, the company's cloud costs and energy usage, just so developers can crap out software slightly faster.
We are providing users with valuable features at a faster rate, saving them and us time, which is a far more valuable asset than "the user's battery, CPU, memory, bandwidth, the company's cloud costs and energy usage".
Doing 'the cloud' right at scale has to involve running your own cloud at some point. We should not pollute the good ideas around API-delivered infrastructure with the more questionable idea of outsourcing your infrastructuree.
OpenStack has been around 15 years powering this idea at scale for huge organizatons, including Wal-Mart, Verizon, Blizzard and more.
Not really. I run several web applications on one 15$ VPS. Although the user count is <5. But I think it would need quite a lot of users for the resource usage to go up to a critical level.
> Every problem can be solved by just giving AWS another $100,000 this month because we don't have time (and don't know how) to make even basically optimized software.
Don't forget the Java + Kafka consultants telling you to deploy your complicated "micro-service" to AWS and you ending up spending tens of millions on their "enterprise optimized compliant best practice™" solution which you end up needing to raise money every 6 months instead of saving costs as you scale up.
Instead, you spin up more VMs and pods to "solve" the scaling issue, which you lose even more money.
Oh, cloud native. For a few years people used to look at you funny if you were ...gasp... using battle-tested open source software instead of the overpriced AWS alternative. I'm so glad we're finally seeing pushback.
This is overblown fear mongering, especially for desktop apps.
There are only a few applications with exposed attack surface (i.e. accept incoming requests from the network) and a large enough install base to cause "massive damage all of the Internet". A desktop screen recorder app has no business being constructed in a manner that's "wormable", nor an install base that would result in significant replication.
The software that we need the "average user" to update is stuff like operating systems. OS "manufacturers" have that mostly covered for desktop OS's now.
Microsoft, even though their Customers were hit with the "SQL Slammer" worm, doesn't force automatic updates for the SQL Server. Likewise, they restrict forcing updates only to mainstream desktop OS SKUs. Their server, embedded, and "Enterprise" OS SKUs can be configured to never update.
Hrm... might depend on the purpose of the update. "New feature X" announcements every few days... I hate and disable. "Warning - update now - security bug"... I want to be notified of those pretty quickly.
Ironically the only real call for an update check every 5 mins would be so you can quickly fix a problem like everyone downloading the update every 5 mins.
I honestly lost so much respect for the author after reading this that I completely bailed on the article. Every 5 minutes update check is entirely unhinged behaviour.
It's a fucking screen recorder. Why does it need to check for updates more than once a month? Why does it need to check for updates at all? It's an appliance. It either records your screen, or it doesn't.
I would also put into question if you _really_ need to check for updates every 5 minutes. Once per startup is already enough, and if you're concerned about users who leave it on for days, it could easily be daily or even less often.
A 5 minute update check interval is usage-reporting in disguise. Way fewer people would turn off a setting labeled “check for updates” than one labeled “report usage statistics”.
Eh, this one is probably ignorance over malice. It's super common to see people who need to make an arbitrary interval choice go with 300 out of habit.
Screen Studio can collect basic usage data to help us improve the app, but you can opt out of it during the first launch. You can also opt out at any time in the app settings.
In fact, assume the opposite unless you have a reason to assume otherwise (aka a close personal relationship). Giving strangers/businesses that you have no connection to the benefit of the doubt when they harm you is a good way to get taken advantage of.
Never contort your reasoning to attribute to incompetence what is much better explained by malice. Especially when politics or money is involved, malice should be the assumed default.
Their users do not care about their screen recording studio anywhere near as much as the devs who wrote it do.
Once a month is probably plenty.
Personally, I disable auto-update on everything wherever possible, because the likelihood of annoying changes is much greater than welcome changes for almost all software I use, in my experience.
To be as user friendly as possible, always ask if user wants automatic background updates or not. If you can’t update without user noticing it, please implement manual updates as two mechanisms:
1) Emergency update for remote exploit fixes only
2) Regular updates
The emergency update can show a popup, but only once. It should explain the security risk. But allow user to decline, as you should never interrupt work in progress. After decline leave an always visible small warning banner in the app until approved.
The regular update should never popup, only show a very mild update reminder that is NOT always visible, instead behind a menu that is frequently used. Do not show notification badges, they frustrate people with inbox type 0 condition.
This is the most user friendly way of suggesting manual updates.
You have to understand, if user has 30 pieces of software, they have to update every day of the month. That is not a good overall user experience.
> You have to understand, if user has 30 pieces of software, they have to update every day of the month. That is not a good overall user experience.
That's not an user issue tho, it's a "packaging and distribution of updates" issue which coincidentally has been solved for other OS:es using a package manager.
I'd also question if the updater needs to download the update before the user saying they want it. Why not check against a simple endpoint if a newer version is available and if so, prompt the user that an update could be downloaded and then download it. This would also allow the user to delay the update if they are on metered connections.
First thing I thought as well. Every 5 minutes for a screen recording software is an absurd frequency. I doubt they release multiple new versions per day.
It's near-instant now not usually because of more incessant polling, but because it simply keeps the connection open (can last many hours without sending a single byte, depending also on the platform) and writes data onto it as needed (IMAP IDLE). This has gotten more efficient if anything
In the early days, it was a long distance call to either Lisbon or Porto, I only got a modem, when regional numbers were available to the district capitals.
However on BBS days was much worse, it was mostly long distace calls to someone around the country, and they usually only had a couple of connections available like five or so.
Ah another thing is that they adopted the same model as mobile phones, so at least we could pre-pay the calls, and went we run out of cash there was it, no surprise bills, even if frustated.
Yeah but that should be a variable anyways. Maybe even a variable provided by the server. But in this case it should be on demand. with the old version cached and only downloading the new one when there is a new version once a day.
I am always kind of a stickler about code reviews. I once had a manager tell me that I should leave more to QA with an offhand comment along the lines of "what is the worst that could happen" to which I replied without missing a beat "We all lose our jobs. We are always one bad line of code away from losing our jobs"
The number of times I have caught junior or even experienced devs writing potential PII leaks is absolutely wild. It's just crazy easy in most systems to open yourself up to potential legal issues.
Code reviews kill velocity - introduce context switching, and are make work, it feels like you’re doing something to make a PR etc but your not.
The context it makes the most sense is accepting code from strangers in a low trust environment.
The alternative to trying to prevent mistakes is making it easy to find and correct them. Run CI on code after it’s been merged and send out emails if it’s failed. At the end of a day produce a summary of changes and review them asynchronously. Use QA, test environments, etc.
This feels like a strange sense of priorities which would be satirised in a New Yorker/Far Side single-panel comic: “Sure, my mistake brought down the business and killed a dozen people, but I’m not sure you appreciate how fast I did it”.
Code should be correct and efficient. Monkeys banging their heads against a keyboard may produce code fast, but it will be brittle and you’ll have to pay the cost for it later. Of course, too many people view “later” as “when I’m no longer here and it’s no longer my problem”, which is why most of the world’s software feels like it’s held together with spit.
> would be satirised in a New Yorker/Far Side single-panel comic:
Thanks for taking my experience and comment seriously and challenging your preconceptions.
> Code should be correct and efficient.
When it ships to customers. The goal is to find the bugs before then. Having a stable branch can be accomplished in many ways besides gating each merge with a review.
Do you have any studies to show how effective synchronous code review is in preventing mistakes? If they are such a good idea why not do 2 or 3?
> Thanks for taking my experience and comment seriously and challenging your preconceptions.
I apologise if my comment read as mean. I wanted to make the joke and it may have overshadowed the point.
> Do you have any studies to show how effective synchronous code review is in preventing mistakes?
I could’ve been clearer. I’m not advocating for code reviews, I’m advocating for not placing “velocity” so high on the list of priorities.
> If they are such a good idea why not do 2 or 3?
This argument doesn‘t really make sense, though. You’ve probably heard the expression “measure twice, cut once”—you don’t keep measuring over and over, you do it just enough to ensure it’s right.
> Thanks for taking my experience and comment seriously and challenging your preconception
I'm honestly curious what you're experience level is? I've never known a developer with more than a couple years of experience valuing velocity over financial stability
The up-front cost of code review can be easily be tripled or quadrupled when it’s distributed over several weeks after the fact in the form of unplanned work, each instance of which incurs its own cost of context switching, as well as the cost of potential rework.
The purpose of such a review is a deliberate bottleneck in the earlier stage of development to stop it becoming a much larger bottleneck further down the line. Blocking one PR is a lot cheaper than blocking an entire release, and having a human in the loop there can ensure the change is in alignment in terms of architecture and engineering practices.
CI/CD isn’t the only way to do it but shifting left is generally beneficial even with the most archaic processes.
To catch bugs and avoid the bottleneck that is code review, you’re just moving the bottleneck to some point after merging the code, not before.
Like it or not you still have to stop what you’re doing to identify a bug and then fix it, which takes time away from planned feature work. You’re not optimising anything, you’re just adding fragility to the process.
As I said before, an issue localised to a PR in review blocks one person. An issue that has spread to staging or prod blocks the entire team.
> Code reviews kill velocity - introduce context switching, and are make work
This is the same point three times, and I don't agree with it. This is like saying tests kill velocity, there's nothing high velocity about introducing bugs to your code base.
Everything introduces context switching, there's nothing special about code reviews that makes it worse than answering emails, but I'm not going to ignore an important email because of "context switching."
Everyone makes mistakes, code reviews are a way to catch those. They can also spread out the knowledge of the code base to multiple people. This is really important at small companies.
CI is great, but I have yet to see a good CI tool that catches the things I do.
> Why not require two or three reviews if they are so helpful at finding mistakes?
Places do? a lot of opensource projects have the concept of dual reviews, and a lot of code bases have CODEOWNERS to ensure the people with the context review the code, so you could have 5-10 reviewers if you do a large PR
> Why not require two or three reviews if they are so helpful at finding mistakes?
For secure software, e.g. ASIL-D, you will absolutely have a minimum 2 reviewers. And that’s just for the development branch. Merging to a release branch requires additional sign offs from the release manager, safety manager, and QA.
By design the process slows down “velocity”, but it definitely increases code quality and reduces bugs.
Once again let me reframe the mindset. Trying to get a perfect change where you anticipate every possible thing that will go wrong beforehand is impossible - or at least extremely costly. The alternative is to spend your effort on making it easy to find and fix problems after.
You are not anticipating every possible bugs. It's mostly a learning experience for you and the team if it's done correctly. Someone may proposes another approach, highlight certain aspects that needs to be done "right" (definition may vary), let you know possible pitfalls, etc... It's not always LGTM.
> No it isn’t. Fake work, synchronization, and context switching are all separate problems
Context switching is a problem because it...kills velocity. Fake work is a problem because it kills velocity. You're saying it's time that could be better spent elsewhere, but trying to make it sound wider. I disagree with the premise.
Synchronization is a new word, unrelated to what you originally wrote.
> How many times have you gone back to address review comments and introduced a regression because you no longer have the context in your head?
Never? I am not unable to code in a branch after a few days away from it. If I were, I would want reviews for sure! Maybe you have had reviews where people are suggesting large, unnecessary structural changes, which I agree would be a waste of time. We're just looking for bug fixes and acceptably readable code. I wouldn't want reviewers opining on a new architecture they read about that morning.
I guess they say never, because if you have descriptive commit messages and write good PR description, it's easy to regain the context surrounding the change. It's all about communication, both to others and your future self.
When I work on my own code, at home, with no-one to assist or review, I write tests, and open a PR anyway, and review it myself, sometimes the next day with fresh eyes, or even 10 minutes later after a quick walk in and out of the room and a glass of water.
If you're not confident you can review a piece of code you wrote and spot a potentially disastrous bug like the one in OP, write more tests.
These days you can dump your code/diff to ChatGPT and ask it to criticize. Often most of it is junk -- irrelevant or incorrect suggestions. But the 1 in 10 suggestions is actually useful and correct, which can be immensely helpful.
What about the bandwidth burned needlessly for thousands of users on their data plans.
At some scale such careless mistakes are going to create real effects for all users of internet through congestion as well.
If this was not a $8000 mistake but was somehow covered by a free tier or other plan from Google Cloud, would they still have considered it a serious bug and fixed it as promptly?
How many such poor designs are out there generating traffic and draining common resources.
I worked on a product where there was basically no automated testing, just a huge product surface to click around with a bunch of options. Because of technical debt some of the options would trigger different code paths, but it was up to the developer to memorize all the code paths and test accordingly.
After I shipped a bug the Director of Engineering told me I should "test better" (by clicking around the app). This was about 1 step away from "just don't write bugs" IMO.
Yep, my first job was at a company like that. Huge Windows desktop app built in Delphi. No automated testing of any kind. No testing scripts either. Just a lot of clicking around.
My first job was exactly that, selling windows app in Delphi. I joined the new team working on .net windows apps and we had an army of people clicking on UI all day long.
They maintained their "test plan" on a custom software where they could report failures.
TBH, that was well done for what it was but really called for automation and lacked unit-testing.
I am forced to use a custom kv store for my current t project. That pos has a custom dsl, which can only be imported through a swing ui, by clicking five buttons. Also, the ui is for 1024 screens, they are tiny in my 4k monitor
Contrarian approach: $8000 is not a lot in this context. What did the CEO think of this? Most of the time it is just a very small speed bump in the overall finances of the company.
Avoidable, unfortunate, but the cost of slowing down development progress e.g. 10% is much higher.
But agree that senior gatekeepers should know by heart some places where review needs to be extra careful. Like security pitfalls, exponential fallback of error handling, and yeah, probably this.
I'm sure it cost a lot more than $8000. That was only the direct visible cost to them. There were likely users hit with costs for the additional downloads, who never even knew what was the issue. Users working on a mobile hotspot who had to pay for extra data etc etc.
I doubt there’s a CEO. Despite the use of “we”, pretty sure this is one guy building the app. All the copyright notices and social media go back to one person.
Imagine if that was Meta that had over 1B users with their messenger desktop app update functionality that did just that. The loss would be in the hundreds of millions.
> But agree that senior gatekeepers should know by heart some places where review needs to be extra careful. Like security pitfalls, exponential fallback of error handling, and yeah, probably this.
The lesson here is much better use of automated tests (The app likely has no tests at all) and proper use of basic testing principles like TDD would prevent such junior-level embarrassing bugs creeping up in production paid software.
That is the difference between a $100 problem vs a $200M problem.
See the case of Knight Capital [0] who lost $460M, due to a horrific deploy.
How do you adjust your testing approach to catch cases like this? In my experience, timing related issues are hard to catch and can linger for years unnoticed.
I would mock/hook/monkey patch/whatever the functions to get the current time/elapsed time, simulate a period of time (a day/week/month/year/whatever), make sure the function to download the file is called the correct amount of times. That would probably catch this bug.
Although, after such a fuck up, I would be tempted to make a pre-release check that tests the compiled binary, not any unit test or whatever. Use LD_PRELOAD to hook the system timing functions(a quick google shows that libfaketime[0] exists, but I've never used it), launch the real program and speed up time to make sure it doesn't try to download more than once.
Similar to doix said, consider reading the time as IO and then rewrite the code in sans-IO style so you can inject the time.
Then it's a unit test that looks too obvious to exist until you read the ticket mentioned in the comment above it
No need for monkey patching or hooking or preload
But before that you add a couple checkmarks to the manual pre-release test list: "1 hour soak test" and "check network transfer meters before and after, expect under 50 MB used in 1 hour (see bug #6969)"
I don't think the author is wrong for saying that certain kinds of code should be written carefully. I object to the implication that other code shouldn't.
From TFA: "Write your auto-updater code very carefully. Actually, write any code that has the potential to generate costs carefully." So the focus is on code that "generate[s] costs". I think this is a common delusion programmers have; that some code is inherently unrelated to security (or cost), so they can get lazy with it. I see it like gun safety. You have to always treat a gun like it's loaded, not because it always is (although sometimes it may be loaded when you don't expect it), but because it teaches you to always be careful, so you don't absent-mindedly fall back into bad habits when you handle a loaded one.
Telling people to write code carefully sounds simplistic but I believe for some people it's genuinely the right advice.
In comparison, when I shipped a Mac desktop application:
We used Sparkle, https://sparkle-project.org/, to do our updates. IMO, it was a poor choice to "roll their own" updater.
Our application was very complicated and shipped with Mono... And it was only about ~10MB. The Windows version of our application was ~2MB and included both 32-bit and 64-bit binaries. WTF are they doing shipping a 250MB screen recorder?
So, IMO, they didn't learn their lesson. The whole article makes them look foolish.
People are willing to trade performance/size for convenience. Writing your application using Electron + React means it is going to probably ship a > 500mb app that will suck up 500mb ram while running, but you have a much easier dev experience and can deliver a "flashy" UI with minimal effort.
I'm pretty conservative about adopting third-party libraries (due to the long-term issues each one has the potential to cause), but an app updater is probably worth it.
It's just tricky, basically one fat edge case, and a critical part of your recovery plan in case of serious bugs in your app.
(This bug isn't the only problem with their home-grown updater. Checking every 5 min is just insane. Kinda tells me they aren't thinking much about it.)
> I'm pretty conservative about adopting third-party libraries (due to the long-term issues each one has the potential to cause), but an app updater is probably worth it.
Especially for a Mac-only application where Sparkle (https://sparkle-project.org/) has been around for almost two decades now and has been widely used across all sorts of projects to the point that it's a de facto standard. I'd be willing to bet that almost every single Mac "power user" on the planet has at least one application using Sparkle installed and most have a few.
As far as system package managers go, yeah. That's part of the price you pay for choosing Apple (Knows Best) TM. There is brew, nix and the like for applications on MacOS too though.
Apple doesn't "know best" - it's just that that is what the system package manager is.
You can use whatever you want outside of the App Store - most will use Sparkle to handle updates https://sparkle-project.org/. I presume Windows is similar.
> Apple doesn't "know best" - it's just that that is what the system package manager is.
The fact that that is what the system package manager is is why I said Apple "knows best". You can pick from dozens of system packages managers hooked up to hundreds, if not thousands of different repos on Linux.
I'm running an anti-censorship proxy service which uses Proxy Auto-Configuration (PAC) file which you can configure OS-wide or in the browser.
If the file contains invalid JS (syntax error, or too new features for IE on Win7/8), or if it's >1MB (Chromium-based browsers & Electron limit), and the file is configured system-wide, then EVERY APP which uses wininet starts flooding the server with the requests over and over almost in an endless loop (missing/short error caching).
Over the years, this resulted in DDoSing my own server and blackholing its IP on BGP level (happened 10+ times), and after switching to public IPFS gateways to serve the files, Pinata IPFS gateway has blocked entire country, on IPFS.io gateway the files were in top #2 requests for weeks (impacting operational budget of the gateway).
All of the above happens with tight per-IP per-minute request limits and other measures to conserve the bandwidth. It's used by 500 000+ users daily. My web server is a $20/mo VPS with unmetered traffic, and thanks to this, I was never in the situation as the OP :)
> We decided to take responsibility and offer to cover all the costs related to this situation.
Good on them. Most companies would cap their responsibility at a refund of their own service's fees, which is understandable as you can't really predict costs incurred by those using your service, but this is going above and beyond and it's great to see.
Sloppy coding all around. If you don't want to program something right, why don't you just direct users to the website to manually update it?
On one hand it's good that the author owns up to it, and they worked with their users to provide remedies. But so many things aren't adding up. Why does your screen recorder need to check for updates every 5 minutes? Once a day is more than enough.
This screams "We don't do QA, we shorts just ship"
Or, given it's a Mac app, just have the Mac app store take care of updates. That's part of the value that using the app store service gives you, the other one being not spending thousands in accidental data transfer when you do auto updates wrong.
> Or, given it's a Mac app, just have the Mac app store take care of updates. That's part of the value that using the app store service gives you,
And pay Apple their 30% cut on your revenue? No thanks.
> the other one being not spending thousands in accidental data transfer when you do auto updates wrong.
Or just actually write proper automated tests for basic features first, before a large refactor to prevent introducing issues like this from happening again?
While I respect the author's honesty in this mistake, the main takeaway here is not mentioned and that is just writing proper automated tests as their impression on this post is that there aren't any.
The same Netflix that's been spats with Apple over avoiding app store fees by flouting policies that had other apps kicked out of the store? I'd suggest that it was, in fact, not good enough for Netflix.
Software doesn't need to check for updates at all. If I want to update my software, I'll update it. I don't need or want the software to be doing it on its own. All OS's have a native package manager at this point that can handle updates. We don't need applications going around it.
A quick warning "Hi User, your out of date, please update." Is fair.
What's really scary here is the lack of consent. If I want to record videos I don't necessarily have an extra 250mb to spend( many users effectively pay by the gig) everytime the developer feels like updating.
> One of our users, who lived in a house, had their internet provider cancel their contract due to enormous traffic generated during a month. It was extremely problematic as there was no other internet provider available around.
>Add special signals you can change on your server, which the app will understand, such as a forced update that will install without asking the user.
I understand the reasoning, but that makes it feel a bit too close to a C&C server for my liking. If the update server ever gets compromised, I imagine this could increase the damage done drastically.
> The app checks for the update every 5 minutes or when the user activates the app. Normally, when the app detected the update - it downloaded it and stopped the 5 minutes interval until the user installed it and restarted it.
This is still bad. I was really hoping the bug would have been something like "I put a 5 minute check in for devs to be able to wait and check and test a periodic update check, and forgot to revert it". That's what I expected, really.
We just put a header for version in our app, and when we deploy new code the client checks against the version header and upgrades if the version is mismatched. No extra get requests required. Bonus: we just use the last git commit hash as the version. Stupid simple.
I saw some project that used a DNS TXT field to check its version
That way I guess you get the caching of the DNS network for free, it uses basically one packet each way, encryption is still possible, and it can reduce the traffic greatly if a big org is running a thousand instances on the same network
I think it was written in Go. Might have been Syncthing
Probably product owner wants to show off a nice chart at their next meeting showing how quickyl users upgrade, as some kind of proxy metric for "engagement"
The scale is astounding. I was briefly interested in the person that caused the error then immediately realized it was irrelevant because if a mechanism doesn't exist to catch an issue like that, then any company is living on borrowed time.
Well when I follow a desktop link on my phone, it redirects me to the mobile version, despite the URL specifically asking to serve the desktop site, it just doesn't work the other way around. Plus I never asked to see the mobile site, I followed a link someone else posted
Totally. I live in a place where many (most?) ISP plans have limited monthly downloads. I'd be so pissed off if my monthly allowance was blown by this series of boneheaded decisions.
More anecdata that commercial software is garbage, especially if it's targetting consumers.
I'll stick with open source. It may not be perfect, but at least I can improve it when it's doing something silly like checking for updates every 5 minutes.
This 1000 times. It takes courage to open up to mistakes. As a relatively young industry, we have a lot to learn still to move away from the instinctive blaming culture surrounding such failures. In this case, it's only a file being downloaded a couple of times, nobody died or got injured.
For those interested in this topic, and how other industries (e.g. Airline industry) deal with learning from or preventing failure: Sidney Dekker is the authority in this ___domain. Things like Restorative Just Culture, or Field guide to understanding human error could one day apply to our industry as well: https://sidneydekker.com/books.
These articles are great, but I have to one-up the blog: I recently helped a small dev team clean up a one-line mistake that cost them $95,000... which they didn't notice for three months.
The relevance is that instead of checking for a change every 5 minutes, the delay wasn't working at all, so the check ran as fast as possible in a tight loop. This was between a server and a blob storage account, so there was no network bottleneck to slow things down either.
It turns out that if you read a few megabytes 1,000 times per second all day, every day, those fractions of a cent per request are going to add up!
What might be fun is figuring out all the ways this bug could have been avoided.
Another way to avoid this problem would have been using a form of “content addressable storage”. For those who are new, this is just a fancy way of saying make sure to store/distribute the hash (ex. Sha256) of what you’re distributing and store it on disk in a way that content can be effectively deduplicated by name.
It’s probably not so easy as to make it a rule, but most of the time, an update download should probably do this
Note that billing alerts protect against unexpected network traffic, not directly against bugs and bad design in the software. Update checking remains a terrible idea.
I have Screen Studio and I don't leave it open but all I wish for now is that you disable the auto updater. Provide an option for it to be disabled and allow for manual update checking. Checking for an update every 5 minutes is total overkill and downloading the update automatically is just bad. What if I was on mobile internet and had limited bandwidth and usage. The last thing I want is an app downloading it's own update without my consent and knowledge.
You can still get delta updates with Sparkle in an electron app. I am using it, and liking it a lot more than Electron Updater so far: https://www.hydraulic.dev
Yes, a single line of code is in the stack trace every time a bug happens. Why does every headline have to push this clickbait?
All errors occur at a single line in the program - and every single line is interconnected to the rest of the program, so it's an irrelevant statement.
I read that as "A single line of code costs $8000" which, from the comments seems like a few others had the same thought. Reading the article it is not costs and the original title is "One line of code that did cost $8,000", so as some others have pointed out it is a bug that cost $8000.
They had no cost usage alerts. So they even did not know that the thing was happening, just realized with the first bill.
I think that is the essence of what is wrong with the cloud costs. Defaulting to possibility for everyone to scale rapidly while in reality 99% have quite predictable costs month over month.
I assume most of that 2PB network traffic was not egress, right? Otherwise how did it "only" cost you $8k on Google Cloud?
Even at a cost of 0.02$ per GB, which is usually a few times lower than the actual prices I could find there, that would still result in an invoice of about $40k...
The first 500TB should have cost $35k already. At that point pricing goes from $0.06/GB to "contact us". So I'd have expected google to charge $80k or so for the whole thing. (Unless google decided to forgive most of the cost)
Bug notwithstanding, checking for updates every 5 minutes is exactly the wrong way to do it.
You want to spread out update rollouts in case of a catastrophic problem. The absolute minimum should be once a day at a random time of day, preferably roll out updates over multiple days.
From the article:
"While refactoring it, I forgot to add the code to stop the 5-minute interval after the new version file was available and downloaded.
It meant the app was downloading the same 250MB file, over and over again, every 5 minutes."
Ignoring the obvious question of "why does a screen recorder that checks for updates every 5 minutes need to be installed if macOS already has a screen recorder built in"—writing your own (buggy) auto updater for a macOS app, in 2025, is nuts considering you also have two existing options for auto updates at your disposal, the Mac App Store and Sparkle (https://sparkle-project.org/), both of which are now nearly two decades old.
Why is nobody talking about what a shady business practice it is that cloud providers don't alert you to this kind of overage by default? Sure, you can set up alerts, but when you go 10x over your baseline in a short period of time, that should trigger an alert regardless of your configured alerts.
In the grand scheme of things, $8k is not much money for a business, right? Like we can be pretty sure nobody at Google said “a-ha, if we don’t notify the users, we will be able sneak $8k out of their wallets at a time.” I think it is more likely that they don’t really care that much about this market, other than generally creating an environment where their products are well known.
While this unfortunate, I am sure I also have single lines in production with greater cost and equivalent value (close to none) -- and I've only worked at small companies. I am sure some of y'all can beat this by ~2 orders of magnitude.
CI/CD at it's finest :p I guess the 5-minutely updates is correlated with the rate of bug fixes they need to push... Surely, that can't be for new features.
Looking at the summary section, I'm not convinced these guys learned the right lesson yet.
For me this shows once again that proper testing is neglected by many developers and companies. Sadly, it is not even mentioned in the advice at the end of the article.
Exactly. No mention of writing automated tests or even TDD at all.
It's best to save everyone by writing tests that prevent a $100 issue on your machine from becoming a costly $10M+ problem in production as the product scales after it has launched.
This won't be the last time and this is what 'vibe coding' doesn't consider and it it will introduce more issues like this.
$8000 also seems pretty cheap for 2PB of traffic? Looking at google cloud storage egress rates, $0.02/GiB (which is on the lower end, since it depends on destination) would be about $40k for 2PB.
And in that case, the problem would not be discovered until 1) someone opened a bug report, which rarely happens, because any competent user would just disable auto-updates, and 2) that bug report would be investigated, which also rarely happens.
Truth to be told, Google Cloud console is horrible mess for new people, who just wants quickly setup API, pay and don't have time to care about it anymore.
Well, you should hire contractor to set console for you.
"Designed for MacOS", aah don't worry, you will have the money from apes back in the no time. :)
> While refactoring it, I forgot to add the code to stop the 5-minute interval after the new version file was available and downloaded.
I’m sorry but it’s exactly cases like these that should be covered by some kind of test, especially When diving into a refactor. Admittedly it’s nice to hear people share their mistakes and horror stories, I would get some stick for this at work.
A giant ship’s engine failed. The ship’s owners tried one ‘professional’ after another but none of them could figure out how to fix the broken engine.
Then they brought in a man who had been fixing ships since he was young.
He carried a large bag of tools with him and when he arrived immediately went to work. He inspected the engine very carefully, top to bottom.
Two of the ship’s owners were there watching this man, hoping he would know what to do. After looking things over, the old man reached into his bag and pulled out a small hammer. He gently tapped something. Instantly, the engine lurched into life. He carefully put his hammer away and the engine was fixed!!!
A week later, the owners received an invoice from the old man for $10,000.
What?! the owners exclaimed. “He hardly did anything..!!!”.
So they wrote to the man; “Please send us an itemised invoice.”
I’ve done worse. At my very first job I wrote some slow login rewards calculation code for a mobile game that caused a black screen on startup for long enough that users thought the app was broken and closed it out. (I was simulating passing time one minute at a time in lua or some BS. Oops!) It cost the company some large fraction of my salary at the time. My boss very kindly said that it was okay, everyone ends up mucking up like that at some point in the career, and no I wasn’t fired because the company just spend a large sum teaching me a lesson. We sat down at a whiteboard and I quickly came up with a solution that could just calculate the rewards one should get between two dates - there was some complexity that made this harder than it sounds on paper, but simulating time manually was not the answer.
Here is the example of why LLMs need to replace some douchbag SWE. But then again, unless there is a nutcase product owner involved, update check every 5min and downloading automatically is pure symptom of bad engineering practices and fear driven development(if anything breaks update is 5min away and we can laugh at the user when their app is updated and they are baffled).
Good thing, this was not shopify/Duolingo/Msft, else the news would be, how AI saved us $8k by fixing a dangerous code and why AI will improve software quality.
> As a designer, I value the experience product I create provides to the users. And this was not even a bad experience; it was actually harmful.
$229 per year on a closed source product and this is the level of quality you can expect.
You can have all the respect for users in the world, but if you write downright hazardous code then you're only doing them a disservice. What happened to all the metered internet plans you blasted for 3 months? Are you going to make those users whole?
Learning from and owning your mistake is great and all, but you shouldn't be proud or gloating about this in any way, shape, or form. It is a very awkward and disrespectful flex on your customers.
We've had single characters cost, you know, millions of $. (If you're familiar with C++ and the auto keyword, it's relatively obvious why that character is "&".)
This 'single line of code' headline trend is dumb. Of course a single line of code can fuck everything up, code is complicated and thats how it works. Its not knitting.
Honestly, that's a rough one. I've written dumb bugs like that before, cost nowhere near $8k tho lmao. Kinda makes me extra paranoid about what my code's actually doing in the background.
screen.studio is macOS screen recording software that checks for updates every five minutes. Somehow, that alone is NOT the bug described in this post. The /other/ bug described in this blog is: their software also downloaded a 250MB update file every five minutes.
The software developers there consider all of this normal except the actual download, which cost them $8000 in bandwidth fees.
To re-cap: Screen recording software. Checks for updates every five (5) minutes. That's 12 times an hour.
I choose software based on how much I trust the judgement of the developers. Please consider if this feels like reasonable judgement to you.
reply