In the AI era, I often deal with the situation that I ask AI for a python script then need to setup an environment to run it. It's frustrated because I only need to run it once but need to deal with 15-30mins setup.
That why I developed a small helper program that allow me execute any python script. It's not perfect but it has been serve my needs very well for months.
If you are Go developer, let's install it with command
go install github.com/kanthorlabs/runit/cmd/runit@latest
Greate article! I have learned about block/page long time ago when I needed to debug performance issue but not as deep as this article. Will share it with my teammate and its funny to see their emotional face :D
The funny story behind KanthorQ is that it stemmed from a challenge I faced with another project: an open-source webhook gateway (https://github.com/kanthorlabs/kanthor). After completing that project and starting some benchmarks, I realized I was dealing with issues I had never encountered before.
When something went wrong, I had to scan through a list of events to identify the failed ones. However, doing that with traditional message brokers like NATS.io or RabbitMQ was challenging. I needed an additional layer to store the events and analyze the data. As a result, a simple system required two high-availability components to function properly.
Additionally, backup and restoration were not straightforward with traditional message brokers like NATS.io or RabbitMQ. It's difficult to back up and restore these systems—or at least, I couldn't find a reliable way to do it.
That's why I developed KanthorQ. Now, I have a message broker where data is stored in a database, making it easy to browse, analyze, back up, and restore.
I agree that we are overthinking about infrastructure. Boring stack like traditional RDMS, single server with regular backup, few bash script for deployment is fine for normal startup that targets to non-tech customer. They will serve you well at least one or two years, then you will know what should be improve.
One of the big surprise is database like PostgreSQL can handle like 100tps very well with cheap hardware cost. That mean you can handle up to 86 millions transaction per day.
I'm using KSUID (https://sotergreco.com/why-choose-ulids-over-traditional-uui...) for a while and totally happen with it
One tiny note about using sortable id is you should not expose it to public use without authentication because it can be scan easily
Database is the most interesting topic that I could not stop curious about it. I used to hosted a techtalk for my teammate about how to write a db with LSM tree ans SSTable but I could not go so far as your article
Sent it to my teammate and hope they can use your article as a note to explore more topics about the database
Thanks for amazing article
Absolutely Posgtres!!! Yesterday I had to migrate from postgres to mysql and I found some disadvantage things about mysql in my use case
There is no returning for insert/update/delete what forces me do a second query to get new data. Why dont just return what I have updated? Because it is some thing like: UPDATE wallets SET balance = balance + 1
I have to give up json function because its hard to work with it on mysql. Have to do the aggregation in my code instead
I'm considering host a gitea instance backup all of my repos.
I have an important fix that need to be deployed right now but there is no way to deploy it in a normal way with our CI which one was setup with Github Action. Fortunately I have a instruction to bypass CI and build the source by myself.
But again, Github defeat me because our release workflows are depend on GitOps which are effected by Github issue. Ahhhhhhhhhh I have to build the docker image, push it to ECR then update a YAML template to make EKS apply the new changes
It's 9PM in my timezone and I'm waiting for my patches are up. A frustrating incident
Gitea's ability to create a local repository as mirror of a remote repository is great for this. You can stay on Github and have your code regularly mirrored locally.
I have this setup running on a Synology NAS at home. I'm currently syncing all of my starred github repos to local storage using a short bash script that runs once a week. Once a repo is in gitea, it pulls any new updates from github every 6 hours or so. It's mostly for archival purposes, just in case something majorly bad happens to github.
Or use Gitlab. I remember doing a mirror syncing from Github with one of my gitlab repository. The main reason I used to do it is Gitlab offer free built-in CI at that time
In my case one of most important thing is the GitOps workflow. It's single source of truth so it's also single point of failure ;(
Thank you for the feedback and advice you provided, especially regarding the target audience. In light of this, I plan to start by publishing some of my smaller projects first, as they are relatively easier to complete. This will involve documenting the process, testing the functionality, and creating blog posts about them.
I am also curious about your motivation to work on a project for an extended period of time, such as five years. While I have found excitement and fulfillment in completing the core functions of a project, I have also experienced burnout in continuing to work on it. So, I would like to know what keeps you motivated throughout the long haul of a project.
I'm not sure, TBH... Maybe I'm just a glutton for punishment?
I'm currently partway through refactoring all of the portable unit tests for Concise Encoding ( https://github.com/kstenerud/go-concise-encoding/tree/master... ) and it is a SLOG! I mean, so goddamn boring and tedious that I wanna stick an ice pick through my skull. There's easily another 200 hours of this terrible work ahead and I'll be right back to it the moment Dogma v1 is published in a few weeks.
Do I dread it? Yes. Am I still going to do it? Yes. Because at the end of the day I want to be able to stand back and say "I made that. I completed it - ALL of it. It's not perfect, but it's doing its job and people are using it."
I do take vacations from my projects from time to time. It's good to do so regardless, because you need the distance to gain a better perspective. Coming at it a bit cooled off means you need to refresh your memory - and that uncovers many of the UX problems your project has.
1. Does the retry function work based on the logic that the same input will always return the same output, even if the code uses non-deterministic methods like "new Date()" or "Math.random"? If not, your retry logic may produce unpredictable results and therefore not be safe.
2. According to the article "How Defer Works" (https://docs.defer.run/platform/how-defer-works/), you will be running our code on your platform. How do you ensure the protection of sensitive data during the execution process? For example, how will you prevent access tokens or user/password information for the database from showing up in logs due to exceptions or bad configurations?
1. Yes, we currently provide a simple approach to retries but are already working on providing features for idempotency.
2. All sensitive data (tokens, env vars) are encrypted on our side, however, we don't prevent users to print their values in the logs yet - it's planned in our next releases.
I see. I hope you have a great journey. I used to want to implement a system like that for my import/export tasks because my workload heavily involves importing and exporting CSV files. But the deadline defeated me ;) I have no time to design a system like that
By the way, I really love how simple the defer API is to use
That why I developed a small helper program that allow me execute any python script. It's not perfect but it has been serve my needs very well for months.
If you are Go developer, let's install it with command
go install github.com/kanthorlabs/runit/cmd/runit@latest
Otherwise you can download the binary and use it
https://github.com/kanthorlabs/runit/releases/tag/v25.4.19