I feel like we never see HAProxy in these reverse proxy comparisons. Lots of nginx, Apache, Caddy, Traefik, Envoy, etc.
The HAProxy configuration is just as simple as Caddy for a reverse proxy setup. It's also written in C which is a comparison the author makes between nginx and Caddy. And it seems to be available on every *nix OS.
I am not sure I would agree with the assertion that config for HAProxy is just as easy.
In fact I use HAProxy in production pretty regularly because it is solid but its config one of the main reasons I would choose something else.
A basic HAProxy config is fine but it feels like after a little bit each line is just a collection of tokens in a random order that I have to sit and think about to parse.
I feel the same way. I'm not a fan of haproxy's configuration system. It's really difficult for me to understand it, whereas I feel I can read most nginx/apache configs and immediately know what is supposed to be happening. I still maintain servers under load in production that use all three to this day and I always go back to nginx because of the configuration alone.
I can't comment on haproxy because I haven't used it enough, but I think that the "nginx's config is easy to grasp" posture has a bit of Stockholm syndrome in it.
- Do you want to add a header in this "___location" block? Great, you better remember to re-apply all the security headers you've defined at a higher level (server block for instance) because of course adding a new header will reset those.
- Oh, you mixed prefix locations with exact ___location with regex locations. Great, let's see if you can figure out by which ___location block will a request end up being processed. The docs "clearly" explain what the priority rules for those are and they're easy to grasp [1].
- I see you used a hostname in a proxy_pass directive (e.g.: http://internal.thing.com). Great, I will resolve it at startup and never check again, because this is the most sensible thing to do of course.
- Oh... now you used a variable (e.g.: http://$internal_host). That fundamentally changes things (how?) so I'll respect the DNS's TTL now. Except you'll have to set up a DNS resolver in my config because I refuse to use the system's normal resolver because reasons.
- Here's an `if` directive for the configuration. It sounds extremely useful, doesn't it? Well.. "if is evil" [2] and you should NOT use it. There be dragons, you've been warned.
I could go on... but I think I've proved my point already. Note that these are not complaints, it's just me pointing out that nginx's configuration has its _very_ significant warts too.
All that may be true, but for a lot of us old timers we were coming from apache to nginx and apache's configs can eat a bag of dicks.
Unfortunately it's likely I worked in the same building as one of the people responsible for either creating or at least maintaining that mess, but I didn't know at the time that he needed an intervention.
Exactly all of this. I've mentioned the first point about add_header redefining instead of appending in a previous HN comment of mine: https://news.ycombinator.com/item?id=27253579. As mentioned in that comment, HAProxy's configuration is much more obvious, because it's procedural. You can read it from top to bottom and know what's happening in which order.
Disclosure: Community contributor to HAProxy, I help maintain HAProxy's issue tracker.
Yes, also a lighttpd fan here. It's not as refined a reverse proxy, but for the halfway trivial use case it's doing fine, and as a web server it's light (duh) and fast. Much more readable configuration than nginx or apache.
To be clear I never said it was easy. I have a LOT of issues with Nginx's configuration, I just find it to be significantly less bad than the other options.
Other than Caddy, Caddy has been great so far but I have only used it for personal projects.
For simple things, Caddy is nice and easy, but I've struggled with Caddy quite a bit, too, especially for more complex setups. I usually break out haproxy or nginx for really challenging setups, because caddy's documentation and examples are quite sparse (esp v2)
What do you struggle with about the documentation or "more complex setups"? I was just on the phone recently with Stripe who has a fairly complex, large-scale deployment, and they seemed to have figured it out with relative ease.
I'm not the GP, but I've had a similar experience, with the Caddy docs seeming good for simple configurations, but lacking for more complex ones.
I last setup a Caddy config maybe 6-
9 months ago, and everything related to client certificates was either scantily documented, wrongly documented, or not documented at all. It might be I got unlucky, as some of the client cert features were fairly new, but it wasn't a great experience.
Still, I much prefer Caddy's config system to Nginx or HAProxy.
Oh, something else I'd really love to see are more fully-featured example configs, as it can be hard to know how to start sometimes.
Examples of "fully-featured" what though? The more features you add, the more combinations of ways to configure things there are.
The problem with "fully-featured" examples is that people copy and paste instead of learn how the software works. I'd rather our user base be skilled crafting configuration.
I agree completely. Fully-featured examples would be extremely helpful, especially with things like proxying to different servers for different paths and domains, static file serving set up differently for different paths and domains, various plugin setups, etc.
Oh, just saw this. You wrote your comment while I wrote mine. If you can enumerate specifically what you want to see, please submit it to our issue tracker: https://github.com/caddyserver/website
Generally we encourage examples in our community wiki though: https://caddy.community/c/wiki/13 -- much easier to maintain that way.
FWIW I'm a big fan of HAProxy as well, but I was just constrained by the sheer volume of testing and how rigorous I intended to be. Maybe once my testing is a little more generalized I can fan out to additional proxies like HAProxy without too much hassle, as I'd love to know as well.
I think one reason a lot of benchmarks don't include TLS termination is because it's often impractical in the real-world, where most clients reuse the connection and the TLS session for many requests, thus making them negligible in the long run. And given hardware optimizations for cryptographic functions combined with network round trips, you end up benchmarking the network and the protocol more than its actual implementation, which is often upstream from the server itself anyway.
Go's TLS stack is set to be more efficient and safer in coming versions thanks to continued work by Filippo and team.
Sure, we've already done this very real test in production a number of times and Caddy doesn't even skip a beat. (IMO that's the best kind of benchmark right there. No need to simulate with pretend traffic!)
h2o [1] was excellent when I tried it for TLS termination, beating hitch in my unscientific tests. And it got http/2 priorities right. It's a shame they don't make regular releases.
I’m surprised Varnish is not mentioned much either. For a while there it had a reputation as the fastest reverse proxy. I think its popularity was harmed by complex config and refusal to handle TLS.
It's always been blisteringly fast when we've used it, and I like the power of the configuration (it has its quirks but so do most powerful systems). But the overhead of setting it up and maintaining it due to having to handle TLS termination separately puts me off using it when other software is 'good enough'. If Varnish Enterprise was cheaper I would have bought it, but at their enterprise prices no way.
I'm keeping a watching brief on https://github.com/darkweak/souin and its Caddy integration to see if that can step up and replace Varnish for short-lived dynamic caching of web applications. Though I've lost track of its current status.
Amazing that you're talking about Souin and it's possible usage to replace Varnish. Let me know if you have question about the configuration or implementation. ATM I'm working on the stabilization branch to have a more stable version and merge the improvements into the caddy's cache-handler module.
Varnish is a must for my production apps. The grace period ability to serve stale cache while passing a request through to get the latest is just huge and I can’t live without it.
Personally I still recommend the config file. Even when they are simple, it gives you one single source of truth that you can refer to, it will grow as you need it, and it can be stored in source control.
Where and how parameters are configured is a bit more of a wild card and dependent on the environment you are running in.
That's something Matt and I tend to disagree on - I agree that a config file is better almost always because it gives you a better starting point to experiment with other features.
I still cannot make myself to try Caddy.. in things like this looks sweet but just may be 5% of the functionality [I care about]. Not saying it's not possible, but with Nginx I already know how to do list of CORS, OPTIONS , per ___location & cookie name caching. Issuing certs is probably simplest and the last thing in config setups of reverse proxying.
It does not, because HAProxy does not perform any disk access at runtime and thus would be unable to persist the certificates anywhere. Disks accesses can be unpredictably slow and would block the entire thread which is not something you want when handling hundreds of thousands of requests per second.
That issue has some good explanation, thanks. I wonder if a disk-writing process could be spun out before dropping privileges?
> Disks accesses can be unpredictably slow and would block the entire thread which is not something you want when handling hundreds of thousands of requests per second.
This is not something I see mentioned in the issue, but I don't see why disk accesses need to block requests, or why they have to occur in the same thread as requests?
When reading along: Keep in mind that I'm not a core developer and thus are not directly involved in development, design decisions, or roadmap. I have some understanding of the internals and the associated challenges based on my contributions and discussions on the mailing list, but the following might not be entirely correct.
> I wonder if a disk-writing process could be spun out before dropping privileges?
I mean … it sure can and that appears the plan based on the last comment in that issue. However the “no disk access” policy is also useful for security. HAProxy can chroot itself to an empty directory to reduce the blast radius and that is done in the default configuration on at least Debian.
> but I don't see why disk accesses need to block requests
My understanding is that historically Linux disk IO was inherently blocking. A non-blocking interface (io_uring) only became available fairly recently: https://stackoverflow.com/a/57451551/782822. And even then it's a operating system specific interface. For the BSD's you need a different solution.
> or why they have to occur in the same thread as requests?
“have” is a strong word, of course nothing “has” to be. One thing to keep in mind is that HAProxy is 20 years old and apart from possibly doing Let's Encrypt there was no real need for it to have disk access. HAProxy is a reverse proxy / load balancer, not a web server.
Inter-thread communication comes with its own set of challenges and building something reliable for a narrow use case is not necessarily worth it, because you likely need to sacrifice something else.
As an example at scale you can't even let your operating system schedule out one of the worker threads to schedule in the “disk writer” thread, because that will effectively result in a reduced processing capacity for some fractions of a second which will result in dropped requests or increased latency. This becomes even worse if the worker holds an important lock.
HAProxy does not serve static files (AFAIK), so for some stacks you need to add nginx or caddy after haproxy as well to serve static files and forward to a fastcgi backend.
The HAProxy configuration is just as simple as Caddy for a reverse proxy setup. It's also written in C which is a comparison the author makes between nginx and Caddy. And it seems to be available on every *nix OS.