How is this a replacement for LangSmith? I browsed the source and I could only find what appear to be a few small helper functions for emitting structured logs.
I’m less familiar with LangSmith, but browsing their site suggests they happen to offer observability into LLM interactions in addition to other parts of the workflow lifecycle. This just seems to handle logging and you have to pass all the data yourself- it’s not instrumenting an LLM client, for example.
> in addition to other parts of the workflow lifecycle
FWIW this is primarily based on the LangChain framework so it's fairly turnkey, but has no integration with the rest of your application. You can use the @traceable decorator in python to decorate a custom function in code too, but this doesn't integrate with frameworks like OpenTelemetry, which makes it hard to see everything happens.
So for example, if your LLM feature is plugged into another feature area in the rest of your product, you need to do a lot more work to capture things like which user is involved, or if you did some post-processing on a response later down the road, what steps might have had to be taken to produce a better response, etc. It's quite useful for chat apps right now, but most enterprise RAG use cases will likely want to instrument with OpenTelemetry directly.
Lots of vendors dismiss support issues without strong data. But if you go to the length of decoding the request, outlining the steps to reproduce etc you can have a much faster experience. Especially with network vendors where 99%+ of their support workload is dealing with client or reseller misconfiguration.
Its maybe a bit trumped up, it smells a bit like MSP marketing but its also at least a little bit warranted?
Actually I was in a similar place with Palo 24 months or so ago, and despite handing them everything they could possibly expect they handed us a workaround (Just bounce your vpn sessions manually when they fail) instead of issuing a patch. However there was a strong argument there that our customer was a bit too dedicated to their wacky vpn architecture and should be doing things differently. Really the kudos here is getting the vendor to perform which I feel is a huge skill these days.
felt the same tbh.. - u won't believe how many trouble previously was needed to go through to get this vendor to acknowledge a CVE >.> - hard to believe this is the whole story.. maybe they learnt finally not to try and shove shit under the carpet. who knows. I doubt it.
Op here. It was but a very muddy timeline... there was also a wildfire cve which I didn't detail due to how it was discovered (long story) but playing responsible is good!
Long story short. We patched the firewall things broke... we spoke to TAC they had no idea kept just saying it was client so we went log digging realised it was a DoS and sent it over to PSIRT and they confirmed it and raised the CVE
I’m curious why there appears to be a pretty significant delta between their data and yours.
Also, if you’re open to suggestions, I had trouble finding the pricing for gridstatus. Entirely possible I was missing something obvious, but I wanted to see how much it might cost to get the long term Ercot generation-by-source dataset and couldn’t seem to get a clear answer.
Regardless, building things like this takes a ton of effort, and I appreciate all you’re doing. Keep up the great work!
we report the instantaneous load records while that page is reporting the hourly average. I wouldn’t say the results are that different. They point to about the same times even if numbers have a small delta.
My focus right now is to make this data more available and easier to use. Haven’t done much to monetize yet. If you just want that data source, happy to get it to you for free. Shoot me an email at [email protected] or sign up for a free api key and grab it from the site
Congrats on the launch! Could you expand a bit on how you differentiate your product from other products in the space like Tailscale and Nebula?
Edit - I see you mention that Tailscale uses userland WireGuard. Is that the biggest difference between the two? Do you foresee yourselves running into issues by not using the userland implementation?
The biggest differences with Tailscale are that you can use the kernel version (speed) and that we are self-hostable. We actually noticed a major contributor to the speed difference between us and them was a lot of times, they'll route traffic through their relays which can eat up a good amount of time.
With Nebula, they're a lot closer on speed, but also we've got a management GUI which makes things a lot easier. We were very close to using Nebula in our early days. The main thing that stopped us was, we decided WireGuard was going to be the standard in the future, and wanted to be based on WireGuard.
That leads to a bit more fundamental of a difference which is a bit harder to quantify. Our aim is to really be a "WireGuard controller." You should be able to shut down our server and agents and your network should still run fine, and you should be able to manually modify all your WireGuard interfaces if necessary. We're getting close to that vision but aren't quite there yet.
That last point leads back to the kernel thing. We use kernel by default, but really, Netmaker can use any WireGuard implementation. If users are scared of the security implications of using kernel, they can use the userland version, and Netmaker should be able to pick that up just fine. They can even run it in a docker container on their machine.
We briefly considered building something atop Wireguard in the early days of Nebula, but decided not to do so because of scaling. Wireguard's protocol necessitates that all nodes have existing keypairs for each other ahead of time. At Slack's scale, that means every time a fresh node is launched, you would have to tell 50,000 other nodes it exists.
Obviously you can smarten this up and tell only hosts it might talk to. But this adds complexity. Using PKI eliminates this key distribution problem and means that you don't have the same scaling limitations as something built on WG.
Wireguard is a very very good VPN, but I cannot imagine trying to run something on the scale of tens of thousands of nodes when you need such complex coordination systems to exchange keys/trust, especially in a dynamic environment where nodes are coming and going all the time.
I totally get that it solvable overall, but Slack has had 4 years of nearly perfect uptime on Nebula, whilst using it to pass >95% of all backend traffic. These considerations may seem simple to address, but there are fundamentals that mattered and led us to writing Nebula. We didn't want to create something new, but to do what Slack needed, we had to.
This is great! Can you also compare to Headscale and Zerotier?
How do you handle NAT compared to these other options? I've run into issues with WireGuard when both sides are behind residential NATs (or AWS EC2 IGW); as you know, Nebula solves this with "lighthouse" servers (which are self-hosted but externally accessible), and Tailscale uses its third-party intro devices.
Do the Gravitl servers have to always listen on an exposed port?
It is way, way faster than Zerotier. Headscale I am less familiar with, and they do make Tailscale self-hostable, but they're a community-driven project I'm not sure how reliable it'll be long term (can't speak for the author).
On the NAT side, we provide 3 layers of traversal options:
#1 (default) port forwarding: This actually works in a surprisingly well, for about 90% of environments, but does require an exposed port.
#2 UDP Hole Punching: The server acts similarly to a Nebula "lighthouse" and will tell clients where to reach each other. This covers that small situation of dualing NAT's, and doesn't require the exposed port.
#3 Relay: In situations where neither works such as CGNAT, you can set a public node as a relay to route traffic to the "hidden" node
> but they're a community-driven project I'm not sure how reliable it'll be long term
Don't take this the wrong way, but subtly/implictly implying that some software is not going to be reliable in the long run because it only has a community behind is low-key FUD.
Nahh you're right, that's not a fair criticism at all, and we rely on tons of community-driven stuff. I only meant that as a differentiator because to some companies, they need to know there's a company behind the project for support purposes or they're not gonna use it (whether or not that's fair of them).
Tailscale has some functionality to automatically determine what flavor of NAT traversal is necessary, this isn’t a big deal but do I have to configure the nodes to use these methods or will your client figure this out itself?
Right now you need to choose your own option. We're planning to automate this in the next couple of months. I will say, this isn't always a bad thing. We have a lot of users who switched from Tailscale because it was frequently falling back to relay, often via public relays that were hundreds of miles away from them, causing big increases in latency. We want to give people the automation, but also the choice.
> You should be able to shut down our server and agents and your network should still run fine, and you should be able to manually modify all your WireGuard interfaces if necessary.
Mainly we give a lot more configuration options as well as a GUI. I'm also not sure if they run on anything other than Linux (I think not?). We can run on Mac, FreeBSD, and Windows. If you're just looking to create a pure mesh network with linux-based devices, they're definitely comparable and I think you'd just have to try out both and see what you like more.
> - Healthcare staff have great animosity towards the unvaccinated patients.
In addition to this, I’ve also heard anecdotally that many of the unvaccinated (by choice) patients have animosity towards the healthcare workers themselves because the patients see this virus as politicized.
Overall I can imagine that it’s resulted in a more-hostile-than-average working environment which is bound to be stressful.
Unfortunately they are correct on this, this pandemic has been heavily politicized in the States, which imo was a very wrong thing to do. They tried the same thing in my country (Eastern-Europe), with a political party blasting their logos on “vaccine information” tents installed on the sidewalks but fortunately it hasn’t caught on that well.
This argument, while technically true, is relatively dismissive of all that we have learned about the virus.
Of primary importance, COVID-19 appears to be easily and completely cleared by the human immune system and does not appear to become latent by settling in immune privileged areas or by integrating itself into the human genome or body in any other way (like Epstein-Barr, Herpesvirus, HIV, etc.)
Outside of the mainstream media and in the medical research literature COVID-19 has been thought to be easily "curable" since early on in the pandemic. In my opinion there appeared to be more uncertainty around the FDA and the rapid development timelines than there ever was around whether or not an effective and complete treatment could be developed.
A severe infection or cytokine storm may obviously cause lasting damage. I do not mean to suggest that everyone will be off the hook in the future!
What do you think about all these people with long covid then? I've seen studies suggesting that it's pretty common, as well as anectodal observations, although I don't have any links handy off the top of my head.
I'm not a doctor but as I said in my earlier comment, I think the literature supports that most of this is just damage from the directly infected cells (lung, organ, nerve) that the body will eventually mostly repair given appropriate nutrition and time. But certainly for some people certainly there may be permanent damage.
A better way to state my point was to say that other diseases can cause exactly the same types of cell damage, and having, say, reduced lung function or some neuropathy going forward after a COVID infection are not themselves new and novel medical problems.
I have had COVID despite my best attempts to do everything possible not to contract it. For complete transparency here, I have a strongly vested interest in knowing the truth of this matter. I agree that there is not a strong consensus yet. Sorry I do not have great references at hand to share right now either.
You're correct in principle, but when weighing the "unknown long term risks" of a vaccine versus the "unknown long term risks" of COVID itself it really needs to be an apple to apples comparison.
We know both the virus and the vaccine will be out of your system within a short period of time. So any long term effects would have to be derived from what happens while they're in your system. The vaccine generates a few proteins, which your immune system learns to fight off in short order. The virus generates similar proteins, but continues to replicate many times over, attacking your cells and triggering a much more severe immune response as it goes.
So while there are unknown long-term risks for both, I think it is pretty safe to say the number of vectors for long term consequences are much more numerous with the virus.
I think my original comment was maybe misconstrued. I stated plainly that COVID could cause lasting damage just as any other virus. It just doesn't seem there is much to support that it causes damage above and beyond that of other viruses are capable of and for which long term therapies and treatments exist and are well understood. Certainly there isnt anything that I have seen which substantiates that COVID itself can persist in the body and continue to wreck havoc after the body is able to clear the infection, but no doubt it can do serious and potentially lasting damage.
I absolutely cannot agree more strongly that the long term consequences of any of the COVID vaccines will be far fewer than those of the virus itself.
1) The “observable window” is the entire installation time. If they make installs take forever, that’ll affect everyone which should raise alarms pretty quick.
2) The conditional execution is possible but the installation is done using a vanilla alpine container which will match many legitimate hosts too. And any fingerprinting activities that involve syscalls would be detected in the process.
All this to say, there’s always room to continue raising the bar!
Evil-me is thinking "So I need to check for the existence of super common but not core modules before I run my exploit code, so a vanilla environment never runs the code"...
That's the thing. If we're watching syscalls, we see these checks. These would be things like attempted file-reads. Would they be enough to set off alarms? Maybe, maybe not.
This is generally the cat/mouse game of malware detonation in general. There are attempts to make sandboxes appear realistic, but I'd argue that our use case is even simpler since running commands or making network connections during installation is not a normal thing. It might be benign, but it's abnormal enough to warrant investigation.
There will always be ways to try and get around the system, but I'm pretty firm that this will significantly raise the bar which is a Good Thing.
Usually in a containerization environment all the requirements are installed at the same time, which may or may not make that kind of introspection difficult.
On a dev machine though... diabolical. That might show up in a syscall, but perhaps not obviously enough that it sets off alarms.
I've been in touch with folks from the Open Source Security Foundation [0] who is interested in making this a centralized service.
I'm a big believer that functions like this should be centralized under a foundation like that, and have really close connections to package manager maintainers so that we can work together towards solving the problem.
I’m less familiar with LangSmith, but browsing their site suggests they happen to offer observability into LLM interactions in addition to other parts of the workflow lifecycle. This just seems to handle logging and you have to pass all the data yourself- it’s not instrumenting an LLM client, for example.