I'm only talking about the AuthN problem, by the way. I'm not making a general argument against circuit breaker architectures.

Basically we 100% agree with you that an authentication service should do this job. The HSM is extra credit. Although it does help in cases where the auth service's DB is leaked through some other means (e.g. backups).

I will say that I'd depart with you on the return value of that service. It shouldn't be a bool. It's better to return a token that downstream services can use to independently verify that the authentication service verified the user. Its better for your infrastructure if you aren't passing around direct user IDs but rather a cryptographically signed, short lived token that is only valid for the life of a specific request.

Mutual TLS can be a bit of work to get set up but leads to huge security wins over time as every RPC within your infrastructure is mediated by an authorization layer. We've helped out a bit with the SPIFFE project which is looking to make mutual TLS easy: https://spiffe.io/

This was an explicit omission, at least for now. We left update out of scope because it's better handled by the infrastructure provisioning system (in our case, infrakit). We'll use infrakit to supply updates (and the dm-verity hash, for that matter). Thus we treat infrastructure provisioning system as the trusted 'bootloader' for a cluster of machines. Most datacenter clusters end up having an infrastructure provisioning system, so it makes more sense for those systems to have the OS update responsibility. This ends up meaning less attack surface on the host itself, and serves as a good separation of concerns and least privilege design.

In short:

Kernel Security Incubator - We want to push linux kernel security as much as possible. In service of that, we want linuxkit to be a place where leading-edge linux kernel security patches can land and incubate. Feature examples are Landlock, Wiregurd, okernel, etc. We'll also incubate KSPP and container hardening improvements, like hardening the kernel eBPF JIT and namespacing the IMA subsystem.

Modern and Securely Configured Kernels - Latest kernel, following all KSPP recommendations.

Minimal Base - No extra dependences, just what's needed to run containerd. Absolutely no package manager.

Type Safe, Containerized System Daemons - many linux privescs happen due to escalations using root system daemons. These daemons should be written in typesafe language like OCaml and Rust. We have an Ocaml dchpcd and look to invite more. If you're convinced by https://tonyarcieri.com/it-s-time-for-a-memory-safety-interv..., linuxkit is a place to contribute to the solution.

Built With Hardened Toolchains and Containers - uses notary signing for all dependencies and intermediate builds, uses musl libc for hardened libc implementation + hardened compiler options for building system packages.

Immutable Infrastructure - Linuxkit follows the principle of immutable infrasructure. The filesystem contains a read-only root FS and boots with dm-verity. Trusted boot via infrakit + notary hash lookup is a next step.

All in all, this multi-pronged approach should lead to a much more secure linux base. As is our tradition, we will track progress here: https://github.com/linuxkit/linuxkit/blob/master/docs/securi..., where we'll catalog Linux CVEs and how LinuxKit holds up.

To run a database, attach non-ephemeral storage and write data there.

Default ACLs are clearly the most important line of defense in an orchestrator's security model, because whether a container escape can happen is not something the orchestration system has control over.

Being able to trigger node compromise should have nothing to do with being able to schedule.

I expect Red Hat to issue a retraction shortly. We notified them last night that this post was incorrect.

Source: Security at Docker.

I realize you're busy, but it would be much more helpful than a curt statement that simply claims they are wrong.

But I think there is an interesting topic to address here, that we deal with a lot at Docker.

The problem in a nutshell: if you choose to take the moral high ground and only promote your products with positive marketing (and that is our strategy at Docker - you will never see any Docker marketing content criticizing competitors), you are vulnerable to bullying and unfair criticism by competitors who don't mind hitting under the belt. Then the question is: do you allow yourself to respond and set the record straight? Or would that just legitimize the criticism by bringing more attention to it? On the other hand, not responding is also risky because it emboldens the trolls to take more and more liberties with facts and ethics. This dilemma becomes more and more pressing as you become more successful and more incumbents start considering you a possible threat to their business. Some of these incumbents have been defending their turf for decades by perfecting negative messaging. Like one competing executive once told me - "we eat startups like yours for breakfast". This situation can be bad for morale also, because your team sees their work and reputation dragged in the mud, and can interpret their employer's silence as a failure to stand up and defend them.

The most perverse variation of this problem is when trolls start preemptively painting you as bullies. If that narrative sticks, then you're in trouble, because any attempt to set the record straight will be interpreted as hostility. Now you have two problems: defending yourself against the bullies AND defending yourself against unfair accusations of being a bully.

The root cause of the problem, I think, is the diminishing importance of facts and critical reasoning in the tech community. We are all guilty of this: when was the last time you repeated a factoid about "X doesn't scale", "Y isn't secure", "I heard Z is really evil" without fact-checking it yourself? Be honest. Because of this collective failure to do our own thinking and researching, bullies have a huge first-mover advantage.

I see an direct parallel between the problem of corporate bullying in tech and the problem of partisan bullying in politics. And I think in both cases, there is a big unresolved problem: how do you succeed and do the right thing? How do we collectively change the rules of the game to make bullying and negative communication a less attractive strategy?

I tried really hard to make this a constructive post about a topic I care about. If you interpret any of this as hostile or defensive, that is not at all the intention.

I posted this entry and I work at Red Hat.

Also, if hypothetically the full details made Red Hat look bad, is it fair to assume you would be calling Nathan hostile for sharing them? In that scenario is there any course of action we could follow that would satisfy you?

The article explicitly states the CVE number and the fact that updated packages are available.

The article IMHO doesn't attack nor provoke Docker and its people. Yet the first comment posted here DOES contain direct accusations against Red Hat. I don't think that's helpful nor needed. That's all.

I still think that SELinux and Docker are a good combination and this article helps in understanding why.

Then the text of the article states:

"This CVE reports that if you exec'd into a running container, the processes inside of the container could attack the process that just entered the container. If this process had open file descriptors, the processes inside of the container could ptrace the new process and gain access to those file descriptors and read/write them, even potentially get access to the host network, or execute commands on the host. ... It could do that, if you aren’t using SELinux in enforcing mode."

So, not only does the title make this suggestion, but the text of the article downright says it.

If the claim is wrong, then Docker's security team is right to correct it. However, I think they should do so in a forum other than in the comments of a HN post, be thorough in their explanation, and maintain a professional, polished tone in any communications.

And, of course, Red Hat should correct and/or clarify the post as well.

Honestly, Docker stepped in it here. This appears to be a theme, and with your posts in particular. I personally can't think of another project that causes more drama on HN.

This makes Docker look worse.

The only way SE Linux can look bad is if you were on the fence about its efficacy, and are only now hearing it can't even stop Docker's problems.

"A number of developers from RedHat were once very involved in the project. However, these developers had a very arrogant attitude towards Docker: They wanted docker changed so that it would follow their design ideas for RHEL and systemd. They often made pull requests with poorly written and undocumented code and then they became very agressive when those pull requests were not accepted, saying "we need this change we will ship a patched version of Docker and cause you problems on RHEL if you don't make this change in master." They were arogant and agressive, when at the same time, they had the choice of working with the Docker developers and writting quality code that could actually be merged. Another thing they often said was along the lines of "systemd is THE ONE AND ONLY INIT SYSTEM so why do you care about writing code that might be portable?" Or "even though the universal method works with systemd now. A redesign has already been planned in systemd, so do it the new systemd way and don't be portable."(This is in response to the fact that Docker writes to cgroups, and systemd would like to be the "sole writter to cgroups" some time in the future.) I think everyone got fed up with those people, and Docker has rightly pushed them out."

Not to mention that this is a fairly gnarly CVE - a great catch by SUSE and Docker - and claiming the software is terrible because it contained this seems like a real stretch.

As far as disagreements on the best way to handle a security disclosure, this one is pretty straightforward and was entirely avoidable. The vulnerability in question is a runC vulnerability, it affects equally all products with a dependency on runC (not just Docker). The vulnerability had already been patched in runC, and an update to Docker had already been released and announced. So it was not a zero day. A few vendors (not just Red Hat) have incorrectly announced to their users that they didn't need to upgrade to the latest version of Docker because their enterprise-grade commercial platform would "stop the vulnerability cold". In the case of Red Hat, the commercial differentiator is what they call "security-enhanced Linux" using SELinux.

These vendors are under a lot of pressure to justify the high cost of their enterprise subscription by demonstrating concrete value. A great way to do that is to describe a scary vulnerability in a well-known product like Docker, and show that buying their product is the best protection against it. That is why the article talks about a "Docker vulnerability" instead of a "runC vulnerability" - Docker is a better-known product so the story will be more impactful that way. And it's also why the vulnerability is qualified as a "zero-day" even though it wasn't: it makes the vulnerability scarier.

Red Hat was privately contacted to inform them of their mistake. They privately acknowledged the mistake. When the article hit the Hacker News front page, they were again privately informed of that as well. In spite of multiple requests, after several days they have still not corrected the article. This puts the security of Red Hat users at risk by continuing to tell them that an upgrade to Docker 1.12.6 is not necessary. This is especially disappointing because of the obvious conflict of interest. It was a perfect opportunity for Red Hat to set the bar high for themselves and remove any doubts that they might put the security of their users before commercial interest.

The saddest part is that RHEL and SELinux have genuine security benefits that could be explained in very compelling ways without these shady marketing tactics.

Because if the answer is "No", and there's some other way to bypass SELinux and exploit this bug, it raises more grave accusation of RedHad - false statement about the vulnerability workaround.

Can you please give a link to the announce from Red Hat or someone else urging their users that they don't need to upgrade? It would be the last thing closing the question.

First post saved by archive.org: http://web.archive.org/web/20170114090437/http://rhelblog.re... Latest post: http://web.archive.org/web/20170117054512/http://rhelblog.re...

  $ wdiff -n -3 first latest
  
  ======================================================================
  [-Docker 0-Day Stopped Cold by-] SELinux
  ======================================================================
   SELinux {+Mitigates docker exec Vulnerability+}
  ======================================================================
   Fixed packages [-have been-] {+are being+} prepared and shipped for RHEL
  ======================================================================
   [-Centos.-] {+CentOS.+}
  ======================================================================
  
  
  
  [-Stopping 0-Days with-] SELinux
  ======================================================================
   SELinux {+Reduces Vulnerability+}
  ======================================================================
  
  
  [-How about a more visually enticing demo? Check out this animation:-]
  ======================================================================
   we were glad to see that our customers were [-safe-] {+safer+} if running containers with setenforce 1
  ======================================================================
   {+Even with SELinux in enforcement, select information could be leaked, so it is recommended that users patch to fully remediate the issue.+}
  {++}
  {+This post has been updated to better reflect SELinux’s impact on the Docker exec vulnerability and the changing threat landscape facing Linux containers.+}
  ======================================================================

You as a Docker developer see the post as an attack on your project. But most of sysadmins and kernel developers see it as a nice example of the fruits of invisible long work - when well cared system with accurately configured security restrictions saves from some vulnerabilities.

Anyway, it not means underestimation of the Docker and you great job. Sorry you've got stressed by all this noise.

Giving people time to patch before releasing the details of how it can be exploited isn't a bad security practice.

http://img.scoop.it/vr-SoyYI8yKYsOf0vxriWrnTzqrqzN7Y9aBZTaXo..., from http://www.docker.com/sites/default/files/WP_IntrotoContaine... (page 9)

Don't know if it was marketing material when you published it in that whitepaper, but it definitely became marketing material when the @Docker twitter account tweeted it (https://twitter.com/docker/status/768232653665558528).

- The comparison table is part if an independent study, not authored or commissioned by Docker.

- The table shows the strengths and weaknesses of different container runtimes; weaknesses are highlighted for all of them, including Docker

- The table is used in Docker material to illustrate the point that independent security researchers consider Docker secure. Nowhere do we make the point that other products are insecure. I encourage you to read the whole material and decide for yourself.

- The context for this material was to respond to a massive communication campaign painting Docker as insecure.

(Disclaimer: I implemented some, but definitely not all, of those security features in rkt, and I currently work at CoreOS)

"This was independent, not authored or commissioned by us." - "the fact that we posted it on our Twitter as "why your containers are safer with Docker", posted a lengthy blog article in no way means we were criticizing competitors..."

"Nowhere do we make the point that other products are insecure." No, just "less secure".

"The context we were in was responding to a massive communication campaign ... " - so you were responding to criticism by what, exactly? Oh, yeah, that independent study that you just decided to post about?

If I were your counselor, I'd advise you to do nothing other than stick to the facts; make the best product you can; delight your customers; take pride in the great work you do; and apologize openly and honestly when you make avoidable mistakes. You can't make everyone happy, so focus on the people you can, and aim to exceed their expectations.

This is a naive perspective that doesn't bear out in the real world. It's important to know that by taking the "high road", you are putting yourself at a competitive disadvantage. Someday, those with fewer scruples may have to pay the piper and their dubiously-maintained prosperity may disintegrate ... but then again, maybe not.

Most often, the truth is that large companies are pretty ruthless, and have consolidated such a huge amount of control that it's extremely difficult to do anything about anything they do or have done. They control the messaging, they have a reputation that supersedes any complaint an individual may make, etc. Those companies do slowly atrophy, but usually it's more because they've lost sight of the founder's vision that originally connected with the masses than that they're engaging in questionable tactics.

If you're taking a position out of principle, that has to suffice for itself, because it probably will cost you in material terms.

You start with a grain of truth — something that actually happened in reality. In this case, it was a joke protesting systemd hegemony.

Perhaps you thought that joke was in poor taste. But lets leave that aside for now.

So you start with an actual fact. Then, you exaggerate/falsify it, changing the details pretty wildly, and present this story of something that supposedly happened. In fact, nothing like that happened... but it sort of feels like something that might have happened. It vaguely resembles the actual event that did happen (in that, a Docker employee did wear a badge with an opinionated phrase on it at a conference).

The key thing, though, is that what you describe is completely and utterly different from the thing that actually happened in reality world[1].

You might not even be the person who changed the details to make the story more compelling (and false). Maybe you got this information from a post shared on Facebook, or from an email forwarded by your uncle.

Either way, though, the impact of your comment is to pollute the body of discussion and degrade the collective understanding of this topic. (If this process feels familiar, it's because it is exactly the process that eventually caused the failure of the American democracy... just at a much smaller scale).

Personally I don't have any stake in the Docker/RedHat relationship and I don't care about it. I only looked up what actually happened[1] because the idea of a Docker employee wearing an official badge that says "I reject red hat patches" seemed so unlikely to have occurred that it sent my bullshitometer into the red.

Suggestion: when something smells like bullshit, don't eat it without conducting a bit of research.

[1]: https://www.facebook.com/hackerspace.budapest/photos/a.40703...

But not nearly as furious as I'd be had it actually read "I reject red hat patches" as claimed above.

Given krakensden's posting (https://news.ycombinator.com/item?id=13399853):

> Especially given Docker Inc.'s history of counterproductive Red Hat hostility.

I think that it is clear that andrewguenther meant 'hostile' instead of 'history' in (https://news.ycombinator.com/item?id=13400383):

> Docker has been very openly history towards Red Hat in the past.

Let he who has never typed a passing thought rather than the word he meant cast the first stone.

That said, the solution is the same as with every other piece of software -- update to latest to get security fixes.

In case it is not obvious, the comment above is by Nathan McCauley, who is the Director of Security for Docker.

Source: https://news.ycombinator.com/user?id=bigmac

Please don't sign comments; they're already signed with your username. If other users want to learn more about you, they can click on it to see your profile.

Also, there is an assumption that the signature contains up to date information and/or does not change over time. The latter situation would else impact historical purpose. The signature has changed and does not refer to the position/information related to the moment of writing.

I agree with how both jwildeboer (Jan) and shykes (Solomon) approached this. Much appreciated in this case.

But yes, in a normal situation, this is irrelevant and the username signature is sufficient.

The comment was signed with his username, and his Docker affiliation was disclosed under said username. That was all that was needed to add validity to the claims in the comment.

All HN comments have that "generic signature". All HN users are free to disclose information about themselves on their profile, and all HN readers are free to click usernames to learn more about the the people who comment on HN.

It really is that simple.

Give it a rest. This is a semi-anonymous forum where people's identities aren't tied to their usernames. This isn't name dropping, it's providing helpful context.

Some of the comments from Red Hat previously implied that they thought the vulnerability could only be exploited via ptrace, which SELinux denied by default for Docker containers. That's definitely not true; ptrace was used in the PoC because it's easy and likely to win the race condition, but you can also grab file descriptors out of /proc/$pid/fd.

However, the blog post appears to show SELinux stopping attacks that don't involve ptrace, because SELinux forbids writing to an open file or an open network socket that has the wrong context. If Docker believes there are attack vectors that aren't covered by the default SELinux policies (such as writing to something that's not a regular file or network socket), they might be unwilling to disclose that too loudly until Red Hat gets around to saying "Uh, actually please patch".

Edit: Don't downvote people trying to help me improve my english. :(

Just kidding.

It's relevant and vital to know the background of people who are making statements like this.

And sorry but not everyone is a kernel engineer who can navigate the truth between RedHat and Docker.

Disclaimer: I work at Red Hat

Where is this explicitly mentioned in the post?

I got the opposite impression reading this story titled "Docker 0-Day Stopped Cold by SELinux", with the closing statement "When we heard about this vulnerability we were glad to see that our customers were safe".

I'm sure your customers are glad to hear that as well, but it sounds like the Docker folks have reason to believe SELinux doesn't fully mitigate this vulnerability.

FTA

"Fixed packages have been prepared and shipped for RHEL as well as Fedora and Centos."

So updates were made, tested and made available. Our customers typically implement these security related updates very fast.

with that out of the way, the article explains how SELinux can mitigate this and similar issues.

And I am 100% sure that we coordinated the update and changes with Docker because that's how Open Source works.

That sounds like a sensible approach and very much related to why I raised concerns about the original title and closing statement earlier. Someone could easily have assessed that there was zero impact (based on the first revision* of your marketing material) if SELinux was enabled and, consequently, find no need to "define a solution and start to work" - why would you update when your OS vendor explicitly says you're safe?

It would have been extremely easy to recommend your customers to install the updated packages, but you didn't do that initially - despite from such a recommendation being quite standard, and despite being notified by the people who found and fixed this vulnerability warning that customers should still update.

Instead you seem to have used this as an marketing opportunity at the expense of your own customers' security. As it turns out, SELinux did in fact not fully mitigate the issue (by Red Hat's own admission in the updated blog post and CVE).

---

* I'm referring to the first revision because the post and CVE have since been updated several times (as pointed out elsewhere in this discussion). A recap of some of the changes that are relevant to this exchange and the phrasing I mentioned earlier:

1. The title "Docker 0-Day Stopped Cold by SELinux" has been renamed to "SELinux Mitigates container Vulnerability" -- accurately reflecting the fact that it was not:

  a) a Docker (it was runc)
  b) 0-Day (patches were released for runc afaik)
  c) Stopped Cold (it was mitigated but still leaking information)

3. The sentence you referred to "Fixed packages have been prepared and shipped for RHEL as well as Fedora and Centos." has been changed to "Fixed packages are being prepared and shipped for RHEL as well as Fedora and CentOS.". Honestly haven't looked into whether the packages were actually released at the time this post was published (?), but I'll assume Red Hat didn't change the wording here for no reason - there's quite a difference between updates that "are being prepared" rather than "have been prepared".

Instead, install using instructions here: https://docs.docker.com/engine/installation/linux/centos/#/i...

  export DOCKER_CONTENT_TRUST=1
  docker pull ubuntu

However, there's some usability problems here that I'd like to bring to your attention:

- There is no indication that the pull is different; no output from trust verification that it is indeed signed. This means I have to trust that Docker did the right thing, with no means of verification.

- Inspecting the image after pulling gives no indication that the image is signed, and gives me no way to do my own signature validation on the image.

- It does fail properly when pulling an unsigned image. Yay!

- Docker run initiates a connection back to notary, and fails if it can not connect. This makes me uneasy - it makes me wonder what data is passed, how it's being used, what changes are being made according to the response, and so forth.

- Using trust is a per-command decision, instead of a daemon setting.

- There's no clear method to revoke a signing key if identified to be malicious.

- This seems like a good candidate for "secure by default" once some of the usability issues are resolved.