Three large-scale campaigns have targeted Docker Hub users, planting millions of repositories designed to push malware and phishing sites since early 2021.
I don’t think that’s possible to know definitively, but I will say that this research doesn’t indicate that any images or repositories were compromised, only that a bunch of malware-ridden images were pushed to new repositories. In other words, it’s like they’re implying Google Drive is compromised because someone uploaded malware to their own account and are sending links to it. AFAIK, there wasn’t an attack on Docker as a platform or any existing repositories, these are just new repositories set up to host malware.
The proper mitigation here is to inform users how to find “official” images and repositories. Pulling a Docker image is like downloading any form of software, make sure you trust the source before pulling it down.
The problem is that rootless docker is a pain and no one does it
Nobody does it because it’s generally not necessary.
The reason to run system services as non-root is because there’s no sandbox to keep you safe. Docker provides that sandbox. Yeah, rootless containers are better security-wise since it provides another level of protection, but just using any sandbox follows the Pareto principle (80% benefit for 20% effort). Yeah, you probably should use rootless containers, but the focus should be on making sure the configuration for the container doesn’t expose more than necessary.
The following needs to happen for a user to be impacted:
download untrusted docker image - similar to downloading random executables from the internet
configure untrusted docker image with way more access than is necessary - as in, essentially eliminate the sandbox
run the container on a system with valuable data on it w/o backups - production systems should separate data from applications (i.e. mount points, S3-like buckets, etc), and development systems should have backups
There’s no Docker vulnerability here, so the title is unnecessarily alarmist. Docker hub is still absolutely safe to use, just make sure you trust the repositories and images you use.
What you are saying is not new but you don’t seem to grasp the difference in risk when you run someone else’s configured environment on your system vs. manually setting them up yourself. You save a lot of time by using docker images but it comes with a price.
There’s no docker vulnerability
No need to. Like sudo doesn’t need a vulnerability when you let contributors of some repository use it on your box.
Things like snyk exist for a reason but it’s not mitigation, just monitoring.
You should stop telling people that using docker is no security problem because that’s wrong, as it adds attack surface to even the most secure projects.
Sure, it saves time but things like OPs news will keep popping up in the future like it did in the past. It can’t be fixed other than just not using it in production. At least build your own containers.
I said it’s not a Docker problem in the same way that installing a malware-laden executable isn’t an OS problem. Yes, you can install malware through Docker if you opt-in, the trick is to not opt-in. Check where they’re coming from, look at multiple examples online and see if they’re all using the same image, etc. Do your due diligence, especially if you’re not a developer and thus looking at the Dockerfiles is impractical.
But unless there’s an actual vulnerability in Docker (i.e. exploit in the daemon, credential breach of Docker hub, etc), the finger should be pointed elsewhere. Maybe that’s the kernel (i.e. breaking out of the sandbox), but it’s likely blog posts and users that are at fault.
I looked through those links, and here are my thoughts:
Yes, Docker vulnerabilities are a problem, but an attacker would need to trigger the vulnerability and break out of the sandbox, so it’s not as serious imo; users should stay up to date though, especially if they’re using a popular base image
This comes down to user configuration; if you confine a volume properly, it shouldn’t be an issue.
This is really bad, and it’s heartening to see Docker take it seriously. A breach is by far my biggest concern when it comes to hosted services like this.
I’ll need to find a follow-up to see how Docker handles account deletions, because if names can be reused, that’s ripe for exploits.
Yes, there are plenty of opportunities for exploits in the Docker ecosystem, but I think the risk is pretty low given Docker has it’s own sandbox, which adds a layer of complexity to exploits. The focus shouldn’t necessarily be on something Docker should be doing, but teaching users how to configure containers properly. That means:
one process per container - nudges users to configure things properly, which reduces risk (limit access to data, executables, etc)
rebuild frequently - using more general names (e.g. python:3.10) instead of specific names (python:3.10.3) means you’re more likely to get security updates when you rebuild; this also means scans targeting big repos are more effective since that’s just one org to contact
do regular backups - ransomware attacks should be a complete non-issue, and other malware attacks should be reasonably easy to recover from
No platform is perfect, and practicing the above is probably easier than implementing non-root containers, and non-root itself isn’t a cure-all. Do the above first, and then consider non-root containers in production.
in the same way that installing a malware-laden executable isn’t an OS problem
except no one is doing that. Every major distro hast mechanisms for software supply chain security and reproducible builds.
Do your due diligence, especially if you’re not a developer and thus looking at the Dockerfiles is impractical.
You’re on to something here. If you automate that process, you end up with something we call a package manager.
it’s likely blog posts and users that are at fault.
Exactly. And sincer reviewing Dockerfiles is impractical, there’s no way docker prevents you from shooting your own foot. Distros learned that long ago: Insecure default configs or injected dependencies are a thing of the past there. With docker, those get reintroduced.
Given the number of posts I’ve seen online and adding PPAs or RPM repos, or installing things from source, I’d say that number is a lot higher than 0.
Docker contains that nonsense in a way that’s easy to update. And it allows developers to test production dependencies on a system without those production libs.
something we call a package manager.
Package managers don’t provide a sandbox. You can get one with SELinux, but that requires proper configuration, and then we’re back to the original issue.
Docker provides a decent default configuration. So I think the average user who doesn’t run updates consistently, may add sketchy dependencies, and doesn’t audit things would be better off with Docker.
So yeah, you may have more vulnerabilities with Docker, but they’re less likely to cause widespread issues since each is in its own sandbox. And due to the way its designed, it’s often easier to do things “the right way” (separate concerns) than the wrong way (relax security until it works).
adding PPAs or RPM repos, or installing things from source, I’d say that number is a lot higher than 0.
Nothing wrong with that. Unlike docker that’s cryptographically protected toolchain/buildchain/depchain. Thus, a PPA owner is much less likely to get compromised.
Installing things from source in a secure environment is about as safe as you can get, when obtaining the source securely.
Docker contains that nonsense in a way that’s easy to update.
Really? Ist there already a builtin way to update all installed docker containers?
What’s uneasy about apt full-upgrade?
Package managers don’t provide a sandbox.
I didn’t say that.
average user who doesn’t run updates consistently, may add sketchy dependencies, and doesn’t audit things would be better off with Docker.
That’s false.
but they’re less likely to cause widespread issues since each is in its own sandbox.
Also false. Sandbox evasion is very easy and the next local PE kernel vulnerability only weeks away. Also VM evasion is a thing.
Basically one compromised container giving local execution is enough to pwn your complete host.
Thus, a PPA owner is much less likely to get compromised.
Again, I have yet to see evidence of a Docker repo being compromised.
Repos are almost never compromised. Most issues are from users making things insecure because it’s easier that way. As in:
opening too many ports
enabling passwordless sudo
removing SELinux rules instead of just the parts they need
And so on. Docker makes it really easy to throw away the entire VPS and redeploy, which means those bad configs are less likely to persist in production.
Installing things from source in a secure environment is about as safe as you can get, when obtaining the source securely.
Sure, but that’s only true from the moment you release.
Source-compiled packages rarely get updated, at least in my experience. However, putting source packages in a Docker container increases those chances because it’s really easy to roll back if something goes wrong, and if you upgrade the base, it rebuilds the source with those new libraries, even if statically compiled.
If you’re talking about repository packages built from source (I’m not, I’m talking about side-loading packages), that’s not really a thing anymore with reproducible builds.
Ist there already a builtin way to update all installed docker containers?
In our CI pipeline, yes, I just run a “rebuild all” script and it’s ready for the next deploy to production. Or the script run could be totally automated (I’m currently fighting our devOPs to enable it). We do “<thing>:<version>-<base>” pattern, so worse case, we update from “bullseye” to “bookworm” or whatever separately from the version and ship that. Total process is like 20s per repo (edit Dockerfile out docker-compose.yml, make PR, then ship), then whatever time it takes to build and deploy.
But I don’t necessarily want to upgrade everything. If I upgrade the OS version, I need to check that each process I use is compatible, so I’m more likely to delay doing it. With Docker, I can update part of my stack and it won’t impact anything else. At work, we have dozens of containers, all at various stages of updates (process A is blocked by dependency X, B is blocked by Y, etc), so we’re able to get most of them updated independently and let the others lag until we can fix whatever the issues are.
When installing to the host, I’d need to do all of it at once, which means I’m more likely to stay behind on updates.
That’s false.
Source? I’m coming from ~15 years of experience, and I can say that servers rarely see updates. Maybe it happens in larger firms, but not in smaller shops. But then, larger firms can also run security audits of docker images and whatnot.
Basically one compromised container giving local execution is enough to pwn your complete host.
Maybe. That depends on if the attacker is able to get out of the sandbox. If it was a vulnerability in a process on the host, there’s no sandbox to escape.
So while your Docker containers will probably lag a bit, they come with a second layer of protection. If your host lags a bit, there’s probably no second layer of protection.
It’s just pointing out the risk of letting someone you don’t know with no legal obligations setup your complete environment.
Probably as likely as someone cracking your really secure ssh password. Still, any sane expert will recommend disabling password auth.
How do you know they weren’t compromised?
The problem is that rootless docker is a pain and no one does it. Privileged software sideloading other software is a huge risk.
That risk now became an incident. Even if you’re not affected, the risk still remains.
I don’t think that’s possible to know definitively, but I will say that this research doesn’t indicate that any images or repositories were compromised, only that a bunch of malware-ridden images were pushed to new repositories. In other words, it’s like they’re implying Google Drive is compromised because someone uploaded malware to their own account and are sending links to it. AFAIK, there wasn’t an attack on Docker as a platform or any existing repositories, these are just new repositories set up to host malware.
The proper mitigation here is to inform users how to find “official” images and repositories. Pulling a Docker image is like downloading any form of software, make sure you trust the source before pulling it down.
Nobody does it because it’s generally not necessary.
The reason to run system services as non-root is because there’s no sandbox to keep you safe. Docker provides that sandbox. Yeah, rootless containers are better security-wise since it provides another level of protection, but just using any sandbox follows the Pareto principle (80% benefit for 20% effort). Yeah, you probably should use rootless containers, but the focus should be on making sure the configuration for the container doesn’t expose more than necessary.
The following needs to happen for a user to be impacted:
There’s no Docker vulnerability here, so the title is unnecessarily alarmist. Docker hub is still absolutely safe to use, just make sure you trust the repositories and images you use.
What you are saying is not new but you don’t seem to grasp the difference in risk when you run someone else’s configured environment on your system vs. manually setting them up yourself. You save a lot of time by using docker images but it comes with a price.
No need to. Like sudo doesn’t need a vulnerability when you let contributors of some repository use it on your box.
Things like snyk exist for a reason but it’s not mitigation, just monitoring.
You should stop telling people that using docker is no security problem because that’s wrong, as it adds attack surface to even the most secure projects. Sure, it saves time but things like OPs news will keep popping up in the future like it did in the past. It can’t be fixed other than just not using it in production. At least build your own containers.
Don’t forget various past issues:
I said it’s not a Docker problem in the same way that installing a malware-laden executable isn’t an OS problem. Yes, you can install malware through Docker if you opt-in, the trick is to not opt-in. Check where they’re coming from, look at multiple examples online and see if they’re all using the same image, etc. Do your due diligence, especially if you’re not a developer and thus looking at the Dockerfiles is impractical.
But unless there’s an actual vulnerability in Docker (i.e. exploit in the daemon, credential breach of Docker hub, etc), the finger should be pointed elsewhere. Maybe that’s the kernel (i.e. breaking out of the sandbox), but it’s likely blog posts and users that are at fault.
I looked through those links, and here are my thoughts:
Yes, there are plenty of opportunities for exploits in the Docker ecosystem, but I think the risk is pretty low given Docker has it’s own sandbox, which adds a layer of complexity to exploits. The focus shouldn’t necessarily be on something Docker should be doing, but teaching users how to configure containers properly. That means:
No platform is perfect, and practicing the above is probably easier than implementing non-root containers, and non-root itself isn’t a cure-all. Do the above first, and then consider non-root containers in production.
except no one is doing that. Every major distro hast mechanisms for software supply chain security and reproducible builds.
You’re on to something here. If you automate that process, you end up with something we call a package manager.
Exactly. And sincer reviewing Dockerfiles is impractical, there’s no way docker prevents you from shooting your own foot. Distros learned that long ago: Insecure default configs or injected dependencies are a thing of the past there. With docker, those get reintroduced.
Given the number of posts I’ve seen online and adding PPAs or RPM repos, or installing things from source, I’d say that number is a lot higher than 0.
Docker contains that nonsense in a way that’s easy to update. And it allows developers to test production dependencies on a system without those production libs.
Package managers don’t provide a sandbox. You can get one with SELinux, but that requires proper configuration, and then we’re back to the original issue.
Docker provides a decent default configuration. So I think the average user who doesn’t run updates consistently, may add sketchy dependencies, and doesn’t audit things would be better off with Docker.
So yeah, you may have more vulnerabilities with Docker, but they’re less likely to cause widespread issues since each is in its own sandbox. And due to the way its designed, it’s often easier to do things “the right way” (separate concerns) than the wrong way (relax security until it works).
Nothing wrong with that. Unlike docker that’s cryptographically protected toolchain/buildchain/depchain. Thus, a PPA owner is much less likely to get compromised.
Installing things from source in a secure environment is about as safe as you can get, when obtaining the source securely.
Really? Ist there already a builtin way to update all installed docker containers?
What’s uneasy about
apt full-upgrade?I didn’t say that.
That’s false.
Also false. Sandbox evasion is very easy and the next local PE kernel vulnerability only weeks away. Also VM evasion is a thing.
Basically one compromised container giving local execution is enough to pwn your complete host.
Again, I have yet to see evidence of a Docker repo being compromised.
Repos are almost never compromised. Most issues are from users making things insecure because it’s easier that way. As in:
And so on. Docker makes it really easy to throw away the entire VPS and redeploy, which means those bad configs are less likely to persist in production.
Sure, but that’s only true from the moment you release.
Source-compiled packages rarely get updated, at least in my experience. However, putting source packages in a Docker container increases those chances because it’s really easy to roll back if something goes wrong, and if you upgrade the base, it rebuilds the source with those new libraries, even if statically compiled.
If you’re talking about repository packages built from source (I’m not, I’m talking about side-loading packages), that’s not really a thing anymore with reproducible builds.
In our CI pipeline, yes, I just run a “rebuild all” script and it’s ready for the next deploy to production. Or the script run could be totally automated (I’m currently fighting our devOPs to enable it). We do “<thing>:<version>-<base>” pattern, so worse case, we update from “bullseye” to “bookworm” or whatever separately from the version and ship that. Total process is like 20s per repo (edit Dockerfile out docker-compose.yml, make PR, then ship), then whatever time it takes to build and deploy.
But I don’t necessarily want to upgrade everything. If I upgrade the OS version, I need to check that each process I use is compatible, so I’m more likely to delay doing it. With Docker, I can update part of my stack and it won’t impact anything else. At work, we have dozens of containers, all at various stages of updates (process A is blocked by dependency X, B is blocked by Y, etc), so we’re able to get most of them updated independently and let the others lag until we can fix whatever the issues are.
When installing to the host, I’d need to do all of it at once, which means I’m more likely to stay behind on updates.
Source? I’m coming from ~15 years of experience, and I can say that servers rarely see updates. Maybe it happens in larger firms, but not in smaller shops. But then, larger firms can also run security audits of docker images and whatnot.
Maybe. That depends on if the attacker is able to get out of the sandbox. If it was a vulnerability in a process on the host, there’s no sandbox to escape.
So while your Docker containers will probably lag a bit, they come with a second layer of protection. If your host lags a bit, there’s probably no second layer of protection.
*ouch*