Or: How To Be Helpful Rather Than Annoying
Tools that scan Docker containers for vulnerabilities have been around for a while; Trivy, Snyk, Grype, Clair, etc. all give you the option of throwing an image at them and in response they'll dump out a list of all the known CVEs they find. With the publication of the log4j vulnerabilities over Christmas we've seen a big spike in users making use of these tools to scan the images they use, and subsequently opening Issues against our repos to address these "critical" vulnerabilities.
Unlike a lot of people who publish Docker images, we do regularly update our base images and containers to ensure they're getting patched - you can read about our build pipeline triggers in great detail if that's your thing - and while it can take several days for an upstream patch in an Alpine or Ubuntu package to propagate through our base images and down to the individual application images, it's typically less than 10 days.
Just as (I hope) if you were ill, you wouldn't Google your symptoms, print out the first 3 pages of results, and hand them to your doctor saying "treat me for all of these please", dumping the unfiltered output from a scanning tool into a Github Issue does more harm than good. Let's step through an example to show what I mean.
I'm going to use grype to scan our latest Jellyfin image, which is based on Ubuntu Focal (20.04), the most recent LTS release.
$ grype lscr.io/linuxserver/jellyfin:latest
✔ Vulnerability DB [updated]
✔ Pulled image
✔ Loaded image
✔ Parsed image
✔ Cataloged packages [212 packages]
✔ Scanned image [69 vulnerabilities]
NAME INSTALLED FIXED-IN VULNERABILITY SEVERITY
bash 5.0-6ubuntu1.1 CVE-2019-18276 Low
coreutils 8.30-3ubuntu2 CVE-2016-2781 Low
krb5-locales 1.17-6ubuntu4.1 CVE-2021-36222 Medium
krb5-locales 1.17-6ubuntu4.1 CVE-2018-5709 Negligible
libasn1-8-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low
libass9 1:0.14.0-2 CVE-2020-36430 Medium
libc-bin 2.31-0ubuntu9.2 CVE-2020-6096 Low
libc-bin 2.31-0ubuntu9.2 CVE-2021-3326 Low
libc-bin 2.31-0ubuntu9.2 CVE-2016-10228 Negligible
libc-bin 2.31-0ubuntu9.2 CVE-2021-35942 Medium
libc-bin 2.31-0ubuntu9.2 CVE-2021-33574 Low
libc-bin 2.31-0ubuntu9.2 CVE-2021-38604 Medium
libc-bin 2.31-0ubuntu9.2 CVE-2020-27618 Low
libc-bin 2.31-0ubuntu9.2 CVE-2021-27645 Low
libc-bin 2.31-0ubuntu9.2 CVE-2020-29562 Low
libc-bin 2.31-0ubuntu9.2 CVE-2019-25013 Low
libc6 2.31-0ubuntu9.2 CVE-2020-6096 Low
libc6 2.31-0ubuntu9.2 CVE-2021-3326 Low
libc6 2.31-0ubuntu9.2 CVE-2016-10228 Negligible
libc6 2.31-0ubuntu9.2 CVE-2021-35942 Medium
libc6 2.31-0ubuntu9.2 CVE-2021-33574 Low
libc6 2.31-0ubuntu9.2 CVE-2021-38604 Medium
libc6 2.31-0ubuntu9.2 CVE-2020-27618 Low
libc6 2.31-0ubuntu9.2 CVE-2021-27645 Low
libc6 2.31-0ubuntu9.2 CVE-2020-29562 Low
libc6 2.31-0ubuntu9.2 CVE-2019-25013 Low
libcairo2 1.16.0-4ubuntu1 CVE-2017-7475 Low
libcairo2 1.16.0-4ubuntu1 CVE-2019-6461 Low
libcairo2 1.16.0-4ubuntu1 CVE-2017-9814 Low
libcairo2 1.16.0-4ubuntu1 CVE-2018-18064 Low
libcairo2 1.16.0-4ubuntu1 CVE-2019-6462 Low
libfl2 2.6.4-6.2 CVE-2019-6293 Low
libgmp10 2:6.2.0+dfsg-4 CVE-2021-43618 Low
libgssapi-krb5-2 1.17-6ubuntu4.1 CVE-2021-36222 Medium
libgssapi-krb5-2 1.17-6ubuntu4.1 CVE-2018-5709 Negligible
libgssapi3-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low
libhcrypto4-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low
libheimbase1-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low
libheimntlm0-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low
libhx509-5-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low
libk5crypto3 1.17-6ubuntu4.1 CVE-2021-36222 Medium
libk5crypto3 1.17-6ubuntu4.1 CVE-2018-5709 Negligible
libkrb5-26-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low
libkrb5-3 1.17-6ubuntu4.1 CVE-2021-36222 Medium
libkrb5-3 1.17-6ubuntu4.1 CVE-2018-5709 Negligible
libkrb5support0 1.17-6ubuntu4.1 CVE-2021-36222 Medium
libkrb5support0 1.17-6ubuntu4.1 CVE-2018-5709 Negligible
libpcre3 2:8.39-12build1 CVE-2020-14155 Negligible
libpcre3 2:8.39-12build1 CVE-2017-11164 Negligible
libpcre3 2:8.39-12build1 CVE-2019-20838 Low
libroken18-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low
libsqlite3-0 3.31.1-4ubuntu0.2 CVE-2020-9991 Low
libsqlite3-0 3.31.1-4ubuntu0.2 CVE-2020-9849 Low
libsqlite3-0 3.31.1-4ubuntu0.2 CVE-2020-9794 Medium
libtasn1-6 4.16.0-2 CVE-2018-1000654 Negligible
libwind0-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low
locales 2.31-0ubuntu9.2 CVE-2020-6096 Low
locales 2.31-0ubuntu9.2 CVE-2021-3326 Low
locales 2.31-0ubuntu9.2 CVE-2016-10228 Negligible
locales 2.31-0ubuntu9.2 CVE-2021-35942 Medium
locales 2.31-0ubuntu9.2 CVE-2021-33574 Low
locales 2.31-0ubuntu9.2 CVE-2021-38604 Medium
locales 2.31-0ubuntu9.2 CVE-2020-27618 Low
locales 2.31-0ubuntu9.2 CVE-2021-27645 Low
locales 2.31-0ubuntu9.2 CVE-2020-29562 Low
locales 2.31-0ubuntu9.2 CVE-2019-25013 Low
login 1:4.8.1-1ubuntu5.20.04.1 CVE-2013-4235 Low
passwd 1:4.8.1-1ubuntu5.20.04.1 CVE-2013-4235 Low
perl-base 5.30.0-9ubuntu0.2 CVE-2020-16156 MediumTerrifying, right? 69 vulnerabilities? The image must be a disaster.
Well, perhaps not. For a start you'll notice the empty "FIXED-IN" column, that's because none of these vulnerabilities have patches available in the version of Ubuntu being used. This could be because they're brand-new vulnerabilities, or it could be because the maintainers have determined that the vulnerability isn't practically exploitable, or that it's not serious enough to justify the engineering time, or it could just be an oversight. You'll also notice that they're all Negligible, Low, or Medium severity, no High or Critical, which makes sense given the above.
So straight up you can see that reporting this to us as-is would be pretty useless as we can't patch any of the listed vulnerabilities anyway, but maybe there are some workarounds that could be applied as with log4j? Let's dig a bit deeper and look at one of those Medium severity vulnerabilities
perl-base 5.30.0-9ubuntu0.2 CVE-2020-16156 MediumThis is a vulnerability in perl affecting CPAN. Jellyfin doesn't use Perl, it's just a built-in part of a basic Ubuntu install, but it's possible it could still be an issue so we'll keep looking. Reading the write-up on the Perl site we discover that:
All three issues involve someone setting up a malicious CPAN mirror, and getting people to start using it.
It's clear at this point that nobody is going to be exploiting this vulnerability in the Jellyfin image unless they somehow get local access to the running container, in which case you've got waaay bigger problems than someone tricking you into installing malicious CPAN modules.
Let's look at another:
libass9 1:0.14.0-2 CVE-2020-36430 MediumThis CVE could be more of a problem; the wonderfully named libass is a portable subtitle renderer for the ASS/SSA subtitle format, which sounds like something Jellyfin might use. However, there's no patch available and no workaround to be found - it's also not remotely exploitable (it would require a user to download and use malicious subtitle files) and only results in a crash, not code execution, which makes it more of an annoyance than an actual risk.
Providing us with a long list of CVEs that are variously not exploitable, not patchable, and/or not actually creating a risk for users of the image doesn't help us fix anything, it just dumps a lot of verification work onto our team and makes you look lazy. As an example of our automated package update process, again with our Jellyfin container, this is a grype scan from yesterday - this time I'm filtering to only show vulnerabilities with patches available, to reduce the amount of noise.
$ grype --only-fixed lscr.io/linuxserver/jellyfin:10.7.7-1-ls149
✔ Vulnerability DB [no update available]
✔ Pulled image
✔ Loaded image
✔ Parsed image
✔ Cataloged packages [212 packages]
✔ Scanned image [71 vulnerabilities]
NAME INSTALLED FIXED-IN VULNERABILITY SEVERITY
libsystemd0 245.4-4ubuntu3.14 245.4-4ubuntu3.15 CVE-2021-3997 Medium
libudev1 245.4-4ubuntu3.14 245.4-4ubuntu3.15 CVE-2021-3997 Medium One medium severity patchable CVE, which isn't exploitable because our containers don't use systemd. The patch in question was released on the 13th of January, we picked it up automatically, updated our base images, and subsequently updated our Jellyfin image, which was then published today, the 20th of January. A 7 day turnaround, without human intervention.
So, How Should This Work?
If you're concerned about the security of one of our images, or think you've found a vulnerability how should you go about addressing it?
DON'T
- Blindly run vulnerability scanning tools and dump the output into a Github Issue.
- Publicly disclose vulnerability information, proof of concept code, or anything else that could be harmful to other users of the image.
- Make demands, or generally behave aggressively, in service of getting vulnerabilities addressed.
DO
- Check if the vulnerability is in the container itself or the upstream application - we can't usually fix other people's software but we may be able to apply a workaround.
- Verify that a vulnerability is actually applicable and exploitable for the image in question.
- Contact us privately to disclose anything you've found that you believe to be exploitable.
In summary: It's healthy to be aware of the security of the images you're running, by all means run image scanning tools against them to help you assess your position, but please don't just take those scan results and throw them at us while demanding we "fix" it. If you think you've found a genuine issue or exploitable vulnerability then please disclose it to us, or the upstream project, in a responsible fashion.