Or: How To Be Helpful Rather Than Annoying
Tools that scan Docker containers for vulnerabilities have been around for a while; Trivy, Snyk, Grype, Clair, etc. all give you the option of throwing an image at them and in response they'll dump out a list of all the known CVEs they find. With the publication of the log4j vulnerabilities over Christmas we've seen a big spike in users making use of these tools to scan the images they use, and subsequently opening Issues against our repos to address these "critical" vulnerabilities.
Unlike a lot of people who publish Docker images, we do regularly update our base images and containers to ensure they're getting patched - you can read about our build pipeline triggers in great detail if that's your thing - and while it can take several days for an upstream patch in an Alpine or Ubuntu package to propagate through our base images and down to the individual application images, it's typically less than 10 days.
Just as (I hope) if you were ill, you wouldn't Google your symptoms, print out the first 3 pages of results, and hand them to your doctor saying "treat me for all of these please", dumping the unfiltered output from a scanning tool into a Github Issue does more harm than good. Let's step through an example to show what I mean.
I'm going to use grype to scan our latest Jellyfin image, which is based on Ubuntu Focal (20.04), the most recent LTS release.
$ grype lscr.io/linuxserver/jellyfin:latest ✔ Vulnerability DB [updated] ✔ Pulled image ✔ Loaded image ✔ Parsed image ✔ Cataloged packages [212 packages] ✔ Scanned image [69 vulnerabilities] NAME INSTALLED FIXED-IN VULNERABILITY SEVERITY bash 5.0-6ubuntu1.1 CVE-2019-18276 Low coreutils 8.30-3ubuntu2 CVE-2016-2781 Low krb5-locales 1.17-6ubuntu4.1 CVE-2021-36222 Medium krb5-locales 1.17-6ubuntu4.1 CVE-2018-5709 Negligible libasn1-8-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low libass9 1:0.14.0-2 CVE-2020-36430 Medium libc-bin 2.31-0ubuntu9.2 CVE-2020-6096 Low libc-bin 2.31-0ubuntu9.2 CVE-2021-3326 Low libc-bin 2.31-0ubuntu9.2 CVE-2016-10228 Negligible libc-bin 2.31-0ubuntu9.2 CVE-2021-35942 Medium libc-bin 2.31-0ubuntu9.2 CVE-2021-33574 Low libc-bin 2.31-0ubuntu9.2 CVE-2021-38604 Medium libc-bin 2.31-0ubuntu9.2 CVE-2020-27618 Low libc-bin 2.31-0ubuntu9.2 CVE-2021-27645 Low libc-bin 2.31-0ubuntu9.2 CVE-2020-29562 Low libc-bin 2.31-0ubuntu9.2 CVE-2019-25013 Low libc6 2.31-0ubuntu9.2 CVE-2020-6096 Low libc6 2.31-0ubuntu9.2 CVE-2021-3326 Low libc6 2.31-0ubuntu9.2 CVE-2016-10228 Negligible libc6 2.31-0ubuntu9.2 CVE-2021-35942 Medium libc6 2.31-0ubuntu9.2 CVE-2021-33574 Low libc6 2.31-0ubuntu9.2 CVE-2021-38604 Medium libc6 2.31-0ubuntu9.2 CVE-2020-27618 Low libc6 2.31-0ubuntu9.2 CVE-2021-27645 Low libc6 2.31-0ubuntu9.2 CVE-2020-29562 Low libc6 2.31-0ubuntu9.2 CVE-2019-25013 Low libcairo2 1.16.0-4ubuntu1 CVE-2017-7475 Low libcairo2 1.16.0-4ubuntu1 CVE-2019-6461 Low libcairo2 1.16.0-4ubuntu1 CVE-2017-9814 Low libcairo2 1.16.0-4ubuntu1 CVE-2018-18064 Low libcairo2 1.16.0-4ubuntu1 CVE-2019-6462 Low libfl2 2.6.4-6.2 CVE-2019-6293 Low libgmp10 2:6.2.0+dfsg-4 CVE-2021-43618 Low libgssapi-krb5-2 1.17-6ubuntu4.1 CVE-2021-36222 Medium libgssapi-krb5-2 1.17-6ubuntu4.1 CVE-2018-5709 Negligible libgssapi3-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low libhcrypto4-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low libheimbase1-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low libheimntlm0-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low libhx509-5-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low libk5crypto3 1.17-6ubuntu4.1 CVE-2021-36222 Medium libk5crypto3 1.17-6ubuntu4.1 CVE-2018-5709 Negligible libkrb5-26-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low libkrb5-3 1.17-6ubuntu4.1 CVE-2021-36222 Medium libkrb5-3 1.17-6ubuntu4.1 CVE-2018-5709 Negligible libkrb5support0 1.17-6ubuntu4.1 CVE-2021-36222 Medium libkrb5support0 1.17-6ubuntu4.1 CVE-2018-5709 Negligible libpcre3 2:8.39-12build1 CVE-2020-14155 Negligible libpcre3 2:8.39-12build1 CVE-2017-11164 Negligible libpcre3 2:8.39-12build1 CVE-2019-20838 Low libroken18-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low libsqlite3-0 3.31.1-4ubuntu0.2 CVE-2020-9991 Low libsqlite3-0 3.31.1-4ubuntu0.2 CVE-2020-9849 Low libsqlite3-0 3.31.1-4ubuntu0.2 CVE-2020-9794 Medium libtasn1-6 4.16.0-2 CVE-2018-1000654 Negligible libwind0-heimdal 7.7.0+dfsg-1ubuntu1 CVE-2021-3671 Low locales 2.31-0ubuntu9.2 CVE-2020-6096 Low locales 2.31-0ubuntu9.2 CVE-2021-3326 Low locales 2.31-0ubuntu9.2 CVE-2016-10228 Negligible locales 2.31-0ubuntu9.2 CVE-2021-35942 Medium locales 2.31-0ubuntu9.2 CVE-2021-33574 Low locales 2.31-0ubuntu9.2 CVE-2021-38604 Medium locales 2.31-0ubuntu9.2 CVE-2020-27618 Low locales 2.31-0ubuntu9.2 CVE-2021-27645 Low locales 2.31-0ubuntu9.2 CVE-2020-29562 Low locales 2.31-0ubuntu9.2 CVE-2019-25013 Low login 1:4.8.1-1ubuntu5.20.04.1 CVE-2013-4235 Low passwd 1:4.8.1-1ubuntu5.20.04.1 CVE-2013-4235 Low perl-base 5.30.0-9ubuntu0.2 CVE-2020-16156 Medium
Terrifying, right? 69 vulnerabilities? The image must be a disaster.
Well, perhaps not. For a start you'll notice the empty "FIXED-IN" column, that's because none of these vulnerabilities have patches available in the version of Ubuntu being used. This could be because they're brand-new vulnerabilities, or it could be because the maintainers have determined that the vulnerability isn't practically exploitable, or that it's not serious enough to justify the engineering time, or it could just be an oversight. You'll also notice that they're all Negligible, Low, or Medium severity, no High or Critical, which makes sense given the above.
So straight up you can see that reporting this to us as-is would be pretty useless as we can't patch any of the listed vulnerabilities anyway, but maybe there are some workarounds that could be applied as with log4j? Let's dig a bit deeper and look at one of those Medium severity vulnerabilities
perl-base 5.30.0-9ubuntu0.2 CVE-2020-16156 Medium
This is a vulnerability in perl affecting CPAN. Jellyfin doesn't use Perl, it's just a built-in part of a basic Ubuntu install, but it's possible it could still be an issue so we'll keep looking. Reading the write-up on the Perl site we discover that:
All three issues involve someone setting up a malicious CPAN mirror, and getting people to start using it.
It's clear at this point that nobody is going to be exploiting this vulnerability in the Jellyfin image unless they somehow get local access to the running container, in which case you've got waaay bigger problems than someone tricking you into installing malicious CPAN modules.
Let's look at another:
libass9 1:0.14.0-2 CVE-2020-36430 Medium
This CVE could be more of a problem; the wonderfully named libass is a portable subtitle renderer for the ASS/SSA subtitle format, which sounds like something Jellyfin might use. However, there's no patch available and no workaround to be found - it's also not remotely exploitable (it would require a user to download and use malicious subtitle files) and only results in a crash, not code execution, which makes it more of an annoyance than an actual risk.
Providing us with a long list of CVEs that are variously not exploitable, not patchable, and/or not actually creating a risk for users of the image doesn't help us fix anything, it just dumps a lot of verification work onto our team and makes you look lazy. As an example of our automated package update process, again with our Jellyfin container, this is a grype scan from yesterday - this time I'm filtering to only show vulnerabilities with patches available, to reduce the amount of noise.
$ grype --only-fixed lscr.io/linuxserver/jellyfin:10.7.7-1-ls149 ✔ Vulnerability DB [no update available] ✔ Pulled image ✔ Loaded image ✔ Parsed image ✔ Cataloged packages [212 packages] ✔ Scanned image [71 vulnerabilities] NAME INSTALLED FIXED-IN VULNERABILITY SEVERITY libsystemd0 245.4-4ubuntu3.14 245.4-4ubuntu3.15 CVE-2021-3997 Medium libudev1 245.4-4ubuntu3.14 245.4-4ubuntu3.15 CVE-2021-3997 Medium
One medium severity patchable CVE, which isn't exploitable because our containers don't use systemd. The patch in question was released on the 13th of January, we picked it up automatically, updated our base images, and subsequently updated our Jellyfin image, which was then published today, the 20th of January. A 7 day turnaround, without human intervention.
So, How Should This Work?
If you're concerned about the security of one of our images, or think you've found a vulnerability how should you go about addressing it?
- Blindly run vulnerability scanning tools and dump the output into a Github Issue.
- Publicly disclose vulnerability information, proof of concept code, or anything else that could be harmful to other users of the image.
- Make demands, or generally behave aggressively, in service of getting vulnerabilities addressed.
- Check if the vulnerability is in the container itself or the upstream application - we can't usually fix other people's software but we may be able to apply a workaround.
- Verify that a vulnerability is actually applicable and exploitable for the image in question.
- Contact us privately to disclose anything you've found that you believe to be exploitable.
In summary: It's healthy to be aware of the security of the images you're running, by all means run image scanning tools against them to help you assess your position, but please don't just take those scan results and throw them at us while demanding we "fix" it. If you think you've found a genuine issue or exploitable vulnerability then please disclose it to us, or the upstream project, in a responsible fashion.