Config Based CVE Matching for Linux Kernel

Japanese version here.

This article is a blog version of that I presented as Config Based CVE Matching for Linux Kernel at OSS Summit Japan 2022.

In the embedded Linux industory, the CVE search method is widely used to fix their software vulnerabilities. They search the NVD by package name and version number to obtain a list of vulnerabilities in that package. They check for real impact based on the product use case and build configuration, and if so, fix it with a cherry-picked patch.

The problem is, too many CVEs are reported for Linux kernel, compared to other software. About 100 CVEs a year are reported for Linux kernel, while others are normally 0 to 10 a year.

People say “Cherry-picking is wrong, just use the latest LTS kernel”. That’s true, but how often should we deliver software updates? LTS kernel is released twice a week on average. In many cases we can say “Our kernel is not affected by the CVE”, and doing so will reduce the frequency of software updates.

In this article, I will describe two methods to determine whether a Linux kernel is affected by a given CVE.

Triage method 1: Using the Ubuntu CVE Tracker to identify the vulnerable version range

For many Linux kernel CVEs, only the fixed major version is registered to the NVD.

If your kernel is 5.15.y (5.15 is released on Oct, 2021), because most CVEs found after Nov 2021 are fixed in Linux 5.16 or later and NVD only describes “linux < 5.16 is affected”, CVE search tools report as your kernel is affected by these CVEs, including the CVEs related to the new kernel features implemented in linux >= 5.16. What we need is the proper version range that is affected by a given CVE.

The Ubuntu CVE Tracker provides the information of commit ids that introduced and fixed the bug for each CVE. I call these break-commit and fix-commit. For CVE-2021-45480, you can find break-fix-commits at https://ubuntu.com/security/CVE-2021-45480. This human-readable page is generated from a machine-readable text file, this: https://git.launchpad.net/ubuntu-cve-tracker/tree/active/CVE-2021-45480.

Patches_linux:
 break-fix: aced3ce57cd37b5ca332bcacd370d01f5a8c5371 5f9562ebe710c307adc5f666bf1a2162ee7977c0

For CVE-2021-45480, the bug was introduced in aced3ce57cd37b5ca332bcacd370d01f5a8c5371 and fixed in 5f9562ebe710c307adc5f666bf1a2162ee7977c0. These are mainline commit ids. To find out the released version of these commit ids, you can try the following command.

$ git remote -v
origin  https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux.git (fetch)
origin  https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux.git (push)
$ git fetch
$ git commit-graph write --reachable
$ git tag --contains aced3ce57cd37b5ca332bcacd370d01f5a8c5371 | sort -V | head -1
v5.13
$ git tag --contains 5f9562ebe710c307adc5f666bf1a2162ee7977c0 | sort -V | head -1
v5.16

Note that you can skip git commit-graph command, but this makes git tag --contains command blazing fast.

In this case, the bug for CVE-2021-45480 is introduced in 5.13 and fixed in 5.16, so your 5.15.y kernel may affected by the CVE. But, because 5.15 is still maintained, the fix commit 5f9562ebe710c307adc5f666bf1a2162ee7977c0 must be backported to the linux-5.15.y branch. How can we determine the vulnerable version range in linux-5.15.y branch? According to the stable kernel maintenance rules, the backported commits should have an annotation of the corresponding mainline commit.

The upstream commit ID must be specified with a separate line above the commit text, like this:

commit <sha1> upstream.

https://www.kernel.org/doc/html/v5.15/process/stable-kernel-rules.html

So, if the fix commit 5f9562ebe710c307adc5f666bf1a2162ee7977c0 is backported to the linux-5.15.y branch, we can search the commit log by the upstream commit id 5f9562ebe710c307adc5f666bf1a2162ee7977c0 to find backported commit, like this:

$ git log --grep=5f9562ebe710c307adc5f666bf1a2162ee7977c0 v5.15..linux-5.15.y
commit 68014890e4382ff9192e1357be39b7d0455665fa
Author: Hangyu Hua <hbh25y@gmail.com>
Date:   Tue Dec 14 18:46:59 2021 +0800

    rds: memory leak in __rds_conn_create()

    [ Upstream commit 5f9562ebe710c307adc5f666bf1a2162ee7977c0 ]

    __rds_conn_create() did not release conn->c_path when loop_trans != 0 and
    trans->t_prefer_loopback != 0 and is_outgoing == 0.

    Fixes: aced3ce57cd3 ("RDS tcp loopback connection can hang")
    Signed-off-by: Hangyu Hua <hbh25y@gmail.com>
    Reviewed-by: Sharath Srinivasan <sharath.srinivasan@oracle.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>
    Signed-off-by: Sasha Levin <sashal@kernel.org>
$ git tag --contains 68014890e4382ff9192e1357be39b7d0455665fa | sort -V | head -1
v5.15.11

The CVE-2021-45480 is fixed in 5.15.11, in linux-5.15.y branch. The version range affected by the CVE-2021-45480 are:

In this article I grep’d the branch history by commit id, but you can find more practical regular expression at CIP project repo for automation.

# https://gitlab.com/cip-project/cip-kernel/cip-kernel-sec/-/blob/master/scripts/import_stable.py

RE_USE = {'hash': r'[0-9a-f]{40}'}
BACKPORT_COMMIT_TOP_RE = re.compile(
    r'^(?:' r'commit ({hash})(?: upstream\.?)?'
    r'|'    r'\[ [Uu]pstream commit ({hash}) \]'
    r'|'    r'\[ commit ({hash}) upstream \]'
    r')$'
    .format(**RE_USE))
BACKPORT_COMMIT_ANYWHERE_RE = re.compile(
    r'^(?:' r'\(cherry[- ]picked from commit ({hash})\)'
    r'|'    r'\(backported from(?: commit)? ({hash})\b.*'  # Ubuntu
    r')$'
    .format(**RE_USE))

Triage method 2: Check if the vulnerable code is compiled with koverage command

Most CVEs for Linux kernel are related to arch or drivers. For embedded Linux platform, we normally optimize build configuration for a given hardware, so the most CVEs can be classified as not-for-us problem.

You can use koverage command to determine whether a given patch is compiled or not. As you know the commit id that introduced the bug for a given CVE, if the patch is not compiled with your .config, you can say that your kernel is not affected by the CVE.

As an example, I try CVE-2021-45480 again. The break commit is aced3ce57cd37b5ca332bcacd370d01f5a8c5371, so execute the following command:

$ pip install kmax
$ cd linux-5.15.4
$ git revert aced3ce57cd37b5ca332bcacd370d01f5a8c5371
$ git revert HEAD
$ git format-patch HEAD^
$ export CROSS_COMPILE=aarch64-linux-gnu-
$ koverage --config ../config.txt \
           --arch arm64 \
           --check-patch 0001-Revert-Revert-RDS-tcp-loopback-connection-can-hang.patch \
           --cross-compiler make \
           -o result.json

In the above command, I’m git-revert-ing the break commit twice. This fixes hunk offsets. Because the koverage command is based on line numbers, we need the diff of the break commit with no hunk offset.

If you passed a patch file with --check-patch to the koverage command, it firstly parse the given file in unified diff format, and lists all “filename:linenumber” pairs that modified by the patch. Then it inserts line number markers to the modified lines of the kernel code, and finally run C preprocessor. If the line number markers are deleted by the C preprocessor, it determines the line is not compiled.

With the above command, the koverage save a report to result.json. It looks like as below:

{
  "headerfile_loc": {
    "net/rds/tcp.h": [
      [61, "FILE_EXCLUDED"],
      [62, "FILE_EXCLUDED"],
      [63, "FILE_EXCLUDED"]
    ]
  },
...

If the all lines are either FILE_EXCLUDED or LINE_EXCLUDED_FILE_INCLUDED, it means all code introduced by the break-commit are not compiled, so your kernel is not affected by the CVE.

The triage flow

To sum up, the CVE triage flow is as follows. It can be done by hand, and of course fully automated.

  1. Firstly check the break-fix commit ids at the Ubuntu CVE Tracker repo. e.g. https://git.launchpad.net/ubuntu-cve-tracker/tree/active/CVE-2021-45480
  2. Check the vulnerable version range in mainline with git tags --contains command. If your kernel major version is out of range, your kernel is not affected by the CVE.
  3. Find the fix-commit in the stable branch you are using with git log --grep command.
  4. Check the vulnerable version range in the stable branch you are using with git tags --contains command. If your kernel version is out of range, your kernel is not affected by the CVE.
  5. Check the break-commit is compiled or not considering your .config with the koverage command. If the all lines modified by the break-commit are not compiled with your .config, your kernel is not affected by the CVE.

Benchmark 1

I took some benchmarks to evaluate the above two methods.

I generated CVE lists of given three kernel versions with a CVE search tool, go-cve-dictionary, and classified the CVEs by the above two methods.

  • The total height represents the number of CVEs reported by CVE search tool
  • The dark green and light yellow area represents the number of CVEs identified as false positives by the method 1
  • The light yellow and light green area represents the number of CVEs identified as false positives by the method 2
  • The light yellow area represents the number of CVEs identified as false positives by the both method 1 and 2
  • The target kernel version 5.4.221, 160, 80 are chosen so that these release date are a year apart
  • I used arm64 defconfig for the method 2

The number of CVEs determined as false positives by the method 1 are almost linear to the release date. As the NVD only records the major version number fixed the bug for each CVE, the CVE search tool reports almost identical CVE list for these minor versions. However, about 100 CVEs are reported and fixed annually, thus the number of false positives continues increase 100 CVEs a year.

The number of CVEs determined as false positives by the method 2 are almost constant to minor version. Whether a CVE is determined to be a false positive by the method 2 is depends on .config and the CVE list reported by the CVE search tool. As I described above, these two parameters are constant to minor version.

Benchmark 2

I took the same benchmark for the latest LTS kernes still maintained.

  • Because these are latest version, of course all serious bugs are fixed, more than 80% are determined as false positives
  • More CVEs are reported for older major versions, because the CVE search tool only consider major version, as the NVD does so

Conclusion

  • For latest LTS kernel, despite most CVEs are already fixed, CVE search tools report hundreds of false positives
  • For old kernels, some parts of CVEs, ~20% in this case, could be ignored by testing whether the patch is effective regarding .config

References


Leave a Reply

Your email address will not be published. Required fields are marked *