1. Overview

This site collects three related artifacts:

  1. CodeQL databases built from open-source projects.
  2. CodeQL analysis results for those projects.

The intended uses are:

  1. Provide ready-made databases for CodeQL query development.
  2. Provide large ready-made analysis result sets for work on presenting and disseminating results that may number in the millions.
  3. Provide databases for the MRVA project.

Future work includes summary pages that explain the workflow, data locations, and comparisons of full tracing builds to --build-mode=none

The corpus is based on deployed open-source software rather than arbitrary GitHub repositories. The databases were built from a random sample of 7000 Debian projects drawn from a much larger Debian package population. Of those 7000 projects, 2487 successfully produced a C/C++ CodeQL database using full build tracing. Of those projects, 843 also successfully produced a C/C++ CodeQL database using --build-mode=none, where CodeQL reads the source tree without tracing a build. No other languages are included yet.

This document follows the standard, but often under-explained, end-to-end CodeQL path:

  1. Convert source code into CodeQL database bundles. The database is the artifact that CodeQL queries examine.
  2. Apply query suites to one or more CodeQL databases. Each analyzed database produces one SARIF file containing zero or more results.
  3. Build reports and comparisons from the SARIF results.

The same path applies to one database and to large corpora, but the scale changes the engineering problem.

  1. Single database: choose one source tree, build one database, and run one query suite. If 20 queries produce results, the output is small enough for direct review and direct SARIF inspection.
  2. Corpus scale: choose 2487 source trees, build two databases per project when comparing BMF and BMN, and run one query suite. If 20 queries produce results per database, that is about 2487 * 2 * 20 = 99480 result groups before considering multiple findings per query. Direct SARIF inspection is no longer practical; SQL ingestion becomes useful.
  3. Large recurring scale: choose 120000 source trees, build one database per project, and run one query suite weekly. If 20 queries produce results per database, one run can produce about 120000 * 20 = 2400000 result groups. Keeping recent weekly runs plus monthly snapshots quickly reaches tens of millions of result groups. At that point, a high-performance SQL engine such as DuckDB is required for comparison and reporting.

The following sections describe the general workflow used across the source repository corpus. For two concrete end-to-end examples, see the detailed QEMU and DPDK directories. They follow the same create/analyze workflow, with additional logging for CPU, RAM, disk use, package events, SQLite import, and gnuplot graph generation.

2. From code to CodeQL DB

Start with a source tree source. Create a CodeQL database codeql_DB_type from source, where type is either full build tracing or build-mode=none. The resulting database directory is zipped so it can be analyzed later or moved between hosts.

The two database creation modes are:

  • bmf, build-mode full: CodeQL traces a real build with codeql database create ... --command'…' …=. Successful database bundles are under data/codeql-db-zips-bm-full/.
  • bmn, build-mode none: CodeQL scans visible C/C++ source without running the build with codeql database create ... --build-mode=none .... Successful database bundles are under data/codeql-db-zips-bm-none/.

2.1. Results for Full Build Trace

This is labeled bmf, for build-mode full. CodeQL observes a real build, so the database reflects files and compiler options that actually participated in that build. 

codeql database create DB_DIR \
       --language=cpp \
       --source-root=SOURCE_DIR \
       --command='debian/rules build' \
       --threads="$THREADS" \
       --ram="$RAM_MB"

codeql database bundle --output PROJECT.zip DB_DIR

The corpus-level BMF database bundles are available at data/codeql-db-zips-bm-full/; a paginated searchable listing follows. There are 2487 successful BMF DB zip bundles.

2.2. Results for build-mode=none

This is labeled bmn, for build-mode=none. CodeQL does not run the build; it infers extraction from the source tree.

The core CodeQL command used to build every one of these DBs is 

codeql database create \
       --language=cpp \
       --source-root="$SOURCE_ROOT" \
       --threads="$THREADS" \
       --ram="$RAM_MB" \
       --build-mode=none \
       "$DB_PATH"

The corpus-level BMN database bundles are available at data/codeql-db-zips-bm-none/; a paginated searchable listing follows. There are 843 successful BMN DB zip bundles.

3. From CodeQL DB to results

After database creation, DBs from both modes are analyzed with a CodeQL query suite; standard preinstalled query suites useful for C/C++ include:

cpp-security-and-quality.qls
broad security and quality suite used here.
cpp-security-extended.qls
security-focused suite with extended checks.
cpp-code-scanning.qls
default code-scanning-oriented suite.
(no term)
Individual query packs or query paths under the installed CodeQL checkout.

Here, CodeQL version 2.23.6 was used.

The analysis steps are identical for bmf and bmn; only the DB zip and output SARIF directory change.

  1. Choose the DB zip.
  2. Extract it to a temporary directory.
  3. Find the directory containing codeql-database.yml.
  4. Run codeql database analyze with the selected suite.
  5. Store the SARIF and retain stdout, stderr, resource logs, and status.

The core CodeQL command used to analyze every CodeQL DB is 

codeql database analyze \
       --format=sarif-latest \
       --rerun \
       -j"$THREADS" \
       --ram="$RAM_MB" \
       --output "$SARIF" \
       -- "$DB_DIR" cpp-security-and-quality.qls

3.1. SARIF files produced for bm-full

The work here used the C/C++ cpp-security-and-quality suite. The BMF SARIF outputs are available at data/bmf-db-cpp-security-and-quality-sarifs/. There are 2486 BMF C/C++ cpp-security-and-quality SARIF files.

3.2. SARIF files produced for bm-none

This section also used the C/C++ cpp-security-and-quality suite. The BMN SARIF outputs are available at data/bmn-db-cpp-security-and-quality-sarifs/.

There are 832 BMN C/C++ cpp-security-and-quality SARIF files, significantly fewer than the analysis of databases from full builds.

4. Comparing CodeQL database build options

There are two choices when building a C/C++ CodeQL database for this corpus. A full build trace uses --command to observe a real build. A database created with --build-mode=none examines source code without running the build.

The SARIF file size comparison and the largest BMF/BMN SARIF size differences already suggest that the two database modes can produce substantially different analysis output. File size is only a proxy, though, so the more direct comparison is at the result level.

After converting SARIF results to a SQL database, using the result identity 

file_name, kind, rule_id, rule_name, uri, start_line, start_column

gives the following counts of distinct result keys. 

kind          comparison  result_count
path-problem  common             4,149
path-problem  only_full         32,685
path-problem  only_none          3,595
problem       common           126,238
problem       only_full        699,833
problem       only_none        843,338

4.1. Interpretation

Across more than 2,400 CodeQL databases, the overlap between build-mode=none and a full build is only 10.3% for path-problem results and 7.6% for problem results. These differences are orders of magnitude larger than would be expected from random variation, indicating that the two build modes produce substantially different result sets.

For path-problem results:

Comparison Count
common 4,149
only_full 32,685
only_none 3,595

Only about 10.3% of distinct path results are common: 

4,149 / (4,149 + 32,685 + 3,595) ~= 10.3%

For problem results:

Comparison Count
common 126,238
only_full 699,833
only_none 843,338

Only about 7.6% of distinct problem results are common: 

126,238 / (126,238 + 699,833 + 843,338) ~= 7.6%

These values are low enough that the two modes should be treated as substantially different analyses, not as equivalent runs over the same effective program.

More specifically:

  • For path-problem results, the full build produces many more unique results than build-mode=none, but there is still a nontrivial only_none set.
  • For problem results, build-mode=none produces more unique results than the full build, but both modes have very large exclusive sets.

The surprising part is the small intersection. That suggests one or more of:

  • database creation mode affects database contents substantially.
  • result identity may be too location-sensitive.
  • rule behavior changes because extracted code or dependencies differ.
  • many alerts are sensitive to build configuration.
  • file_name or uri normalization may differ between modes.

A useful summary metric is Jaccard similarity: ignoring which build mode is "correct," how much do the two analyses agree on the set of findings? 

common / (common + only_none + only_full)

Current values:

Kind Jaccard similarity
path-problem 0.103
problem 0.076

4.2. Treating tracing build as the reference

If the full build is treated as the best available approximation of the true result set, then the build-mode=none results can be evaluated as an approximation of that reference. Let

\[ T = F \]

be the reference ("truth"), and

\[ A = N \]

be the approximation produced by build-mode=none.

Then

\[ TP = |N \cap F| \]

\[ FP = |N \setminus F| \]

\[ FN = |F \setminus N| \]

4.2.1. Path problems

Given

\[ TP = 4149,\quad FP = 3595,\quad FN = 32685, \]

we obtain

\[ \text{Precision} = \frac{TP}{TP+FP} = \frac{4149}{4149+3595} \approx 0.536, \]

and

\[ \text{Recall} = \frac{TP}{TP+FN} = \frac{4149}{4149+32685} \approx 0.113. \]

Thus, build-mode=none reproduces only about 11.3% of the path problems reported by the full build.

4.2.2. Problems

Given

\[ TP = 126238,\quad FP = 843338,\quad FN = 699833, \]

we obtain

\[ \text{Precision} = \frac{126238}{126238+843338} \approx 0.130, \]

and

\[ \text{Recall} = \frac{126238}{126238+699833} \approx 0.153. \]

Thus, build-mode=none reproduces only about 15.3% of the problem results reported by the full build.

4.2.3. Summary

Kind Precision Recall
path-problem 53.6% 11.3%
problem 13.0% 15.3%

Overall, if the full build is taken as the best approximation of the true result set, then build-mode=none is a poor approximation.

For path-problem, it achieves moderate precision but very low recall: when it agrees with the full build it is often correct, but it misses nearly nine out of ten findings.

For problem results, both precision and recall are low, indicating that the two analyses report substantially different sets of findings.

5. Appendix

5.1. Precision and recall

Precision and recall are asymmetric measures: one result set is chosen as the reference ("truth"), and the other is evaluated against it.

For this discussion, let

\[ T = F \]

denote the results obtained from the full build, which we treat as the best available approximation of the true result set. Let

\[ A = N \]

denote the results obtained from --build-mode=none.

The results naturally partition into three disjoint sets:

\[ TP = A \cap T, \]

the results reported by both analyses (true positives),

\[ FP = A \setminus T, \]

the results reported only by --build-mode=none (false positives with respect to the reference), and

\[ FN = T \setminus A, \]

the results reported only by the full build (false negatives).

5.1.1. Precision

Precision answers the question:

Of the results reported by --build-mode=none, what fraction also appear in the reference analysis?

It is defined as

\[ \operatorname{Precision} = \frac{TP}{TP+FP}. \]

A precision of 1.0 means every reported result also appears in the reference. A low precision indicates that many results are unique to --build-mode=none.

In this study, a result counted as a false positive is not necessarily incorrect; it simply does not appear in the full build. Without knowing the true set of defects, we cannot determine whether these findings are genuine or spurious.

5.1.2. Recall

Recall answers the complementary question:

Of the results reported by the reference analysis, what fraction were also found by --build-mode=none?

It is defined as

\[ \operatorname{Recall} = \frac{TP}{TP+FN}. \]

A recall of 1.0 means that --build-mode=none reproduced every finding from the full build. A low recall indicates that many findings from the full build are missing.

Again, because the full build is only an approximation of the true result set, these "false negatives" are simply findings absent from --build-mode=none, not necessarily true defects that were missed.

5.1.3. Interpretation

Precision and recall measure different properties.

  • High precision, low recall: --build-mode=none reports relatively few unique findings, but misses many of those reported by the full build.
  • Low precision, high recall: --build-mode=none finds most results from the full build, but also reports many additional findings.
  • High precision, high recall: The two analyses produce very similar result sets.

In this work, the distinction is important because neither build mode is known to produce the true set of defects. The full build is used solely as a reference, allowing us to quantify how closely --build-mode=none reproduces its results.