pdf-reporting/README.md

# report_generator

Builds the PDF report of one stored scan, from the artifacts that scan put in the bucket.

## Installing

The package lives in `report_generator/`, beside the `pyproject.toml` that builds it. Install it
from the repository root — the directory holding this file, not the package directory:

```
pip install -e .            # add [dev] for the test and lint tooling
```

Installing from inside `report_generator/` does not work: `packages.find` looks for the package as a
subdirectory, so it finds nothing, and the `report_generator` console script is then installed
pointing at a module that was never put on the path.

## Running

Two ways to say where the artifacts are.

**Out of the store** — Postgres for the rows, the bucket for the files:

```
report_generator generate --scan_id <uuid> [--path <file-or-directory>] --config config.json
    [--project <name>] [--manufacturer <name>] [--version <version>]
```

**Out of a folder** — neither Postgres nor the bucket, just the artifacts as the pipeline exported
them:

```
report_generator from-folder --artifacts ./sample [--path <file-or-directory>]
    [--project <name>] [--manufacturer <name>] [--version <version>]
```

`--version` sets the version shown on the cover and the project page. Without it the report uses
`repository_version` from `metadata.json`, else the commit the scan was run on.

`--scan_id` is optional there: the pipeline writes its own id into the paths it records in
`metadata.json`, so an export names its own run. Pass it for a folder assembled by hand, or one
whose metadata is gone.

The inputs are one SARIF file (`merged.sarif`, with one run per tool) and the vulnerability report
(`vdr.json`); either is enough on its own. `metadata.json` and `cloc.txt` (cloc's plain
table, for the lines of code and the three to five main languages) are optional; without either the
project page is left out. The report has these parts:

1. the cover and a table of contents,
2. a summary and the most critical issues across all sources,
3. the dependency vulnerabilities from the VDR, as CipherCell SCA,
4. one section per SARIF run, named after its tool: CipherCell Lynkeus, then CipherCell SAST, then
   any other tool in the order its run appears,
5. at the end, *False Positives and Suppressed Findings*: every result Polygraph classified as a
   false positive or that carries a SARIF suppression (such as the areas Lynkeus ruled out), with
   the reason.

Every finding is listed; only the summary charts and the most-critical table are capped.

The two produce the same document. Everything after the artifacts are in hand is shared code, and
the sample export in `sample/` was checked against the report built from the store for the same
scan: 17 of 17 pages identical. The tests cover the folder path; comparing the two needs a store
and so is not among them.

`--scan_id` takes either id a run is known by. A run is recorded under two: the **pipeline id**,
which the GUI shows, cipherverifier names its working directories after, and archon's `core_scan`
row is keyed by; and the **stored scan id**, minted when the artifacts are uploaded, which `scans`,
`artifacts` and the S3 keys are keyed by. `core_scanresult` is where they meet — its own id is the
stored scan, its `scan_id` the pipeline — and it is translated in both directions:

- Reading always uses the stored scan id, because that is what the store is keyed by.
- The report always *identifies* itself by the pipeline id — the file name, the `Scan Identifier`
  field and the page footers — because that is the id the GUI shows and so the only one a reader
  can match against anything else.

So both ids produce the same report, filed under the pipeline id. A run with no pipeline row falls
back to being reported under its stored id, which is then the only name it has. Passing a pipeline
id used to fail with a bare `No row was found when one was required`.

Without `--path` the report is written into `reports/`. The S3 secret comes from the environment:
`PERSISTENT_STORAGE_STORAGE_CONFIG_S3_SECRET_ACCESS_KEY` (and the database password from
`PERSISTENT_STORAGE_STORAGE_CONFIG_DATABASE_PASSWORD`). The name is not this tool's to choose: it is
derived from the config path `inject_secrets` fills in under — `persistent_storage.storage_config` —
by the one naming convention in `cc_common.config.env_var_name`, so it stays whatever the monorepo
calls it. A `.env` beside the `pyproject.toml` is read too. `--config` is resolved relative to the
working directory.

## The summary report

A shorter document, written by hand in Markdown (like `SUMMARY.md` beside the scan's artifacts) and
set in the dress of the full report: the same cover, header, footers, table of contents, headings
and tables.

```
report_generator from-markdown --markdown SUMMARY.md [--path <file-or-directory>]
    [--project <name>] [--manufacturer <name>] [--scan_id <uuid>]
```

How the Markdown is read (`report_generator/summary.py`):

- The first `#` heading is the title. Written as `<title>: <subject>`, the subject is the project
  unless `--project` is given.
- What comes before the first section is the introduction, and goes on the cover.
- Every further `#` or `##` heading is a section: it starts a new page, is listed in the table of
  contents and keeps the anchor GitHub gives it, so links such as `[…](#1-static-analysis-sast)`
  work in the PDF. `###` and below become the report's subheadings.
- A paragraph that is only bold text (`**Remaining findings**`) is set as a subheading.
- Tables are drawn as the report's grids. A number in a column headed *Severity*, *CVSS* or *Score*
  carries its severity colour, and a row starting with `**Total**` is set as a total.

The scan id is optional here; without it the cover leaves it out and the footer names the project.
The file is written as `summary_<scan-id>.pdf`, or `summary_<project>.pdf` without one.

## How it is put together

Four steps, each its own module, so that the one that will change is the one that is separate:

| Module | Does |
| --- | --- |
| `report_generator/sources.py` | Reads the scan and its files out of the store. **The only module that knows which store.** |
| `report_generator/rendering.py` | Decides which stored file feeds which part of the report, and parses it. |
| `report_generator/context.py` | Maps the parsed artifacts onto the page, and writes `<xxx>` where they say nothing. |
| `report_generator/layout.py` | Draws `template/report.html.j2` through WeasyPrint. |
| `report_generator/summary.py` | Sets a Markdown summary in `template/summary.html.j2`, drawn the same way. |

No parsing of its own. SARIF and the vulnerability report are read by the summarisers in
`cipherverifier.core.services`, which are written to be free of Django and of any store; the CVSS
vectors by `persistent_storage.utils.cvss_utils`.

### What it cannot fill in

A field the artifacts do not answer is written as `<xxx>`, never guessed or quietly dropped. Today
that is the manufacturer (unless given with `--manufacturer`), and, when there is no `cloc.txt`, the
analysed lines of code and the language breakdown.

A measured zero survives: `0 filtered` is a reading, and a blank is a gap.

## Moving it to archon

`sources.collect()` is the whole of it. It returns files by name and the rows a report lists;
everything after that point works on bytes and cannot tell which store handed them over. Pointing it
at archon's tables is a change to that function and to nothing else.

## Its relationship to the monorepo

This repository is the split of `src/persistent_storage/report_generator/` out of the monorepo. The
package names nothing about itself — every import inside it is relative — so the split was a move
and not a rewrite: the sources sit unchanged under `report_generator/`, with `pyproject.toml` beside
them rather than inside them.

`report_generator/vendor/` supplies everything the package borrows from the monorepo — `cc_common`,
`persistent_storage.shared`, and the two summarisers. `vendor/__init__.py` puts the copies on the
path when the real packages are not installed, and stays out of the way when they are, so the same
sources run in both places. See `report_generator/vendor/README.md`.

Still to come across: `tests/persistent_storage/test_report_*.py` from the monorepo, with
`persistent_storage.report_generator` changed to `report_generator` in their imports. Until they
land, `testpaths = ["tests"]` in `pyproject.toml` points at a directory that does not exist yet.

On the monorepo side, `tests/persistent_storage/test_report_vendor.py` performs this same split into
a temporary directory on every run and renders a report from it, so the copies cannot quietly stop
being sufficient.

### Native dependencies

WeasyPrint needs Pango and cairo, which pip does not install:

```
brew install pango          # macOS
apt-get install libpango-1.0-0 libpangoft2-1.0-0    # Debian/Ubuntu
```