125 lines
6.1 KiB
Markdown
125 lines
6.1 KiB
Markdown
# report_generator
|
|
|
|
Builds the PDF report of one stored scan, from the artifacts that scan put in the bucket.
|
|
|
|
## Installing
|
|
|
|
The package lives in `report_generator/`, beside the `pyproject.toml` that builds it. Install it
|
|
from the repository root — the directory holding this file, not the package directory:
|
|
|
|
```
|
|
pip install -e . # add [dev] for the test and lint tooling
|
|
```
|
|
|
|
Installing from inside `report_generator/` does not work: `packages.find` looks for the package as a
|
|
subdirectory, so it finds nothing, and the `report_generator` console script is then installed
|
|
pointing at a module that was never put on the path.
|
|
|
|
## Running
|
|
|
|
Two ways to say where the artifacts are.
|
|
|
|
**Out of the store** — Postgres for the rows, the bucket for the files:
|
|
|
|
```
|
|
report_generator generate --scan_id <uuid> [--path <file-or-directory>] --config config.json
|
|
```
|
|
|
|
**Out of a folder** — neither Postgres nor the bucket, just the artifacts as the pipeline exported
|
|
them:
|
|
|
|
```
|
|
report_generator from-folder --artifacts ./sample [--path <file-or-directory>]
|
|
```
|
|
|
|
`--scan_id` is optional there: the pipeline writes its own id into the paths it records in
|
|
`metadata.json`, so an export names its own run. Pass it for a folder assembled by hand, or one
|
|
whose metadata is gone.
|
|
|
|
The two produce the same document. Everything after the artifacts are in hand is shared code, and
|
|
the sample export in `sample/` was checked against the report built from the store for the same
|
|
scan: 17 of 17 pages identical. The tests cover the folder path; comparing the two needs a store
|
|
and so is not among them.
|
|
|
|
`--scan_id` takes either id a run is known by. A run is recorded under two: the **pipeline id**,
|
|
which the GUI shows, cipherverifier names its working directories after, and archon's `core_scan`
|
|
row is keyed by; and the **stored scan id**, minted when the artifacts are uploaded, which `scans`,
|
|
`artifacts` and the S3 keys are keyed by. `core_scanresult` is where they meet — its own id is the
|
|
stored scan, its `scan_id` the pipeline — and it is translated in both directions:
|
|
|
|
- Reading always uses the stored scan id, because that is what the store is keyed by.
|
|
- The report always *identifies* itself by the pipeline id — the file name, the `Scan Identifier`
|
|
field and the page footers — because that is the id the GUI shows and so the only one a reader
|
|
can match against anything else.
|
|
|
|
So both ids produce the same report, filed under the pipeline id. A run with no pipeline row falls
|
|
back to being reported under its stored id, which is then the only name it has. Passing a pipeline
|
|
id used to fail with a bare `No row was found when one was required`.
|
|
|
|
Without `--path` the report is written into `reports/`. The S3 secret comes from the environment:
|
|
`PERSISTENT_STORAGE_STORAGE_CONFIG_S3_SECRET_ACCESS_KEY` (and the database password from
|
|
`PERSISTENT_STORAGE_STORAGE_CONFIG_DATABASE_PASSWORD`). The name is not this tool's to choose: it is
|
|
derived from the config path `inject_secrets` fills in under — `persistent_storage.storage_config` —
|
|
by the one naming convention in `cc_common.config.env_var_name`, so it stays whatever the monorepo
|
|
calls it. A `.env` beside the `pyproject.toml` is read too. `--config` is resolved relative to the
|
|
working directory.
|
|
|
|
## How it is put together
|
|
|
|
Four steps, each its own module, so that the one that will change is the one that is separate:
|
|
|
|
| Module | Does |
|
|
| --- | --- |
|
|
| `report_generator/sources.py` | Reads the scan and its files out of the store. **The only module that knows which store.** |
|
|
| `report_generator/rendering.py` | Decides which stored file feeds which part of the report, and parses it. |
|
|
| `report_generator/context.py` | Maps the parsed artifacts onto the page, and writes `<xxx>` where they say nothing. |
|
|
| `report_generator/layout.py` | Draws `template/report.html.j2` through WeasyPrint. |
|
|
|
|
No parsing of its own. SARIF and the vulnerability report are read by the summarisers in
|
|
`cipherverifier.core.services`, which are written to be free of Django and of any store; the CVSS
|
|
vectors by `persistent_storage.utils.cvss_utils`.
|
|
|
|
### What it cannot fill in
|
|
|
|
A field the artifacts do not answer is written as `<xxx>`, never guessed or quietly dropped. Today
|
|
that is the report and artifact ids, the manufacturer, the analysed lines of code, the language
|
|
breakdown, and the tools and profile of each pipeline stage — the metadata records that a stage ran
|
|
and how long it took, not what it ran.
|
|
|
|
A measured zero survives: `0 filtered` is a reading, and a blank is a gap.
|
|
|
|
## Moving it to archon
|
|
|
|
`sources.collect()` is the whole of it. It returns files by name and the rows a report lists;
|
|
everything after that point works on bytes and cannot tell which store handed them over. Pointing it
|
|
at archon's tables is a change to that function and to nothing else.
|
|
|
|
## Its relationship to the monorepo
|
|
|
|
This repository is the split of `src/persistent_storage/report_generator/` out of the monorepo. The
|
|
package names nothing about itself — every import inside it is relative — so the split was a move
|
|
and not a rewrite: the sources sit unchanged under `report_generator/`, with `pyproject.toml` beside
|
|
them rather than inside them.
|
|
|
|
`report_generator/vendor/` supplies everything the package borrows from the monorepo — `cc_common`,
|
|
`persistent_storage.shared`, and the two summarisers. `vendor/__init__.py` puts the copies on the
|
|
path when the real packages are not installed, and stays out of the way when they are, so the same
|
|
sources run in both places. See `report_generator/vendor/README.md`.
|
|
|
|
Still to come across: `tests/persistent_storage/test_report_*.py` from the monorepo, with
|
|
`persistent_storage.report_generator` changed to `report_generator` in their imports. Until they
|
|
land, `testpaths = ["tests"]` in `pyproject.toml` points at a directory that does not exist yet.
|
|
|
|
On the monorepo side, `tests/persistent_storage/test_report_vendor.py` performs this same split into
|
|
a temporary directory on every run and renders a report from it, so the copies cannot quietly stop
|
|
being sufficient.
|
|
|
|
### Native dependencies
|
|
|
|
WeasyPrint needs Pango and cairo, which pip does not install:
|
|
|
|
```
|
|
brew install pango # macOS
|
|
apt-get install libpango-1.0-0 libpangoft2-1.0-0 # Debian/Ubuntu
|
|
```
|