Updates to new format #1

Open
fklement wants to merge 17 commits from CipherCell/pdf-reporting:updates-to-new-format into main
pull from: CipherCell/pdf-reporting:updates-to-new-format
10 changed files with 573 additions and 92 deletions
Showing only changes of commit 864e2ff824 - Show all commits

No files matched your search

+27
View File
@@ -85,6 +85,32 @@ by the one naming convention in `cc_common.config.env_var_name`, so it stays wha
calls it. A `.env` beside the `pyproject.toml` is read too. `--config` is resolved relative to the
working directory.
## The summary report
A shorter document, written by hand in Markdown (like `SUMMARY.md` beside the scan's artifacts) and
set in the dress of the full report: the same cover, header, footers, table of contents, headings
and tables.
```
report_generator from-markdown --markdown SUMMARY.md [--path <file-or-directory>]
[--project <name>] [--manufacturer <name>] [--scan_id <uuid>]
```
How the Markdown is read (`report_generator/summary.py`):
- The first `#` heading is the title. Written as `<title>: <subject>`, the subject is the project
unless `--project` is given.
- What comes before the first section is the introduction, and goes on the cover.
- Every further `#` or `##` heading is a section: it starts a new page, is listed in the table of
contents and keeps the anchor GitHub gives it, so links such as `[…](#1-static-analysis-sast)`
work in the PDF. `###` and below become the report's subheadings.
- A paragraph that is only bold text (`**Remaining findings**`) is set as a subheading.
- Tables are drawn as the report's grids. A number in a column headed *Severity*, *CVSS* or *Score*
carries its severity colour, and a row starting with `**Total**` is set as a total.
The scan id is optional here; without it the cover leaves it out and the footer names the project.
The file is written as `summary_<scan-id>.pdf`, or `summary_<project>.pdf` without one.
## How it is put together
Four steps, each its own module, so that the one that will change is the one that is separate:
@@ -95,6 +121,7 @@ Four steps, each its own module, so that the one that will change is the one tha
| `report_generator/rendering.py` | Decides which stored file feeds which part of the report, and parses it. |
| `report_generator/context.py` | Maps the parsed artifacts onto the page, and writes `<xxx>` where they say nothing. |
| `report_generator/layout.py` | Draws `template/report.html.j2` through WeasyPrint. |
| `report_generator/summary.py` | Sets a Markdown summary in `template/summary.html.j2`, drawn the same way. |
No parsing of its own. SARIF and the vulnerability report are read by the summarisers in
`cipherverifier.core.services`, which are written to be free of Django and of any store; the CVSS
+2
View File
@@ -40,6 +40,8 @@ dependencies = [
# needs the native Pango and cairo libraries, which pip does not install -- see the README.
"jinja2>=3.1.6",
"weasyprint>=69.0",
# The summary report, which is written in Markdown and set in the same template.
"markdown-it-py>=4.0.0",
]
[project.optional-dependencies]
+7
View File
@@ -5,6 +5,7 @@ from .vendor.cc_common.config import load_environment
from .vendor.persistent_storage.shared import inject_secrets
from .commands.generate import FolderProperties, GenerateProperties, from_folder, generate
from .commands.summary import SummaryProperties, from_markdown
def run() -> None:
@@ -25,6 +26,12 @@ def run() -> None:
FolderProperties,
help="Builds the report from a folder of exported artifacts, without Postgres or the bucket.",
)
cli.register_command(
"from-markdown",
from_markdown,
SummaryProperties,
help="Builds a summary report from a Markdown file, in the style of the full report.",
)
cli.run()
+94
View File
@@ -0,0 +1,94 @@
"""The summary report: a Markdown summary, set as a PDF in the style of the full report."""
from pathlib import Path
from uuid import UUID
from pydantic import BaseModel, Field
from ..vendor.cc_common.api import PipelineError, context
from ..vendor.cc_common.utils import print
from ..summary import render_summary
from .generate import _destination
class SummaryProperties(BaseModel):
"""Configuration properties for the from-markdown CLI command.
Attributes:
markdown: The Markdown summary to set.
scan_id: The scan it is about. Named on the cover and in the footers
when given, and left out when not.
path: Where to write the PDF. A directory is written into under the name
the renderer chose; leaving it out writes into ``reports/``.
project: The name of the scanned project. Taken from the summary's title
when not given.
manufacturer: Who makes the scanned product, as the summary states it.
"""
markdown: Path = Field(description="The Markdown summary to build the PDF from.")
scan_id: UUID | None = Field(default=None, description="The scan the summary is about.")
path: Path | None = Field(default=None, description="Where to write the PDF summary.")
project: str | None = Field(
default=None, description="The name of the scanned project. Read from the title when omitted."
)
manufacturer: str | None = Field(default=None, description="The manufacturer of the scanned product.")
class GeneratedSummary(BaseModel):
"""The summary that was written.
Attributes:
source: The Markdown it was set from.
path: The file written.
size: How large it came out, in bytes.
"""
source: Path
path: Path
size: int
def print_as_cc_common_result(self) -> None:
"""Say what was written and where."""
print(f"[bold]Summary from {self.source}[/bold]")
print(f" {self.size / 1024:.0f} kB written to {self.path}")
def from_markdown(properties: SummaryProperties) -> GeneratedSummary:
"""Build the summary report from a Markdown file.
Args:
properties: The Markdown file, what to say about it, and the destination.
Returns:
The summary that was written.
Raises:
PipelineError: The Markdown cannot be read, or the destination cannot be written.
"""
try:
markdown = properties.markdown.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError) as error:
message = f"Cannot read the summary '{properties.markdown}': {error}"
raise PipelineError(message) from error
name, content = render_summary(
markdown,
scan_id=properties.scan_id,
project=properties.project,
manufacturer=properties.manufacturer,
)
destination = _destination(properties.path, name)
try:
destination.parent.mkdir(parents=True, exist_ok=True)
destination.write_bytes(content)
except OSError as error:
message = f"Cannot write the summary to '{destination}': {error}"
raise PipelineError(message) from error
context.logger.info(f"Summary written to {destination} ✔")
return GeneratedSummary(source=properties.markdown, path=destination, size=len(content))
+8 -5
View File
@@ -39,11 +39,12 @@ CONTEXT = {
}
def render_html(context: dict[str, Any]) -> str:
def render_html(context: dict[str, Any], template: str = TEMPLATE_NAME) -> str:
"""Render the report as an HTML document.
Args:
context: The page-ready context; see :data:`CONTEXT`.
template: Which template in :data:`TEMPLATE_DIR` to draw it with.
Returns:
The rendered HTML.
@@ -55,14 +56,16 @@ def render_html(context: dict[str, Any]) -> str:
lstrip_blocks=True,
)
return environment.get_template(TEMPLATE_NAME).render(**context)
return environment.get_template(template).render(**context)
def render_pdf(context: dict[str, Any]) -> bytes:
def render_pdf(context: dict[str, Any], template: str = TEMPLATE_NAME, title: str = TITLE) -> bytes:
"""Render the report as a PDF.
Args:
context: The page-ready context; see :data:`CONTEXT`.
template: Which template in :data:`TEMPLATE_DIR` to draw it with.
title: What the document calls itself in its metadata.
Returns:
The PDF.
@@ -78,12 +81,12 @@ def render_pdf(context: dict[str, Any]) -> bytes:
# Relative URLs in the template and the stylesheet -- the mark, the font files -- resolve
# against the template directory.
document = HTML(string=render_html(context), base_url=str(TEMPLATE_DIR)).render(
document = HTML(string=render_html(context, template), base_url=str(TEMPLATE_DIR)).render(
stylesheets=[CSS(filename=str(STYLESHEET), font_config=fonts)],
font_config=fonts,
)
document.metadata.title = f"{TITLE} - {context.get('cover', {}).get('subject', '')}".strip(" -")
document.metadata.title = f"{title} - {context.get('cover', {}).get('subject', '')}".strip(" -")
document.metadata.authors = [AUTHOR]
return bytes(document.write_pdf())
+239
View File
@@ -0,0 +1,239 @@
"""Turns a Markdown summary into the summary report.
The summary is written by hand, beside the scan's artifacts, and reads as a document of its own: a
title, a few lines saying what it covers, then one section per source. This module sets it in the
dress of the full report -- the same cover, running header and footers, table of contents, headings
and tables -- so the two sit together as one series.
No design of its own. The Markdown is rendered to HTML whose elements are the ones ``report.css``
already styles, by moving them onto the full report's heading levels and classes:
- The first level-one heading is the title. Written as ``<title>: <subject>``, the subject is the
project, unless one is given.
- What comes before the first section is the introduction, and goes on the cover.
- Every further level-one or level-two heading is a section: it starts a new page, as the full
report's sections do, is listed in the table of contents and keeps the anchor GitHub would give it, so the links a
summary makes to its own sections go on working in the PDF.
- A paragraph that is nothing but bold text is the subheading it is being used as.
- Tables are drawn as the full report's grids. A number in a severity or CVSS column carries its
severity colour, and a row that opens with a bold ``Total`` is set as a total.
"""
from __future__ import annotations
import re
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any
from markdown_it import MarkdownIt
from markupsafe import Markup
from .context import UNKNOWN, _fact, _stated
from .layout import render_pdf
from .severity import _score_band
if TYPE_CHECKING:
from uuid import UUID
from markdown_it.token import Token
TEMPLATE_NAME = "summary.html.j2"
# What the document calls itself when the Markdown does not say.
TITLE = "Security Scan Summary"
# The shape the summary template expects.
CONTEXT = {
"cover": "subject, footer, footer_date, logo, title, identity[], intro?",
"contents": "[{id, title}]",
"body": "the rendered sections",
}
# What the file is called: after the scan where there is one, else after the project.
NAME = "summary_{name}.pdf"
# A column whose numbers are CVSS scores, to be coloured by the band each falls in.
_SCORED_COLUMN = re.compile(r"severity|cvss|score", re.IGNORECASE)
_SCORE = re.compile(r"\d+(?:\.\d+)?")
# Where Markdown's heading levels land on the full report's: a section is its ``h1``, and what is
# under it one and two steps down.
_LEVELS = {"h1": "h1", "h2": "h1", "h3": "h2", "h4": "h3", "h5": "h3", "h6": "h3"}
def render_summary(
markdown: str,
*,
scan_id: UUID | None = None,
project: str | None = None,
manufacturer: str | None = None,
) -> tuple[str, bytes]:
"""Render a Markdown summary as a PDF.
Args:
markdown: The summary.
scan_id: The scan it is about, if it is to say so.
project: The project name; overrides the subject of the title.
manufacturer: Who makes the scanned product.
Returns:
The file name the summary should be saved under, and its bytes.
"""
context = build_summary_context(markdown, scan_id=scan_id, project=project, manufacturer=manufacturer)
name = str(scan_id) if scan_id else _slug(context["cover"]["subject"]) or "report"
return NAME.format(name=name), render_pdf(context, TEMPLATE_NAME, context["cover"]["title"])
def build_summary_context(
markdown: str,
*,
scan_id: UUID | None = None,
project: str | None = None,
manufacturer: str | None = None,
) -> dict[str, Any]:
"""Map a Markdown summary onto the page.
Args:
markdown: The summary.
scan_id: The scan it is about, if it is to say so.
project: The project name; overrides the subject of the title.
manufacturer: Who makes the scanned product.
Returns:
The page-ready context; see :data:`CONTEXT`.
"""
parser = MarkdownIt("commonmark").enable(["table", "strikethrough"])
tokens = parser.parse(markdown)
heading = _title(tokens)
title, _, subject = heading.partition(":") if heading else (TITLE, "", "")
project = project or subject.strip() or None
contents = _sections(tokens, parser)
_subheadings(tokens)
_tables(tokens)
start = next((index for index, token in enumerate(tokens) if token.meta.get("section")), len(tokens))
intro, body = tokens[:start], tokens[start:]
render = parser.renderer.render
identity = [
_fact("Manufacturer", manufacturer),
_fact("Project", project),
_fact("Report Created", datetime.now(UTC).strftime("%A, %B %d, %Y")),
]
if scan_id:
identity.append(_fact("Scan Identifier", scan_id, kind="mono"))
return {
"cover": {
"subject": _stated(project),
"footer": f"Scan: {scan_id}" if scan_id else _stated(project),
"footer_date": datetime.now(UTC).strftime("%d %B %Y"),
"logo": "cc_logo.png",
"title": title.strip(),
"identity": identity,
"intro": Markup(render(intro, parser.options, {})) if intro else None, # noqa: S704
},
"contents": contents,
"body": Markup(render(body, parser.options, {})), # noqa: S704
}
def _title(tokens: list[Token]) -> str | None:
"""Take the first level-one heading out of the document, and say what it read."""
for index, token in enumerate(tokens):
if token.type == "heading_open" and token.tag == "h1":
text = tokens[index + 1].content
del tokens[index : index + 3]
return text
return None
def _sections(tokens: list[Token], parser: MarkdownIt) -> list[dict[str, Any]]:
"""Move every heading onto the full report's levels, and list the sections for the contents."""
contents: list[dict[str, Any]] = []
taken: set[str] = set()
for index, token in enumerate(tokens):
if token.type not in ("heading_open", "heading_close"):
continue
section = token.tag in ("h1", "h2")
token.tag = _LEVELS[token.tag]
if token.type == "heading_close" or not section:
continue
inline = tokens[index + 1]
anchor = _unique(_github_anchor(inline.content), taken)
token.attrSet("id", anchor)
token.meta["section"] = True
# As in the full report, every section opens a page of its own.
token.attrSet("style", "break-before: page")
title = Markup(parser.renderer.render(inline.children or [], parser.options, {})) # noqa: S704
contents.append({"id": anchor, "title": title})
return contents
def _subheadings(tokens: list[Token]) -> None:
"""Set a paragraph that is nothing but bold text as the subheading it stands in for."""
for index, token in enumerate(tokens):
if token.type != "paragraph_open" or token.hidden:
continue
children = tokens[index + 1].children or []
kinds = [child.type for child in children]
if kinds == ["strong_open", "text", "strong_close"]:
token.tag = tokens[index + 2].tag = "h2"
tokens[index + 1].children = [children[1]]
def _tables(tokens: list[Token]) -> None:
"""Draw tables as the full report's grids: scores coloured, totals set as totals."""
columns: list[str] = []
column = 0
for index, token in enumerate(tokens):
if token.type == "table_open":
token.attrSet("class", "grid")
columns = []
elif token.type == "th_open":
columns.append(tokens[index + 1].content)
elif token.type == "tr_open":
column = 0
cell = tokens[index + 2] if tokens[index + 1].type == "td_open" else None
if cell is not None and cell.content.strip() == "**Total**":
token.attrSet("class", "total")
elif token.type == "td_open":
header = columns[column] if column < len(columns) else ""
content = tokens[index + 1].content.strip()
band = _score_band(content) if _SCORED_COLUMN.search(header) and _SCORE.fullmatch(content) else None
if band:
token.attrSet("class", f"sev sev-{band.lower()}")
column += 1
def _github_anchor(text: str) -> str:
"""The anchor GitHub gives a heading, so the links a summary makes to itself still land."""
return re.sub(r"[^\w\- ]", "", text.strip().lower()).replace(" ", "-")
def _unique(anchor: str, taken: set[str]) -> str:
"""The anchor, numbered as GitHub numbers a second heading of the same name."""
candidate, number = anchor or "section", 0
while candidate in taken:
number += 1
candidate = f"{anchor}-{number}"
taken.add(candidate)
return candidate
def _slug(text: str) -> str:
"""A name fit for a file."""
return "" if text == UNKNOWN else re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")
+92
View File
@@ -0,0 +1,92 @@
{#
The pieces the report templates share: `report.html.j2`, the full report, and `summary.html.j2`,
the summary set from Markdown. Kept apart so that the two documents draw a label, a value or a
table the same way, and a change to one is a change to both.
#}
{# A value is drawn by what it is, not by where it sits: an identifier, a path or a CVSS vector is
set in the monospace face wherever it appears, and a severity carries its colour. #}
{% macro value(row) %}
{% if row.token %}<span class="token">{{ row.token }}</span> — {% endif %}
{% if row.kind == "severity" %}
<span class="sev sev-{{ row.value|lower }}">{{ row.value }}</span>
{% elif row.kind == "mono" %}
<span class="mono">{{ row.value }}</span>
{% else %}
{{ row.value }}
{% endif %}
{% endmacro %}
{% macro facts(rows, wide=false) %}
<table class="facts{% if wide %} wide{% endif %}">
{% for row in rows %}
<tr{% if row.gap %} class="gap"{% endif %}>
<td>{{ row.label }}:</td>
<td>{{ value(row) }}</td>
</tr>
{% endfor %}
</table>
{% endmacro %}
{% macro card(title, rows, note="") %}
{# Wrapped, because break-inside on the table does not hold its caption: the title would stay on
one page while the rows it titles moved to the next. #}
<div class="block">
<table class="card">
<caption>{{ title }}</caption>
{% for row in rows %}
<tr{% if row.aside %} class="aside"{% endif %}>
<th>{{ row.label }}</th>
<td>{{ value(row) }}</td>
</tr>
{% endfor %}
</table>
{% if note %}<p class="note card-note">{{ note }}</p>{% endif %}
</div>
{% endmacro %}
{# Scaled against the busiest row, not against the total: what is read off a row of these is which
one is the outlier, and a scale set by the sum flattens all of them. #}
{% macro distribution(rows, legend) %}
{# The bars and the legend that decodes them are one thing; a break between them leaves a reader
with coloured segments and no key. #}
<div class="block">
<table class="bars">
{% for row in rows %}
<tr>
<td{% if row.mono %} class="mono"{% endif %}>{{ row.label }}</td>
<td class="count">{{ row.count }}</td>
<td class="track">
<div class="bar">
{% for part in row.parts %}
<span class="seg-{{ part.severity|lower }}" style="width: {{ part.share }}%"></span>
{% endfor %}
</div>
</td>
</tr>
{% endfor %}
</table>
<div class="legend">
{% for item in legend.entries %}
<span class="item"><span class="key seg-{{ item.severity|lower }}"></span>{{ item.label }}</span>
{% endfor %}
{% if legend.caveat %}<span class="caveat">— {{ legend.caveat }}</span>{% endif %}
</div>
</div>
{% endmacro %}
{% macro grid(table) %}
<table class="grid">
<thead>
<tr>{% for column in table.columns %}<th>{{ column }}</th>{% endfor %}</tr>
</thead>
<tbody>
{% for row in table.rows %}
<tr{% if row.total %} class="total"{% endif %}>
{% for cell in row.cells %}
<td>{% if cell is mapping %}{{ value(cell) }}{% else %}{{ cell }}{% endif %}</td>
{% endfor %}
</tr>
{% endfor %}
</tbody>
</table>
{% endmacro %}
+67
View File
@@ -470,3 +470,70 @@ p.note {
.toc a::after {
content: leader(".") target-counter(attr(href), page);
}
/* --- Prose (the summary report) ----------------------------------------------------------- */
/* The summary is written in Markdown, so it brings what the full report never has: running text,
lists and links. Set to the full report's measure and rhythm, so the two read as one series. */
.prose p,
.prose ul,
.prose ol,
.prose blockquote {
max-width: 165mm;
}
/* Every section of the summary opens a page, so the space a heading keeps above it would only push
the page down. */
.prose h1 {
margin-top: 0;
}
.prose ul,
.prose ol {
margin: 0 0 3mm 0;
padding-left: 5mm;
}
.prose li {
margin-bottom: 1.4mm;
}
.prose li > p {
margin: 0;
}
.prose a {
color: var(--ink);
text-decoration: underline;
text-decoration-color: var(--border);
}
.prose blockquote {
margin: 0 0 3mm 0;
padding-left: 3mm;
border-left: 0.8pt solid var(--border);
color: var(--muted);
}
.prose pre {
background: var(--label-fill);
border: 0.4pt solid var(--border);
padding: 2mm;
white-space: pre-wrap;
margin: 0 0 3mm 0;
}
.prose hr {
border: none;
border-top: 0.4pt solid var(--border);
margin: 5mm 0;
}
/* The severity colour, where it marks a score in a table, is on the cell itself. */
.grid td.sev {
white-space: nowrap;
}
.cover-intro {
margin-top: 9mm;
}
+1 -87
View File
@@ -8,93 +8,7 @@
The context this expects is described by `layout.CONTEXT`, and `sample.sample_context()` builds a
complete one.
#}
{# A value is drawn by what it is, not by where it sits: an identifier, a path or a CVSS vector is
set in the monospace face wherever it appears, and a severity carries its colour. #}
{% macro value(row) %}
{% if row.token %}<span class="token">{{ row.token }}</span> — {% endif %}
{% if row.kind == "severity" %}
<span class="sev sev-{{ row.value|lower }}">{{ row.value }}</span>
{% elif row.kind == "mono" %}
<span class="mono">{{ row.value }}</span>
{% else %}
{{ row.value }}
{% endif %}
{% endmacro %}
{% macro facts(rows, wide=false) %}
<table class="facts{% if wide %} wide{% endif %}">
{% for row in rows %}
<tr{% if row.gap %} class="gap"{% endif %}>
<td>{{ row.label }}:</td>
<td>{{ value(row) }}</td>
</tr>
{% endfor %}
</table>
{% endmacro %}
{% macro card(title, rows, note="") %}
{# Wrapped, because break-inside on the table does not hold its caption: the title would stay on
one page while the rows it titles moved to the next. #}
<div class="block">
<table class="card">
<caption>{{ title }}</caption>
{% for row in rows %}
<tr{% if row.aside %} class="aside"{% endif %}>
<th>{{ row.label }}</th>
<td>{{ value(row) }}</td>
</tr>
{% endfor %}
</table>
{% if note %}<p class="note card-note">{{ note }}</p>{% endif %}
</div>
{% endmacro %}
{# Scaled against the busiest row, not against the total: what is read off a row of these is which
one is the outlier, and a scale set by the sum flattens all of them. #}
{% macro distribution(rows, legend) %}
{# The bars and the legend that decodes them are one thing; a break between them leaves a reader
with coloured segments and no key. #}
<div class="block">
<table class="bars">
{% for row in rows %}
<tr>
<td{% if row.mono %} class="mono"{% endif %}>{{ row.label }}</td>
<td class="count">{{ row.count }}</td>
<td class="track">
<div class="bar">
{% for part in row.parts %}
<span class="seg-{{ part.severity|lower }}" style="width: {{ part.share }}%"></span>
{% endfor %}
</div>
</td>
</tr>
{% endfor %}
</table>
<div class="legend">
{% for item in legend.entries %}
<span class="item"><span class="key seg-{{ item.severity|lower }}"></span>{{ item.label }}</span>
{% endfor %}
{% if legend.caveat %}<span class="caveat">— {{ legend.caveat }}</span>{% endif %}
</div>
</div>
{% endmacro %}
{% macro grid(table) %}
<table class="grid">
<thead>
<tr>{% for column in table.columns %}<th>{{ column }}</th>{% endfor %}</tr>
</thead>
<tbody>
{% for row in table.rows %}
<tr{% if row.total %} class="total"{% endif %}>
{% for cell in row.cells %}
<td>{% if cell is mapping %}{{ value(cell) }}{% else %}{{ cell }}{% endif %}</td>
{% endfor %}
</tr>
{% endfor %}
</tbody>
</table>
{% endmacro %}
{% from "macros.html.j2" import value, facts, card, distribution, grid %}
{# ------------------------------------------------------------------------------------------ #}
+36
View File
@@ -0,0 +1,36 @@
{#
The summary report, set from a Markdown file in the dress of the full report: the same cover,
running header and footers, table of contents and headings, with the body in between.
The body arrives already rendered -- :mod:`summary` turns the Markdown into HTML whose elements
are the ones `report.css` already styles -- so this file only places it. The context this expects
is described by `summary.CONTEXT`.
#}
{% from "macros.html.j2" import facts %}
<div class="page-header"><img src="{{ cover.logo }}" alt="CipherCell" /></div>
<span class="footer-scan">{{ cover.footer }}</span>
<span class="footer-date">{{ cover.footer_date }}</span>
<div class="cover-mark">
<img src="{{ cover.logo }}" alt="CipherCell" />
</div>
<div class="cover-title">{{ cover.title }}</div>
{{ facts(cover.identity) }}
{% if cover.intro %}
<div class="prose cover-intro">{{ cover.intro }}</div>
{% endif %}
{% if contents %}
<h1 style="break-before: page">Table of Contents</h1>
<ol class="toc">
{% for entry in contents %}
<li><a href="#{{ entry.id }}">{{ entry.title }}</a></li>
{% endfor %}
</ol>
{% endif %}
<div class="prose">{{ body }}</div>