Pipeline Steps Reference¶

Detailed reference for each of the nine pipeline steps. For an overview and flow diagram, see Architecture. For configuration keys, see the Configuration Reference.

Step 1 — Output Preparation¶

Module: pipeline/prepare_output.py

Prepares the output directory before the pipeline run begins. This step runs before any data is processed.

Configuration keys read:

Behavior:

• If the output directory does not exist, creates it and the logs subdirectory.
• If clear_output_directory: true, removes all contents of output/ except output/logs/ (logs are always preserved for audit trail).
• If clear_output_directory: false, prompts the user interactively before deleting.
• Returns False (cancels the pipeline) if the user declines the prompt.

Output: Clean output/ directory with output/logs/ ready for writing.

Step 2 — Preprocessing¶

Module: pipeline/preprocess.py

Reads the raw Excel input, validates the schema, normalizes all client and vaccination data, and writes a single canonical JSON artifact.

Configuration keys read:

Inputs:

• Excel file from input/ (single worksheet, .xlsx)
• config/vaccine_reference.json — maps vaccine codes to disease names
• config/disease_normalization.json — normalizes raw disease name variants
• config/phix_mapping.json — PHU-keyed school name → PHIX facility ID mapping (when PHIX validation enabled)

Outputs:

• output/artifacts/preprocessed_clients_<run_id>.json — canonical client artifact
• output/logs/preprocess_<run_id>.log — processing log
• phix_exact.csv, phix_inexact.csv, phix_no_match.csv — school match audit CSVs (when PHIX validation enabled)

Processing:

1. Validates school/daycare names against phix_mapping.json for the configured PHU (when phix_validation.enabled: true)
2. Reads and validates Excel schema (required columns, data types)
3. Normalizes disease names using disease_normalization.json
4. Expands vaccine codes to disease names using vaccine_reference.json
5. Filters diseases against chart_diseases_header; collapses unlisted diseases to "Other"
6. Computes client ages relative to date_notice_delivery (determines parent vs. student addressing)
7. Sorts clients deterministically: school → last name → first name → client ID
8. Assigns stable sequence numbers (00001, 00002, …)
9. Synthesizes missing school/board identifiers where needed
10. Writes the canonical JSON artifact

Step 3 — QR Code Generation (optional)¶

Module: pipeline/generate_qr_codes.py

Generates QR code PNG images for each client from a configurable URL payload template. Skipped entirely when qr.enabled: false.

Configuration keys read:

Inputs: output/artifacts/preprocessed_clients_<run_id>.json

Outputs: PNG files in output/artifacts/qr_codes/<sequence>.png

All template placeholders are validated at config-load time against the TemplateField enum. See the Template Field Reference for available placeholders.

Step 4 — Notice Generation¶

Module: pipeline/generate_notices.py

Renders Typst source files (.typ) for each client by combining the preprocessed JSON artifact with PHU-specific language templates. This step is always run.

Configuration keys read:

Inputs:

• output/artifacts/preprocessed_clients_<run_id>.json
• output/artifacts/qr_codes/<sequence>.png (if QR codes were generated)
• Language template module (templates/ or phu_templates/<name>/)
• config/translations/{lang}_diseases_chart.json
• config/translations/{lang}_diseases_overdue.json

Outputs: .typ source files in output/artifacts/typst/

Templates are loaded dynamically at runtime. The --template CLI argument selects a directory under phu_templates/. When omitted, the built-in templates/ directory is used. Each template module must define a render_notice() function.

Disease names are translated into the target language in Python before being passed to Typst — no runtime lookups occur in the Typst templates themselves.

Step 5 — PDF Compilation¶

Module: pipeline/compile_notices.py

Compiles the generated .typ Typst source files into individual PDF notices by invoking the typst command-line tool as a subprocess.

Configuration keys read:

Inputs: .typ files from output/artifacts/typst/

Outputs: PDF files in output/pdf_individual/

This step is fail-fast: if any .typ file fails to compile, the pipeline halts immediately. Typst v0.14.2 is required; the binary must be on PATH or configured via typst.bin.

Step 6 — PDF Validation¶

Module: pipeline/validate_pdfs.py

Validates compiled PDFs against configurable rules using invisible markers embedded by the Typst templates and pypdf text extraction.

Configuration keys read:

Severity levels: disabled (skip), warn (log only), error (halt pipeline).

Inputs:

• PDFs from output/pdf_individual/
• output/artifacts/preprocessed_clients_<run_id>.json (source of truth for expected client IDs)

Outputs:

• Console summary with per-rule pass/fail counts
• output/metadata/<lang>_validation_<run_id>.json — per-PDF results and measurements

For a full explanation of how markers work and how to add new rules, see PDF Validation.

Step 7 — PDF Encryption (optional)¶

Module: pipeline/encrypt_notice.py

Encrypts individual PDFs using a per-client password generated from a configurable template. Skipped when encryption.enabled: false.

Configuration keys read:

Inputs: PDFs from output/pdf_individual/

Outputs: Encrypted PDFs in output/pdf_individual/ (replace originals or alongside them)

Per-item recovery: if individual PDF encryption fails, the failure is logged and processing continues for remaining PDFs.

Step 8 — PDF Bundling (optional)¶

Module: pipeline/bundle_pdfs.py

Combines individual PDFs into multi-client bundles, optionally grouping by school or board. Skipped when bundling.bundle_size: 0.

Configuration keys read:

Inputs: PDFs from output/pdf_individual/

Outputs: Bundle PDFs in output/pdf_combined/ with a manifest JSON

This step runs independently of encryption — both can be enabled simultaneously. Per-item recovery applies.

Step 9 — Cleanup¶

Module: pipeline/cleanup.py

Removes intermediate files after a successful pipeline run. Behavior is controlled entirely by configuration flags.

Configuration keys read:

When remove_unencrypted_pdfs: true and neither encryption nor bundling is enabled, individual PDFs are assumed to be the final output and are preserved regardless of this setting.