Pipeline Architecture¶

Overview¶

The ImmuKnow pipeline transforms raw vaccination records exported from Panorama/PEAR into bilingual (EN/FR) PDF immunization notices and history charts for children overdue under CCEYA and ISPA.

The pipeline follows a sequential, stateless step architecture: each step reads its inputs fresh from disk and writes its outputs to disk. No in-memory state is passed between steps through the orchestrator. This design ensures that steps are independently testable and re-runnable — if Step 5 fails, you can fix the issue and re-run Steps 5–9 without reprocessing the input data.

Pipeline flow¶

flowchart LR
    A[Excel Input] --> S1[1. Prepare Output]
    S1 --> S2[2. Preprocess]
    S2 --> S3[3. Generate QR Codes]
    S3 --> S4[4. Generate Notices]
    S4 --> S5[5. Compile Notices]
    S5 --> S6[6. Validate PDFs]
    S6 --> S7[7. Encrypt PDFs]
    S7 --> S8[8. Bundle PDFs]
    S8 --> S9[9. Cleanup]
    S9 --> Z[Output Directory]

    style S3 stroke-dasharray: 5 5
    style S7 stroke-dasharray: 5 5
    style S8 stroke-dasharray: 5 5

Steps shown with dashed borders are optional — they are skipped when disabled in config/parameters.yaml.

Step summary¶

Design principles¶

Stateless steps Each step reads its inputs from disk and writes outputs to disk. The orchestrator never passes in-memory objects between steps. This means same input always produces the same output, and any step can be re-run independently.

Normalized JSON artifact Preprocessing produces a single preprocessed_clients_<run_id>.json artifact that serves as the canonical source of truth for all downstream steps. Client records are deterministically ordered by school → last name → first name → client ID, and each client receives a stable sequence number (00001, 00002, etc.) that persists through all downstream operations.

Bilingual support Both English and French are first-class concerns. Disease names, notice text, and date formatting are all localized before being passed to Typst. The language argument selects the full rendering path; both languages share the same pipeline steps and configuration file.

Fail-fast vs. per-item recovery Critical steps (Preprocessing, Notice Generation, Compilation, PDF Validation) implement fail-fast: any error halts the pipeline immediately. Optional steps (QR Codes, Encryption, Bundling) implement per-item recovery: individual item failures are logged and skipped, and the pipeline continues processing remaining items.

Module organization¶

pipeline/
├── orchestrator.py         # CLI entry point (viper), coordinates 9 steps
├── prepare_output.py       # Step 1
├── preprocess.py           # Step 2
├── generate_qr_codes.py    # Step 3 (optional)
├── generate_notices.py     # Step 4
├── compile_notices.py      # Step 5
├── validate_pdfs.py        # Step 6
├── encrypt_notice.py       # Step 7 (optional)
├── bundle_pdfs.py          # Step 8 (optional)
├── cleanup.py              # Step 9
├── config_loader.py        # Configuration loading and validation
├── data_models.py          # Dataclasses for client records and artifacts
├── enums.py                # Language, BundleStrategy, TemplateField enums
├── translation_helpers.py  # Disease name normalization and translation
├── validate_phix.py        # PHIX school name validation (called from preprocess)
└── utils.py                # Template rendering and context building utilities

templates/                  # Built-in Typst templates (EN/FR)
phu_templates/              # PHU-specific template overrides (gitignored)