04 — Model Strategy & Fine-Tuning#

Document set: Agentic-Native SDLC for Regulated Medical Device Engineering Status: Controlled engineering reference · Revision date: May 2026 Owning function: ML Platform & Quality Engineering Cross-references: 01-requirements · 02-maturity-model · 03-reference-architecture · 05-evaluation-and-validation · 06-agentic-workflows · 07-security-and-compliance · 08-token-and-gpu-economics · 09-adoption-roadmap

Regulatory anchors: IEC 62304 · ISO 13485 / QMSR · ISO 14971 · FDA CSA · GAMP 5 · 21 CFR Part 11 · ISO/IEC 42001 · FDA AI-enabled device guidance + PCCP

Note on thresholds: All numeric thresholds in this document (e.g., ≥99.9%, abstention rates, eval gates) are placeholders pending calibration in 05. They denote intent and governance structure, not yet-ratified acceptance criteria.

1. Strategy rationale: why a tiered, multi-model fleet of open-weight models#

The toolchain is the regulated artifact, not the device. Per Principle P5 (the harness is the product), the models are components inside a verification harness whose system-level correctness target is ≥99.9% at the release gate (P1). No single model — however large — is the source of that guarantee; the guarantee emerges from Generate → Verify → Repair → Gate loops where deterministic checks (P2: determinism wraps probabilism) bound probabilistic generation.

Given that, model strategy optimizes for fitness-per-task at lowest defensible cost, not for a single frontier model.

1.1 Why tiered multi-model instead of one big model#

The fleet is a portfolio: route each task to the smallest capable model (§10), escalate on low confidence, and let deterministic verifiers — not model scale — own the correctness guarantee.

1.2 Why open-weight + self-hosted (P7: self-hosted, sovereign, reproducible)#

HARD CONSTRAINT (restated): Self-hosted, fine-tuned, open-weight models only. No SaaS LLM APIs anywhere in the SDLC toolchain.

2. The fleet specification#

Five tiers. All tiers are served from the shared stack: K8s + NVIDIA GPU Operator, vLLM / Triton + TensorRT-LLM / KServe with multi-LoRA hot-swap. See 03 for serving topology.

Routing summary. A Tier-S classifier/router triages every request: trivial → answer; ambiguous/high-risk → escalate to Tier-M/L; visual → Tier-V; retrieval → Tier-E first. Routing policy is governed by IEC 62304 class of the affected artifact (P3: autonomy by class A/B/C) and recorded as evidence (P4).

3. Base-model selection criteria & governance#

A base model is a supplier-provided component under ISO 13485 / QMSR supplier controls and GAMP 5 categorization. No base model enters the fleet without passing the gate below and landing in the Approved Base-Model Registry.

3.1 Selection criteria#

3.2 License review (open-weight ≠ unrestricted)#

"Open-weight" describes weight availability, not unrestricted rights. Each license is reviewed individually; the table below is engineering guidance, not legal advice — Legal sign-off per model is mandatory and recorded in the registry.

3.3 Approved Base-Model Registry#

Maintained in the MLflow registry with signed entries. Schema:

Only approved bases may be parents of a fine-tune. Revocation propagates to all derived adapters (§8).

4. The fine-tuning pipeline#

flowchart TD
    subgraph SRC["Sourced & governed inputs (§6)"]
        A1[Internal sanitized corpus<br/>code · docs · tickets]
        A2[Task / instruction datasets]
        A3[Preference pairs<br/>chosen / rejected]
        A4[Verifier-filtered synthetic data]
    end

    B[("Approved Base-Model<br/>Registry (§3)")] --> S1

    A1 --> S1["Stage 1: DAPT<br/>Domain-Adaptive Continued Pretraining<br/>(usually full FT or large LoRA)"]
    S1 --> S2["Stage 2: SFT<br/>Instruction / task tuning<br/>(LoRA or full FT)"]
    A2 --> S2
    A4 --> S2
    S2 --> S3["Stage 3: Preference alignment<br/>DPO / ORPO (TRL)"]
    A3 --> S3
    S3 --> S4["Stage 4: Task/Domain LoRA adapters<br/>(PEFT/QLoRA) — one per specialization (§5)"]

    S4 --> E["Eval gate (05)<br/>deterministic suites + abstention checks"]
    E -->|pass| R[("MLflow registry<br/>signed adapter + lineage (§7)")]
    E -->|fail| X[Repair / re-tune / reject]

    R --> SERVE["Multi-LoRA serving<br/>vLLM/Triton hot-swap on shared base"]

    classDef gate fill:#eef,stroke:#446;
    class E gate;

4.1 When to use each stage#

4.2 LoRA vs full fine-tuning — decision rule#

Default posture: prefer LoRA. Full FT is the exception and requires a documented justification plus its own registry entry as a derived base.

5. Domain specialization#

Each domain ships as a named LoRA adapter over an approved (optionally DAPT'd) base, independently versioned, eval-gated, and signed. Multi-LoRA serving hot-swaps the right adapter per request.

5.1 Multimodal angle (Tier-V)#

Design inputs in this domain are inherently visual. Tier-V adapters target:

• Design PDFs / design history files → extract structured requirements/specs (feeds 06).
• Schematics / block diagrams → derive interfaces, signal lists, architecture facts.
• UI screenshots → verify UI against spec; detect drift.
• Imaging artifacts → describe/triage visual anomalies (advisory only; never a clinical claim).

All Tier-V outputs are advisory inputs to deterministic verifiers, not autonomous decisions; class-C-affecting outputs always require human confirmation (P3).

6. Data strategy for fine-tuning#

Training data is a controlled, versioned, signed artifact. The dataset is as much a regulated input as the model.

Contamination is a correctness-and-evidence risk, not a metric nuisance. A model trained on its own eval set produces inflated scores that cannot support a defensible ≥99.9% claim (P1/P4). Contamination control is a release gate (§11 anti-pattern).

7. Reproducibility & validation as first-class (P7)#

A fine-tune must be reproducible and defensible as CSA evidence. We lock every input and sign the full lineage so an auditor can re-derive the artifact.

7.1 What is locked#

7.2 Signed lineage chain#

flowchart LR
    D["dataset_uid<br/>(signed digest)"] --> A
    BM["base model_uid<br/>(registry digest)"] --> A
    CFG["config + seeds + env<br/>(digest)"] --> A
    A["adapter / model artifact<br/>(SHA-256)"] --> EV
    EV["eval run<br/>(suite ver + scores)"] --> REG
    REG[("MLflow registry entry<br/>cosign-signed, SLSA provenance")]

Each edge is a cosign attestation; the registry entry carries SLSA provenance. The chain answers the auditor's question — "show me exactly how this model was produced and that nothing changed" — and is the artifact-level realization of Part 11 (P4) and CSA.

7.3 How this satisfies P7 and D2-L4#

• P7 (reproducible): Any registered fine-tune is byte-reproducible from locked inputs; re-running the pipeline yields the same artifact digest (modulo documented nondeterminism, which is itself bounded and recorded).
• D2-L4 (02): Validated Autonomous Agents require validated models. The signed lineage + eval-gated promotion (§8) is the model-side evidence package that lets an agent operate autonomously within its IEC 62304 class envelope.

8. Model lifecycle & governed evolution#

Models evolve under a PCCP-style predetermined change control applied to the toolchain models (reusing the FDA PCCP concept; the device itself is governed separately). Promotion is eval-gated (05); no promotion bypasses the gate.

flowchart LR
    C["candidate<br/>(new adapter / base)"] --> SH["shadow<br/>(mirror traffic, no effect)"]
    SH --> CN["canary<br/>(small % real, guarded)"]
    CN --> PR["promote<br/>(default for tier/domain)"]
    PR --> DP["deprecate<br/>(retain weights + lineage)"]
    SH -->|fail gate| RJ[reject]
    CN -->|regression| RB[rollback]

8.1 PCCP-style predetermined change control (toolchain models)#

The Predetermined Change Control Plan for the model fleet specifies, in advance: the allowed change types (e.g., new domain adapter, refreshed SFT data), the fixed eval protocol that gates them, the rollback triggers, and the autonomy class affected. Changes inside the envelope flow through the lifecycle without re-opening the whole validation; changes outside it require plan revision. This ties to D1-L5 (02) (self-optimizing under governance) and the evaluation regime in 05. Base-model revocation (§3) forces immediate deprecation of all derived adapters.

9. Abstention & calibration#

The ≥99.9% system property (P1) depends on models that decline rather than guess on out-of-distribution or high-risk inputs, escalating to a larger tier or a human. Abstention is a trained and served behavior, validated in 05.

Calibration target: in the operating region, a high-confidence answer is correct ≥99.9%; everything else abstains and routes to verification or a human. A miscalibrated-but-accurate model is not acceptable — abstention behavior is itself an eval gate.

10. Right-sizing & cost linkage#

Smallest-capable-model principle (P6): route every task to the smallest model that passes the gate; escalate only on abstention.

Distillation note. The verifier-filtered synthetic pipeline (§6) is the distillation substrate: only Tier-L outputs that pass deterministic verification become training data for smaller adapters, so distillation transfers verified behavior, not hallucinations. The primary metric for right-sizing decisions is cost-per-green-PR (P6), owned in 08.

11. Anti-patterns#

Summary#

The model strategy is a portfolio of small, specialized, signed, open-weight models governed as regulated components: tiered for cost/latency/specialization, fine-tuned through a locked-and-signed pipeline (DAPT → SFT → DPO/ORPO → LoRA), specialized per domain via hot-swappable adapters, and promoted only through eval-gated, PCCP-style change control. The correctness guarantee lives in the harness (P5) and its deterministic verification (P2), not in model scale; the models contribute calibrated capability and disciplined abstention (P1, §9), and reproducible, signed lineage (P7, §7) is what makes every fine-tune defensible CSA evidence. Implementation detail for evaluation lives in 05 and for economics in 08.