Architecture / Security Architecture Operating Models / Measurement
05 · Evidence
Measurement and data lineage
Instrument the lifecycle, trace outcomes to trustworthy sources and expose data quality.
17leading indicators
18lineage definitions
1illustrative scorecard
Workbook source · ODM Contributing Indicators
Through-life instrumentation
Leading and diagnostic phase-gate indicators.
Tier-2 leading and diagnostic indicators instrumented at Service Through Life phase gates. These do not carry PLAs; they explain ODM movement and give ESA/SSA/OSA leads early warning. Phase numbers refer to the Through Life tab.
| CI ID | Through Life Phase | Architecture Layer | Indicator | Measure | Role Owner | Feeds ODM(s) | Source System |
|---|---|---|---|---|---|---|---|
| CI-01 | 2. Planning | Business | Security requirements defined pre-build | % of in-scope initiatives with security requirements baselined before Development phase entry; median days to produce requirements for procurement. | SSA | ODM-04, 05, 06 | Requirements repository; procurement workflow |
| CI-02 | 2–3. Planning/Design | Information Systems | Traceability established at design | % of Tier-1 designs with requirement → control objective → control mapping recorded in the architecture repository at Design-gate exit. | SSA | ODM-04, 07, 10 | Architecture repository |
| CI-03 | 3. Design | All layers | Design-gate engagement rate | % of in-scope changes passing the Design gate with completed security architecture review; median review turnaround (working days). | SSA | ODM-01, 15 | Delivery pipeline gates; review log |
| CI-04 | 3. Design | All layers | Exceptions raised at design | Count and severity of design exceptions accepted at gate; % with owner, expiry and remediation path recorded. | ESA | ODM-02, 11 | Exception register |
| CI-05 | 4–5. Development/Deployment | Information Systems | Escape defects | Security defects found in build/test/deployment that should have been caught at design (escape rate); security test pass rate at go-live gate. | SSA | ODM-01, 02 | Defect tracker; SAST/DAST results |
| CI-06 | 5. Deployment | Technology | Pattern conformance at go-live | % of deployments matching the specifying reference pattern at first configuration scan post go-live. | OSA | ODM-03, 10 | Configuration compliance scanning; CMDB |
| CI-07 | 5–6. Deployment/Transition | Information Systems | Verification evidence at handover | % of security requirements with verification evidence attached at operational acceptance; completeness of security runbook/monitoring handover. | OSA | ODM-01, 04 | Test evidence; service transition records |
| CI-08 | 7–8. Operation/Maintenance | Technology | Drift detection | Mean days from configuration drift (vs pattern/standard baseline) to detection; count of unremediated drift items >30 days. | OSA | ODM-03, 09 | Configuration compliance scanning |
| CI-09 | 7–8. Operation/Maintenance | Business | Supplier assurance currency | % of critical/high suppliers whose assurance is in date; count of contracts past re-assessment due date. | ESA | ODM-05 | Supplier assurance platform; contract register |
| CI-10 | 7. Operation | All layers | Control assurance results | % of sampled controls on critical domains operating as designed (control testing); audit finding pipeline by root-cause class. | OSA | ODM-07, 08 | GRC platform; control testing results |
| CI-11 | 8–9. Maintenance/Optimisation | Technology | Debt identified vs retired | Debt items identified per assessment cycle vs items retired; net debt movement on critical domains. | ESA | ODM-11 | Technical debt inventory |
| CI-12 | 1–2. Strategy/Planning | Business | Roadmap milestone health | Milestone RAG across the security capability roadmap; slippage >1 quarter flagged. | SSA | ODM-12 | Portfolio tooling |
| CI-13 | 7. Operation | All layers | Post-incident architectural review | % of P1/P2 incidents with architectural RCA completed within 30 days; pattern/standard changes raised per incident. | SSA | ODM-13 | Incident records; standards register |
| CI-14 | 7–9. Operation–Optimisation | Technology | Recovery exercise completion | Critical services exercised in rolling 12 months; RTO/RPO achievement per exercise. | OSA | ODM-14 | BC/DR test records |
| CI-15 | 1–3. Strategy–Design | Business | Early engagement signals | % of initiatives engaging security architecture at Strategy/Planning (before Design); repeat-engagement rate by delivery team. | SSA | ODM-15, 16 | Engagement log; pipeline gates |
| CI-16 | 9. Optimisation | All layers | Self-service pattern usage | Pattern downloads/instantiations without ESA escalation; accredited practitioner count by team. | ESA | ODM-16 | Pattern repository telemetry; training records |
| CI-17 | 10. Retirement | All layers | Secure decommissioning completion | % of retired services with completed secure decommissioning checklist (access revoked, data sanitised, dependencies removed, artefacts updated). | OSA | ODM-10, 11 | CMDB; decommissioning records |
Workbook source · ODM Data Lineage
Operational data to decision evidence
Sources, extraction, calculations and quality ownership.
Operational Data → Metrics path per ODM (Risk Ontology). An ODM without a reliable automated source should not enter a PLA until its lineage is fixed - data quality kills credibility faster than a bad number.
| ODM ID | ODM Name | Operational Data Required | System of Record | Extraction | Frequency | Transformation / Calculation | Quality Checks & Data Owner |
|---|---|---|---|---|---|---|---|
| ODM-01 | Design-stage defect capture | Security design defects by detection phase; incident RCAs tagged “design-attributable” | Defect tracker; incident management | API / export | Monthly | Defects detected ≤ Design gate ÷ all security design defects in period | Phase tagging mandatory at defect entry; RCA tagging QA sampled 10%. Owner: SSA lead |
| ODM-02 | Security rework cost | Change records flagged security-driven post-build; effort/cost per change | Delivery tooling; finance cost codes | API + finance extract | Quarterly | Σ security-driven rework cost ÷ total in-scope delivery spend | Security-driven flag reviewed at CAB; finance code reconciliation. Owner: ESA lead |
| ODM-03 | Pattern adoption & conformance | Deployment records; pattern baselines; config scan results | CMDB; compliance scanning; architecture repository | Automated scan | Monthly | Conformant deployments ÷ pattern-specified deployments; drift age from scan timestamps | Pattern baselines version-controlled; scan coverage ≥ 95% of estate. Owner: OSA lead |
| ODM-04 | Requirements traceability integrity | Requirements with links to controls and verification evidence | Requirements repository; test management | API | Quarterly | Tier-1 changes with complete req→control→evidence chain ÷ all Tier-1 changes at go-live | Broken-link report zero at gate; evidence attachments mandatory. Owner: SSA lead |
| ODM-05 | Third-party assurance coverage | Supplier tier, assessment date, score, remediation status | Supplier assurance platform; contract register | API | Monthly | Critical suppliers in-date and in-threshold ÷ all critical suppliers | Tiering reviewed annually; assessment expiry alerts. Owner: ESA lead |
| ODM-06 | Procurement security cycle time | Engagement and requirements-issue timestamps per procurement | Procurement workflow | API | Quarterly | Median working days engagement → issue | Timestamp completeness check; outliers reviewed. Owner: SSA lead |
| ODM-07 | Regulatory trace completeness | Obligation register; control mappings; architecture links | GRC platform; architecture repository | API | Quarterly | Obligations with full evidence chain ÷ in-scope obligations | Obligation register refresh on regulatory change; orphan-control report. Owner: ESA lead |
| ODM-08 | Audit findings attributable to design | Audit findings with root-cause classification | Audit findings register | Export | Per audit | Count where root cause = architecture/design | Root-cause class agreed with audit function. Owner: ESA lead |
| ODM-09 | Standards currency | Standard, class, last-review date, review cycle | Standards register | Repository query | Monthly | Standards within cycle ÷ all standards | Review-date mandatory field; auto-expiry flags. Owner: OSA lead |
| ODM-10 | Control-objective coverage | Control objectives per Tier-1 domain; realising controls; assurance status | Architecture repository; CMDB; control testing | Repository query + API | Quarterly | Objectives with ≥ 1 implemented, operating control ÷ all Tier-1 objectives; gaps require risk-acceptance record | Paper-trace audited against config scans (pairs with ODM-03); acceptance records checked. Owner: ESA lead |
| ODM-11 | Security debt exposure | Debt items and exceptions: severity, age, domain, owner, expiry | Exception register; debt inventory | API | Monthly | Σ (severity weight × age factor) on critical domains, indexed to baseline = 100 | No item without owner and expiry; monthly orphan report. Owner: ESA lead |
| ODM-12 | Roadmap delivery | Milestone plan vs actual per capability | Portfolio tooling | API | Quarterly | Milestones delivered within +1 quarter ÷ milestones due | Baseline change control on plan dates. Owner: SSA lead |
| ODM-13 | Architectural lessons closed | P1/P2 incidents; RCA outputs; pattern/standard change records | Incident management; standards register | API | Quarterly | Incidents with published change ≤ 90 days ÷ closed P1/P2 incidents | RCA completion enforced at incident closure. Owner: SSA lead |
| ODM-14 | Resilience design verification | Exercise records with RTO/RPO results per critical service | BC/DR test records; service catalogue | Export | Quarterly | Critical services exercised ≤ 12 months meeting RTO/RPO ÷ all critical services | Criticality list owned by Resilience Board; results signed by service owner. Owner: OSA lead |
| ODM-15 | Early engagement rate | Initiative register with engagement timestamps vs build start | Pipeline gates; engagement log | API | Quarterly | Initiatives engaged pre-build ÷ in-scope initiatives | In-scope definition (Tier-1) reviewed annually. Owner: SSA lead |
| ODM-16 | Capability uplift | Accreditations; pattern usage without escalation | Training records; pattern telemetry | Export + telemetry | Annual | Composite trend: practitioner count; self-served design % | Accreditation register verified by HR. Owner: ESA lead |
| MP-01 | Metric automation & data quality | Pipeline run logs; manual-adjustment log | Metrics pipeline | Pipeline logs | Quarterly | ODMs produced without manual adjustment ÷ all ODMs | Adjustment log reviewed at Design Authority. Owner: OSA lead |
| MP-02 | Decision uptake | Governance decision records with evidence citations | Decision log / minutes | Export | Quarterly | Decisions citing ODM evidence ÷ all recorded decisions | Citation field mandatory in decision template. Owner: ESA lead |
Workbook source · ODM Dashboard
Illustrative scorecard
A worked portfolio view; current values are placeholders.
Current values (blue) are illustrative placeholders - replace with pipeline feeds. RAG and rollups are formula-driven from the thresholds in the ODM Register.
| ODM ID | ODM Name | Type | Unit | Direction | Target (Green) | Amber Threshold | Current (illustrative) | RAG |
|---|---|---|---|---|---|---|---|---|
| ODM-01 | Design-stage defect capture | KPI | % | Higher | 80 | 65 | 72 | Amber |
| ODM-02 | Security rework cost | KRI | % | Lower | 2 | 4 | 2.8 | Amber |
| ODM-03 | Pattern adoption & conformance | KCI | % | Higher | 85 | 70 | 81 | Amber |
| ODM-04 | Requirements traceability integrity | KCI | % | Higher | 95 | 85 | 93 | Amber |
| ODM-05 | Third-party assurance coverage | KRI | % | Higher | 100 | 90 | 96 | Amber |
| ODM-06 | Procurement security cycle time | KPI | days | Lower | 10 | 15 | 9 | Green |
| ODM-07 | Regulatory trace completeness | KRI | % | Higher | 95 | 85 | 88 | Amber |
| ODM-08 | Audit findings attributable to design | KRI | count | Lower | 2 | 4 | 3 | Amber |
| ODM-09 | Standards currency | KCI | % | Higher | 90 | 75 | 92 | Green |
| ODM-10 | Control-objective coverage | KRI | % | Higher | 95 | 85 | 91 | Amber |
| ODM-11 | Security debt exposure | KRI | index | Lower | 100 | 130 | 112 | Amber |
| ODM-12 | Roadmap delivery | KPI | % | Higher | 80 | 65 | 83 | Green |
| ODM-13 | Architectural lessons closed | KPI | % | Higher | 100 | 80 | 100 | Green |
| ODM-14 | Resilience design verification | KCI | % | Higher | 95 | 85 | 89 | Amber |
| ODM-15 | Early engagement rate | KPI | % | Higher | 90 | 75 | 86 | Amber |
| ODM-16 | Capability uplift | KPI | trend | Higher | 10 | 5 | 12 | Green |
| MP-01 | Metric automation & data quality | KCI | % | Higher | 80 | 60 | 65 | Amber |
| MP-02 | Decision uptake | KPI | % | Higher | 70 | 50 | 55 | Amber |
Portfolio rollup
| Green | 5 |
|---|---|
| Amber | 13 |
| Red | 0 |
| Total ODMs | 18 |