Cost, efficiency and AI

From 1.1 Core claim and hooks

Three-sentence version (for the opening)

• AI is no longer optional. The State Council set the timetable: over 70% adoption of smart devices and agents by 2027.
• But using it is not earning from it. 88% of organisations use it; only 13% report enterprise value.
• Where is the gap? Nobody translated those three saved hours into a number on the income statement.

Why this claim lands with a CEO

Hook 3: the −19% experiment

Claim   ── The gap is not in the technology, it is in the operating loop
Figures ── 88% use it, 13% get enterprise value → 75 points of arbitrage

Hooks   ── ① The 75-point gap: everyone is running, most are running on the spot
           ② Superstars: +33.5% vs +163% — AI widens gaps, it does not close them
           ③ +55% or −19%: same tool, two directions, and the difference is management

Close   ── Not how to use AI — how to sign AI off:
           where the value is · how strong the evidence is · who owns it · how to scale without losing control

From 2.1 Three numbers on the value gap

Three supporting pillars

Leave this line out and an informed CFO takes you apart. Put it in and you look more rigorous than the consultancy.

Supporting layer (other angles on the same conclusion)

From 2.5 Talent and organisation

The superstar effect (the most striking numbers in the course)

Skill premium: the talent market has already voted with money

A two-track labour market (talk role redesign, not layoffs)

[R025] AI is splitting jobs onto two tracks:

• Skill requirements in high-AI-exposure roles change twice as fast as in low-exposure roles
• New tasks relying on empathy, judgement and creativity are growing 2.5× faster

The upheaval in entry-level roles (what management should watch most)

The global labour picture (WEF; for the macro backdrop slide)

From 2.4 Productivity evidence

Task level: evidence that holds (all controlled or quasi-experimental)

One rule (the conclusion here; put it on the slide in large type)

From 2.3 China policy and industry

The State Council set the timetable (the hardest page)

Guidance on Deepening the "AI Plus" Initiative, issued August 2025 — [R013]

The NDRC's framing: the phase has changed

Market size: the market is already big enough

Capability jump: domestic is no longer the weak link

The number that stings most: data quality

From 2.2 Global trends at a glance

Money: investment is doubling, and not on short-term returns

People: the CEO gets personally involved

Speed: one of the fastest-spreading technologies ever

From 2.2 Global trends at a glance

Rollout: from pilot to production

Money ── 94% keep investing even without a return; only 6% pull back
People ── 72% of CEOs say "I am the decision-maker"
Speed ── 53% adoption in three years, faster than the PC or the internet
Soon  ── Production deployment goes from 25% to 54% within six months

Chart suggestion

From 2.7 BCG value tiers

Three tiers (works as a single slide)

How big the gap is ⭐

From 2.7 BCG value tiers

What the top tier does differently

These two sets are not the same thing; do not mix them:

• Open with 88/36/13 to establish that using it is not earning from it
• Use 5/35/60 in the self-assessment so they place themselves
• ⚠️ Never plot the two sets on one chart — different bases invite challenge

From 2.7 BCG value tiers

The good news (don't leave only anxiety)

5%   Future-built ── 5× revenue growth · 3× cost reduction · 1/3 already using agents
35%  Scaling      ── Value starting to show · 12% using agents
60%  Lagging      ── Substantial spend, almost no real value · barely any agents

What the top tier does:
  IT spend +26% · AI up to 64% of the IT budget · returns reinvested · 15% to agents
  → By 2028: twice the revenue growth of laggards, 40% deeper cost reduction

Mechanism ── Compounding: returns → reinvest in capability → larger returns

From 5.6 Maturity self-assessment

How to use it

Dimension 1: strategy and accountability

□ 1. There is a clear AI priority order (what to do, what not to)        ___
□ 2. Every AI application has a named business owner (not IT)            ___
□ 3. The executive team explains "why we do AI" the same way             ___
□ 4. AI outcomes appear in someone's performance review                  ___
                                                  Dimension average ___

Dimension 2: cost and value

□ 1. You can state the unit cost of your main processes                  ___
□ 2. Every AI project has baseline data before approval                  ___
□ 3. At least one AI benefit has been confirmed by finance               ___
□ 4. ROI includes human review and data governance costs                 ___
                                                  Dimension average ___

Dimension 3: data foundation

□ 1. Master data is standardised (material, customer, vendor codes)      ___
□ 2. Event logs export for key processes (case ID + activity + time)     ___
□ 3. Data is classified and access is least-privilege                    ___
□ 4. You know how much complete, labelled, usable data you have          ___
                                                  Dimension average ___

Dimension 4: process and use cases

□ 1. You know the gap between the standard and the real process          ___
□ 2. You know which step waits longest and reworks most                  ___
□ 3. Five or fewer AI projects running, plus a do-not-do list            ___
□ 4. Pilots have a control group or historical baseline                  ___
                                                  Dimension average ___

Dimension 5: organisation and people

□ 1. There is a roadmap for how AI reshapes roles                        ___
□ 2. At least 25% of the AI budget goes to people and change             ___
□ 3. AI-capable key staff have a retention mechanism                     ___
□ 4. Staff know the company's position: augment or replace               ___
                                                  Dimension average ___

From 5.6 Maturity self-assessment

Dimension 6: governance and risk

□ 1. Written AI usage guidance exists and has been communicated          ___
□ 2. A department or body clearly owns governance                        ___
□ 3. High-risk output requires a human signature                         ___
□ 4. AI-assisted decisions are logged and auditable                      ___
□ 5. There is a vendor exit plan (you can switch, and take the data)     ___
                                                  Dimension average ___

Scoring and placement

Radar chart (drawn live)

        Strategy &
       accountability
            5
 Governance 4    Cost &
  & risk    3     value
            2
  People    1     Data
            0
      Process & use cases

The fastest fix for each of the six dimensions

Six dimensions ── strategy · cost & value · data · process · people · governance
Scoring        ── 1–5 each, sum the six averages, 30 maximum
Placement      ── 24–30 future-built (5%) / 15–23 scaling (35%) / 6–14 lagging (60%)

⭐ Core        ── Read the shortest spoke, not the longest
                  Scaling is a series circuit: break one link and everything after it fails

Fixes          ── Four of the six cost nothing and can be done this week

• Anonymity is the point. Put names on it and everyone scores a 4.
• When reading a few out, give the score spread only; never comment on a company.
• If the room scores under 10 across the board, that is good — they are being honest. Say at once: "60% of companies worldwide are in this tier. You are not alone."
• If someone scores 28, ask privately which AI benefit the CFO has signed off. It is usually an overestimate.

From 2.1 Three numbers on the value gap

Counter-evidence (makes the 13% look reachable)

Do not present only the gap, or the boss concludes "let's wait". Follow immediately with: the gap can be crossed, and the conditions are known.

Three bars ── 88% adopted → 36% scaled → 13% getting enterprise value
⚠️ Say it   ── The three surveys use different bases; this is not one funnel

Support    ── Only 34% are genuinely redesigning the business (Deloitte)
           ── Technology budget : people budget = 3 : 1 (Accenture)
           ── Only 35% of executives have a workforce roadmap

Causes     ── Costs are opaque · processes are invisible · benefits are not recognised

13% is not luck ──
  ×2.5  all five priorities in place    ×3    go deep on one core process
  ×4.5  strategic bet on agents         +88%  score on reshaping ways of working

From 2.1 Three numbers on the value gap

Chart suggestion

• Chart: three bars (88/36/13) falling off a cliff, joined by dashed arrows labelled "different bases".
• Colour: 88% grey, 36% mid-grey, 13% in the accent colour (red or orange).
• Footer — Sources: Stanford HAI, AI Index 2026; Accenture, Making Reinvention Real with Gen AI. The three surveys differ in sample and definition and do not form a single funnel. Accessed 2026-08-05.
• Mermaid and ECharts source is in 6.2.

From 2.5 Talent and organisation

The biggest organisational gap

Hiring   ── The most AI-exposed firms add people twice as fast as the least
Amplify  ── Top 20%: revenue per head +163% vs +33.5% for the group (5× gap)
Premium  ── 62% wage premium for AI skills (57% last year)
Two tracks ── 22% of roles professionalise (+42% pay) / 52% democratise
Newcomers ── 52% of new entry-level skill demands are senior capabilities (7% in low-exposure roles)
Global   ── Net +78 million roles; of every 100 people, 59 need training and 11 will not get it
Gap      ── Budget is 3:1 technology to people, but the gap opens on the people side

From 4.1 The seven-step operating loop

The seven steps in detail

Which section of this library each step maps to

Mapped to the CAICT framework (essential for SOEs and upward reporting)

Where's the money → Where's it blocked → What first → Does it work → Did it earn → Who owns it → Can it scale
      ①                    ②                  ③            ④              ⑤            ⑥              ⑦
                                                                            ↑
                                                              Most companies die here

From 4.2 Cost transparency

Three open-source tools (demonstrable in class, no production access needed)

From 4.2 Cost transparency

Cost tree (participant output 1)

Total cost
├── People        → split by process/role  → labour unit cost per process
├── Cloud & compute → split by app/team    → unit cost per application (OpenCost)
├── Software licences → split by usage     → share of idle seats
├── Outsourcing & procurement → by category → unit price vs market benchmark
└── Cost of quality → rework / missed defects / stoppages / claims  ← the big one most often missed

From 4.2 Cost transparency

Three unit costs you must ask about (take away)

□ What does one unit of business cost?  (per order / ticket / conversation / quote)
□ How does that compare with peers?     (no benchmark, no target)
□ How much of it is repetitive work?    (that is what AI can actually reach)

From 4.11 FOCUS cost definitions

⭐ The four costs (FOCUS's core concept)

What FOCUS is

Before: one field set for AWS, another for Azure, another for Alibaba Cloud, more for each SaaS
        → to answer "what did we spend on AI" you hand-align dozens of fields

Now:    export everything as FOCUS → one field set → one table shows all of it

43 standard fields (grouped by purpose)

From 4.11 FOCUS cost definitions

Three things you can do immediately

□ 1. Have the CIO export cloud bills as FOCUS (all major providers support it)
□ 2. Check tag coverage — how much spend can be attributed to a business line?
□ 3. Standardise all cost reporting on EffectiveCost, and write it into policy

Classroom demo (no real invoices)

⭐ Four costs ── ListCost (list price) · ContractedCost (negotiated)
               · BilledCost (invoice) · EffectiveCost (after amortisation)
               → unit economics, allocation and ROI may use EffectiveCost only

Trap        ── Allocating on BilledCost: the prepayment month is punished, later months ride free

FOCUS       ── One cost vocabulary across cloud, SaaS and on-premise; 43 standard fields
Key field   ── Tags (they decide whose budget it is)
First KPI   ── Not money saved — tag coverage

Three moves ── Export as FOCUS · check tag coverage · standardise on EffectiveCost

From 4.2 Cost transparency

Accountability matrix (participant output 2)

From 4.3 Process diagnosis

Four questions to take away

□ 1. How many steps in the standard process? How many variants in reality?
□ 2. Which step waits longest? Who is it waiting on?
□ 3. What is the rework rate? Is it always the same problem?
□ 4. How many transactions bypassed a mandatory approval?

From process diagnosis to AI use cases (where this section lands)

After diagnosis, run it through three sieves:

Highly repetitive → automation / agent execution
Long waits        → AI pre-fill / smart routing / parallelisation
High rework       → AI rule checking / right first time

From 4.3 Process diagnosis

Participant output: target process and bottleneck map

• The real process today (with the share of main variants)
• The three biggest bottlenecks (waiting / rework / bypass)
• Candidate actions per bottleneck: standardise / automate / apply AI

Problem ── The process in the policy ≠ the process in the system
Method  ── Process mining: reconstruct the real process from event logs
Bar     ── Three columns is enough: case ID + activity + timestamp
Tool    ── PM4Py (GPL — fine for demos, commercial use needs legal review)
Four Qs ── How many variants? Waiting on whom? Rework rate? Who bypassed approval?
Landing ── Repetition→automate · waiting→pre-fill/route · rework→rule checking
Rule    ── See it clearly → standardise what can be standardised → then apply AI

• Process variant chart: the standard path vs the top five real variants, with shares
• Waiting-time waterfall: time per step, with the longest wait marked
• Before-and-after funnel: which step's waiting, rework or drop-off fell after AI

From 4.4 Use case priority matrix

Value dimension (1–5 each)

Feasibility dimension (1–5 each)

From 2.4 Productivity evidence

Macro level: a J-curve — don't rush to read the totals

Note the +4%, stronger where training was provided: the hardest evidence you have for a training budget (and it prices your own course nicely).

✅ Works    Service +14–15% / coding +26–55% / marketing +50% / office work +29%
❌ Counter  Senior developers −19% — and they believed they were faster
📏 Rule     Clear task boundary + repeatable + quality monitored = strongest gain
👤 Who gains  Novices > experts (AI narrows the skill gap)
📈 Macro    12,000 European firms +4% (stronger with training); US productivity 2.7% in 2025
⏳ Mindset  A J-curve: pay first, gain later — but be able to prove you are climbing

From 4.4 Use case priority matrix

Risk dimension (sets bubble size)

From 2.3 China policy and industry

Where AI lands across industry

[R015] Industrial models are deepening at both ends and breaking through in the middle:

Governance: regulators are flagging risk too

Rollout methodology: CAICT's official framework (teachable as is)

Capability analysis → needs discovery → solution design → development & testing
  → application build → performance assessment → monitoring → operations management

Policy  ── Over 70% adoption by 2027 — under 17 months left
Framing ── From "experiment" to "value creation"; regulators want results
Market  ── ¥900bn core industry in 2024 (+24%), heading for ¥1.2tn in 2025
Capability ── Domestic chips are broadly on par; the technical reason to wait is disappearing
Data    ── 82.5% of industry datasets lack content density ← your real starting point
Landing ── Operations 45.8% > R&D and design 28.3% > production 25.9%
Method  ── CAICT's four phases and eight steps — the best language for reporting upward

Copyright note

Government documents and CAICT reports are natively Chinese and may be summarised, but:

• Government infographics (such as the NDRC explainers): quote in full; never crop the source, date or emblem
• CAICT reports are copyrighted: cite report and page when extracting data, and clear permission for full charts
• See 7.3 Licensing red lines

From 4.4 Use case priority matrix

Three hard screening rules (from the evidence in 2.4)

✅ Clear task boundary   ← you can describe what "done right" looks like
✅ Repeatable            ← high-frequency and patterned
✅ Quality monitorable   ← a person or a rule can check it

From 4.4 Use case priority matrix

Pre-judged common use cases (to save time)

From 4.4 Use case priority matrix

Workshop run sheet (standard 75-minute version)

Hard rule ── Clear boundary · repeatable · verifiable (missing one? fix it first)
Quota     ── Start now: 1–2 · quick win: 1 · strategic bet: 1 (groundwork only)
Weighting ── Data readiness is 40% of feasibility (82.5% of datasets have quality problems)
Deliver   ── Priority use case list + do-not-do list + business owner's signature
Counter-intuitive ── Strategic decisions, where the boss most wants AI, are what to avoid now

From 5.5 AI capability demo script

The six-step translation method (every demo follows it)

1. Start from a business problem   ← not from the technology
2. Pick the smallest example       ← understandable in five minutes
3. Swap in company data            ← anonymised / synthetic / public; never real sensitive data
4. Record before and after         ← baseline, result, human review rate, running cost
5. Translate into management terms ← beside the architecture write: owner, risk, cost, conditions to extend
6. Keep licences and sources       ← note the repository, version, commit date and licence in the deck

Preparation

From 5.5 AI capability demo script

Four screens on stage (all four required)

① Input        ← what the user asked
② Output       ← what it answered, and which passage of which document it cited
③ Time & tokens ← this is the invoice
④ Human check  ← is it right, and who decides

Management question (ask immediately after the demo)

□ Who is accountable when it answers wrongly?
□ When it doesn't know, does it invent something?
□ When a document is updated, how long until answers reflect it?
□ Who may see which documents? Can the AI answer across permission boundaries?

Demo point: the human is driving

AI advises → a person judges → a person accepts or rejects → the person is accountable

Side-by-side on stage (the version that works best)

From 5.5 AI capability demo script

Management question

• Controlled experiment: about 55% faster on a coding task [R049]
• But another study found senior developers 19% slower [R019] (with a note that it was not replicated)

Demo point: it plans and calls tools by itself

Receives a goal → breaks it into steps → calls tools → judges the result → continues or adjusts → delivers

Management question (the most important one here)

□ Can it perform write actions? (change data, place orders, send messages)
□ Do write actions need approval? Who approves?
□ How do you roll back an error?
□ If step seven of ten goes wrong, how do you find it? Is there a log?
□ Where does its autonomy end? Does it stop and ask when it reaches that edge?

From 5.5 AI capability demo script

When the demo fails ⭐

From 5.5 AI capability demo script

Demo discipline (red lines)

🔴 Never upload real customer data, contracts or personal information to a public AI service
🔴 Never let a key, internal address or real client name appear in a demo
🔴 Never promise "we can achieve the same result"
✅ Always use anonymised, synthetic or public sample data
✅ Ask a management question straight after each demo to pull it back to the business
✅ Note in the deck: repository, version, commit, licence

Three demos ── RAG (it answers, you judge) · Copilot (it suggests, you decide) · agent (it acts for you)
Risk line   ── Wrong answer → you stop judging → it acts wrongly and you don't know (rising, and harder to spot)

Key design:
  RAG      Deliberately ask one it cannot answer → demonstrate hallucination
  Copilot  Time it live, side by side → task level ≠ role level
  Agent    Deliberately show a failure → explain write-action approval

Rule     ── Read-only is open; write actions need approval and a log
Backup   ── Record it beforehand; a bad answer is a gift; if challenged, be straight

From 2.6 Agents and the next wave

This is where the money is going

Who is using it, and how big the gap is

Evidence from China

Embodied / physical AI (essential for manufacturing clients)

Management implication: agents raise three new questions

Trend    ── AI moves from "can think" to "can do" (CAICT's phrasing)
Money    ── Leading organisations put 60% of the AI budget into agents
Value    ── Agents go from 17% of total AI value (2025) to 29% (2028)
Divide   ── Usage: future-built 33% / scaling 12% / lagging ≈0
Counter-intuitive ── A well-packaged general agent can beat the frontier model itself
Physical AI ── 58% → 80% within two years; Asia-Pacific leads

Management ── AI used to give you a wrong answer; now it does the wrong thing for you
Rule       ── Read-only is open; write actions need approval and a log

From 4.9 Technology selection decision table

Decision 1: open-weight or closed

□ Do we have a team that can maintain an open-weight model? (No → go closed)
□ Can data leave the company? (No → open-weight, self-hosted)
□ What would switching vendors cost in three years? (A lot → prefer open-weight)

Decision 2: how to make the model understand your business (pick one or combine)

□ Start with RAG, run it three months, then see what is missing  ← where 90% should begin
□ If the gap is jargon            → instruction tuning
□ If the gap is missing knowledge → extend the knowledge base, not fine-tuning
□ If the gap is an unusual task pattern → only then consider fine-tuning

Decision 3: where to deploy

From 4.9 Technology selection decision table

Decision 4: how large a model

Recommended combinations of the four decisions, by company type

Four forks (the boss decides these):
① Open-weight or closed   → sets team commitment and exit difficulty
② RAG or fine-tuning      → 90% of companies should do RAG first
③ Public / private / hybrid → sets data security and upfront cost
④ Large model or small     → the easiest decision to save money on

⚠️ The official wording: RAG mitigates hallucination only "to a certain extent"
   → human review cost can never be deleted from the budget

From 4.12 The five-layer AI cloud architecture

The five layers (CAICT's original naming)

• AIIaaS: reshapes intelligent infrastructure for efficient training and inference
• AIPaaS: reshapes AI development services and accelerates AI+ applications at scale
• MaaS: end-to-end service capability, the main vehicle for putting models into production
• AISaaS: brings intelligence to enterprise applications and lifts operating efficiency
• AIMSP: professional AI cloud services connecting end-to-end model application delivery

⭐ Break the budget down these five layers (the core tool here)

Three decision questions (for the CIO to settle)

□ AIPaaS layer: prefer platforms on open standards where the model is swappable
□ MaaS layer: connect at least two model vendors and keep the ability to switch
□ AIMSP layer: write data and knowledge-base export terms into the contract

How it echoes the market trend

Five layers ── AIIaaS infrastructure → AIPaaS platform → MaaS models → AISaaS applications → AIMSP delivery
              (CAICT Cloud Computing Blue Book 2025, AI cloud architecture)

⭐ One more row ── Human review (outside the technical architecture, often the largest running cost)

Build rule  ── Build only when you are large enough and it is your differentiator
               For most companies only the data and use cases are yours; buy the rest

Anti-lock-in ── AIPaaS and AIMSP lock you in fastest
               → choose open standards · connect two model vendors · put export terms in the contract

Biggest saving ── The MaaS layer: tier large and small models by use case

From 3.1 Case quick-reference

Full table

Indexed by industry

Indexed by management topic

The four-step formula for presenting a case

Use these four steps for every case. Participants remember the formula — worth more than the case itself.

1. What was it stuck on?      ← the baseline. A case without one is not worth telling
2. What did it change?        ← the process, or only the tool?
3. Where do the numbers come from? ← who measured, on what basis, grade A, B or C
4. What would it take for you? ← bring it back to the audience

⚠️ Three rules for presenting a case

• Talk business results, not headcount replaced. Follow Klarna's "700 FTE equivalent" immediately with resolution time and repeat-contact rate — headcount alone invites "and what happened to service?"
• Explain the range. The textile case's 15%–50% is wide; say it reflects different companies' baselines, not one company fluctuating.
• Treat Grade C cases as hypotheses. Glodon (C+) and Digiwin (C) come from media and vendors — say plainly that these are hypotheses, not commitments.

See 3.4 Evidence grading and the debunking checklist.

From 3.2 International benchmark cards

Figure

Watch out for

• ⚠️ Never say only "replaced 700 people" — always give resolution time, repeat contacts and satisfaction too
• ⚠️ Klarna later adjusted its human-service strategy. If asked, answer: "That is precisely the point — AI service is not a one-off replacement but a continuously tuned human-machine mix."

Figure

Figure

Why this card is worth having

Large sample, measured in real workplaces rather than a lab; fits admin, sales, finance and knowledge work.

From 3.3 China practice cards

The figures (five dimensions move together — that is what makes it valuable)

Why this card is the trump

• Real financial figures — revenue and profit to the decimal — not a vague "XX% better"
• Five dimensions improve at once — a perfect illustration of the cost-of-quality model
• 98% domestically sourced — decisive for SOE, defence and supply-chain-sensitive clients
• Connected to ERP and MES — proof it is not an isolated pilot

Figure

Figure

From 3.3 China practice cards

Additional cases (optional, use as needed)

How to combine the China cases

State Grid (7) → Textiles (6) → Capacitor inspection (5)
 digitise first   a price you can       all five dimensions
                  actually account for   connected

From 3.2 International benchmark cards

Figure

How to teach it: the four-rung metric ladder (this card's core value)

Adoption ── 100,000 people got the tool
   ↓
Frequency ── 70% genuinely use it weekly   ← most companies die here
   ↓
Time     ── About 3 hours saved per person per week
   ↓
Business result ── ???                     ← the case does not disclose this

Watch out for

• ⚠️ Grade B: a customer case page; the figures are self-reported
• ⚠️ The "80% efficiency gain" is the best selected process, not an average, and is not a company-wide figure

How to combine the three cards (recommended order)

Copilot (2) → M365 (3) → BBVA (4) → Klarna (1)
 task-level    real working  organisation  confirmed by
 experiment    conditions    scale         finance

From 3.4 Evidence grading and the debunking checklist

The three-grade evidence standard (the core tool taught here)

Four figure labels (finer than the three grades; for reported data)

Every figure in this deck belongs to one of these:

Debunking checklist: eight questions a CEO can take away

□ 1. What is the baseline?   Without a "before" number, no percentage means anything
□ 2. Who measured it?        Vendor / customer / third party — three grades apart
□ 3. Was there a control?    Without one you cannot tell whether AI did it
□ 4. How big, how long?      Three people for a week is not 3,000 people for a year
□ 5. Only keen users?        Interviewing volunteers filters out the failures automatically
□ 6. Was quality measured?   Halve any number that reports speed but not quality
□ 7. Are all costs in?       Model, integration, data governance, human review, operations — miss one and the books are false
□ 8. Who signed it off?      A benefit without a CFO signature is still just a slide

From 3.4 Evidence grading and the debunking checklist

The five most common number traps (each gets a laugh)

The figures in this course face the same test

Grade A = regulator / financial report / experiment → board resolutions
Grade B = company self-report                      → project approval papers
Grade C = media / vendor retelling                 → brainstorming only

Eight questions: baseline? who measured? control? sample? cherry-picked? quality? all costs? who signed?

Five traps: extrapolating from one task · salary conversion · reporting only the best · survivorship bias · average accuracy only

From 4.6 ROI calculation template

Five exit conditions (the most valuable page in this course)

For saved time to become a financial benefit, at least one must hold:

□ 1. Less overtime, outsourcing or new hiring
      → direct cost reduction; the hardest evidence

□ 2. More orders, cases or customers handled by the same people
      → capacity released; you need incremental demand to absorb it

□ 3. Shorter time to market, quoting, collection or development, producing incremental revenue
      → revenue-side benefit; usually the largest and the hardest to prove

□ 4. Capacity redeployed onto clearly measurable high-value work
      → you must say where it went and how it is measured, or it does not count

□ 5. Fewer errors, rework, incidents, compliance failures or lost customers
      → cost of quality; the most consistently underestimated

The ROI formula

        annualised realised benefit − annualised total cost
ROI =  ───────────────────────────────────────────────────
                     annualised total cost

From 4.6 ROI calculation template

Suggested benefit discount (how to apply the risk haircut)

From 4.5 The five-layer KPI tree

The five-layer metric tree

┌──────────────────────────┐
        │  ① Financial results      │  ← the board reads only this layer
        ├──────────────────────────┤
        │  ② Operational results    │
        ├──────────────────────────┤
        │  ③ Customer results       │
        ├──────────────────────────┤
        │  ④ People results         │
        ├──────────────────────────┤
        │  ⑤ Risk results           │  ← the only layer read when something goes wrong
        └──────────────────────────┘
   (Call volume and active users sit outside the tree — that is health monitoring, not value)

From 4.5 The five-layer KPI tree

The pairing rule for each layer (the heart of this section)

The BBVA four-rung ladder (why you cannot skip a rung)

Adoption  ── 100,000 staff got the tool
   ↓  (most companies start boasting here)
Frequency ── 70% weekly active
   ↓  (most companies die here)
Time      ── About 3 hours saved per person per week
   ↓  ← ⚠️ the hardest jump
Business result ── ?

From 4.5 The five-layer KPI tree

Metric tree template (participant output)

• If the baseline column is blank, it does not enter ROI review.
• At least one metric per layer, plus a paired counter-metric.
• Only the CFO or a finance business partner may fill the last column.

Outside the tree ── call volume and active users are health monitoring, not value
① Financial  EBIT / cost avoided / profit / cash flow / payback
② Operational cycle time / throughput / first-pass yield / rework / unit cost
③ Customer   response / first-contact resolution / conversion / retention / NPS
④ People     weekly active / proficient share / task coverage / skill transfer
⑤ Risk       error rate / hallucination rate / data exposure / review rate / audit findings

Hard rule ── Every positive metric needs a paired counter-metric, or it may not be reported
Hard rule ── No baseline, no ROI review

• Governance maturity radar: score all five layers and find the weak one (pairs with 4.7)
• Four-rung funnel: adoption → frequency → time → business result, with the drop-off at each rung

From 4.6 ROI calculation template

Full template (participant output 3)

From 4.6 ROI calculation template

ROI waterfall (recommended chart)

Gross benefit ██████████████████  100
  Models and compute   ▼ −12
  Integration          ▼ −18
  Data governance      ▼ −15
  Human review         ▼ −22   ← the largest single piece
  Change and training  ▼  −8
  Ongoing operations   ▼ −10
Net benefit   ███                15

Five exits ── less hiring/outsourcing · throughput · cycle→revenue · capacity redeployed · losses avoided
             (none of them ticked = the benefit path was never designed)

Eight cost items ── compute · licences · data governance · integration · compliance · training · human review · operations
             (human review is missed most, and can eat 60% of the gross benefit)

Three numbers ── theoretical potential / risk-discounted / finance-confirmed → executives read only the last two

Hard rule ── No baseline, no review; no CFO signature, no benefit

From 2.8 Evidence on hallucination and governance

Measured hallucination rates (26 models)

⭐ The most dangerous finding: say something wrong and the model follows

Four more measured findings (each with a management action)

From 2.8 Evidence on hallucination and governance

The standards companies cite most (for procurement and compliance)

Hallucination ── 26 frontier models, rates from 22% to 94% (not 2% to 9%)
              ── A low rate means it knows better when to admit it doesn't know
              ── First selection rule: pick the one that knows what it doesn't know, not the most fluent

⭐ Most dangerous ── A false premise framed as "others believe" → the model corrects you
                    Framed as "I believe" → performance collapses
                    → an employee with a wrong premise gets agreement, delivered more professionally
                    → human review must check the input premise, not only the output

Four findings ── Dialects cost nearly half the accuracy · every model's defences fall once jailbroken
              ── Transparency went backwards (58→40) · safety, fairness and accuracy fight each other

Incidents ── Under 100 before 2022 → 233 in 2024 → 362 in 2025
Governance ── Firms with no policy fall from 24% to 11%; the biggest barrier is a knowledge gap, at 59%
Standards ── GDPR 60% · ISO/IEC 42001 36% · NIST AI RMF 33%

• Hallucination-rate bars: 26 models ranked, 22%–94%, both ends highlighted
• Premise-contamination comparison: "others believe" vs "I believe", the latter collapsing in red
• Incident growth curve: <100 → 233 → 362
• Governance catching up: firms with no policy fall from 24% to 11%

From 4.10 Risk checklist: NIST and CAICT

NIST's twelve generative AI risks

NIST's three groupings (easier to present upward)

From 4.10 Risk checklist: NIST and CAICT

The four that matter most (only these four are taught)

The CAICT "two horizontals, three verticals" framework

            Development   Deployment    Use
              ┌──────────┬──────────┬──────────┐
 Management   │  policy · clear accountability · audit trail  │
              ├──────────┼──────────┼──────────┤
 Technology   │  capability · guardrails · monitoring         │
              └──────────┴──────────┴──────────┘
                     ↓ multi-stakeholder governance (ecosystem coordination)

From 4.10 Risk checklist: NIST and CAICT

Five practice modules

Four things internal governance must do (CAICT wording; copy straight into policy)

NIST's twelve ── confabulation · privacy · bias · human-AI configuration · information integrity · information security · IP · value chain…
NIST's four   ── Govern → Map → Measure → Manage

Watch four    ── 🔴 hallucination · 🔴 data leakage · 🟡 over-reliance · 🟡 opaque vendors
Most insidious ── Human-AI configuration: risk peaks in month six, not on launch day

CAICT framework ── "Two horizontals, three verticals"
  Horizontals: management + technology    Verticals: development + deployment + use
  Five modules: risk management · model development · system deployment · application operation · industry ecosystem

Cheapest first step ── ① publish usage guidance  ② name an owning department

From 4.7 Governance and risk gates

Human-AI division of responsibility (one swimlane explains it)

┌──────────────┬──────────────────┬──────────────────┐
│  AI executes │  Human reviews    │  System automates │
├──────────────┼──────────────────┼──────────────────┤
│ Drafting     │ High-risk clauses │ Archiving         │
│ Extraction   │ Amounts and terms │ Validation rules  │
│ Classifying  │ Escalations       │ Access control    │
│ First pass   │ Final signature   │ Audit logging     │
└──────────────┴──────────────────┴──────────────────┘

• AI drafts, a person signs. External commitments, financial amounts and legal terms remain a human's responsibility.
• The review rate is a manageable metric, not a fixed cost — bringing it down from 100% is itself proof of maturity.
• Exceptions need an escalation path, with a real person on it.

From 4.10 Risk checklist: NIST and CAICT

Procurement questionnaire (take away; use it on vendors)

□ Which model is underneath? Open-weight or closed? Which version?
□ Where did the training data come from? Any copyright issues?   ← NIST #10, #12
□ Will our data be used to train your models?                    ← NIST #4
□ Where is it deployed? Does data leave our network?
□ Are there content safety guardrails? Who sets the blocking rules? ← NIST #2, #3
□ What protects against prompt injection and data poisoning?     ← NIST #9
□ How is liability for wrong output divided? What does the contract say?
□ If we switch vendors, can we export the data and knowledge base? ← exit plan

• The two-horizontals, three-verticals framework (Mermaid grid)
• Heat map of the twelve risks (rows = risk, columns = likelihood / impact)
• Governance maturity radar (see 4.7)

From 4.7 Governance and risk gates

Governance maturity radar (self-assessment tool)

Six dimensions, scored 1–5 each:

     Policy & accountability
              │
   Agent ─────┼───── Data
   control    │      governance
              │
   Staff ─────┴───── Model
   training          monitoring
     Security & compliance

Five gates ── approval → prototype → pilot → finance sign-off (CFO) → scale-up
              ①           ②            ③        ④ ⭐                  ⑤

Seven risks ── hallucination · leakage · privilege overreach · shadow AI · supply-chain lock-in · compliance · unclear accountability

Division ── AI drafts, a person signs; the review rate is a metric you can lower; escalations need a real person

Framework ── NIST GenAI Profile [R027] + the CAICT safety governance blue book [R016]
             Govern → Map → Measure → Manage

Line ── A racing car has brakes so it can go fast, not slowly

From 4.8 The CAICT rollout methodology

Four phases · eight steps · five layers

Diagnose ─────→ Build ─────→ Apply ─────→ Manage
 │               │             │            │
Capability      Solution      Application   Monitoring
analysis        design        build
Needs           Development   Performance   Operations
discovery       and testing   assessment    management

Five layers (check all five at every step):
infrastructure · data resources · algorithms and models · application services · security and trust

1. Completeness of core resources

2. Balance of the talent mix

3. Fit with strategic planning ⭐

Apply phase: the five-layer maturity assessment

CAICT's assessment dimensions, usable directly as a self-assessment sheet:

From 4.8 The CAICT rollout methodology

Manage phase: three things

Four phases ── diagnose → build → apply → manage
Eight steps ── capability analysis · needs discovery | solution design · development and testing | application build · performance assessment | monitoring · operations management
Five layers ── infrastructure · data resources · algorithms and models · application services · security and trust

Three diagnostics ── completeness of core resources · balance of the talent mix · fit with strategic planning ⭐
Maturity layers ── infrastructure / data resources / algorithms and models / application services / security and trust
Key metrics ── use case penetration · optimisation rate · return on investment (the official set)

Use ── The common language for reporting upward; nobody questions the framework itself

• Four-phase, eight-step flow (Mermaid; see 6.2)
• Three-readiness radar (self-scored 1–5)
• Five-layer maturity heat map (rows = layers, columns = maturity levels)

From 5.4 Workshop output template

Section A: scoring candidate use cases

□ Clear task boundary   □ Repeatable   □ Quality monitorable
(Missing one → fix it first; missing two or more → onto the do-not-do list)

Section B: priority use cases (3 maximum)

Baseline (leave this blank and it does not enter ROI review)

Benefit exit (tick at least one)

□ 1. Less overtime, outsourcing or new hiring
□ 2. More throughput from the same people   → where does the extra demand come from? ______
□ 3. Shorter cycles producing revenue       → how is that revenue measured? ______
□ 4. Capacity redeployed to high-value work → redeployed where? measured how? ______
□ 5. Fewer errors / rework / incidents / churn → what is the current loss? ______

Five KPI layers (at least one positive and one paired counter-metric each)

The three ROI numbers

Table 3 | 90-day action plan (the five gates)

From 9.1 CXO glossary

Capability

Practice terms (tied to the four decisions)

Cost and deployment

Governance

Evaluation

The five words you are most likely to be sold ⭐

Billing unit ── the token: double the usage, double the money
Starting point ── RAG: the right place for 90% of companies to begin
Trap ── Hallucination never goes away → human review cost cannot be deleted
Saving ── Small model vs large: costs can differ by an order of magnitude
Security ── Write actions need approval; read-only can be opened up
Hard metrics ── baseline · control group · first-pass yield · unit economics
Sales-proofing ── "accuracy" / "zero hallucination" / "out of the box" / "industry model" / "80% more efficient"