Steven Yang · Outcome Delivery
Steven Yang.
01 / 11  

01 · 我是誰Who I am

顧問出身。
對結果負責。Consultant by training.
Accountable for the result.

七年只做一件事：把承諾的數字，變成客戶系統裡真的出現的數字。Seven years, one job: turn the number we promised into the number that shows up in the client's own system.

0年，顧問與企業 AIyears, consulting & enterprise AI
0Zero Trust 已簽約營收Zero Trust signed revenue
0Deloitte 營收來自續約of Deloitte revenue from renewals
0同時進行的案子（高峰）concurrent engagements at peak
0iKala 企業 AI pipeline，12 個月enterprise AI pipeline in 12 months
0家醫院上線的 SaMD（共同創辦）hospitals live on a SaMD I co-founded
INSEAD MBA '26Kellogg exchangeISO/IEC 42001 LAISO/IEC 23894ISO/IEC 27001 LACISA (exam passed)ISO/IEC 27701ISO/IEC 27799NVIDIA AI Infra & OpsJHU AI for CybersecurityMcKinsey ForwardEU AI Act

從方法開始 →Start with the method →

steven.yang26j@insead.edu · linkedin.com/in/sy8888 · Singapore / Taipei

Steven Yang
Steven J. Yang · Singapore
AIOM Outcome Cockpit可操作的子專案：Monday brief、結果 vs baseline、震動偏差、KPI 象限、生命週期。示範資料。A working sub-project: Monday brief, outcome vs baseline, vibration deviation, KPI quadrant, lifecycle. Sample data.
開啟 ↗Open ↗

02 · KPI 方法論The method

從結果往回推的五個問題Five questions,
asked backwards from the result

五個問題，順序不能換。AI 在第四題才出現。Five questions, always in this order. AI only shows up at question four.

每一步都有存在的理由，但「必要」不等於「只能這樣做」。先問它產出什麼，再問最直接的路。Every step exists for a reason. But "necessary" doesn't mean "only this way." Ask what it produces, then ask for the most direct route.

1 · 它產出什麼？1 · What does it produce?日期、數量、門檻 → 規則。分類、預測 → 資料與標籤。判斷、生成 → 才用模型。A date, a count, a threshold → a rule. Classify or predict → data and labels. Judge or generate → then a model.
2 · 最直接的路？2 · The most direct route?先算不用新技術的做法要多少錢。沒有這個基準，什麼 AI 方案都看起來划算。First cost out the no-new-tech option. Without that baseline, every AI option looks worth it.
3 · 多付的錢買到什麼？3 · What does the extra cost buy?往上一階，要講得出多動多少數字、多扛多少風險。講不出來就別上去。Going up a rung has to buy a number I can state, and a risk I can state. If I can't, I stay down.

不是標準，是提問的順序。越下面越便宜，先從下面問。It's not a standard, it's an order of questions. Cheaper at the bottom, so start there.

從數字開始。找到產生它的流程。找到最值得改的那一步。選最便宜能改它的東西。然後決定星期一誰要做不一樣的事。Start with the number. Find the process behind it. Find the one step worth changing. Pick the cheapest thing that changes it. Then decide who does something different on Monday.

03 · 每週一早上的一頁The page I read on Monday morning

Outcome Dashboard

一頁只看一個數字，其他都在解釋它為什麼會動。示範資料：捷運手扶梯預測性維修。One number per page. Everything else explains why it moves. Illustrative data: metro escalator maintenance.

這一章是靜態示意；完整互動版在This chapter is a static sketch; the working version is the AIOM Outcome Cockpit ↗

KPI vs 承諾KPI vs commitment

採用（leading）Adoption (leading)

舊表單還有 35% 在用：舊流程不退場，數字會回彈。35% still on the old sheet. If it doesn't retire, the number bounces back.

Baseline: Jan–Jun avg, SOW §4.2Source: CMMS + sensor gateway, client system of recordReviewed weekly with client示範資料Illustrative data
#承諾（SOW）Promised (SOW)做出來的（我驗證）Built (verified by me)狀態Status行動Action
M112 個月感測與維修歷史資料進入 pipeline12 months of sensor + maintenance history in the pipeline9 個月；2024 Q1 檔案格式不同未解析9 months; 2024 Q1 files in a different format, unparsedDrift我自己寫了解析腳本補齊，下週重跑 baselineWrote the parser myself; baseline re-run next week
M2模型 v1 precision ≥ 0.70 on holdoutModel v1 precision ≥ 0.70 on holdout0.74，已核對 holdout 定義0.74, holdout definition checkedOn promise—
M3警示自動建立 CMMS 工單Alerts auto-create CMMS work orders目前為每日 CSV 匯出，人工建單Daily CSV export, manual ticketingDriftIT 的 API 權限卡 6 週；先用 n8n 寫入，API 開通後切換Client IT API access stuck 6 weeks; n8n writes for now, switch when API opens
M420 台試行，技師訓練完成20 units piloted, technicians trained20 台上線，18 位技師完訓20 live, 18 technicians trainedOn promise—
M5舊 Excel 巡檢表退場Old Excel inspection sheet retired進行中，35% 仍在用In progress, 35% still using itIn progress與站務主管約談，設定退場日期Meeting station managers to set a retirement date

Built 那欄是我自己跑過、看過、打開過之後寫的，不是狀態報告抄來的。I write the Built column after I've run it, read it and opened it myself. Not from a status report.

升級紀錄（一定附解法）Escalation log (always with a fix)

W問題Issue建議解法Recommended fix狀態Status
19CMMS API 權限延誤，M3 將影響 Q2 數字CMMS API delay will hit the Q2 numbern8n 暫代寫入，兩週可上n8n as interim writer, live in two weeksOpen
21兩站技師不信任警示，手動覆核Two stations don't trust alerts, re-check manually先給他們看前 10 次警示的命中紀錄Show them the hit record of the first 10 alertsOpen
15Baseline 期間含一次電力中斷Baseline window included a power outage依 SOW exclusions 剔除，重算Excluded per SOW, baseline recalculatedClosed

證據鏈與下一個機會Evidence trail & next opportunity

機會Opportunity為什麼是現在Why now估值Est.
電梯模組，同站點Elevator module, same stations同一套感測與工單流程可複用Same sensor + work-order flow reusableSGD 180K
第二線，20 台Second line, 20 units試行 vs 未試行對照已成立Pilot vs control comparison establishedSGD 220K
  • 每月 KPI 由客戶 Facilities 出具，不是我們算的Monthly KPI issued by the client's Facilities team, not computed by us
  • 避免的停機損失用客戶財務的每小時成本換算Avoided downtime valued at the client's own hourly cost

哪個 KPI 先做？三個分數：影響力、業務衝擊（越低越好）、導入成本（越低越好）。改分數、拉權重，即時排名。Which KPI first? Three scores: impact, disruption (lower is better), cost (lower is better). Change a score, drag a weight, the ranking updates.

Net Gain = Impact × 20  ·  Investment Cost = (wD·Disruption + wC·Cost) ÷ (wD+wC) × 20  ·  Priority = wI·Net Gain − (1−wI)·Investment Cost  ·  ROI% = Priority ÷ Investment Cost × 100
#候選 KPICandidate KPI影響Impact衝擊Disrupt成本CostNet GainInv. CostPriorityROI%

左上角是 Net Gain 高、Investment Cost 低：先做。氣泡越大 ROI 越高，藍色是前兩名。Top-left is high Net Gain, low Investment Cost: do first. Bigger bubble, higher ROI; blue marks the top two.

04 · 每個承諾數字一張卡One card per committed number

KPI CardKPI Card

交付物是我給你的東西。成果是你的數字動了。SOW 兩個都要有。A deliverable is what I hand you. An outcome is your number moving. The SOW needs both.

一條線，三個點：起點、承諾、現在。One line, three marks: where we started, what we promised, where we are.

Baseline 120 h上半年平均，SOW 簽認。沒有 baseline，就沒有「動了」。First-half average, signed in the SOW. No baseline, no "it moved."
Target 84 h · Q4−30%。要連續一季都達標才算。−30%. It has to hold for a full quarter.
Guardrails誤報 ≤ 15%，計畫性維修最多 +10%。不准用多排維修來「消滅」停機。False alerts under 15%, planned maintenance up no more than 10%. You can't fix downtime by scheduling more downtime.

leading 指標畫不到 KPI、KPI 畫不到錢的，就是虛榮指標。If I can't draw a line from the indicator to the KPI, and from the KPI to money, it's a vanity metric.

客戶不肯承諾數字？Client won't commit a number? 我先承諾量測方法。四週建 baseline，然後一起定數字。Then I commit to the measurement. Four weeks for a baseline, then we set the number together.
每月非計畫停機時數Unplanned downtime · h / monthMRT escalators · 範例example
Business question
停機造成多少服務中斷與緊急出勤成本How much service disruption and emergency call-out cost does downtime create
Metric & formula
Σ 非計畫停機分鐘 ÷ 60，每月，排除計畫性維修Σ unplanned downtime minutes ÷ 60, monthly, excluding planned maintenance
System of record
CMMS 工單時間戳；客戶 Facilities 部門擁有CMMS work-order timestamps; owned by the client's Facilities team
Grain & window
每台 / 每月；月報，季度判定Per unit / per month; monthly report, quarterly judgement
Baseline
Jan–Jun 平均 120h，SOW §4.2 簽認Jan–Jun average 120h, signed in SOW §4.2
Target & deadline
84h · Q4 月底Q4 month-end
Exclusions
電力中斷、人為破壞、供應商召回Power outage, vandalism, vendor recall
Guardrails
誤報率 ≤ 15%；計畫性維修時數不增加超過 10%False-alert rate ≤ 15%; planned maintenance hours up no more than 10%
Leading indicators
警示 24h 內處理率、工單來自模型的比例Alerts handled within 24h; share of work orders from the model
Owner (client)
Head of Station Facilities
Attribution rule
試行 20 台 vs 未試行 20 台對照20 piloted units vs 20 untouched units

05 · 當 KPI 是一段爬坡，不是一個點When the KPI is a climb, not a point

漸進式 KPI：爬梯子The progressive KPI:
climbing the ladder

有些成果是爬梯子。每爬一階才看得到下一階，所以 KPI 要跟著梯子調。Some outcomes are a climb. Each rung shows you the next one, so the KPI moves with the ladder.

示範：Zero Trust 的 Identity 支柱，CISA ZTMM 四級。Example: the Identity pillar of Zero Trust, on CISA's four levels.

梯子怎麼讓 KPI 動起來How the ladder keeps the KPI honest

  • 每一階一張 KPI card。量的是這一階的條件完成了多少。One KPI card per rung. It measures how much of that rung is done.
  • 只承諾下一階。爬到了再估下一階。合約用 change order 往上加。Commit to the next rung only. Re-estimate when you get there. Extend by change order.
  • 護欄防偷爬。不准改定義、不准把風險評低。Guardrails stop cheating. No redefining levels, no rating risk down.
  • 用在：監管成熟度、Zero Trust、AI 治理、AI 採用。Works for: regulatory maturity, Zero Trust, AI governance, AI adoption.

美國分行：先量固有風險，再定每個領域要爬到哪一級。CAT 2025 年退場，換成 NIST CSF 一樣用。US branches: measure inherent risk first, then set the level each domain has to reach. CAT retired in 2025; the same logic works on NIST CSF.

固有風險檔案：這家分行為什麼要爬到這裡Inherent risk profile: why this branch has to climb this high
技術與連線類型Technologies & connection types
服務通路Delivery channels
線上 / 行動產品與技術服務Online/mobile products & tech services
組織特性Organizational characteristics
外部威脅External threats

此分行整體 Moderate。風險越高，要爬得越高。This branch: Moderate. Higher risk, higher climb.

領域 3 的一年四個 gateDomain 3, four gates in a year
Q1Q2Q3Q4
Evolving 的陳述 100% 為「是」，證據可查100% of Evolving statements "Yes", evidence on file Intermediate 完成 40%，每個缺口有 owner 與日期40% of Intermediate; every gap has an owner and a date 完成 80%，內稽抽測通過80%; internal-audit sampling passed 100%，獨立驗證，向董事會與監管報告100%; independently verified; reported to board and regulator

我做過的另一座山：Deloitte 時期起草的四構面 ESG 成熟度模型，拿了 Deloitte 創新首獎，後來成為台灣證交所 ESG 評鑑的基礎。Another mountain I've drawn: the four-dimension ESG maturity model I drafted at Deloitte won the firm's innovation first prize and became the basis of the Taiwan Stock Exchange's ESG evaluation.

同一套梯子套在 AI 上。每一階有自己的 KPI 和 gate。The same ladder, applied to AI. Each rung has its own KPI and gate.

兩張參考圖。FSC Zero Trust 指引參考的就是 CISA 與美國國防部的分級。Two reference figures. The FSC Zero Trust guidelines drew on CISA's and the US DoD's levels.

Implementing the CISA Zero Trust Maturity Model: seven steps from assess and benchmark to partner with expert
Implementing the CISA Zero Trust Maturity Model · reference figure (Napkin)
Zero Trust maturity journey from Traditional to Initial, Advanced and Optimal drawn as a mountain climb
Zero Trust Maturity Journey · reference figure

06 · 我過去怎麼做How I have done it

Proposal → SOW → 交付Proposal → SOW → delivery

Deloitte：先談 proposal，簽約後寫 SOW，我算人天。iKala：PoC 先行，簽約金加訂閱。共同點：proposal 的每一塊都要變成可驗收的。Deloitte: negotiate the proposal, sign, then write the SOW. I priced in person-days. iKala: PoC first, signing fee plus subscription. Either way, every block of the proposal has to become something you can verify.

兩個外部基準：McKinsey 轉型辦公室的 L0–L5 閘門，和 SaaS 客戶成功的 value realization。我的八步對得上，差別在我把「簽字」寫成一步。Two outside benchmarks: McKinsey's L0–L5 transformation gates, and SaaS value-realization practice. My eight steps line up with both. The difference is that I make the signature its own step.

−70%初估價值到 L5 實現時平均掉七成，所以 pipeline 要有目標的 3 倍Initial value estimates fall ~70% by L5, so the pipeline needs 3× the targetMcKinsey, 18 APAC transformations
~30%只有三成轉型達到原定目標Only about 30% of transformations hit their intended targetsMcKinsey
4每個 initiative 約四個里程碑最好：早期預警，又不微管理About four milestones per initiative: early warning without micromanagingMcKinsey
2–4每個帳戶 2–4 個 SMART KPI，各帶 baseline、資料來源、target、deadline2–4 SMART KPIs per account, each with baseline, data source, target, deadlineValue realization practice

我已經在做的Already in my method

  • KPI 先寫 baseline、資料來源、target、deadline、owner（KPI card）KPI with baseline, data source, target, deadline, owner (the KPI card)
  • 量測在 onboarding 就接好，不事後找資料（instrumentation 是交付物）Instrument during onboarding, don't hunt for data afterwards (instrumentation is a deliverable)
  • 每週 tactical、每月 KPI、每季 business review（dashboard 的節奏）Weekly tactical, monthly KPI pulse, quarterly review (the dashboard cadence)
  • 採用率當 leading 指標，看 activation、頻率、深度Adoption as a leading indicator: activation, frequency, depth
  • 報給預算擁有者，不只報給使用者Report to the budget owner, not only the users

我從基準學到、要加進去的What I'm taking from the benchmarks

  • 初估要打折：把每個候選 KPI 的預估價值乘以 0.3 再排優先Discount early estimates: multiply each candidate's estimated value by 0.3 before ranking
  • 每週每個 initiative 要有一個具體動作，不等里程碑One concrete action per initiative every week, not just at milestones
  • 砍指標：對結果貢獻不到 0.01% 的指標合併掉Prune metrics: anything under 0.01% of impact gets folded in
  • 每個 owner 最多三個 initiative，避免 20% 的人扛 80% 的事Cap owners at three initiatives, so 20% of people don't carry 80% of the load
  • QBR 前先把資料對清楚，不在主管面前吵數字Reconcile the data before the QBR, never in front of the executive

來源：Sources: McKinsey, Keeping transformations on target · Umbrex, Value Realization & Success Management

先設目標，再看現況。最後要有人簽字。上面是 Evlos 五階段，藍框是我多加的三步。Target first, then the current state. And someone signs at the end. Evlos phases on top; the blue ones are the three steps I add.

最常斷在 5 和 6 之間：慶祝完上線，舊表單還開在第二個螢幕。所以退場日期寫進 SOW，放上 dashboard。It usually breaks between 5 and 6. We celebrate go-live and the old sheet is still open on the second screen. So the retirement date goes in the SOW and on the dashboard.

Proposal 裡的東西In the proposalSOW 裡要變成什麼Becomes, in the SOW我過去的做法My past practice
價值主張、預期效益Value propositionKPI card：數字、baseline、target、deadline、owner、量測方法KPI card: number, baseline, target, deadline, owner, method板階 roadmap 的每個 initiative 都有 named owner 與 KPIEvery initiative on a board roadmap had a named owner and KPI
Deliverables 清單Deliverables list每個配 acceptance criteria：怎樣算交付、誰簽收、幾天內回覆Each with acceptance criteria: what counts, who signs, response days政府案：期初 / 期中 / 期末報告各有審查會與修正期Government: inception / interim / final reports, each with review and revision window
時程TimelineMilestone 配 gate，過不了不往下走Milestones with gatesZero Trust：assessment 過了才進 implementationZero Trust: implementation only after the assessment gate
範圍ScopeIn / out of scope / client obligationsIn / out of scope / client obligations資料、權限、人、決策時限都寫成客戶義務Data, access, people and decision deadlines as client obligations
報價Pricing付款綁 acceptance；outcome-based 的部分要有爭議處理條款Payment tied to acceptance; dispute clause for any outcome-based part人天計價（Deloitte）；bundle 成本與毛利模型（iKala）Person-day pricing (Deloitte); bundle cost and margin model (iKala)
假設AssumptionsAssumptions & dependenciesAssumptions & dependencies資料可取得、API 可開通、使用者可參與訓練Data accessible, API available, users available for training
—Change control政府案走公文變更；企業案走 change orderOfficial correspondence for government; change orders for enterprise

07 · 交付是在哪裡學會的Where the delivery discipline came from

從政府到銀行到企業 AIGovernment, banks,
enterprise AI

政府教我文件，銀行教我稽核與跨境，企業 AI 教我 on-site 與續約。Government taught me paperwork. Banks taught me audit and cross-border work. Enterprise AI taught me on-site builds and renewals.

創業：先做 100 次訪談，再寫第一行程式The startup: 100 interviews before the first line of code

門診跑太晚，醫生重複問同樣的病史。我們訪了 100 多位醫師、跑了 40 多家診所，才開始做。Clinics ran late and doctors kept asking the same history questions. We interviewed over 100 doctors and visited 40-plus clinics before we built anything.

做了 AIQ：病人在平板上先答專科問卷，醫生看一頁摘要。帶 3 人團隊，4 家醫院 2 家診所上線。胸腔科試點把一節門診從 208 分鐘降到 183 分鐘，病人等候時間少了三分之二。U-Start 第一階段 NT$50 萬。We built AIQ: patients answer a specialty questionnaire on a tablet, the doctor sees a one-page summary. A team of three, live in four hospitals and two clinics. The chest-medicine pilot cut a session from 208 to 183 minutes and patient waiting by two thirds. U-Start phase one, NT$500K.

這是我第一次學到：先找瓶頸，再放工具。後來所有案子都是這樣開始的。First time I learned it: find the bottleneck, then place the tool. Every engagement since has started the same way.

第一次帶團隊：衛生福利部的標案（2021）First time leading a team: a Ministry of Health and Welfare tender (2021)

第一次帶 5 個人。政府案教會我：交付物不等於成果，每一條都有人查。My first team of five. Government work taught me that a deliverable isn't an outcome, and every line gets checked.

留下的習慣：承諾寫得越具體，驗收越不會吵。The habit I kept: the more specific the promise, the quieter the acceptance.

私人企業與跨境：併購後的全行合規審查Private sector and cross-border: a post-acquisition bank-wide compliance review

中信買下泰國第五大銀行之後，第一次全行合規審查。兩套制度、兩個監管、兩種語言。CTBC had just bought Thailand's fifth-largest bank. First bank-wide compliance review. Two operating models, two regulators, two languages.

我把 KYC 和交易監控從開戶到申報整條對映，看分行實際做法、SOP 和系統設定有沒有對上。I mapped KYC and transaction monitoring end to end, and checked whether what the branch did, what the SOP said and what the system was set to actually agreed.

海外交付的細節，當面聊。The overseas details are better in person.

海外專案：美國分行的 AML/KYC 與 IT 控制審查Overseas delivery: AML/KYC and IT control reviews for US branches

兆豐美國四個分行：矽谷、舊金山、洛杉磯、芝加哥。十次以上的 AML/KYC 與 IT 控制審查。Mega Bank's four US branches: Silicon Valley, San Francisco, LA, Chicago. Ten-plus AML/KYC and IT control reviews.

同一套評分套在總行和每個分行，結果才能比。第 05 章的成熟度爬坡就是從這裡來的。One scoring rubric across head office and every branch, so results actually compare. The maturity climb in chapter 05 comes from here.

官股民營：資策會（2022–2023，18 個月）Government-affiliated, privately run: III (2022–2023, 18 months)

民營的組織，官方的背景。用民間的速度做事，用公部門的紀律留證據。Private organisation, public mandate. You work at private-sector speed but keep public-sector paperwork.

18 個月。像新加坡的國家級機構，兩邊都要顧。Eighteen months. A bit like Singapore's national bodies: you serve both sides.

從一個案子到 US$4M：台灣金融業第一個 Zero Trust 計畫From one project to US$4M: Taiwan's first Zero Trust programme in financial services

一個諮詢案，變成五大銀行的多年合約。US$4M 已簽約，路線圖到 2028。One advisory project became multi-year accounts with the five largest banks. US$4M signed, roadmaps through 2028.

先跟客戶鎖定可驗證的成果，再跟主管機關一起寫指引，讓它變成全業的 baseline。評估、建置、維運包在一起。I locked a verifiable outcome with the client, then co-wrote the guidelines with the regulator so they became the whole sector's baseline. Assessment, build and maintenance in one bundle.

企業 AI：on-site 建置與續約（iKala, Dec 2024 – Jan 2026）Enterprise AI: on-site builds and renewal (iKala, Dec 2024 – Jan 2026)

12 個月開出 US$2M+ pipeline，親手成交 6 家。跟 FDE 一起進客戶現場。A US$2M+ pipeline in twelve months, six accounts closed myself. On site with the engineers.

下一章是怎麼做的。The next chapter shows how.

08 · On-site 與 FDE 一起做On site with the forward-deployed engineer

Nexus one-boxNexus one-box

一台機器：GPU、私有 LLM、RAG、agent，給不能上公有雲的客戶。Leadtek 出硬體，iKala 出模型和交付；夥伴圈有 NVIDIA、Google、Qualcomm、Wistron。能上雲的客戶走混合架構。六個月出貨。One box: GPUs, a private LLM, RAG and agents, for clients who can't use public cloud. Leadtek brought the hardware, iKala owned the model and delivery; NVIDIA, Google, Qualcomm and Wistron in the partner circle. Clients who could use cloud got a hybrid design. Shipped in six months.

One-box單機、少量 GPU。單一部門的知識問答與 agent。首單：醫療健檢中心，約 US$100K。Single unit, few GPUs. One department's Q&A and agents. First deal: a health-screening provider, ~US$100K.
Rack-mounted multi-GPU依模型大小與併發使用者加 GPU。第二單：資安服務商，AI 伺服器 + WAF + 軟體 bundle，約 US$600K。GPUs added for model size and concurrent users. Second deal: a security services firm, AI server + WAF + software bundle, ~US$600K.
Cabinet多台伺服器，配合客戶機房空間與電力。規則：模型大小 × 預期併發 × 機房條件，再回推價格。Several servers, matched to the client's data-centre space and power. Rule: model size × expected concurrency × room conditions, then back into price.
工程師管 pipeline 和檢索。我管放進去的東西，和答案對不對。The engineer owns the pipeline and retrieval. I own what goes in, and whether the answers are right.

AI 答錯了？先問落在哪個桶子。每個桶子的 owner 和修法都不一樣。AI got it wrong? First question: which bucket. Each one has a different owner and a different fix.

content~70%
來源沒寫或寫錯 → 我補內容、改 metadataSource missing or wrong → I fix content and metadata
retrieval
有寫但沒抓到 → FDE 調 chunking、索引、參數Written but not found → FDE tunes chunking, index, parameters
prompt
抓到但答錯方向 → 一起改 promptFound but misdirected → we fix the prompt together
model
前三者都對仍錯 → 換模型或加規則。最少見。All three right and still wrong → swap model or add rules. Rarest.

比例為經驗值，示意用。Split is an experience-based illustration.

09 · 一個知識型生成系統怎麼被規格化How a knowledge-based generation system gets specified

AI 使用手冊生成系統AI user-manual
generation system

iKala 2024 年的提案：50 份手冊建知識庫，一鍵生成新手冊初稿。我帶走四件事：驗收指標先寫、gate 有停損、資料條件進範圍、方案有階梯。An iKala proposal from 2024: build a knowledge base from 50 manuals, generate a first draft in one click. I took four things from it: metrics before build, gates with a stop, data readiness in scope, and an options ladder.

三種導入策略，選了 ShaperThree adoption strategies; Shaper was chosen

TTaker直接用市面 GenAI 方案接進既有工作流，不客製。Consume off-the-shelf GenAI via API inside existing workflows; no customisation.
SShaper針對場景部分客製或微調既有模型，整合工作流。本案：自架開源模型（Self-host），需要時才 fine-tune，以 Docker 容器交付，部署在 Azure。Partly customise or fine-tune an existing model for the scenario and integrate it. Here: self-hosted open-source model, fine-tuned only if needed, delivered as a Docker container on Azure.
MMaker從頭訓練全新模型。成本與人才門檻最高。Train a new model from scratch. Highest cost and talent bar.

選 Shaper 的理由跟第 02 章一樣：多付的錢要買得到講得出來的東西。Same reason as chapter 02: the extra money has to buy something you can say out loud.

範圍怎麼收How the scope was bounded

試行標的Pilot target工業電腦產品線，50 份使用手冊（含系列）；成效確認後才擴到其他產品線Industrial-computer line, 50 user manuals (incl. series); other lines only after results
資料就緒條件Data readiness客戶提供 Word 原檔，不收 PDF 掃描，避免 OCR 誤差進入訓練集Client supplies Word source files, not scanned PDFs, so OCR error never enters the training set
模型與所有權Model & ownership開源模型；是否 fine-tune 取決於初步產出是否貼近需求；以 Container 形式交付Open-source model; fine-tune only if first outputs miss the need; delivered as a container
部署DeploymentMicrosoft Azure；Azure AD 登入與權限Microsoft Azure; Azure AD sign-in and permissions
產出形式Output一鍵生成整份初版手冊，使用者保留編修權One-click first draft of the whole manual; the user keeps editorial control

虛線左邊是 AI 模型，右邊是軟體工程。這條線就是合約裡誰負責什麼。Left of the dashed line is the AI model, right is the software. That line is who owns what in the contract.

資料怎麼走How data moves

  1. 使用者在 UI 建立新手冊：定義範圍、章節、規格；上傳參考文件User creates a manual in the UI: scope, chapters, specs; uploads reference files
  2. Backend 把上傳文件與勾選需求組成 prompt，送給 AI SystemBackend composes the prompt from uploads and selections and sends it to the AI system
  3. AI System 比對 Vector DB，依關聯性回傳索引值The AI system matches against the vector DB and returns relevance-ranked indexes
  4. Backend 用索引到 Relational DB 取出文件段落；完整文件與歷史手冊存在 Cloud StorageBackend pulls passages from the relational DB by index; full files and past manuals live in cloud storage

為什麼這張圖重要Why this diagram matters

  • 索引回到 Relational DB 再取原文，所以每段生成內容都能追到來源：這是 audit trail 的雛形。Indexes go back to the relational DB for the original text, so every generated passage traces to a source: the seed of an audit trail.
  • Azure AD 在最前面，權限決定看得到哪些手冊：知識來源治理從登入開始。Azure AD sits in front; permissions decide which manuals are visible. Knowledge-source governance starts at sign-in.
  • 模型以容器交付，客戶擁有部署，不綁 API 供應商。The model ships as a container; the client owns the deployment and is not tied to an API vendor.

上面是文件進來的路，下面是問題進來的路。Top row: how documents come in. Bottom row: how a question comes in.

Pre-processing 做什麼What pre-processing does

  • 以目錄（TOC）為分界切文件，每章一個可檢索單位Split by table of contents; one retrievable unit per chapter
  • 建 metadata：規格（Spec）、元件（Components）、產品線Build metadata: specs, components, product line
  • Parser、OCR、版面分析、表格結構辨識，再 embedding 進 Vector DBParser, OCR, layout analysis, table-structure recognition, then embed into the vector DB

Retrieval 與生成Retrieval and generation

  • Query analyze：分詞、去停用詞、理解結構與意圖Query analysis: tokenise, drop stop-words, read structure and intent
  • Task dispatch 把任務分給對應的 executor；Multi-way recall 從多個模型與來源召回；Re-rank 依關聯性排序Task dispatch routes to the right executor; multi-way recall pulls from several models and sources; re-rank orders by relevance
  • LLM 把召回的 chunk 轉成人可讀的草稿，盡可能保留原格式；每段附前三名替代內容The LLM turns recalled chunks into a readable draft in the original format, with the top three alternatives per passage

先寫 User Story，再翻成 Prompt Story。這是「產品經理要什麼」到「模型做什麼」的轉譯。User story first, then a prompt story. That's the translation from what the PM wants to what the model does.

驗收指標在建置前就寫好Acceptance metrics written before the build

第一階段 · 8/2Stage 1 · Aug 2第二階段 · 10/11Stage 2 · Oct 11
F1-score≥ 0.50.6 – 0.7
MRR—≥ 0.3
資料Data50 份訓練；每個產品線另留 3 份做線下驗證；以各產品線平均判定50 for training; 3 more per product line held out for offline validation; judged on per-line averages
評估方式Method線下評估（可短時間多次迭代），不做 A/B 與滿意度調查（時間長、主觀偏誤）Offline evaluation (fast iteration), not A/B or satisfaction surveys (slow, subjective)
停損Stop rule第一階段未過，可決議是否繼續專案If Stage 1 fails, the client may decide whether to continue

階段與交付物Phases and deliverables

0啟動：AI VOC 驗證、架構圖稿、SOW 定稿、啟動會議Kickoff: AI voice-of-customer validation, architecture draft, SOW final, kickoff
1系統設計：需求訪談、wireframe、UI、軟體架構、DB schema、API spec、資料清洗、基礎模型整合驗證；一章內容生成 demoDesign: interviews, wireframes, UI, software architecture, DB schema, API spec, data cleaning, base-model integration check; one-chapter generation demo
2開發測試：模型調整與 prompt 測試、AI 架構施作、模型封裝、API 整合、SIT 報告Build & test: model tuning and prompt tests, AI architecture build, model packaging, API integration, SIT report
3驗收結案：Azure 環境、UAT、正式上線、文件交付、驗收單簽署Acceptance: Azure environment, UAT, go-live, document handover, signed acceptance

四個方案，越左越完整Four scope options, fuller to the left

一 完整系統1 Full systemUI + 模型 + 串接 + 環境；管理層能真實評估成效UI + model + integration + environment; management can measure real effect
二 縮減軟工2 Reduced SW去掉登入與封面頁；標準章節預設英文Drop sign-in and cover pages; standard chapters default to English
三 模型 + API3 Model + API前端由客戶自建；模型與 API 完整Client builds the front end; model and API complete
四 模型 PoC4 Model PoC只驗核心章節；無 API；成效需客戶自行推估Core chapters only; no API; client estimates the effect itself

有疑慮就先做簡化版，再按階段加約。用 change order 往上爬。If in doubt, start simple and add contracts stage by stage. Climb by change order.

提案裡的做法In the proposal對應到我的方法In my method
F1 與 MRR 的門檻在建置前寫進驗收標準，而且有日期F1 and MRR thresholds written into acceptance before build, with datesDefine the number upfront；KPI card 的 metric、target、deadlineDefine the number upfront; the KPI card's metric, target, deadline
第一階段沒過，客戶可以決定不繼續If Stage 1 fails the client may stopGate 不是延後補，是停損；信任從敢寫停損開始A gate is a stop, not a deferral; trust starts with daring to write the stop
只收 Word，不收 PDF 掃描Word only, no scanned PDFs資料就緒條件寫進 client obligations；第 08 章四個桶子裡的 content 問題，在這裡就先堵Data readiness as a client obligation; the content bucket from chapter 08 is blocked here
以 TOC 切塊、建 Spec / Components metadata、索引回到原文Split by TOC, build spec/component metadata, index back to sourceNexus 知識層的切塊與六個 metadata 欄位；來源可追溯Nexus knowledge-layer chunking and six metadata fields; source traceability
User Story → Prompt StoryUser story → prompt storySpec & build literacy：能把業務需求翻成模型要做的 Input / Process / Output，才驗得了工程師的計畫Spec & build literacy: translating business need into input / process / output is what lets me verify an engineer's plan
四個方案階梯，先簡化再按階段加約Four-option ladder; start simple, extend by stageFind the smarter path；第 05 章的「只承諾下一階」Find the smarter path; chapter 05's "commit to the next rung only"
先做 50 份、一條產品線，成效確認再擴50 manuals, one product line, scale only after resultsValidate → Launch → Scale 的順序；Scale 是 gate 之後的事Validate → Launch → Scale; scale comes after the gate

來源：iKala AI Lab 提案 v1.0，2024 年 6 月。圖是我重繪的；金額與客戶未公開。Source: iKala AI Lab proposal v1.0, June 2024. Diagrams redrawn by me; pricing and client withheld.

09 · 找更快、更便宜的路Finding the faster, cheaper path

兩次對「用 AI」說不Two times I said no
to "use AI"

軍品倉儲Defence warehousing

C

國防客戶要一套 AI 倉儲系統管彈藥和軍品。需求書真的寫著「AI 驅動」。A defence client wanted an AI warehouse system for ammunition and supplies. The brief literally said "AI-driven".

A

前兩次會議只談流程。效期是日期，安全庫存是門檻，補貨是規則。沒有一件事需要預測。我說：你們不需要 AI。我們做了資料模型、規則引擎和安全的 dashboard，全部 on-prem。The first two sessions were all about the process. Expiry is a date. Safety stock is a threshold. Reorder is a rule. Nothing needed predicting. So I told them: you don't need AI. We built a data model, a rules engine and a secure dashboard, all on-prem.

R

用一小部分的時間和成本，拿到可用、可稽核的系統。我的規則：答案是日期、數量或門檻，就是 workflow。AI 是給需要判斷的時候。A working, auditable system in a fraction of the time and cost. My rule since: if the answer is a date, a count or a threshold, it's workflow. AI is for when the answer is a judgement.

完整版Full story

我留了一扇門：資料乾淨一年之後，再談需求預測。角色權限和日誌從第一天就做進去，因為在封閉網路裡，稽核軌跡比模型重要。I left one door open: once the data had been clean for a year, we could talk about forecasting. Role-based access and logging went in from day one, because inside a closed network the audit trail matters more than any model.

震動感測器 → 手扶梯Vibration sensors → escalators

C

感測器客戶以為現在的 AI 可以直接預測機械手臂故障。我們收了震動資料，收了第一筆錢，團隊準備從 FFT 從頭做起。A sensor client believed today's AI could predict robotic-arm failures out of the box. We had the vibration data, we'd taken a first payment, and the team was about to build FFT features from scratch.

A

我問了兩件事。一，物理 AI 平台已經有了，為什麼還手工建？二，更大的問題：我們有很多震動資料，但幾乎沒有標好的故障。沒有故障就沒東西可預測。所以我回去提了一個有標籤的案子：捷運手扶梯。I asked two things. One: physics-AI platforms already exist, why are we hand-building this? Two, the bigger one: we had lots of vibration data and almost no labelled failures. No failures, nothing to predict. So I went back with a case that had labels: metro escalators.

R

成交約 US$150K，交付有 baseline 的預測維修模型。機械手臂留到第二階段。一課：最聰明的路很少是更好的模型，而是把問題問對。Closed at about US$150K, delivered a predictive-maintenance model with a real baseline. The robotic arm became phase two. The lesson: the smartest path is rarely a better model. It's a better question.

完整版Full story

我提的是 NVIDIA PhysicsNeMo 這類物理資訊的代理模型，也問過能不能用物理模型生成合成故障樣本。這個提問本身沒有被採用，但它暴露了真正的問題是標籤不足。What I raised was physics-informed surrogate modelling, NVIDIA PhysicsNeMo and the like, including whether physics models could generate synthetic failure samples. That route wasn't taken, but asking is what exposed the real problem: no labels.

10 · 經驗怎麼落到一個 outcome 角色How the experience lands in an outcome role

對應，與前 90 天The mapping,
and the first 90 days

左邊是做過的，右邊是這個角色要的。點右邊，看線。Left: what I've done. Right: what the role needs. Tap the right side to see the lines.

Deloitte →

iKala →

前 90 天First 90 days

Day 1–30接手兩個進行中的案子，重做每個案子的 KPI card，找出沒有 baseline 或沒有 client owner 的，把 dashboard 立起來。Take over two live engagements, redo each KPI card, find the ones with no baseline or no client owner, stand up the dashboard.
Day 31–60每個案子做一次 built vs promised 稽核，親自驗證至少一個技術 milestone（跑一次 pipeline、看一次 eval set）。有 drift 的，帶解法升級。One built-vs-promised audit per engagement; personally verify at least one technical milestone (run the pipeline, read the eval set). Escalate drift with a fix attached.
Day 61–90為每個案子寫 evidence pack 與下一個機會，至少一個進入提案；把 KPI card 與 dashboard 模板變成團隊共用。Evidence pack and next opportunity per engagement, at least one into proposal; make the KPI card and dashboard templates team-wide.

12 · 開工前的釐清Before any build starts

我會問客戶的十二個問題Twelve questions
I ask a client

第一次見面就問。答案決定整合怎麼做、交付怎麼切。每題附我為什麼問，和什麼答案是警訊。I ask these in the first meeting. The answers decide how we integrate and how we cut the delivery. Each one comes with why I ask, and which answer is a red flag.