How to Run an AI Readiness Assessment

We've watched AI pilots stall for the same reason a building inspection fails: the foundation was never checked. A team picks a flashy use case, wires up a model, demos something impressive, then hits production and finds the data is dirty, nobody owns the audit log, and the one engineer who understood the retrieval layer just left. An AI readiness assessment is the inspection you run before you pour the concrete. Done right, it tells you in two weeks what would otherwise take a six-month failed pilot to learn. Most readiness tools online are self-serve marketing funnels: answer 12 questions, get a vague maturity score, talk to sales. We're going to hand you the actual artifact instead. A six-dimension scorecard, a weighted scoring formula in Python, the rubric as a decision matrix, and a 14-day process you can run with your own team. If you're still deciding whether to build in-house or hire help at all, read the consulting-vs-build decision first. If you've decided AI is coming and you need to know how exposed you are, this is the inspection.

The output of a good assessment is not a number. It's a prioritized gap list with owners and a 90-day roadmap. The number is what executives ask for; the gap list is what changes what your team does on Monday. We'll show you both, and we'll be honest about the trap: a readiness score of 82 means nothing if the assessor scored every dimension generously to win the follow-on engagement. Score yourself before you trust anyone else's score.

What an AI readiness assessment actually is (and what it isn't)

An AI readiness assessment is a structured evaluation of whether your organization can build, deploy, and operate AI systems safely and profitably. It scores six dimensions: data readiness, infrastructure, talent and skills, governance and risk, use-case viability, and change management. Each dimension gets a 0 to 4 maturity score against documented evidence, the scores are weighted by delivery risk, and the result is a single readiness index plus a ranked list of the gaps that will hurt you first.

Here's what it isn't. It isn't an AI maturity model where you slot yourself into 'crawl, walk, run' and feel vaguely behind. It isn't a vendor questionnaire designed to qualify you for a sales call. And it isn't a one-time event you file away. The assessment is a baseline. You re-run it every two or three quarters because data drifts, the team turns over, and the EU AI Act compliance bar moves.

The six dimensions of AI readiness

We score six dimensions because each is a distinct way an AI initiative dies. Skip data readiness and your model hallucinates on dirty inputs. Skip infrastructure and you can't ship past a notebook demo. Skip talent and the system rots the moment the one person who understands it leaves. Skip governance and your first regulated deployment gets blocked by legal. Skip use-case viability and you build something impressive that nobody needed. Skip change management and the people meant to use it route around it.

Notice the weighting. Data readiness carries the most because in 2026 the bottleneck is almost never the model and almost always the data feeding it. Infrastructure is next because shipping past a demo requires real MLOps. The exact weights are yours to adjust, and we show you the code to do it, but start here and change them only when you can defend the change with evidence.

The weighted scoring rubric: how to score each dimension 0 to 4

A score is only as honest as its rubric. Below is the rubric we run. Each dimension has explicit evidence requirements at each level, so two assessors scoring the same organization land within a point of each other. The discipline is in level 3, the ship bar: it requires the practice to be tested and enforced, not just documented. Anyone can write a data-governance policy. Far fewer can show you the CI job that fails a deploy when PII leaks into the embedding pipeline.

Dimension 1: scoring data readiness (the 25% that breaks most pilots)

Data readiness is the highest-weighted dimension because it's the one that quietly fails every pilot. A model is a commodity in 2026; you can swap Claude Sonnet 4 for GPT-4o in a config line. Your data is not a commodity. To score data readiness, gather evidence on four things: do you have a catalog that knows where your data lives, do you measure quality, have you classified PII, and are access controls enforced rather than aspirational. The choice between custom AI versus an off-the-shelf product often comes down to exactly this: off-the-shelf tools assume your data is clean and accessible, and most organizations discover during the pilot that it isn't. Across enterprise AI surveys published through 2025, roughly 70% to 80% of AI projects stall before production, with data quality and integration cited as the top recurring blocker.

Dimension 2: scoring infrastructure and MLOps readiness

Infrastructure readiness is the gap between a demo and a system. A demo runs in a notebook. A system runs behind a serving layer, ships through CI/CD with an eval gate that fails the build when recall drops, emits traces to Langfuse or LangSmith, and tracks per-token cost. To score this dimension, ask your team how a model change reaches production today. If the answer involves a person copying a file, you're at a 1. If there's an eval gate that can block a deploy, you're at a 3.

# Infrastructure dimension — score it by checking each item.
# Each TRUE moves you up. Score 3 (ship bar) requires every item in the
# ship_bar block. Run this as a literal checklist during the assessment.

infrastructure_dimension:
  weight: 0.20

  ship_bar:           # all must be true to score 3+
    model_serving:      true   # vLLM / AWS Bedrock / Azure OpenAI behind an API
    ci_cd_for_models:   true   # model + prompt changes ship through a pipeline
    eval_gate_in_ci:    true   # build fails if recall@5 or faithfulness drops
    tracing:            true   # Langfuse or LangSmith on every production call
    cost_tracking:      true   # per-token + per-request spend on a dashboard
    rollback_path:      true   # one command reverts a bad model/prompt deploy

  optimizing:         # level 4 — measured + improving each quarter
    autoscaling_inference: false
    canary_releases:       false
    otel_full_traces:      false   # OpenTelemetry spans across the agent graph

  scoring:
    0_1: "notebooks only, manual deploy, no observability"
    2:   "cloud capacity exists, no CI/CD for models, no eval gate"
    3:   "all ship_bar items true"
    4:   "all ship_bar + all optimizing items true"

Dimension 3: scoring talent, skills, and the bus-factor problem

The talent dimension is where readiness scores most often get inflated. Leadership sees one brilliant engineer shipping impressive demos and scores talent a 4. The honest score is a 1, because the bus factor is one. The right question isn't 'do we have skilled people?' It's 'if our best AI engineer left next week, would the system keep running and could someone else change it safely?' Score on redundancy, documentation, named ownership, and a real upskilling cadence. A team that contributes to LangGraph or maintains a Ragas eval config in public is a 4. A team where the knowledge lives in one person's head is a 1, no matter how good that person is.

Dimension 4: scoring governance, risk, and compliance readiness

Governance readiness is the dimension legal and risk teams care about and engineering teams forget until it blocks a launch. In 2026 the EU AI Act risk tiering is in force for many enterprise deployments, SOC 2 auditors ask for AI-specific event trails, and any agent touching customer data needs an audit log and a revocation path wired by default. Score this dimension on whether the controls are enforced on every call, not whether a policy document exists. Our guide to what responsible AI means in practice covers the full control set; for the assessment, the two non-negotiables are an immutable audit log and a kill switch that revokes an agent in under 60 seconds.

Dimensions 5 and 6: use-case viability vs change management

The last two dimensions are the ones executives most often conflate, so we score them separately on purpose. Use-case viability asks whether the thing you want to build is worth building and measurable. Change management asks whether the people it touches will actually use it. A use case can be perfectly viable on paper and die because the support team it was built for was never consulted and quietly kept the old process. Score viability on whether each candidate has a baseline and a buyer-grade ROI test. Score change management on whether affected teams co-designed the workflow.

The 14-day process: how to run the assessment end to end

Two weeks is enough. Longer than that and the assessment becomes a project that competes with the work it's supposed to enable. Here is the sequence we run as a discovery audit: scope and stakeholder kickoff, then evidence gathering across the six dimensions, then scoring against the rubric, then a gap workshop, then the readiness report and roadmap. Each phase is a decision point. If data readiness scores a zero on day five, you can stop and fix data before spending another nine days assessing infrastructure you won't be allowed to use yet.

The scorecard as code: weighted scoring formula in Python

Here is the scorecard as something you can actually run. The Python computes the weighted readiness index and enforces the hard rule that any single dimension below 2 caps the verdict at Not Ready, no matter how high the weighted total. The YAML is the scorecard your assessors fill in. The JSON is the report shape your roadmap tooling consumes. Copy these, adjust the weights to your context, and you have a defensible scoring artifact instead of a vibe.

"""readiness_score.py

Compute a weighted AI readiness index from a filled scorecard.
Enforces the hard floor: any dimension below 2 caps the verdict
at NOT_READY regardless of the weighted total.
    python readiness_score.py --input scorecard.yaml
"""
import argparse, json, yaml
from dataclasses import dataclass
from typing import Dict

WEIGHTS = {
    "data_readiness":    0.25,
    "infrastructure":    0.20,
    "talent_skills":     0.15,
    "governance_risk":   0.15,
    "use_case_viability":0.15,
    "change_management": 0.10,
}
MAX_SCORE = 4          # each dimension scored 0-4
HARD_FLOOR = 2         # any dimension below this caps verdict

@dataclass
class Scorecard:
    org: str
    scores: Dict[str, int]

    @property
    def index(self) -> float:
        total = sum((self.scores.get(d, 0) / MAX_SCORE) * w * 100
                    for d, w in WEIGHTS.items())
        return round(total, 1)

    @property
    def verdict(self) -> str:
        if any(self.scores.get(d, 0) < HARD_FLOOR for d in WEIGHTS):
            return "NOT_READY"   # hard floor breached
        if self.index >= 75:
            return "READY"
        if self.index >= 50:
            return "PARTIALLY_READY"
        return "NOT_READY"

    @property
    def gaps(self):
        # rank dimensions by weighted points lost, biggest first
        ranked = sorted(
            WEIGHTS,
            key=lambda d: (MAX_SCORE - self.scores.get(d, 0)) * WEIGHTS[d],
            reverse=True,
        )
        return [d for d in ranked if self.scores.get(d, 0) < 3]

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--input", default="scorecard.yaml")
    args = ap.parse_args()
    raw = yaml.safe_load(open(args.input))
    card = Scorecard(org=raw["org"], scores=raw["scores"])
    report = {
        "org": card.org,
        "readiness_index": card.index,
        "verdict": card.verdict,
        "prioritized_gaps": card.gaps,
    }
    print(json.dumps(report, indent=2))

if __name__ == "__main__":
    main()

# scorecard.yaml
# Filled during the assessment. Each dimension scored 0-4 against
# documented evidence. Do not score from opinion — attach the artifact.

org: "Example Corp — Q2 2026 baseline"

scores:
  data_readiness:     2   # catalog exists, quality unmeasured, access broad
  infrastructure:     3   # serving + CI/CD + eval gate + tracing wired
  talent_skills:      1   # bus factor of 1, no runbooks
  governance_risk:    1   # policy doc only, no audit log, no kill switch
  use_case_viability: 3   # each use case has baseline + ROI test
  change_management:  2   # stakeholders informed, training unfunded

evidence:
  data_readiness:     "link: data-catalog; gap: no quality metrics"
  infrastructure:     "link: ci-pipeline + langfuse dashboard"
  talent_skills:      "gap: single owner, no runbook repo"
  governance_risk:    "link: policy.md; gap: nothing wired in code"
  use_case_viability: "link: usecase-roi-sheet"
  change_management:  "gap: support team not yet co-designing"

{
  "org": "Example Corp — Q2 2026 baseline",
  "readiness_index": 49.6,
  "verdict": "NOT_READY",
  "_why": "talent_skills and governance_risk both below the hard floor of 2",
  "prioritized_gaps": [
    "governance_risk",
    "talent_skills",
    "data_readiness",
    "change_management"
  ]
}

What to do with your score: ready, partially ready, or not ready

A score with no decision attached is a slide, not an assessment. Each verdict maps to a concrete next action. The hard floor stops organizations from averaging away a fatal gap: a 90 on data and infrastructure with a 0 on governance is not 'mostly ready,' it's a launch legal will block. Below is what each band means.

From gaps to a 90-day roadmap: the mapping that makes the score useful

The deliverable that changes behavior is the gap-to-roadmap mapping. Take the prioritized gap list from the scorecard, attach an owner and a sized action to each gap, and sequence them so the highest-weighted gaps close first. The Python ranked the gaps by weighted points lost; the roadmap turns that ranking into a 90-day plan where each gap becomes a concrete first action, not a theme.

# gap-to-roadmap.yaml
# Generated from prioritized_gaps in readiness_report.json.
# Highest weighted-points-lost first. Each gap gets ONE owner and a
# sized, dated first action. No theme without an action.

roadmap:
  org: "Example Corp"
  horizon_days: 90

  gaps:
    - dimension: governance_risk        # score 1, weight 15% — hard-floor breach
      action: "Wire immutable audit log + policy gate + <60s kill switch"
      owner: "Platform lead"
      first_milestone: "Audit log append-only in staging"
      sized: "2-3 weeks"
      unblocks: "any customer-facing pilot"

    - dimension: talent_skills          # score 1, weight 15% — hard-floor breach
      action: "Document runbooks, cross-train a second owner per system"
      owner: "Eng manager"
      first_milestone: "Runbook repo + named backup owner"
      sized: "3-4 weeks"
      unblocks: "removes bus-factor-of-1 risk"

    - dimension: data_readiness         # score 2, weight 25%
      action: "Add data-quality metrics + PII classification gate in CI"
      owner: "Data lead"
      first_milestone: "Quality dashboard + PII gate blocking bad builds"
      sized: "4-6 weeks"
      unblocks: "moves data_readiness 2 -> 3 (ship bar)"

    - dimension: change_management      # score 2, weight 10%
      action: "Bring support team in as co-designers, fund training"
      owner: "Product lead"
      first_milestone: "Co-design workshop + funded training plan"
      sized: "2 weeks"
      unblocks: "adoption risk on the target use case"

  rescore_trigger: "re-run readiness_score.py after governance + talent clear"

Frameworks and tools to run the assessment with

You don't need to invent the vocabulary from scratch. Borrow framing from the McKinsey AI maturity model and the Gartner AI maturity framework for the board narrative, and use the NIST AI Risk Management Framework and the EU AI Act risk tiers for the governance dimension. For the technical evidence, the tools you score against are the same ones a competent team already runs: Ragas and LangSmith for eval gates, Langfuse for tracing, OpenTelemetry for spans, pgvector or Pinecone for retrieval, LangGraph or CrewAI for the agent layer, and AWS Bedrock or Azure OpenAI for serving. If you'd rather have a neutral party run the assessment and reconcile the scores, that's what our AI consulting company does in the discovery-audit phase.

FAQ