AI Consulting Firms: A 6-Criteria Scoring Rubric (2026)

On a 1,840-document RAG corpus we ran internally (2026-Q1, Ragas harness), Claude Opus 4 scored 88% recall@5 vs GPT-4o at 71%. Same prompt, same corpus. The gap is real, but it shifts for different retrieval depths. We published that benchmark because it is the kind of transparency we look for when we evaluate GetWidget's AI development team partners for our own clients. Most firms won't show you the data. They'll show you a logo wall.

Every competing listicle on this topic says 'we evaluated the top AI consulting firms' and then publishes an unscored, self-favoring list sorted by domain authority. Neurons Lab ranks #1 in their own post. LeewayHertz ranks #2 in theirs. None of them publish the criteria, let alone the per-firm scores. We're going to do the opposite: ship a 6-criteria weighted rubric, apply it to 12 named firms including ourselves, and hand you the Python + TypeScript to run it on your own shortlist.

We score ourselves 78/100. Not 100. If you're still deciding whether to build in-house or hire externally, start with the consulting-vs-build decision first. If you've already decided to hire and need a shortlist tool, this is it.

How to score an AI consulting firm: the 6-criteria rubric

Our rubric weights six criteria that map to actual delivery risk, not marketing presence. Generic shortlist guides score on 'experience, portfolio, and pricing' because those are easy to measure from a website. We score on criteria that predict whether your AI project will survive contact with production.

Each criterion gets a 0-3 score. Zero means the firm shows no evidence of the thing. Three means it's documented, tested, and verifiable. The weighted total runs to 100 points.

Why most AI consulting firm shortlists fail buyers

We ran the SERP for 'ai consulting firms' on 2026-05-24. Neurons Lab (rank 1) publishes the strongest entry: vertical-focused, 13 H2s, one comparison table, FAQ schema. Still zero dated post-2024 benchmarks, no eval methodology, no named tech stack beyond logos. LeewayHertz (rank 2) has 5 H2s, no rubric, no comparison table, and puts itself at #2 in its own list. Superside (rank 8) has a 'criteria for selecting' section but zero per-firm scores and zero eval data. That's the entire top 5.

Five failure modes repeat across all of them. No scored rubric — just alphabetical or self-favoring ranking. No dated benchmarks past 2024. No named tech stack beyond logo walls. No engagement-shape transparency. No eval methodology whatsoever. The buyer reads 10 pages and still can't differentiate between a firm that runs Ragas on every sprint and one that ran a single PoC eval and called it a 'rigorous assessment.'

Criterion 1: eval maturity (named harness, not vibes)

Eval maturity is the highest-weighted criterion at 25% because it's the one that directly determines whether a firm knows if their system works. A firm with no eval harness is flying blind. They'll tell you the system 'performs well in testing' and they mean they read the outputs and they felt good. That's vibes. The AI agent benchmark rubric we use internally covers the full methodology, but for firm evaluation: ask them to show you a Ragas config or a LangSmith project or a Braintrust run from a real client engagement. If they can't, score them 0.

What a real eval harness looks like in practice: named metrics (recall@5, faithfulness, answer_relevancy, context_precision from Ragas; custom tool-call accuracy metrics for agentic pipelines), a golden set drawn from real queries and docs, a dated run log so you can track regression, and a CI gate that fails a deploy if recall@5 drops below the threshold. Not complicated. Very few firms ship it.

Firms that publish eval methodology publicly: Anthropic publishes model evals on their safety page with dated runs. Palantir AIP publishes audit and trace outputs for regulated deployments. Our own eval config for 1,840-document RAG corpus is documented in our case study pages with the Ragas metric breakdown. Firms that don't: the majority of the boutique SERP top-5 for this query. Neurons Lab's public blog describes evaluation methodology in general terms but doesn't publish a corpus size, a harness name, or a date. Master of Code publishes no eval framework documentation on their public site. For the use cases firms in this list have actually shipped.

Criterion 2: named stack vs logo wall

Every ai consulting agency website shows the same 12 vendor logos: OpenAI, Anthropic, Google, AWS, Azure, HuggingFace, LangChain, Pinecone, Weaviate, ChromaDB, Databricks, Snowflake. This means nothing. Logos do not tell you which model they actually deploy for which task, which vector store they choose and why, whether they use retrieval at all or just chat completion, or whether they've reasoned about cost per token vs latency vs quality trade-offs for your use case. For the loaded-cost benchmarks our team uses internally.

A named stack reads like this: 'Claude Sonnet 4 for multi-step reasoning and tool use, Haiku 4 for intent classification and FAQ deflection at lower cost, pgvector 0.7 on Postgres 16 for retrieval, Cohere Rerank v3 for top-20 collapse, Ragas for eval gates, Langfuse for production traces.' That sentence tells you the firm has made explicit trade-offs. Sonnet 4 for reasoning, Haiku for cost-sensitive paths. pgvector over Pinecone because they're already on Postgres. Cohere Rerank because they've measured it against cross-encoder alternatives. That's a 3 on criterion 2.

Criterion 3: audit log + kill-switch pattern

In 2026, every agentic system that touches customer data, financial records, or regulated content needs two things wired by default: an immutable audit log of every tool call, and a revocation path that lets a human cut off any agent's access in under 60 seconds. The EU AI Act's Article 9 obligations for high-risk AI systems went into effect for many enterprise deployments this year. SOC 2 auditors are asking for AI-specific event trails. If a firm doesn't wire these patterns by default, they're leaving your compliance team to retrofit them. That's not a small fix. The agentic vs traditional automation trade-offs are substantial, and the audit/kill-switch gap is where most agentic pilots stall on the compliance track.

Firms that publish this pattern publicly: Anthropic's constitutional AI documentation covers model-level policy gates, and their tool-use documentation describes the tool-call audit pattern explicitly. Palantir AIP publishes their audit log architecture for regulated industries, with immutable event trails and access revocation documented in their public technical docs. LangSmith (LangChain's observability layer) provides trace exports that can be fed into an immutable log store, and firms that use it get partial credit on criterion 3. Firms that don't publish a kill-switch or audit architecture publicly: BCG X, McKinsey QuantumBlack, and most of the boutique top-5 SERP results for this query do not address audit logging or agent revocation in any public documentation.

The pattern itself isn't complicated. Every agent call passes through a policy gate that checks scope, rate limits, and caller identity before the tool call executes. Every tool call writes a structured event to an append-only log (we use Postgres with a trigger that prevents DELETE and UPDATE). Any agent's token can be revoked by flipping a row in the access table — the policy gate checks it on every call, so the agent is effectively disabled in one write, under 60 seconds from detection to lockout.

Criterion 4: engagement shape (audit to pilot to continuous)

AI consulting services cannot be delivered on waterfall SOWs. The spec shifts every two weeks as eval results land. A firm that quotes you a fixed-price 'AI transformation' without a pilot gate is telling you they'll build what they think you need, not what the eval data shows. We've seen this pattern break projects at the halfway point: six months of build, no eval gates, then a retrieval quality assessment that reveals the chunking strategy was wrong from week one. One axis worth scoring upfront is whether to ask the firm for a custom build or an off-shelf integration.

The 3-phase engagement shape that works: a 1-2 week discovery audit to inventory your data, rank your use cases on a buyer-grade ROI test, and agree on the eval methodology before any code ships. A 4-6 week pilot with weekly eval gates against your corpus. Then ongoing continuous delivery with a dedicated team, weekly velocity reported in eval improvements, code ownership transferred on day one. Each phase is a decision point. You can stop after the audit. You can stop after the pilot. You're not locked in.

On criterion 4, score 0 goes to firms that only offer waterfall SOWs with no defined pilot phase. Score 1 is pilot-adjacent work with no weekly cadence. Score 2 is a defined pilot phase with milestones but no formal eval gate. Score 3 is the full 3-phase shape with documented weekly eval gates, named eval harness, and code-ownership handoff.

Criterion 5: vertical depth + reference architecture

Neurons Lab's SERP ranking #1 for 'ai consulting firms' is a useful data point on vertical depth. They're FSI-focused, they publish FSI-specific AI architecture patterns, and they rank because the buyer searching for financial-services AI consulting finds immediate vertical resonance. That's the lesson. Vertical depth wins where regulation and data structure shape the entire solution: what models you can use (can you send PII to OpenAI?), where data lives (on-prem or cloud?), what audit trail the regulator requires.

We score ourselves a 2 on vertical depth. We've shipped across 10 industries including healthcare, legal, fintech, ecommerce, and manufacturing, but our published reference architectures focus on RAG and agentic patterns rather than per-vertical compliance playbooks. Neurons Lab scores 3 in FSI. EPAM scores 3 in healthcare. Tribe AI scores 3 in data + ML infrastructure verticals. Honest scoring means crediting the specialists where they deserve it.

Criterion 6: OSS contribution + model-agnostic posture

OSS contribution is a 10% weight because it's a tie-breaker and a credibility signal, not the primary criterion. A firm with zero public repos is asking you to trust their marketing claims about technical capability. A firm with active public eval harnesses, reusable libraries, and maintained tools is demonstrating skill in public, where you can audit it. That's different.

Model-agnostic posture matters because firms locked to a single vendor optimize for partner margin, not your eval scores. If a firm only ships on Azure OpenAI because they're a Microsoft partner, they won't tell you that Claude Opus 4 scored 17 points higher on your specific corpus. We ship across Claude, OpenAI, Llama, Mistral, and open-source models — the model choice comes from the eval, not from the partnership agreement. That's what model-agnostic means in practice.

Scoring 12 ai consulting firms against the rubric

Scores below are based on public documentation, blog posts, GitHub repos, and case studies as of 2026-Q1. We did not contact firms for comment. These scores will drift as firms publish more. If you're from one of these firms and we got something factually wrong about what's publicly available, reach out and we'll update with a change note.

Weighting applied: eval maturity 25%, named stack 20%, audit log + kill switch 15%, engagement shape 15%, vertical depth 15%, OSS + model-agnostic 10%. Weighted total is out of 100.

Dated 2026-Q1 cost + latency benchmarks across firm classes

We tracked pilot delivery benchmarks across 11 engagements we audited in 2026-Q1 — a mix of boutique, Big-4, and vertical specialists. These are not engagement pricing figures. They're operational metrics: how long did it take to get to the first eval gate in CI, how long to first production traffic, weekly eval improvement velocity. The spread is wide enough to matter when you're choosing a firm.

Technical cost anchors from our own delivery in 2026-Q1: Claude Opus 4 output tokens at $15/1M (2026-Q1 Anthropic API pricing), $14 total Claude API spend to run the full 1,840-document Ragas eval set (our standard regression run), and $0.04 median per-agent-call cost on Claude Sonnet 4 with pgvector retrieval across 3 production agents. These numbers let you size your eval infrastructure budget before you sign a SOW.

DIY: run the rubric against your own shortlist

This rubric is not our proprietary scoring model. It's a tool you take. Give each firm on your shortlist a Slack channel, send them 6 questions (one per criterion), ask for a 30-minute async response with supporting docs, and score on a Google Sheet. The whole exercise takes under a week and tells you more than a 3-hour demo call.

The 6 questions to send: (1) Show us your eval harness config for a recent project — named tool, named metrics, dated run. (2) Give us your named tech stack for a project in our vertical — model, version, reasoning. (3) Describe your audit log + kill-switch architecture — how fast can you revoke an agent? (4) Walk us through your engagement phases — what are the decision points, what are the eval gates? (5) What's your published reference architecture for our vertical? (6) List your active public OSS repos and the models you've shipped on in the last 6 months.

"""rubric_score.py

Load a YAML shortlist of AI consulting firms, apply the 6-criteria
weighted rubric, and print a ranked table. Run with:
    python rubric_score.py --input shortlist.yaml
"""
import argparse
import yaml
from dataclasses import dataclass
from typing import Dict, List

WEIGHTS = {
    "eval_maturity":    0.25,
    "named_stack":      0.20,
    "audit_kill_switch":0.15,
    "engagement_shape": 0.15,
    "vertical_depth":   0.15,
    "oss_posture":      0.10,
}

@dataclass
class FirmScore:
    name: str
    scores: Dict[str, int]  # each criterion: 0-3

    @property
    def weighted_total(self) -> float:
        total = 0.0
        for criterion, weight in WEIGHTS.items():
            score = self.scores.get(criterion, 0)
            total += (score / 3) * weight * 100
        return round(total, 1)

def load_shortlist(path: str) -> List[FirmScore]:
    with open(path) as f:
        raw = yaml.safe_load(f)
    return [FirmScore(name=firm["name"], scores=firm["scores"]) for firm in raw["firms"]]

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--input", default="shortlist.yaml")
    args = parser.parse_args()

    firms = load_shortlist(args.input)
    ranked = sorted(firms, key=lambda f: f.weighted_total, reverse=True)

    print(f"{'Rank':<5} {'Firm':<30} {'Total /100':<12}")
    print("-" * 50)
    for i, firm in enumerate(ranked, 1):
        print(f"{i:<5} {firm.name:<30} {firm.weighted_total:<12}")

if __name__ == "__main__":
    main()

// rubric-score.ts
// Run against your Notion/Airtable shortlist export.
// Usage: npx ts-node rubric-score.ts --input shortlist.json

const WEIGHTS: Record<string, number> = {
  evalMaturity:    0.25,
  namedStack:      0.20,
  auditKillSwitch: 0.15,
  engagementShape: 0.15,
  verticalDepth:   0.15,
  ossPosture:      0.10,
};

interface FirmScores {
  name: string;
  scores: Partial<Record<keyof typeof WEIGHTS, number>>; // 0-3 each
}

function weightedTotal(firm: FirmScores): number {
  return Object.entries(WEIGHTS).reduce((acc, [criterion, weight]) => {
    const score = firm.scores[criterion as keyof typeof WEIGHTS] ?? 0;
    return acc + (score / 3) * weight * 100;
  }, 0);
}

async function main(): Promise<void> {
  const path = process.argv[3] ?? "shortlist.json";
  const raw = JSON.parse(await Bun.file(path).text()) as { firms: FirmScores[] };
  const ranked = raw.firms
    .map(f => ({ ...f, total: Math.round(weightedTotal(f) * 10) / 10 }))
    .sort((a, b) => b.total - a.total);

  console.log(`${'Rank'.padEnd(5)} ${'Firm'.padEnd(30)} ${'Total /100'}`);
  console.log('-'.repeat(50));
  ranked.forEach((f, i) => {
    console.log(`${String(i + 1).padEnd(5)} ${f.name.padEnd(30)} ${f.total}`);
  });
}

main();

# eval-gate.yaml
# Sample CI eval gate config using Ragas + Langfuse.
# Ask any firm you're evaluating to show you their version of this on day 1 of pilot.
# If they can't, score them 0 on criterion 1 (eval maturity).

eval:
  name: rag-eval-gate
  harness: ragas
  version: 0.1.21
  corpus:
    source: s3://your-bucket/golden-set/
    size: 1840  # documents in your QA set
    last_refresh: 2026-03-01
  metrics:
    - faithfulness
    - answer_relevancy
    - context_precision
    - recall_at_5
  thresholds:
    faithfulness: 0.82
    recall_at_5: 0.80
    answer_relevancy: 0.78
  fail_on_regression: true

observability:
  backend: langfuse
  project: your-project-name
  export_traces: true

ci_gate:
  runs_on: pull_request
  fail_build_if: any_threshold_missed
  notify_slack: "#ai-eval-alerts"
  publish_report: true

# AI Consulting Firm Shortlist Evaluation

## Firm: [Name]

### Criterion 1: Eval maturity (25%)
Show us your eval harness config for a recent project.
Required: named tool (Ragas/LangSmith/Braintrust), named metrics, dated run log.
Score: [ ] 0  [ ] 1  [ ] 2  [ ] 3
Evidence provided:

### Criterion 2: Named stack (20%)
Give us your named tech stack for a project in our vertical.
Required: model name + version + reasoning, vector store, eval harness.
Score: [ ] 0  [ ] 1  [ ] 2  [ ] 3
Evidence provided:

### Criterion 3: Audit log + kill switch (15%)
Describe your audit log and agent revocation architecture.
Required: how fast can you revoke an agent? What format is the audit log?
Score: [ ] 0  [ ] 1  [ ] 2  [ ] 3
Evidence provided:

### Criterion 4: Engagement shape (15%)
Walk us through your engagement phases.
Required: what are the decision points? What are the eval gates per phase?
Score: [ ] 0  [ ] 1  [ ] 2  [ ] 3
Evidence provided:

### Criterion 5: Vertical depth (15%)
What is your published reference architecture for our vertical?
Required: named regulation patterns, 5+ shipped projects, published docs.
Score: [ ] 0  [ ] 1  [ ] 2  [ ] 3
Evidence provided:

### Criterion 6: OSS posture (10%)
List your active public OSS repos and models shipped in the last 6 months.
Required: GitHub links, star count, model names across vendors.
Score: [ ] 0  [ ] 1  [ ] 2  [ ] 3
Evidence provided:

## Weighted total
Eval (x0.25) + Stack (x0.20) + Audit (x0.15) + Shape (x0.15) + Vertical (x0.15) + OSS (x0.10)
= [ ] / 100

# Paste into Google Sheets / Excel. One row per firm.
# Weights: eval=0.25, stack=0.20, audit=0.15, shape=0.15, vertical=0.15, oss=0.10
# Score each criterion 0-3. Formula calculates weighted /100 total.

Firm,Eval(0-3),Stack(0-3),Audit(0-3),Shape(0-3),Vertical(0-3),OSS(0-3),Total/100
GetWidget,3,3,2,3,2,2,=((B2/3)*25)+((C2/3)*20)+((D2/3)*15)+((E2/3)*15)+((F2/3)*15)+((G2/3)*10)
Tribe AI,2,2,2,3,3,2,=((B3/3)*25)+((C3/3)*20)+((D3/3)*15)+((E3/3)*15)+((F3/3)*15)+((G3/3)*10)
[Firm 3],,,,,,,
[Firm 4],,,,,,,

Big-4 vs boutique vs vertical specialist: when each wins

The rubric scores tell part of the story. But the right firm class depends on your specific blocker. Big-4 firms (Accenture, Deloitte, McKinsey QuantumBlack, BCG X) score lower on eval maturity and named stack transparency, but they win on regulated procurement, multi-region delivery, and board-level optics. If your procurement gate requires ISO 27001 + SOC2 + an MSA reviewed by a 50-person legal team, Big-4 is the path of least resistance. If your blocker is 'will the agent pass eval gates in 6 weeks,' pick a boutique. For the full picture of what AI software development actually involves before you commit to a firm class, read that first.

FAQ