NLP is no longer sentiment analysis on tweets. Six use cases worth piloting - contract review, support triage, sales call extraction - and how to measure each.


Costa6 min
Contents7 sections
  1. What Changed: NLP Is No Longer Hard to Deploy
  2. Six Use Cases Worth Starting With
  3. Commercial LLM vs Open-Source
  4. Why RAG Still Matters With 1M-Token Contexts
  5. Quality Evaluation at Scale
  6. A 60-Day NLP Deployment
  7. The Bottom Line

What Changed: NLP Is No Longer Hard to Deploy

Pre-2023: deploying an NLP system in business required an ML team, labeled training data, MLOps infrastructure, and months of work. The work was building task-specific models (BERT for classification, T5 for summarization, custom NER for extraction).

2026: most business NLP is API calls to a commercial LLM, with prompt engineering and structured output, and the model itself keeps getting cheaper: the price of GPT-3.5-level performance fell more than 280-fold between November 2022 and October 2024. The bottleneck moved from model training to scoping.

Result: NLP use cases that were not economically viable in 2022 are worth a pilot now.

Six Use Cases Worth Starting With

1. Contract Review

LLM compares incoming contract against playbook. Flags deviations. Suggests redlines. Routes to legal for sign-off.

  • Measure: share of flagged deviations a lawyer agrees with, and review time per contract, on a sample of your own past contracts

2. Support Ticket Triage

LLM classifies incoming ticket, drafts first reply from knowledge base, routes to L1 or L2.

  • Evidence: in a study of 5,179 support agents, a generative AI assistant raised issues resolved per hour by 14% on average and 34% for novice agents
  • Measure: share of drafted replies sent without edits, and time to first reply

3. Sales Call Summarization

LLM transcribes (Whisper API), extracts decisions, action items, objections, next steps. Posts to CRM and Slack. The same extraction can run while the call is still live, when a voice agent is the one talking: call a demo agent in your browser and see the CRM record it writes.

  • Measure: action items the rep had to correct, and minutes of CRM entry saved per call

4. Document Q&A / RAG

User asks a question; LLM retrieves relevant docs from vector store, answers with citations. Used for internal knowledge bases, customer support docs, compliance archives.

  • Measure: share of answers whose cited source actually supports them, on a fixed set of real questions

5. Email Categorization

LLM classifies inbound email by intent (sales lead, support, billing, etc.) and routes to the right inbox/team.

  • Measure: misrouted emails per week, counted by the teams that receive them

6. Resume Screening

LLM extracts structured data from resumes (skills, experience years, education) and scores fit against job requirements.

  • Measure: agreement with recruiter decisions on a held-out sample, checked separately for each group
  • Note: bias testing required (EU AI Act, US EEOC) before deployment

Commercial LLM vs Open-Source

The decision matrix in 2026:

FactorCommercial (OpenAI, Anthropic, Google)Open-source (Llama, Mistral, Qwen)
Time to first deploymentFasterSlower (servers to set up)
Quality (general tasks)HigherLower (improving)
Cost per call (small scale)CheaperMore expensive (server cost)
Cost per call (high volume)More expensiveCheaper
Data residencyTrade-offFull control
Fine-tuning flexibilityLimited (vendor offerings)Full control
Operational overheadNoneSignificant (GPU servers)

Default: commercial. Switch to open-source when crossover conditions hit. Most B2B SaaS in 2026 stays on commercial APIs.

Why RAG Still Matters With 1M-Token Contexts

Since February 2024, models with 1M-token context windows have been available to developers. Some teams concluded RAG was obsolete. It is not.

Cost reason. Input is billed per token, so a 1M-token request pays for 200 times the input of the relevant 5K tokens. At scale, this is the difference between a profitable feature and an unaffordable one.

Quality reason. Models answer best when the relevant passage sits at the start or end of a long context and significantly worse when it sits in the middle - the "lost in the middle" problem. A relevant 5K-token chunk puts the answer where the model reads best.

RAG architecture in 2026:

Query
  -> Embedding model (text -> vector)
    -> Vector store retrieval (top-K chunks)
      -> Reranker (optional, refines top-K)
        -> LLM with retrieved chunks as context
          -> Response with citations

Tools: Pinecone, Weaviate, Qdrant for vector stores. Cohere Rerank, Voyage AI for rerankers. The pattern is mature; pick components, not a "RAG platform."

Quality Evaluation at Scale

Three layers, all required:

  1. Automated metrics where ground truth exists. Classification F1 score, extraction accuracy, retrieval recall@K. Run on every deployment.

  2. LLM-as-judge for subjective quality. Use a stronger model to score the production model's output on a rubric (does the summary capture the key points, is the tone appropriate). Validate the LLM judge against human samples once a month.

  3. Human spot-check on 5-10% of production output. Continuously, randomly. The only way to catch issues the first two layers miss.

Skipping any layer creates a quality blind spot. We have seen production NLP systems degrade silently for months because no one was looking at output quality.

A 60-Day NLP Deployment

Week 1-2: Pick the use case. Define the input format, the output format, the success metric (extraction accuracy, classification F1, summarization quality).

Week 3-4: Build the prompt or pipeline. Test on 100 examples. Iterate prompts. Measure accuracy.

Week 5-6: Deploy to production behind a feature flag. Route 10% of traffic. Monitor accuracy and cost daily.

Week 7-8: Scale to 100% of traffic. Set up the three quality evaluation layers. Document the pattern.

By day 60, the team has one production NLP system with measured accuracy, monitored quality, and documented cost. The next one starts from the same pattern. The same one-workflow-at-a-time discipline that compresses cost is described in AI integration in business.

The Bottom Line

NLP in business in 2026 is mostly API engineering, not ML engineering. Six use cases (contract review, support triage, sales call extraction, document Q&A, email categorization, resume screening) are where to start. The constraint is scoping (one workflow, one metric) and quality evaluation (three layers, all required). Commercial LLMs cover most business NLP needs, and the price of a given level of capability keeps falling. Open-source models matter at high scale or under data residency constraints. Pick one use case, ship in 60 days, instrument quality, then move to the next. The support-triage use case in particular has its own playbook in AI chatbots in customer service; the ROI math that boards accept is in measuring AI ROI.

Frequently Asked Questions

  • What changed in NLP between 2022 and 2026?

    Pre-2023: NLP meant training task-specific models (BERT for classification, T5 for summarization, custom NER). Required ML team, labeled data, MLOps. 2023-2024: GPT-3.5 and GPT-4 made it possible to skip training entirely - prompt the model, get the result. 2024-2025: the price of GPT-3.5-level performance fell more than 280-fold between November 2022 and October 2024 (Stanford AI Index 2025), context windows reached 1M tokens, structured output became reliable. The barrier to NLP in business went from 'ML team' to 'API key'.

  • Should we use commercial LLMs or open-source models?

    Default to commercial APIs (OpenAI, Anthropic, Google) for prototyping and most production. Switch to open-source (Llama, Mistral, Qwen) when (a) per-call cost is significant at scale, (b) data residency requirements forbid commercial APIs, or (c) you need fine-tuning beyond what commercial vendors expose. Where the cost crossover sits depends on your volume and on GPU prices, so compute it from your own call counts.

  • How accurate is LLM-based document extraction?

    It depends on how well-defined the field is and how clean the source is: an invoice number from a structured invoice is far easier than the key obligations in a poorly-worded contract. Measure accuracy on a labelled sample of your own documents before go-live, per field. The errors that remain must be either tolerated, caught by validation, or routed to human review. Plan for them in workflow design.

  • Do we still need RAG in 2026 with 1M-token context windows?

    Yes, for two reasons: (1) cost - input is billed per token, so a 1M-token request pays for 200 times the input of the relevant 5K tokens; (2) quality - models answer best when the relevant passage sits at the start or end of a long context and significantly worse when it sits in the middle (Liu et al., 2023, the 'lost in the middle' problem). RAG with retrieval over a vector store remains the right architecture for any document-heavy use case.

  • How do we evaluate NLP output quality at scale?

    Three layers: (1) automated metrics where ground truth is available (extraction accuracy, classification F1); (2) LLM-as-judge for subjective quality (does the summary capture the key points), validated against human samples; (3) human spot-check on 5-10% of production output continuously. Skipping any layer creates a quality blind spot.

NLPMachine LearningLLMDocument AI