LLM Integration, Custom AI, and the AI Layer

A mature enterprise design separates capability intent from provider implementation.

This mirrors other platform abstractions: you call Flow steps, not SQL; you call IntegrationHub actions, not raw sockets. The AI Layer aims to be the same for intelligence.

Design consequence: you build around stable contracts (inputs/outputs/policy) and let providers be swappable behind the curtain.

Core elements: provider connection + credential management, model inventory, per-capability routing rules, and environment-specific settings (dev/test/prod).

Governance: allowlisted providers only, least-privilege credentials, and controlled rollout of new providers via staged testing.

Avoid 'shadow providers': a developer with a personal API key in a script is a governance and audit failure.

Preferred pattern: Flow subflow/tool wraps the AI call and returns a structured response. Scripts call the subflow/tool rather than calling the provider directly.

API design should include: request id, model/provider metadata, confidence/quality flags, and safe failure codes.

Treat AI APIs like any enterprise integration: versioned, monitored, and rate-limited.

Routing inputs: capability type (draft vs extract), sensitivity tier, required context length, and time budget.

Fallback tiers: primary model → alternate model/provider → degraded mode (no AI) → human queue. Fallback must be explicit, logged, and tested.

Avoid silent fallback that changes quality without visibility. When provider changes, log it and surface it in ops dashboards.

Cache by key: policy version + intent + locale. If the source knowledge hasn’t changed, the answer should not be regenerated every time.

Amortize context: generate a summary once and store it on the record; later steps reference the stored summary rather than replaying full history.

Use model routing: cheaper models for extraction/classification, higher-capability models for complex synthesis only when needed.

Minimum telemetry per call: request id, capability, provider/model, latency, input size, output size, status/error code.

Quality telemetry: accept rate, edit distance (for drafts), override rate (for predictions), and user feedback.

Security telemetry: which records/fields were accessed (at a safe abstraction level), redaction confirmations, and policy blocks.

Signals across providers: multi-model routing, tool-use primitives, and enterprise controls (audit, residency, DLP). ServiceNow will converge toward these primitives to keep the platform stable while models evolve.

Expect change: model names, pricing, and capabilities will shift; the stable part is the contract you design (schemas, gates, and logs).

Practical stance: treat the AI Layer as a platform subsystem like IntegrationHub — version it, monitor it, and evolve it deliberately.

Design stable interfaces: 'GenerateSummary(record_id)' returns summary + citations + metadata. The calling flow should not know which provider generated it.

Centralize provider details in one layer: IntegrationHub action, subflow, or script include tool. Downstream workflows call the abstraction.

Testing discipline: run regression suites when changing provider/model. Evaluate quality, cost, and safety signals before promoting.

Provider choice is a policy decision: data residency and procurement constraints often decide provider for a region before capability does.

Capability differences still matter: long-context synthesis, tool-use quality, and structured output reliability vary. Route by task type.

Avoid mixing providers ad hoc per developer. Centralize connections, allowlists, and logging so governance is consistent.

Separate dev/test/prod connections. Never reuse production keys in dev. Enforce rotation, owners, and revocation procedures.

Endpoint governance: allowlist hostnames and paths. For private endpoints, document network boundaries and outbound restrictions.

Logging: store request metadata, not secrets. Redact request/response bodies where policies require.

Route tasks: extraction → smaller model; routing/classification → PI or small model; long-doc reasoning → stronger model; drafts → medium model.

Define acceptance criteria and test harness per task type. If a cheaper model meets the criteria, use it.

Include latency SLOs. A portal bot that takes 12 seconds will be bypassed regardless of quality.

Options: provider-managed private endpoints, cloud-managed platforms (e.g., Bedrock), or self-hosted endpoints. Each changes responsibility boundaries.

Residency: where prompts are processed and where logs are stored. Don’t assume 'private' means 'no data leaves region' without documentation.

Design degraded modes: private endpoints can still fail. Workflows must continue safely without AI.

Fallback hierarchy: primary provider → alternate provider (if allowed) → degraded mode (no AI) → human queue.

Risk gating: for sensitive workflows, do not fallback to a provider that violates residency or policy; degrade instead.

Observability: track fallback rate and reasons. Spikes indicate provider instability or throttling.

Use a fixed evaluation set: 50–200 prompts across workflows (drafts, extraction, compliance). Score quality and safety.

Measure cost: token usage, latency, error rates, and downstream rework (edit distance).

Select providers per task type if needed. Multi-provider routing is often the best solution.

Track spend by: capability (draft/extract), workflow (incident summary), channel (portal/agent), and org unit.

Enforce quotas: per-day call limits, per-user limits, and max payload caps. Add circuit breakers on anomalies.

Optimize spend: caching, summaries, model routing, and batch processing for offline tasks.

Step 1: Create provider connection for dev/PDI with least-privilege key.

Step 2: Define endpoint allowlist and payload caps.

Step 3: Build IntegrationHub action or Flow wrapper returning structured output.

Step 4: Call from a flow; store output + metadata on record.

Step 5: Simulate failures: wrong key, timeout, rate limit; verify degraded mode.

Built-in experiences reduce prompt work, but custom workflows still require prompt contracts: scope, safety, output shape, and failure behavior.

Treat prompts like code: version, review, test, and roll back.

A block template makes prompts maintainable and testable. It also provides consistent injection defenses by separating policy and user content.

Think in stable contracts: define what the model must produce, not just what it should 'write'.

Best practice: provide a curated context bundle per capability (e.g., incident fields + last 5 work notes + CI criticality).

When expecting typed outputs, include schema and allowed values so the model cannot invent categories or groups.

Use 2–5 canonical examples: normal case, ambiguous case, refusal/insufficient context case.

Never embed real PII. Treat example sets like code assets with review and approvals.

Design an explanation schema: evidence_used, checks, uncertainty.

For high-risk actions, rely on approvals and controls, not the model’s internal reasoning.

A good schema includes required fields, enums, range limits, and max lengths — and a clear error behavior when the model is uncertain.

Keep raw responses for debugging under strict retention and access controls.

Create a stable evaluation set per workflow type (e.g., 50 incidents). Score outputs for correctness, safety, and format adherence.

Track not just 'quality' but also cost and latency. A high-quality prompt that doubles tokens may fail budget constraints.

Store prompts like any platform configuration: with owners, versions, and deployment controls (dev/test/prod).

Pair prompts with schemas and evaluation sets so prompt evolution is measurable, not subjective.

Good reasons: fraud/risk scoring, anomaly detection on proprietary telemetry, image/PDF understanding beyond built-in extraction, or tight latency SLOs.

Bad reasons: re-implementing classification/routing that PI already handles with platform-native governance and workbench tooling.

Patterns: sync scoring, async scoring, batch scoring. Choose by latency and user experience constraints.

Choose based on: data residency, IAM integration, observability, version pinning, and your team's SRE maturity.

Use a feature dictionary and purpose limitation: only include what the capability needs (data minimization).

For high-cardinality text, consider embeddings or summaries — but treat them as sensitive derived data with retention rules.

Persist metadata: model version, feature version, request id, latency, confidence. This is your audit trail.

Use confidence bands and policy tables: high confidence auto-apply, medium suggest, low route to review.

Latency levers: small payloads, locality (region), async queues, and caching stable scores.

If you must be synchronous, prefer fast models and hard timeouts with degraded mode.

Maintain a model card: intended use, training data summary, bias checks, metrics, and limitations.

Change control: new model versions require evaluation and rollback plan.

Flow: ingest case → assemble features → score → route high-risk to investigation queue → auto-block only with approval gates and audit logs.

The score was not the decision — policy tables and investigators were.

RAG = retrieve relevant sources → inject into prompt → generate answer with citations. If retrieval is weak, generation is unreliable.

Design: improve KB quality, tag articles, measure zero-result queries, and build a knowledge flywheel (Chapter 3).

Use CMDB for: service ownership, CI criticality, dependency impacts, and change windows. Provide only the necessary fields (data minimization).

Pattern: trigger → fetch docs → chunk → embed → store vectors → retrieve top-k → generate with citations → log evidence.

Must-haves: metadata filtering, encryption, audit logs, backups, and region support.

Guidelines: chunk by sections/headings, keep 200–800 tokens typical, add overlap for continuity, store source ids and titles for citations.

Track: top-3/5 hit rate, citation coverage, hallucination rate, and user feedback/deflection outcomes.

Step 1: Choose corpus (KB for VPN + onboarding).

Step 2: Ensure tagging and AI Search relevance tuning.

Step 3: Create a flow/subflow: retrieve top-k → inject → generate → return answer + citations.

Step 4: Evaluate with 30 queries; log hit rate and citation accuracy.

Translate principles into controls: approval gates, role scoping, retention rules, evaluation, and incident response.

For PI models: evaluate errors across groups (location, org unit) where lawful and appropriate. For GenAI: evaluate tone and consistency.

Use an explanation schema: evidence fields used, rules applied, and confidence band. Store it with the record.

For GenAI, use citations and source references. For PI, use confidence and top drivers where available.

Practical approach: categorize use cases by impact and autonomy, then apply escalating controls: evaluation, documentation, transparency, and human oversight.

Establish a context budget per capability and a redaction policy for sensitive fields before external calls.

Practical: label AI-generated drafts, show citations, and provide feedback controls. For decisions, show confidence band and evidence.

A good gate is lightweight but mandatory. It produces an approval artifact for audit and future changes.

Keep it short but complete. Store it alongside configuration and version it with changes.