Exemplar Learning

What is Exemplar Learning?

Exemplar Learning is StackFlow's mechanism for continuously improving AI quality by learning from real resolved cases. When an incident is resolved with a high customer satisfaction score (4-5 stars), StackFlow captures the incident description and resolution as an exemplar pair. These exemplar pairs are used as few-shot examples in future AI prompts for similar incidents, teaching the model your organization's specific resolution patterns.

⚙️ Minimum Requirements

DynamoDB: StackFlow_AIExemplar table with at least 10 approved exemplar records for meaningful few-shot learning
DynamoDB: StackFlow_AIExemplar GSI on intentType and qualityScore for ranked retrieval
IAM: StackFlowAPIRole with dynamodb:Query on StackFlow_AIExemplar GSI
Feature Flag: exemplar_learning enabled in StackFlow_FeatureFlag

Unlike fine-tuning (which modifies model weights and requires significant data and compute), exemplar learning uses in-context learning — the examples are injected into the prompt at inference time. This means exemplars can be added and removed instantly without any retraining.

Exemplar Collection

Exemplars are collected automatically when an incident is closed with a satisfaction rating of 4 or 5 stars. They can also be manually nominated in AI → Exemplar Learning → Nominate Exemplar. Each exemplar is tagged with the incident's category, subcategory, and relevant CIs to enable targeted retrieval.

curl -X POST https://your-instance.stackflow-tech.com/prod/api/ai/exemplars   -H "Authorization: Bearer $ADMIN_TOKEN"   -H "Content-Type: application/json"   -d '{
    "source_incident_id": "INC0001234",
    "categories": ["database", "connectivity"],
    "exemplar_type": "resolution",
    "quality_rating": 5,
    "notes": "Clear, reproducible steps. Has been validated by 3 similar subsequent incidents."
  }'

Few-Shot Injection

At inference time, the AI engine retrieves the top 3 exemplars most semantically similar to the current request (using Bedrock vector search) and injects them into the prompt as examples. The injection follows the format: "Here are similar past incidents and their resolutions:" followed by the exemplar pairs. This few-shot approach significantly improves resolution suggestion quality for known problem categories.

Exemplar Retrieval: Exemplar retrieval uses the same Bedrock Knowledge Base (BXJGG7PIPS) as general KB search. Exemplars are stored as a special document type with metadata tags that filter them from end-user visible KB search results.

Exemplar Quality Management

Monitor exemplar quality in AI → Exemplar Learning → Quality Dashboard. Key metrics include: exemplar count by category, exemplar utilization rate (how often each is injected), and impact on AI suggestion acceptance rate. Stale exemplars (not used in 30+ days) are highlighted for review. Outdated exemplars should be retired to prevent the model from learning outdated resolution steps.

Privacy Considerations

Exemplars may contain information from incident descriptions. Before collecting an exemplar, the system automatically scans for PII (using the Bedrock Guardrails PII detector) and redacts detected values. Exemplars containing sensitive data about specific users are flagged for manual review before being added to the active exemplar pool.

← Previous

Semantic Cache

Redis-backed query caching

Architecture Overview

Cognitive graph RAG pipeline