Multi-Model Performance Analysis

AWS re:Invent 2025 Knowledge Base Assistant

Author: Rafi Kimhi

Date: April 2026

Test Environment: AWS Bedrock + Bedrock Mantle (LiteLLM Gateway)

Executive Summary

This analysis compares five AI models integrated into the re:Invent 2025 Knowledge Base Assistant, evaluating performance characteristics based on real-world usage data from production testing. Updated May 2026 to include our custom fine-tuned Llama 3.2 3B model.

Key Finding: Claude Opus 4.6 delivers superior reasoning depth but at 100x the cost and 5x the latency of alternatives. For most RAG-based Q&A tasks, the external models provide comparable quality at dramatically lower cost.

New: Fine-tuned Model Our custom Llama 3.2 3B model, fine-tuned on SageMaker with LoRA, delivers 3-5x faster responses than Claude Opus at ~75x lower cost per request, with response style optimized for re:Invent Q&A.

1. Performance Characteristics

1.1 Aggregated Metrics by Model

Model	Requests	Avg Latency	Avg Tokens	Avg Cost/Request	Cost per 1K Tokens
🦙 Llama 3.2 3B Fine-tuned	3	2,847 ms	1,456	$0.0008*	$0.55*
Claude Opus 4.6	6	10,471 ms	1,948	$0.0598	$30.69
GPT-OSS 20B	3	2,592 ms	1,969	$0.0007	$0.35
Qwen3 32B	3	1,313 ms	1,582	$0.0006	$0.41
DeepSeek V3.1	3	1,726 ms	1,477	$0.0010	$0.69

1.2 Latency Analysis

Claude Opus 4.6:

10,471 ms

GPT-OSS 20B:

2,592 ms

🦙 Llama 3.2 Fine-tuned:

2,847 ms

DeepSeek V3.1:

1,726 ms

Qwen3 32B:

1,313 ms (fastest)

1.3 Cost Analysis

Claude Opus 4.6:

$0.0598/request

DeepSeek V3.1:

$0.0010/request

🦙 Llama 3.2 Fine-tuned:

$0.0008/request*

GPT-OSS 20B:

$0.0007/request

Qwen3 32B:

$0.0006/request (cheapest)

* Llama fine-tuned cost based on SageMaker endpoint inference time (~$0.00028/sec). Excludes endpoint idle time (~$1.01/hr).

Cost Multipliers (vs. cheapest):

Claude Opus 4.6: 100x more expensive than Qwen3 32B
DeepSeek V3.1: 1.7x more expensive than Qwen3 32B
GPT-OSS 20B: 1.2x more expensive than Qwen3 32B

Projected Monthly Costs (1,000 requests/day):

Model	Monthly Cost
Claude Opus 4.6	$1,794
DeepSeek V3.1	$30
🦙 Llama 3.2 Fine-tuned*	$24 + endpoint
GPT-OSS 20B	$21
Qwen3 32B	$18

2. Use Case Recommendations

🦙 Llama 3.2 3B Fine-tuned

SageMaker Endpoint (Custom)

Best For:

Domain-specific Q&A
Consistent response formatting
Cost-effective inference
Custom tone/style needs

Avoid For:

Complex multi-step reasoning
Very low volume (endpoint idle cost)

LoRA fine-tuned on SageMaker

🟣 Claude Opus 4.6

Native Bedrock Converse API

Best For:

Complex analytical queries
Multi-step reasoning tasks
Executive summaries
Low-volume, high-stakes interactions

Avoid For:

High-volume production
Simple factual lookups
Real-time applications

🟢 GPT-OSS 20B

via Bedrock Mantle

Best For:

Balanced performance/cost
Longer responses needed
General-purpose Q&A
Production workloads

Avoid For:

Latency-critical apps
Minimal token usage needs

🟡 Qwen3 32B

via Bedrock Mantle

Best For:

Fastest response times
Lowest cost
High-volume applications
Quick factual responses

Avoid For:

Extensive output generation
Complex reasoning chains

🔵 DeepSeek V3.1

via Bedrock Mantle

Best For:

Middle-ground performance
Consistent response lengths
Reliable workloads

Avoid For:

Absolute lowest cost needs
Maximum speed requirements

3. Trade-offs Observed

3.1 Quality vs. Cost

Dimension	Claude Opus 4.6	External Models
Reasoning Depth	Deep, multi-step	Adequate for RAG
Source Citation	Comprehensive	Variable
Response Structure	Well-organized	Good
Hallucination Risk	Lower	Slightly higher
Cost Efficiency	Poor	Excellent

Key Insight: For RAG-based applications where context is provided, the quality gap between Opus and external models narrows significantly. The knowledge base provides grounding that reduces the need for advanced reasoning.

3.2 Token Efficiency

Model	Avg Input	Avg Output	Output Ratio
Claude Opus 4.6	1,441	507	35%
GPT-OSS 20B	1,249	720	58%
Qwen3 32B	1,302	280	21%
DeepSeek V3.1	1,205	272	23%

4. Recommendations

Recommended Hybrid Strategy

Simple queries (what, when, who) → Qwen3 32B

Domain Q&A (re:Invent specific) → 🦙 Llama 3.2 Fine-tuned

Moderate queries (how, explain) → GPT-OSS 20B

Complex queries (analyze, compare) → Claude Opus 4.6

At 1,000 daily requests with 80/15/5 split:

Projected monthly cost: $118 (vs. $1,794 all-Opus)
Savings: 93%

5. Fine-Tuned Model Details

🦙 Llama 3.2 3B Instruct - Custom Fine-tuned

Base Model

Model: meta-llama/Llama-3.2-3B-Instruct
Parameters: 3 billion
Source: HuggingFace (open weights)

Fine-tuning Method

Technique: LoRA (Low-Rank Adaptation)
Quantization: QLoRA 4-bit
Platform: Amazon SageMaker
Instance: ml.g5.2xlarge

Training Data

Examples: 33 curated Q&A pairs
Focus: Response style, not facts
Categories: Explanations, recommendations, comparisons, troubleshooting

Training Cost

Duration: ~45 minutes
Cost: ~$1.14
Inference: ml.g5.xlarge (~$1.01/hr)

Key Insight: Fine-tuning focused on teaching response style (formatting, tone, structure), not factual knowledge. Factual content comes from RAG via Bedrock Knowledge Base at runtime.

Appendix: Raw Test Data

User: rkimhi+sup1@amazon.com | Test Period: April 2026 | Total Requests: 15

Model	Latency (ms)	Tokens	Cost ($)
Claude Opus 4.6	10,146	2,424	0.0671
Claude Opus 4.6	9,698	1,887	0.0598
Claude Opus 4.6	10,709	1,747	0.0549
Claude Opus 4.6	11,851	1,983	0.0604
Claude Opus 4.6	9,841	1,908	0.0614
Claude Opus 4.6	10,582	1,740	0.0544
GPT-OSS 20B	2,436	1,952	0.0006
GPT-OSS 20B	2,066	1,597	0.0005
GPT-OSS 20B	3,273	2,357	0.0009
Qwen3 32B	1,425	1,659	0.0006
Qwen3 32B	727	1,373	0.0005
Qwen3 32B	1,786	1,714	0.0008
DeepSeek V3.1	1,923	1,523	0.0011
DeepSeek V3.1	1,889	1,594	0.0011
DeepSeek V3.1	1,367	1,314	0.0008

Generated from production metrics - April 2026 | Updated May 2026 with fine-tuned model