Exercise 4: Cross-Provider Comparison (Claude / GPT / Gemini)

Corresponds to Stage 1 — LLM Basics Exercise 4.

🎓 How to use this: starter.py is the complete solution, not a TODO skeleton. The active approach works better — mv starter.py starter_reference.py, read the signatures but not the bodies, write your own starter.py from scratch, then run python test.py to check it; if you are stuck for 20 minutes, go back and compare against the reference. Full methodology in docs/HOW_TO_USE.md.

#Why compare

Give the same "explain AGI vs narrow AI" prompt to three LLMs and the three answers come back different:

• Claude: usually leads with structure (definition → example), neutral tone
• GPT: tends to give the short answer first, then expand (type-A style)
• Gemini: tends toward lists / bullets, with lots of examples

Running it once yourself lands harder than reading a paper about it. You also get to measure three dimensions at once: tokens, cost, latency.

#How to run

pip install -r requirements.txt

# Set at least one. Any provider without a key is skipped, not crashed on
export ANTHROPIC_API_KEY=sk-ant-...
export OPENAI_API_KEY=sk-...
export GOOGLE_API_KEY=...

python starter.py

Expected output (sample):

prompt: Explain the difference between AGI and narrow AI in 1-2 sentences.
============================================================
⚠ skipping call_gemini (no API key for it)

[Anthropic / claude-haiku-4-5]  latency=823ms  in=21 out=58
AGI (artificial general intelligence) can learn and solve problems across domains; narrow AI is good at a single task...

[OpenAI / gpt-5-mini]  latency=612ms  in=24 out=49
Narrow AI specializes in a specific task (chess, recognition); AGI by contrast has...

✅ Exercise 4 passed — got responses from 2 providers; compare style / length / cost

#Validate the logic without spending money

python test.py

All 4 tests replace the SDKs with unittest.mock.patch:

✅ test_skip_when_no_key
✅ test_compare_returns_only_valid_replies
✅ test_reply_dataclass_shape
✅ test_compare_one_provider_set

🎉 全部通過 — Cross-provider 邏輯正確（skip-on-missing-key 已驗）

#Program structure walkthrough

#Common pitfalls

1. The three SDKs have very different API shapes — Anthropic uses messages.create, OpenAI uses chat.completions.create, Google uses models.generate_content. Only a shared dataclass makes them comparable
2. The token fields are named differently — Anthropic has input_tokens / output_tokens, OpenAI has prompt_tokens / completion_tokens, Google has prompt_token_count / candidates_token_count
3. A missing key should skip, not raise — production code always needs this guard; a production agent must not die entirely because one provider is down
4. Not capturing latency — you only find out who is slow after the run, and production routing needs that data

#Want more providers?

OpenRouter / Mistral / Cohere / Groq all speak the OpenAI-compatible API, so changing base_url is enough:

client = OpenAI(
    base_url="https://api.groq.com/openai/v1",
    api_key=os.environ["GROQ_API_KEY"],
)

#🦙 Path B — add a local Ollama as a fourth point of comparison

call_openai is already an OpenAI-compatible client, so swapping out base_url and model connects it to Ollama:

def call_ollama(prompt: str) -> Reply | None:
    """Local Ollama (gemma4:e4b or qwen2.5:3b). Returns None if it isn't installed, never crashes."""
    import requests
    try:
        requests.get("http://localhost:11434/api/tags", timeout=2)
    except Exception:
        return None  # Ollama isn't running
    from openai import OpenAI
    client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
    t0 = time.time()
    r = client.chat.completions.create(
        model="gemma4:e4b",
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}],
    )
    return Reply(
        provider="Ollama-local",
        model="gemma4:e4b",
        text=r.choices[0].message.content or "",
        in_tokens=r.usage.prompt_tokens,
        out_tokens=r.usage.completion_tokens,
        latency_ms=int((time.time() - t0) * 1000),
    )

Add call_ollama to the caller list in compare() and you get a 4-way comparison, including a free $0 local model. In practice you will find gemma4:e4b on CPU is typically 5-10x slower than the cloud — but its cost is 0.

#Extensions

• Cost comparison → wire in the PRICING dict from the Stage 1 pricing exercise and print a dollar-cost column
• Run the same prompt N times and average → add a for-loop inside compare() and look at the latency stdev
• Add a quality eval → bring in a fourth LLM as a judge and score each reply (Stage 7 Exercise 2 covers this)