Understanding Analysis Reports

Merit Analysis Report
─────────────────────────────────────
Total Tests:     150
Passed:          125
Failed:          25
Error Clusters:  5
Analysis Date:   2024-12-26 14:30:15

Cluster 1: API Timeout Errors (12 failures)
Pattern: "Request to .* timed out after \d+ seconds"

Problematic Code:
─────────────────
File: src/api_client.py
Lines: 45-52

Suggested Issue:
HTTP timeout set too low (5 seconds) for slow endpoints.

Recommendation:
- Increase default timeout to 30 seconds
- Add configurable timeout per endpoint
- Implement retry logic with exponential backoff

Affected Tests:
- test_api_slow_endpoint (line 23 in test_api.py)
- test_api_large_payload (line 45 in test_api.py)
- test_api_concurrent_calls (line 67 in test_integration.py)

Cluster 2: Hallucination in Summaries (8 failures)
─────────────────────────────────────────────────

Pattern: "Unsupported facts found in summary"

Problematic Behavior:
Generated summaries contain information not present in
the source documents. The model is adding facts that
aren't grounded in the provided context.

Problematic Code:
File: src/summarizer.py
Lines: 23-35

def generate_summary(text: str) -> str:
    # Issue: No fact-checking step
    prompt = f"Summarize: {text}"
    return llm_call(prompt)

Recommendation:
1. Add fact-checking step after summarization
2. Use has_unsupported_facts() predicate
3. Implement RAG pattern with source attribution
4. Lower temperature to reduce creativity

Affected Tests:
✗ test_article_summary (test_summarizer.py:12)
  Input: "Paris is the capital of France"
  Output: "Paris, France's capital, has 50 million people"
  Issue: Population is hallucinated

✗ test_news_summary (test_summarizer.py:28)
  Input: "Tech company announces new product"
  Output: "Company announces product and plans IPO next month"
  Issue: IPO information is hallucinated

Cluster 3: JSON Parsing Failures (15 failures)
──────────────────────────────────────────────

Pattern: "JSONDecodeError.*line \d+ column \d+"

Problematic Behavior:
LLM returns invalid JSON format. Missing quotes, trailing
commas, or malformed structure.

Problematic Code:
File: src/json_generator.py
Lines: 18-22

prompt = "Return JSON with name and age"
response = llm_call(prompt)
data = json.loads(response)  # Fails here

Recommendation:
1. Add JSON validation prompt:
   "Return ONLY valid JSON, no markdown, no explanations"
2. Use JSON mode in API call (if supported)
3. Add retry with format correction
4. Consider structured output libraries

Affected Tests:
✗ test_user_json (test_json.py:15)
  Output: "```json\n{name: 'John'}\n```"
  Issue: Markdown wrapper and unquoted keys

✗ test_profile_json (test_json.py:30)
  Output: "{name: \"Jane\", age: 25,}"
  Issue: Trailing comma

File: src/api_client.py:45

Cluster 1: API Timeouts (12 failures)    ← Fix this first
Cluster 2: Hallucinations (8 failures)
Cluster 3: JSON Errors (3 failures)       ← Then this

Recommendation:
1. Increase timeout to 30 seconds
2. Add retry logic
3. Implement circuit breaker pattern

Problematic Code:
File: src/api.py:45    ← Click to open

# Share via email, Slack, or store in S3
merit-analyzer analyze results.csv --report team_report.html

# Add timestamp to filename
DATE=$(date +%Y%m%d_%H%M%S)
merit-analyzer analyze results.csv --report "report_${DATE}.html"

- name: Upload Analysis
  uses: actions/upload-artifact@v3
  with:
    name: merit-analysis
    path: merit_report.html