How Skyward Chat Knows When to Trust Itself
Knowledge-graph RAG systems tend to be confidently wrong — they produce high-confidence answers even when the retrieved sub-graph is incomplete or unreliable. Skyward Chat's calibration layer, based on the Ca2KG framework, is designed to catch that before the answer reaches a user.
The Ca2KG framework
Every query runs through four stages: generate a baseline answer, stress-test it with counterfactual interventions, re-score all candidates as a panel, then select the answer that is both well-supported and stable.
From the source paper: the LLM generates an initial KG-RAG answer, two counterfactual interventions stress-test it, a panel re-scores all candidates into a 3×N probability matrix, and the Causal Calibration Index selects (or flags) the final answer.
Stage by stage
Initial KG-RAG answer
The standard retrieval-augmented pipeline runs first: retrieve a sub-graph relevant to the query, then generate a baseline answer grounded in it — same as any RAG system.
Counterfactual prompting
Two deliberate interventions run alongside the baseline: one simulates a poor-quality retrieval, the other simulates flawed reasoning over otherwise valid context. If the answer doesn't change, that's a signal of stability.
Panel-based re-scoring
All three candidate answers go through a unified evaluator that merges semantically identical responses and builds a 3×N probability matrix — how each candidate holds up across all three prompts.
Causal Calibration Index
CCI combines average support with cross-intervention stability to pick the final answer — and to flag it as low-confidence when no candidate holds up, instead of asserting a guess.