tl;dr: LLMs generate confidence scores as plausible text continuations, not as measurements of evidence or calibrated probabilities—so the numbers shift with rewording, vary between runs, and have never been scored against actual correctness. A trustworthy confidence score requires both grounding in observable information and calibration against recorded outcomes. The author suggests building an external rule that derives confidence from measurable signals (e.g., search hits, answer consistency, historical accuracy) and continuously checking it against reviewed results, rather than asking the model how sure it is.