This article explores using weak labeling techniques to infer user satisfaction with AI agents by analyzing existing telemetry and trace data instead of relying on explicit feedback. By combining multiple data heuristics as proxies for user sentiment, the authors developed a classification method that helps identify successful or problematic agent interactions. This approach allows teams to gain actionable insights into agent performance even when manual user feedback rates are low.
Key points
Weak labeling generates useful training data by using heuristics or proxies when ground truth is scarce.
User telemetry, such as trace data and interaction duration, can approximate explicit feedback like thumbs-up signals.
Multiple data sources significantly improve the accuracy of weak labeling models compared to single-source heuristics.
Weak labeling does not need to achieve perfect accuracy to be valuable; it serves as an effective filter to narrow down datasets for manual inspection.
Validating weak labels against a golden dataset is essential to measure the reliability of inferred metrics.