arXiv
Language Models
What users actually care about when they delegate to AI—and it's not what you'd expect
When you hand off a task to an AI agent, you're not just worried about whether it *finishes*—you're worried about whether you can check it, whether it'll overstep, whether it's fair, and whether it stays in its lane. This study listened to 73,000 real conversations and found values cluster around the *conditions* users set, not the agent's outputs.
This means AI designers have been measuring the wrong things; success isn't completion rate—it's whether users feel in control and can trust what happened.
Bug reported: No