Source note · Independent commentary
When AI helps until the task changes
It shows that a tool's competence is uneven in ways that are hard to see from inside a task, which makes knowing where the tool fails a form of experience in its own right.
Research reviewed
Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality
- Authors
- Fabrizio Dell'Acqua, Edward McFowland III, Ethan Mollick, Hila Lifshitz-Assaf, Katherine Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, Karim Lakhani
- Institution
- Harvard Business School
- Publication
- HBS Working Paper 24-013; forthcoming in Organization Science
- Date
- 2023-09
- Type
- Working paper
A preregistered field experiment with several hundred management consultants. On tasks inside the model's capability, consultants using GPT-4 finished more tasks, faster, and at higher rated quality. On a task designed to sit outside that capability, those using the tool were more likely to get the answer wrong.
Read the original sourceWhat the source supports
- AI capability is uneven across tasks that look similar in difficulty. The authors call this the jagged frontier.
- For tasks inside the frontier, consultants with the tool performed measurably better on speed and rated quality.
- For a task outside the frontier, tool use was associated with worse accuracy.
What the source does not support
- That the frontier is fixed. The authors describe it as moving and hard to see from the inside.
- That professionals can reliably tell in advance which side of the frontier a task sits on.
- Any general claim about consulting careers, hiring, or the value of experience over time.
What Sougata is doing with it
Sougata uses this to support one specific point: the segment of a workflow that a tool handles well is not the whole workflow, and knowing where the tool fails is itself a form of experienced judgment. The framing of workflow segment versus judgment is Sougata's, not the paper's.
This is an independent Voyagers Beyond source note. Voyagers Beyond is not affiliated with or endorsed by the authors, publishers, universities, organizations, or institutions referenced here unless explicitly stated otherwise. The summary and analysis are written independently; nothing from the work is reproduced.
Why it matters here