Worried that ChatGPT is coming for your job? An old assessment tool may have the answer

• Published date: 09 March 2023
• Publication title: The Korea Times

"AI passes U.S. medical licensing exam." "ChatGPT passes law school exams despite 'mediocre' performance." "Would ChatGPT get a Wharton MBA?"

Headlines such as these have recently touted (and often exaggerated) the successes of ChatGPT, an artificial intelligence tool capable of writing sophisticated text responses to human prompts. These successes follow a long tradition of comparing an AI's ability to that of human experts, such as Deep Blue's chess victory over Gary Kasparov in 1997, IBM Watson's "Jeopardy!" victory over Ken Jennings and Brad Rutter in 2011, and AlphaGo's victory in the game Go over Lee Sedol in 2016.

The implied subtext of these recent headlines is more alarmist: AI is coming for your job. It's as smart as your doctor, your lawyer and that consultant you hired. It heralds an imminent, pervasive disruption to our lives.

But sensationalism aside, does comparison of AI with human performance tell us anything practically useful? How should we effectively utilize an AI that passes the U.S. medical licensing exam? Could it reliably and safely collect medical histories during patient intake? What about offering a second opinion on a diagnosis? These kinds of questions can't be answered by performing comparably to a human on the medical licensing exam.

The problem is most people have little AI literacy ? an understanding of when and how to use AI tools effectively. What we need is a straightforward, general-purpose framework for assessing the strengths and weaknesses of AI tools that everyone can use. Only then can the public make informed decisions about incorporating those tools into our daily lives.

To meet this need, my research group turned to an old idea from education: Bloom's Taxonomy. First published in 1956 and later revised in 2001, Bloom's Taxonomy is a hierarchy describing levels of thinking in which higher levels represent more complex thought. Its six levels are: 1) Remember ? recall basic facts, 2) Understand ? explain concepts, 3) Apply ? use information in new situations, 4) Analyze ? draw connections between ideas, 5) Evaluate ? critique or justify a decision or opinion, and 6) Create ? produce original work.

These six levels are intuitive, even for non-experts, but specific enough to make meaningful assessments. Moreover, Bloom's Taxonomy isn't tied to a particular technology ? it applies to cognition broadly. We can use it to assess the strengths and limitations of ChatGPT or other AI tools that manipulate images, create audio...