AI bot passes exam at business school

Professor Christian Terwiesch of the Wharton School of Business was sitting with his grown children around the dinner table when the subject of artificial intelligence came up. Both of his kids had been experimenting with the nascent technology in their respective fields: “one of them is interested in design … and the other one is interested in computer science.”

Eventually his son prompted ChatGPT to explain a sorting algorithm “using terms from Homer Simpson.”

“It was funny,” Terwiesch said, adding, “it was, from a computer science perspective, correct, and it was so easy as a user interface.”

ChatGPT users can input any prompt — The Inquirer previously asked it to write a story about Gritty — and the tool will respond nearly instantly with an answer written so clearly that it would be easy to believe a human had written it instead of a computer. Some examples show ChatGPT’s ability to write essays, legal briefs, and even full songs that mimic an artist’s writing style.

Terwiesch wanted to test it himself. He fed ChatGPT questions from the final exam for his operations management class — a subject he literally wrote the book on — to gauge whether the tool could pass an MBA-level course.

And it did. Terwiesch rated the bot’s “performance as a B to B-,” but noted in his research summary that some answers contained “surprising mistakes in relatively simple calculations at the level of sixth-grade math.”

He said those mathematical shortcomings prove that we’re far away from using the current technology to replace trained professionals.

“Imagine a medical professional making a decision of what dosage of a drug to give ... If you would automate it as an investment advisor, this thing would be total garbage,” Terwiesch said. In these cases, professionals need to “stay away from this technology.”

Finding utility in an experimental technology

Terwiesch, who teaches in Wharton’s MBA program, encourages fellow educators to consider “opportunities where we can think about improving our learning process” through using A.I. tools in the classroom. That could include prepping testing materials — Terwiesch asked it to write new exam questions to some success, according to his research — and other ways to lighten educators’ workloads across every grade level, including elementary and public schools.

“We all see and admire how all these teachers come to work and work their butts off in a very difficult work environment, and so I think we have to find ways to use technology to help them as opposed to just kind of worrying about the cheating point,” Terwiesch said, referring to instances where students have already used ChatGPT to cheat on tests, a concern for many critics.

When it comes to things like taking a driver’s license test or a CPA exam, Terwiesch supports an outright ban, because the purpose of those exams is to certify the test taker is qualified to perform particular skills. But the purpose of teaching, and of learning, is to “engage with the material,” not simply recite it.

Terwiesch remains optimistic that tech tools can be used for good in the classroom.

“Our job as educators is to use the technology to engage [students] differently,” Terwiesch said. “We have to find ways in the curriculum where the deepest skill that we really want to teach is taught in a novel way, because if, at the end of the day, we end up as in a status that is as good as prior to the release of ChatGPT, I think we have missed an enormous opportunity.”