What we learned from running a practice CPA exam through ChatGPT

What we learned from running ChatGPT through the CPA exam

May 8, 2023 9:00 AM

Accounting Today ran a practice CPA Exam through ChatGPT and found some surprising results. We needed to experiment extensively with the AI chatbot to determine its capabilities and limits, as well as try out different ways to conduct the experiment. Through this process we found there are ways to use ChatGPT to produce results that are, if not optimal, at least better than a completely unguided prompt.

Be as specific as possible

Jakub Porzycki/NurPhoto

ChatGPT is designed to understand your prompts and produce an output based on what you want. But just like a person who is given vague instructions won't do a good job, neither will ChatGPT. When forming a prompt, it's best to be as specific as possible, even if you think it sounds silly. Don't just ask it to write an email to a client about the status of their audit. Specify how long the email should be, the tone and style it should take, important details to include, specific phrasings to use, topics or areas to avoid, and other clarifying instructions. It makes the difference between producing a painfully generic message that could mean anything and a specific, detailed message applicable for your particular situation.

You probably won't get it right in one try

It is a rare thing when ChatGPT completely nails it the very first time. When you read the initial output for your prompt, it's highly likely there will be further instructions and details you forgot to add. For instance, when asking it to write a report on a recent initiative, it may deliver it in the wrong tone or the wrong format. It may narratively explain what you wanted as a bulleted list, or use a bulleted list when you wanted a narrative explanation. It may misunderstand a word or phrase specific to an industry that has a different generic meaning. Getting the desired output is often an iterative process. You can either tell it to regenerate a response, in which case it will try to answer your question in another way, or enter a modified version of your first prompt.

Remember these conversations are not private

While we're all aware internet activity is constantly monitored, remember especially that none of what you say to ChatGPT is private. Everything a user inputs is stored on the OpenAI servers, where it could potentially be used for training future models. Keep this in mind, so you don't enter sensitive information about yourself, your firm or your clients.

Set the right context and scope

It turns out, if you just copy and paste a CPA exam question into ChatGPT with no preparation at all, it will sometimes not realize it's even taking a test and go off in wild directions with its output. We needed to prime ChatGPT beforehand by telling it to assume the role of a student who is taking the CPA exam today. This illustrates the importance of beginning your chat with a clear context and define the scope of the topic you want to discuss. This will help ChatGPT provide more accurate and relevant responses.

Consider breaking down the question into multiple queries

Sometimes ChatGPT won't produce good answers if you ask a complicated question in a single prompt. By using step-by-step or multi-part questions, you can make the process more manageable for ChatGPT. For example, you might ask, "Tell me about climate change," and it will give you an acceptable answer. But the quality will likely be better if you ask several questions in a row that are more specific, like: "What are the sources of climate change?"; "What are the effects of climate change?"; "List major climate legislation from 2000-2015"; and "Describe different approaches to climate change mitigation."

Do not uncritically accept ChatGPT's answers

Keep in mind ChatGPT was trained off data from the internet, which is not exactly known for its accuracy and objectivity. It can use this data to therefore produce information that's just wrong. And that's even if it references something outside itself. ChatGPT is also capable of just making things up out of whole cloth. One Accounting Today writer asked it once the percentage of small CPA firms with fewer than 10 partners who experienced a cyberattack in the past year. ChatGPT confidently responded 40%. When asked about a source, it cited a 2021 AICPA publication. After spending an afternoon reading through virtually everything the AICPA published in 2021, it became apparent this report did not even exist. While such creativity is great for its use as a storytelling aid, it makes its research capabilities a little more suspect. Be sure to factcheck whatever it tells you.

ChatGPT's knowledge cuts off at 2021

Think of all things that have happened since 2021. Now consider that ChatGPT knows none of them. All its knowledge cuts off at 2021 and therefore if you try to ask it about anything past this point, it will just be speculation on its part. This may also affect its knowledge of things like laws, accounting standards or regulations because it won't be aware of the changes that have taken place since then.

5 principles for accountants

Daniel Street, one of the authors of the study that involved running accounting undergrad questions through ChatGPT, developed a set of more accounting-specific principles for ChatGPT use. They include:

1. Accountants should use large language models like ChatGPT as an ability enhancer for human thought and judgment, rather than a replacement. Think of it as more akin to a robotic arm than an autonomous robot.

2. Decompose complex tasks. For instance, instead of just asking it to make a balance sheet, ask it to build first the liability section, then the asset section, then the equity section over the course of three prompts.

3. Exercise professional skepticism.

4. Remember that models like ChatGPT are not very good at math and so are not very effective at quantitative tasks.

5. Provide feedback to ChatGPT to improve its functioning. If you get an output that is not correct, provide feedback by pointing out it was wrong, as well as clarifying what you want to see. You can also use the thumbs up and thumbs down buttons. ChatGPT runs on a reinforcement learning model, so the more you tell it what was and was not a good answer, the more it will adjust itself to your preferences.