Fact Checking AI in Tax Research: Why AI Powered Tax Research Needs Human Judgment

This transcript has been lightly edited for readability.

EPISODE 02: Fact-Checking AI in Tax Research: Why AI Powered Tax Research Still Needs Human Judgment

Sean Bilodeau: That’s just what it told me. If that’s your answer, when a client, a reviewer, or an IRS agent ask why you’ve got a problem.

AI can put a polished, confident-sounding answer in front of you in seconds. But polished isn’t the same as correct. And it’s definitely not the same as complete.

Welcome to 20-20 Insights and Conversations. I’m Sean Bilodeau. Today I’m joined by my colleague Scott Orwig, Senior Manager of Tax Learning at 20-20 Services, who runs his own tax practice and has gone all in on AI.

We’re talking about fact-checking AI and tax research. Why? The research framework hasn’t changed, even though the tools have the traffic light approach to deciding when AI is safe to use and the habits you trust the output without getting burned.

Let’s get into it. And I am joined by my colleague Scott Orwig, Senior Manager of Tax Learning at 20-20 Services, who also clearly got the memo today, and that we’re wearing the same pullovers and quarter-zip.

So I’m excited to see you in the same quarter-zip.

Scott Orwig: I know right. I mean, just when fate calls fate calls.

That’s what I’m saying Sean.

Sean: Yeah. There you go. So as we want to get to know you a little bit before we get into the conversation of fact-checking AI in tax research. So as a proud Johnson & Wales alum, how did you how have you gotten to where you are today from? What’s your story?

Scott: You know what Sean, It’s funny because I remember the whole reason I went to Johnson & Wales in Rhode Island because originally I’m from Colorado, and that’s where I live now as well, just right outside Denver. But long story short, I’m a huge Red Sox fan, and I remember having the opportunity to talk to the CFO of the Rockies one day and just ask him, I want to be a general manager one day, so what do I have to do? And he said, I get an accounting degree or go to law school. I’m like, I’m not doing law school, so I’ll get my accounting degree, go to Johnson & Wales in Rhode Island because I’m like, hey, that’s close enough to Boston, right? I’m going to be the GM there one day.

And I started taking these accounting courses, and all of a sudden I was just like, wow, this makes sense. Like this feels very natural to me. I love the way that you logistically think through these transactions, how you record them and then really understand the language of business. That’s accounting, right? And after that, one of my great professors, her name, Professor Kanata. Love ya, she actually introduced me to the tax director at 20-20 Services. His name is Tom Kiley, but he has his own tax practice. And when I was in college, I was able to intern with them for 2 or 3 years, and then after that returned back to Colorado, worked at what was then Heine Associates, which was then acquired by Moss, who was now Baker Tilly. Right. The train of acquisitions.

From there I really explored a lot of different areas of taxation, really fell in love with partnership, taxation, estate, gift and trust. And yeah, I mean, it’s just all of a sudden, Sean, I just became more and more and more infatuated with the idea of if someone comes to me with a question, comment or concern, I want to be able to figure it out. I want to be that person, right? So I took that approach to Deloitte, and then after that, my my sense there, I came to 20-20 Services. I had my own tax practice in conjunction as well. So I’m still very much so in it to win it. So that’s a little a little spiel about where I’ve come from.

Sean: You know, the Rockies may be looking for another GM at some point in time. So don’t ever give up on those dreams.

Scott: I might have to give up on the Rockies because the Rockies gave up on us.

Sean: Yeah, it’s a fracture has occurred between fracture has now.

Scott: Exactly.

Sean: Well, what I like to start off every episode with, after getting to know you a little bit, is I always like to call the segment two minutes that matter to you. So kind of an overview of what we’re going to talk about today with fact-checking AI. So why should the listener of this episode care about fact-checking AI research?

Scott: The first thing that comes to mind, right. There’s so many different reasons to care, but the first thing that comes to mind is that it’s here. The genie has left the bottle. It’s not going to get put back into the bottle. So whether or not you decide to use AI or not, right. That is obviously a discussion still being discussed today, but it’s here, right? If you’re going to use it, or if you know someone is using it’s right, doesn’t even have to matter if you’re the one using it.

How do you know if the output is correct or how do you actually utilize the output? Right? So it’s not just is this right, but how do we use it and how does this fit into what we’ve traditionally have done right? I’m a big believer, Sean, that the framework of tax research has not even changed minutely. But the tools that we’re using now, the tools that are available, that’s what’s changing. And that’s going to be the big impact that we have to really discover. We really have to go through and understand. And and it’s not just a platform. It’s not just a tool, right. It’s multitude. So how do we how do we do this with confidence?

Sean: One of the things that’s really interesting to me and you kind of brought up. Right. It’s here. It’s it’s not going away. It’s a tool that’s being used. And you look at across the landscape of the industry, right. Firms level of investment in artificial intelligence is significant. Even you look at the bigger firms. Right.

They’re putting in billions of dollars into AI tools, even working your way down in the top 100 firms, you’re going to find some use of AI at that specific area. And so as you kind of talk about how do we utilize it in thinking about how firms are investing heavily at AI, how do you think how well do you think people are using it on the ground right now?

Scott: That’s your another good question. And it depends. Right? It depends on the firm, depends on the investment and it depends on the methodology. Right. There’s certain firms because again I’ve seen I’ve seen this a lot with just the clients that we have. Right where big investment is put in, big announcements are being made as well. But then the idea is, well, yeah, all these college kids that are coming in, they just know how to use it. So it’s going to be fine. And that’s dangerous, right? Because there’s there’s so many different things that are missing there.

How do you measure the AI’s effectiveness, the usefulness? How do you actually see the value it’s creating if there’s no benchmarks being made, if there’s no way of measuring this, how do we have safeguards if we don’t even define what we’re trying to safeguard? Right. So that’s one thing that I’ve, I feel like is out there, there are then the other firms where they’re taking a much more creative aspect to it, where again, there’s not really a definition of what’s being used, how to use it, but they’re doing maybe challenges for creating workflows and processes and agents, which is great. But again, how much oversight is there? I don’t know. And then there’s I think a word in the majority of firms land, okay, we have a platform that we’re using now. What. Right. And it’s going to be this process that’s going to be developed over time. Right. So it really depends where they’re at because I’m sure there are there are some firms that are that the governance in place, they have the process in place. We have we have certain clients that we teach at where there might be a certain department where they have a very defined and clear structure of how, I like to call it.

I call it green light, yellow light, red light. Green light. Just grip it and rip it right. You can just use it like there’s really no oversight. You’re good to go. We’ve defined that task. Whatever it is, you can use it. I would say probably a common example that might be drafting in an email. Right.

Yellow being more of the well, you can use it, but there needs to be some sort of review QC there needs to be some caution, but proceed with caution. Right. Maybe that’s drafting memos. Maybe that’s reviewing documents. Maybe it’s reviewing work papers. And I would say in our case, doing performing tax research can very much exist there.

And then the red light being do don’t even think about it. Don’t do even think about it. Right. So it’s just it’s a lot of firms aren’t just in a lot of different places right when it comes to this.

And there’s just there’s there’s just not a roadmap Sean. Right there. Just the and that’s one of the most unique things about this is that we’re all just trying to figure out together. How do we actually implement this?

Sean: I love that you presented it in the green light, yellow light, red light, because I think anything that can provide a simplification of where we are with regards to using AI tools, I think is extremely helpful. One of the things you mentioned that really stuck out to me is you talked about a lot of these younger folks that are coming on board, you know, with the way the industry is changing, you’re seeing in a lot of respects associates that need to act more like seniors and advisors and be able to, you know, whether it’s audit, whether it’s tax, be able to use the AI tool, evaluate what the outputs of that AI tool are, and then be able to make decisions based upon that. But then I go back to what you said to about there is really no roadmap or no oversight.

So if I’m looking at it on one spectrum, we’re expecting these 21 to 25 year olds to come in and know what they’re doing right. But we don’t have a whole lot of oversight. Right? So when we think about that, like, how do we even consider closing that gap or bridging it a little bit when we get into audit and tax, you know, evaluations or procedures or standards or work?

Scott: I think one of the places you have to start first is that we have to speak the same language. Right. And that can be different for each firm. But in order to close the gap, we all have to figure out how to minimum when we speak about something, when we’re discussing something, we’re trying to create a process for something.

How are we actually going to use the same verbiage and how do we make something measurable, right. For example, if if I am going to talk about we need an agent to create this workflow process, what is our firm definition of an agent? If you there’s so many different versions of an agent out there.

Right. Like my general definition being a or you’re using AI and it has a level of autonomy. Right. But that’s not how everyone uses that language. Right. So how do we define what it is. And then how do we define some sort of governance, some sort of process that allows the creativity to flow or sell, putting it in a constraint that we’re comfortable with. Right. And that’s a that’s a major challenge that we’re all you really running into. And there are there are third parties that that’s their whole solution. Right. Sean and I could be vibe coding over here. And also they’re great things but they’re you’re using, you know, ChatGPT or Codex. I’m using Claude Code and we’re we’re just again, we’re just creating. Right. Yeah. Who knows. There are platforms, our third parties that can help bring the governance. Right. You you upload the code and it goes to the same place and it has the same structure.

And like so there are certain third parties like that that exists to help mitigate some of these issues. But at the same time, right. That’s not going to be the that’s not the fit for everyone. And honestly, a lot of firms probably don’t want to bring in extra third parties. Right? You know, it’s very easy with AI to be like, all right, we need this platform and then this platform and all of a sudden Sean, You have 12 AI platforms, right?

I’m using that very vaguely, but that’s also what we’re trying to go away. Go away from that. Have that be our reality. So at the end of the day, we just have to identify the same issues, use the same language to to really I mean, if you don’t do that. How how can you even talk about bridging a gap if you can’t even explain what the gap is?

Sean: Speaking of kind of speaking the same language and bridging the gap, let’s drill down a little bit to to fact-checking AI and tax research.

I want to start with kind of the process of tax research. So how has AI changed your day to day research process and what’s changed about how much you trust what comes out of the other end from the AI tool?

Scott: Yeah. So like I mentioned earlier, I don’t think the framework has necessarily changed that much. Right. When I’m thinking about performing tax research, I’m thinking of genuinely what is the actual task at hand. What are the different fact patterns? Different client fact patterns. What are what’s the primary sources I can use to create position? What are the editorial materials or secondary sources? Maybe helps educate me to understand the primary sources or how things can be applied in a situation that maybe you don’t think of initially, and then you just kind of do the cycle over and over and over again, because a lot of times you get to one part of your research is like, well, congratulations, Scott. You have seven more questions to answer now, and we really have to drill this down.

And so that that process doesn’t change. What I have noticed from my personal use, both with my tax practice and with the content that we create and the trainings that we’ve put on, is the access to information, just the genuine ability to access information in a much, much condensed time period is here. That is one of the greatest things about tax research. I mean, an example, Sean, is earlier this year, I had a client where there was a foreign tax credit question, right. And had to do with a different country. Obviously there’s treaties involved. It’s a little bit more of a specific and nuanced pattern. And usually that would have taken me let’s, let’s just call it a full day of really diving in, sussing things out, defining, writing, concluding work. Paper references just the whole nine yards. And I was I probably was able to break that down into a few hours, Sean, because like I had it helped me grab all the appropriate documentation, some great editorial materials that really helps me like, oh, wow.

Like I answered a lot of questions. I can see where that backs into the primary sources. It grabbed all the primary sources when I went back and review it. I’m like this, okay, like this. This feels good. This feels correct. I’m going to try to push this a little bit more further. Wow. We have this looks good. Right. So I in a lot of ways, you know, maybe I saved six hours, just for one project because I was able to get all the information aggregated so much quicker and I was able to get my questions answered so much quicker. But the one thing I hope you also hear me saying, Sean, right, is I checked, I validated, I went back and I actually looked. Are they using the right citations? Are we using the right code? You know, maybe I need to actually read a little bit more around the reg. I need to read more around the code to make sure that there isn’t anything that’s being missed. Right.

Sean: It’s interesting that you mentioned that because I see people in different context using AI tools on a regular basis. And as you kind of stated at the end of that, right, you’re going back, you’re checking primary sources, you’re checking everything that’s there. And while AI can make things a lot faster up front because of the information it can provide you, there’s also, in some ways, what I would call an ability to say, you know what? It’s provided me all this information. I don’t need to do those extra steps.

So when you when you see people in training programs that you’re, you’re interacting with or talk to people who are working at these different firms, where do they maybe stop a little bit short of doing what they need to do from from a fact checking standpoint, when they get all that information?

Scott: You know, a really great way of checking to see if we’ve gotten to a place is are they actually answering the why? Tell me more about that. So I get a response. It looks great if you put that extra question. And honestly this happens all the time. So why okay. So this is your response, your answer. Why do you believe that this fits. And tell me three primary sources that’s going to back you with that. Honestly Sean, it is.

A lot of times you might have about 70% of people, but the deer in the headlights of like, I don’t know, that’s just what it told me, and and that’s, that’s the whole point because you’re 1,000% correct. It is incredibly tempting. You get this response looks so nice. And then you’re like, boom, that’s the answer. Have a nice day. But if you can’t answer why that output is correct, if you can’t, actually, I always tell people, can you look into the eyes of a judge? Can you look into the eyes of an IRS agent and confidently plant your flag? If you can’t do that, you haven’t answered the why and you need to go back.

So that’s probably the biggest thing right there. Why? And give me just give me three sources and maybe maybe there’s not that many sources. Right. The the Internal Revenue Code, it’s very nuanced area. But you should you should be able to back into a why why not or it depends. Yeah.

Sean: So you know being able to go in there and really it’s it’s really that critical thinking piece and that professional judgment that we’re talking about here, where you’ve got to lean into that heavily in terms of what you’re what you’re getting out for for an output. Because if you only get 70% of the answer, right, how much more work are you creating for yourself in the end? Because you’re having to go back and you’re you’re you’re going to have to do this at some point in the process anyway. So why not just do it up front?

Scott: Well, and this idea, Sean, I’m like, we’re not, we’re not I’m not an accountant I’m not a tax advisor. I’m a trusted advisor. If someone’s giving me 70% of the correct answer, 70% of the truth, 70% of whatever, that’s not fantastic right there.

That it’s not just what’s the extra work I’m going to have to do? It’s also how much trust am I going to violate in inadvertently And like, I mean, I know I’ve had to do this and I’m sure you’ve had to as well. But you know, when you have when you made a mistake or when you’ve said something wrong and you have to just look the client in the face or write the email, whatever it may be, and say, boy, I, I quoted this incorrectly, I used this incorrectly, or I gave you outdated information or I’m just wrong.

You know, sometimes there’s a little grace, especially when you built up the equity. Maybe it’s something that’s incredibly complicated, but I’ve had I’ve had moments in my career, Sean, where it was. I was just too lazy. I was honestly probably just a little too lazy. I didn’t look into it far enough. And basically having to tell the client that it’s hard, right? It’s very difficult. And I think those opportunities, those traps, I’m going to say, not opportunities, those traps are going to be so much more common unless unless we take that extra step to make sure that we’re bridging the gap. Right. Because the output might not be wrong, but to your point, it might not be complete.

Sean: So it’s kind of like if someone tells you, gives you a statement, but they, you know, they didn’t lie to you, but they didn’t get the whole truth either. Right?

Scott: Right. Exactly.

Sean: So so I want to understand from your perspective, because you have the opportunity to connect with a lot of individuals across the industry, and you have a chance to, you know, be in classrooms and teach them about this process.

So as you’re kind of laying out the process, what how do you really instill it upon the folks that are in your in the classes that you’re teaching about going that extra step, how do you set up your classes, or how do you even set up your case studies to make sure they’re doing the steps necessary?

Scott: So one of the great things that we do, at 20-20 Services for our tax trainings, we have a tremendous amount of AI exercises that embedded throughout our course. Including our we have two different tax research offerings that we give. And so depending on the AI exercise or topic, it depends. Right. But we do have certain exercises where we go through the actual framework, not just a framework for tax research, but how do we use AI and how do we, for example, like how do we prompt engineer something that can really help gather, not just gather the information, not just really clearly defining the the AI’s identity and the purpose, but how do we also make sure that we do some sort of automatic check? You know, maybe, for example, it’s at the end of the prompt.

One thing I love to put in there is a devil’s advocate type of line item where I’m going to have after it’s given me all this information, I’m going to ask the AI, okay, why is this incorrect and why is everything you just gave me dead wrong. So not just incorrect. I also want the dead wrong reason and that helps challenge what AI has produced. It’s also helping me challenge myself, right? Or you can ask, it’s hey, what are seven different client follow up questions to this? Right?

We can include that all in the same prompt. Or what are the different items in this fact pattern that I didn’t ask for isn’t addressed in this response. Right. There’s so many different creative ways of doing that. And that’s something that we actually get to play with when in the classroom setting.

But one thing I would also say is one thing I like to do, I like to challenge, especially in the tax research courses we do. I do like the challenge. Okay. Yes. This is the client fact pattern. This is you know, we’re kind of guiding you where we eventually want to get to. But this is still really the question that we’re trying to answer. And there’s no right or wrong necessarily, Sean, because there are participants that can give a great answer as to yes.

And some they’re like, you know, maybe it could be X, Y, and Z, right? We might do a tax research question on interest expense limitations. But maybe the real question isn’t about interest expense limitations. Maybe the client has a cash flow constraint and need and that’s something we should look for. Maybe their over leveraged or maybe not leveraged enough. Right. So even though we might not have them have the research all the way down to that level, it’s still asking.

It’s still that just curious thought of, is this it?

Sean: One of the things that stuck out to me more than anything in your response is, is the concept of curiosity.

And I think where that really showed up in the example that you provided is you will go back to the AI tool, that idea of playing devil’s advocate, right? That’s that’s an additional layer or an additional step that most professionals may not think to go to.

So you’re using AI tool as a critical thinking tool to say, okay, tell me, tell me where I may be wrong, either in the research or in my interpretation of the research.

And that really strikes me as interesting because, again, it’s something that someone can take away that may not be normally as part of their of their normal thought process.

So if you’re thinking about that as kind of an extra step, what are some extra steps that folks may be able to take in fact, checking the research that they get using AI, that may be that may not seem obvious to them as they’re going through and doing this.

Scott: So there’s one thing I will always, always put out. There is somehow have some sort of check. Right. So the Devil’s advocate is an example of kind of a self check. Right where I’m asking AI to tell me why it’s wrong. Right. And which is obviously counterintuitive. And the AI, they will usually respond. Right. But I also love, for example, a an authority table. Right. All right. So for every citation you use in your response, tell me what is it.

Why did you use it. Why it’s applicable and where’s the limitation. Right. So it’s kind of doing an auto QC of itself or with secondary information. What’s what is it. Why is it use. Where does it support the primary information. Right. The primary sources like where does it to. Where’s the limitations. Right. So those are some of the things that I love to do, I love to implement in there. And honestly one of the ways to also help just verify your answers.

There’s so much low hanging fruit. For example, I mentioned earlier assigning the AI an identity. Sean, you know, because I think about when you think about AI for in a very general sense, it’s like this nebula of just infinite space. There’s there’s really no rhyme or reason as to where the response is going to come from. But then when I say, all right, you are a 20 year tax professional. Okay? Now we’ve gone from the nebula to a galaxy, your 20 year tax professional with a with a plethora of experience in international taxation.

Okay. Now we’re getting to like a planet system. You specifically have have spent most of your career dealing with issues with key fix. Okay. Now we’re on a planet, right? Your goal now your your goal as this professional is to determine whether you know a mark, the mark or a whatever it may be. Right? The more granular you get, the more detailed you get. All of a sudden we go from a nebula to a country to a state, to a city, to your home address.Right? Not really. Hopefully your home address, but but the more granular you get to that, that’s just going to help your responses as well.

So, you know, if I’m getting a response, I feel like it’s a vague response or it’s just like, oh, like this feels like we’re just we’re pulling things from Wikipedia, from Reddit, from everywhere. Well, that’s because we’re not being we’re not defining our problem. We’re not defining the scope enough. So how do we do that?

Sean: I love I love the example you provided working from the solar system down to the home address level, because I think that that provides a nice little context for the listener of this in terms of creating or assigning that identity.

If if I’m someone who maybe doesn’t, who maybe has not done that in the past, and I have trouble thinking about how to assign or from a prompt engineering standpoint, what I’m asking the AI tool to do, what are the ways that I can get better at that so that I can be able to to to assign to that identity that gets me down to that that country, state home address level.

Scott: But what’s super interesting about AI and I, I want to say it’s the first time this has happened. I don’t know, there could be maybe other examples of this, but this is one of the first, if not the first tools that tells you how you can use it better. Right? So I can use any platform and start it off saying, hey, I’m pretty new to this. I don’t really know what I’m doing. This is kind of where I’m at right now, and this is where I want to go. And like, so how do I do that? And if you want to, you could use the nebula to the home address and it’ll just run that. It’s like, all right, we start at the nebula. In order to get to the galaxy. I need you to answer this, this and this, and then to get to the solar system. I need this, this and this. And then to get to your home address, I need your mother’s maiden name and last four of your social. Right. Hopefully not. You’re getting schemed if that’s happening, but but it tells you it can literally tell you what to do. Right. How to do this.

And it’s just it could be a very creative process. Right. And that’s, that’s, that’s probably the first place I would say, Sean. And and if you can bring certain things to that. Yeah, that will make the process better. So if I already have an idea that, you know, I’ve heard that assigning an identity really helps it. Right. And if I, if I give a clear purpose and I clear output and a clear whatever it may be, if I can come into that conversation with that knowledge, I’m just going to even make it better. But the the real thing is, Sean, this is a tool you have to use if you don’t use it ever. And then you’re like, well, this is silly. Like, I’m never I’m, I look at this. This is a terrible response. Like, well, maybe that’s just a signal to suggest that you have you’re not using it correctly. And the only way you learn how to use it better is by using it.

Sean: You know, it’s interesting because as I continue to have conversations in this space and think about the industry as a whole, right? You know, you look at maybe there was 30 years where, you know, leading up to kind of where we are now, where the industry, I mean, it changed, but there wasn’t that rapid change. You know, we went to paperless and there were other things that came up. And now it feels like everything is just moving at this rapid pace. And as you kind of alluded to, you have to use this tool. I mean, firms are investing in it. They’re expecting to gain efficiencies through using the tool. I have one quick question before I go to the fast five and it relates around the standards.

Tax in this country changes constantly. You know we had the one big beautiful bill act you know, last year that forced a lot of changes that folks needed to be aware of. When we think about AI and some of those outputs in our tax research, how how do we be able to to discern between maybe conflicting cases, conflicting regulations to make sure we’re citing the right regulations?

Scott: Yeah, yeah, there’s a lot, a lot of things that come to mind. I think one thing we’re going to see is that and maybe this isn’t the case either, but there are certain platforms that are just, frankly, more appropriate for tax research than others, frankly. And that is almost it’s there’s a real economic component to it, because if you do some of these tax specific platforms, it obviously comes with a bigger price tag than maybe your free ChatGPT subscription. Right. But that’s one thing to think about. Right? What platform are we using? That could be a really great step forward. But regardless of the platform that’s used, it’s it’s it’s why I’ve been saying all along. You’ve got to check your resources. You have to see which code, which code section are we pulling from. Are we doing the pre one big beautiful bill? Are we doing the transition period. Are we doing the after. Right. There’s we have to actually go back to make sure. Are we actually looking at the most up to date code section. Are we going to the most up to date. Like you have to go back and check it because, you know, I’ve used quite a few different AI platforms and even the best I would say in this space, especially when it comes to law changes, this is something that can get tripped up oh so easily. And again, I read something. I’m just like, this is art this is great.

This, this is going to make me sound way smarter than I really am. And it looks great. It looks awesome. But wow, if I didn’t go back and I didn’t see. Oh, this is pulling from the wrong code. Like this is pulling from the outdated code section now because of like to your point, like maybe something like OB three, the one big beautiful bill, like, wow, now this is going to radically change my answer. What?

And another thing is to make sure that we’re putting in the correct time periods. Yeah. Am I doing tax research for 2025, 2026, 2027? Am I doing tax research because I have a client that hasn’t paid their taxes in ten years, and and we have to go through that whole process of trying to get them right. What year am I looking at? What is the correct relief? What is the correct procedures? Because again, gosh, it’s just a crazy world and things change. The switch gets flipped on and off all the time. And so the due diligence of of actually going to and just making sure is that the most up to date, is it the right tax year. Are we the right formation Sean, Like no matter what I don’t care if you’re using a thesaurus, AI, if you’re using your standard enterprise systems that we’ve kind of come up with a long time, or if you’re using whatever, you have to go back and you have to check there’s just no and if’s or buts there’s there’s never going to be a replacement for that.

Sean: Yeah. Always verify I mean no matter what. That’s the big thing that I take from this conversation. Always verify whatever outputs that you’re getting.

So I do want to close real quick with a fast five. So if we’re if if a listener is joining in to this podcast and thinking, man what can I do? What are some habits that I can instill in myself that I can take away in order to be able to use these tools effectively in tax research, what would be the five habits you’d lay out for them?

Scott: So obviously not an exhaustive list, but the first thing that comes to mind be creative. All right.

One example, Sean, of me being creative in my tax practice is I have a client where English is not the first language. Right. So I was able to put together great memo. I do speak Spanish, but that’s not my first language either. But I had the, the AI platform that I use. I said, hey, why don’t you translate this from English to Spanish? And I want you to also mark it down here, and I’m going to read through it, and we’re going to work through it together where I feel comfortable. Right. That that was to me that felt creative, where I’m like instead of having my client maybe potentially struggled through some more technical concepts in a second language, I was able to do it in their primary language. Right? And that that was an extra layer there, which is like, oh, okay, that’s pretty cool. I was not expecting that to be creative. Right. That’s just an example. Right.

Be consistent. I’m a very big believer, especially when it comes to something like prompt engineering. If you’ve read it once, never write it again and write it in a way where it’s robust. I have a standard, for example, prompt that I like to use when I’m doing tax research, and the only thing I have to change is the purpose of the research and the client fact patterns. Everything else is in stone. I might go and I might tweak a thing or two, but it does. I don’t have to rewrite this massive prompt anymore, right?

The other thing I would suggest is keep the output in mind as well. Sean, I don’t know about you, but my biggest frustration I know what people’s frustrations with AI is. What is this? This is what I was looking for. This is and and it’s fair. Like we have this expectation in our mind, but we also need to be fair to the models just like, brother, I can’t. I can’t go on your mind. Not yet at least. So.

Sean: So might be a scary place, right?

Scott: Yeah, yeah, yeah. Right. But if I want this to be an email, like if I want this to eventually be an email that I’m writing to my client who’s not sophisticated in tax, I should have the output in mind that that’s what I’m going towards. If I want to be doing a work paper reference, then I’m also I’m probably going to I want this to be like three sentences with documentation. I need to make sure that is going to be the output in mind.

Another habit I would say is have the AI, especially in a tax research environment, have it actually aggregates the information have it when I do a review. Right. So let’s just say for example, it pulled some sort of edit editorial document. I want it to actually get me that document. I want to tell me where it pulled from and I want them. I wanted to explain why it was used. And then I also want to like go through that document. Right. But I don’t want it to be the one that’s like, all right, now have to go to either be an A or Checkpoint and try to search for it or go to whatever resource pull it. That takes a lot of time, right? One of the beautiful things about AI is that it can automate a lot of those processes where all of a sudden you don’t even realize how many hours, days, weeks, you save, because you’ve had it aggregate the information for you, you’ve had it summarize why it used it, and then you just check it. Which then leads into my last point why I’ve hit home this entire time.

Always verify. It’s so easy, Sean. I’m guilty of the want of as well as just like, wow, this looks really good. Oh, it’s like 6:00. The Sox are start playing. You know the good Sox—the Red Sox, not your White Sox, right?.

Sean: So hey hey hey now

Scott: But but it’s so easy. And so part of it too sean is understanding when to verify it. So one thing I’d like to do, for example, I’m writing some new content for our estate Gift and Trust Bootcamp I do it. I’ve done all this research. It looks good, but I’m not going to review that research. I’m not going to review some of the outputs. I’m not going to review some of the material probably until 24 hours afterwards, because I’m going to be fresher, I’m going to be further away from it, and I’m going to be more objective. So I’m putting my place into I’m putting myself in a place to succeed at verifying, too, because I think that’s one thing people think about. It’s like, well, I just I just want to do it now. It’s like, no, just literally give yourself that space, that time. Come back when you’re more refreshed, you’re ready to look at it and you’ll still you’re still going to be blazing through this potentially compared before.

Sean: Scott, always a pleasure to connect with you and hear your insights, especially in the tax and AI world, because you’ve you’ve you’ve delved yourself. You’ve dove in so far with it. And it’s it’s just great to hear these insights. And I think the listeners will take a lot away from this. Thank you so much for joining us on this episode.

Scott: All right, Sean, great talking to you as well.

Sean: That’s a wrap on this episode of 20-20 Insights and Conversations.

A big thank you to Scott Orwig for showing us how to put AI to work in tax research without taking its word for it. If today’s conversation got you thinking about how your team uses AI with confidence and judgment, that’s exactly what we do here at 20-20 Services.

We partner with accounting and advisory firms across the country, including 25 of the top 100 CPA firms on audit and tax training, leadership development and the power skills that set accountants apart all built around your firm’s goals.

Visit 20-20services.com to explore our programs, see upcoming courses and find every episode of this podcast in one place. And if you got something out of today, subscribe so you don’t miss what’s next?

Then share it with someone in your team who’s navigating all this change to Until next time, stay curious and stay proactive.