AI agents ask better questions than KPMG accountants in audit trial

(europeanbusinessmagazine.com)

98 points | by Betelbuddy 2 hours ago

16 comments

  • prometheus1992 17 minutes ago
    READ TFA- The experiment is conducted by KPMG itself. The result of this experiment is the equivalent of 'most of the code in faang today is written by AI'.

    The task of auditing a business is extremely complicated by virtue of data/information being all over the place. AI had not gotten to a point of where it could outperform humans but in this experiment they find out a group of agents are good enough for this task. I still think there are so many poorly defined tasks that auditors just understand implicitly, eg data discovery; before the audit even starts, that humans will always be needed to initiate and conclude audits just like software projects.

    • mannyv 12 minutes ago
      At some point an audit is literally a checklist of things to look over. If there are anomalies then they get dug into.

      I expect for 80% of the audits AI will be 'good enough.' And those are the bread and butter for the public accounting firms.

      • datsci_est_2015 2 minutes ago
        I, on the other hand, expect for X% of audit activities AI will be good enough. How those activities are distributed across clients will be long tail, and you’ll have a bunch of huge pain in the ass clients, and many, many clients where there’s one sticky thing that’s very difficult to automate.

        The difference is important because I don’t want idiots out there reading this comment and drinking the KoolAid, believing that an entire client’s audit activities can be blindly automated.

        Just like a self-driving car’s entire drive can’t be blindly automated, even though X% of the maneuvers that the car has to perform can be automated.

    • echelon 10 minutes ago
      > 'most of the code in faang today is written by AI'.

      And it's true.

      We're getting to a point where you don't even have to look at the code anymore, and I'm serious.

      > AI had not gotten to a point of where it could outperform humans but in this experiment they find out a group of agents are good enough for this task

      The more this is done, the better it gets. Especially as more resources are poured into making it better.

      It's only a matter of time before AI auditing is the common practice and humans are the backup for the AI.

      • dev_dan_2 2 minutes ago
        As long as the AI is then consequently blamed for the resulting outages and bugs, I don't see the problem here =)
  • trentnix 28 minutes ago
    KMPG accountants are generally inexperienced, overworked, and poorly incentivized. The entire big firm public accounting model is exploitative at multiple levels.
  • origini 1 hour ago
    Even in normal jobs we have such a HUGE difference between two people. The only organization which is working against are pilots with procedures and checklists.

    And every year a new human comes along and has to learn all of it again.

    Of course ONE AI agent and teaching ONE AI is massivly in an advantage as soon as its as good as the avg person in the job and it blows everyone else out of the water as soon as it gets a tag better.

    • joshAg 38 minutes ago
      There are other professions with procedures and checklists. Just off the top of my head there's safe surgical checklists to minimize errors during surgeries [1] and checklists for A+P procedures on planes, checklists for operating/working at nuclear plants, and for factories/manufacturing plants making precision parts or operating within a good safety regulatory regime (like semiconductor fabs or auto manufacturing), to say nothing of checklists in the military, which designs equipment to have easily disableable safety mechanisms if it has any at all because the operator might need to operate it outside normal, safe limits [2].

      [1]: https://www.who.int/teams/integrated-health-services/patient... [2]: https://en.wikipedia.org/wiki/Battleshort

    • lazide 1 hour ago
      Except that ‘one AI’ will have it’s own blindspots which can be exploited universally and all at once, until the magic fix goes into it.

      It’s literally a monoculture.

      For example, if an iteration of the singular AI ends up being crashable with a magic visual pattern or QR code (highly likely at some point), that is a major national security risk if that AI is piloting aircraft, as the entire nations aircraft could be taken down by anyone projecting said image in it’s visual plane until the fix gets pushed.

      And these systems are far too complicated to exhaustively test all possible inputs before release.

      There is a reason why nature rarely has such monocultures - that live for very long, anyway.

      • origini 45 minutes ago
        With RL it will learn from all misstakes once while with humans after a generation, information might just be gone.

        Its a possible as an attack vector for sure.

        But in comparision to nature, compute can speedrun all of this with RL.

        • disgruntledphd2 14 minutes ago
          RL data is really really really expensive to collect and train on, and only works for verifiable tasks, which is not most tasks.

          Even a lot of what software professionals do isn't verifiable.

  • chilmers 1 hour ago
    I think properly-trained AI agents will have a huge advantage against humans in such contexts: They will have no intrinsic tendency towards laziness. I think most humans have a natural inclination to optimise the amount of work we do in a professional setting, unless we really happen to enjoy it doing it for the sake of it. We can fight against this by setting up processes and incentives, but we'll always be going against a tendency deeply ingrained by evolution to minimise our mental energy expenditure and to avoid repetitive, tedious or boring tasks.

    On the other hand, with AI we'll use reinforcement learning to create agents without such evolved-in tendencies. Where (so long as we can solve alignment and cheating) they are trained to do the job, no matter how much effort it takes, or what's involved.

    • wongarsu 34 minutes ago
      LLMs absolutely suffer from lazyness. Both reinforcement- training environments and benchmarks have token limits, which makes giving 100% on every subtask disadvantageous

      They are less lazy than humans though

      • disgruntledphd2 12 minutes ago
        They are differently lazy from humans. They'll happily bang their heads against a wall to make something work, but they'll almost never step back to consider a different approach.
      • jtmarl1n 17 minutes ago
        Is that laziness or just resource constraints
  • chirau 44 minutes ago
    I am not surprised at all.

    AI labs were working with companies like Mercor for more than a year paying the consultants and accountants themselves from these very firms a good amount per typical task they work on everyday, some even providing actual documents.

    The expectations were more rigorous, and the rubrics pretty expansive. so yeah, the agent trained from tens of thousands of consultants would eventually ask better questions than an individual consultant.

  • intended 1 minute ago
    [delayed]
  • marconey 1 hour ago
  • joe_the_user 34 minutes ago
    I think AIs can ask good questions and lead a person to important insights.

    But they can also fail to ask "obvious" questions and get stuck at points that an average person wouldn't be stuck in.

    Which makes them for adversarial processes - like auditing or hacking. But no so good at producing defect-free software or planning without guidance. With guidance, they're great aids in this too. But when people AI processes to "turn off their brains" or act autonomously, then you get problems. And you get a lot of pressure to very lightly if at all scan AI output when AI investment is supposed to show productivity gains.

  • jordanb 59 minutes ago
    KPMG's nickname in Germany is Kinder Prüfen Meine Gesellschaft (children auditing my company). Not exactly a stamp of excellence. The KPMG folks were probably also asking questions they got from gpt.
    • TrackerFF 22 minutes ago
      TBH the same could be said about pretty much any of the big professional white collar firms. Accounting, consulting, etc. these industries are built on a never-ending stream of hungry graduates that will pull long hours for a couple of years.

      The very first time I ever worked with a McKinsey consultant, he turned out to be a jr. consultant half year into his gig at McKinsey. In all likelihood a really sharp and ambitious person, but extremely junior when compared to the domain experts (with decades of experience) in the same room.

  • shye 1 hour ago
    I suspect this says more about KPMG accountants than AI.
    • johndhi 1 hour ago
      this is a cute comment, but the reality is a similar analysis could probably apply across literally every profession
      • munk-a 1 hour ago
        Certainly within large corporations where the best survival skill to have is not asking inconvenient questions.
      • devindotcom 1 hour ago
        literally every profession
        • hall0ween 27 minutes ago
          literally. every. profession.
  • letmeguessok 33 minutes ago
    Let me guess.. auditing and doing “accounting” never was the “hard part” anyway?
  • pembrook 1 hour ago
    Is anybody surprised that AI can do a better job in a field that is literally just counting beans and reading written down rules and rulings (laws, precedent) and following them during the process of bean counting?
    • JumpCrisscross 1 hour ago
      > a field that is literally just counting beans and reading written down rules and rulings (laws, precedent) and following them during the process of bean counting?

      To be clear, this describes the vast majority of coding.

      • letmeguessok 29 minutes ago
        That is literally all cognitive work that can be done on a laptop.

        Rules, rulings and following them. I have an extremely hard time coming up with something that requires intelligence but lacks rules and rulings. The rules and ruling might be complex but they are always there even if in the background (like in, say, visual arts).

        • JumpCrisscross 12 minutes ago
          > That is literally all cognitive work that can be done on a laptop

          No. Not currently.

          > Rules, rulings and following them

          Does not describe an auditor's role in the context of litigation.

          > rules and ruling might be complex but they are always there even if in the background (like in, say, visual arts)

          Uh, sure. Q.E.D.

      • pembrook 1 hour ago
        Yes, hence why all code is written by AI now.

        I'm not trying to denigrate accountants, I'm just amazed that this is a headline finding in 2026.

        I guess for industries heavily tied up by regulatory norms and regimes it doesn't matter how much better AI is at the job. Hence why Anthropic's claims of 15% GDP growth are unserious childlike predictions from nerds who have never interfaced with the world outside silicon valley.

        The ThinkPad army will be manually typing business buzzwords into their powerpoint presentations for another 10 years still, regardless of how fast the actual tech is moving.

    • arctic-true 49 minutes ago
      This is an extremely reductive way to think about accounting, especially audit work. The article is about asking good questions, which is a skill a lot of people don’t have.
    • rithdmc 32 minutes ago
      Understanding and reading are different things.
  • bell-cot 1 hour ago
    From coverage of a fair number of accounting scandals over the decades, my understanding is that a "Big Four" audit of a major enterprise is too often a cut-rate whitewashing exercise - the auditors overlook all the dubious accounting, and in exchange their firm gets various lucrative consulting contracts and such.

    Most famously - https://en.wikipedia.org/wiki/Enron_scandal

    Is anyone in this space, to speak to whether AI agents asking "better" questions is a benefit, vs. a bad product-market fit?

    • hectormalot 56 minutes ago
      > the auditors overlook all the dubious accounting, and in exchange their firm gets various lucrative consulting contracts and such.

      I know some people in the big four world. At least here (NL), it’s prohibited to be the auditor and provide consulting services at the same time.

      • disgruntledphd2 9 minutes ago
        This is prohibited basically everywhere.

        Like, there are a lot of problems with auditing as currently practiced, but to imply widespread corruption is entirely wide of the mark.

        Once upon there used to be a Big 5, they are now four because of one case like you're describing

    • Topfi 1 hour ago
      Much as I hate to play Accountants advocate, let's be fair to the industry, we did go from the Big Five to Four post Enron.
      • fidotron 27 minutes ago
        The accounting side ceased existing, but Accenture is very much around.
    • RobotToaster 1 hour ago
      I imagine your average 10 year old could do a better job of uncovering dodgy accounting than KPMG.
      • fernmyth 59 minutes ago
        Kinder Prüfen Meine Gesellschaft?
      • method_capital 1 hour ago
        I doubt it.

        The reason these things happen is compliance is paid for by the company. The Enron account was massive, so auditors incentives are misaligned.

  • redwood 49 minutes ago
    What this means is we Face a future in which instead of a sample of accounts auditing moves into being a comprehensive assessment. That's what near free at scale Brute Force enables that humans simply could never do.

    There are many downsides to this in that business cannot ever be perfect and a sample-based regime is useful because it allows for a few mistakes here and there it's not waste everyone's time at audit time

    So similar to cybersecurity where you're going to need to apply AI offensively and defensively I suspect we'll see the same in accounting

    • jsty 23 minutes ago
      > It allows for a few mistakes here and there it's not waste everyone's time at audit time

      But even moving from sampling to an exhaustive analysis, the time-wasting mistakes could still be rejected as non-material (i.e. too small to care about). If it wasn't possible to automate this, I doubt an exhaustive approach would take off.

  • jdw64 1 hour ago
    In my country, Korea, a perfect score was achieved on a tax-related exam that even practicing tax professionals find difficult to score perfectly on—and it turned out that AI had been used. Regulations on AI cheating are now being drafted in Korea. Looking at this article, at what I’ve seen and felt around me, and at similar problems in other East Asian countries, I often think that AI is better than humans in areas where specific procedures are fixed. In other words, it feels stronger than humans in problems where the search space is defined.
  • j45 1 hour ago
    Given AI's tendency to start with the average and most common questions, I wonder if that's what's outperforming accountants and management accountants.

    The accountants will not have this ability to use AI because their fantasy of managing and overseeing IT wasn't able to be handled.

    • bigstrat2003 1 hour ago
      > Given AI's tendency to start with the average and most common questions,

      Very much not my experience (which is also why I don't believe the claims in this article). My experience is that when trying to troubleshoot issues with LLMs they are prone to make wild leaps and insistently chase down esoteric edge cases, in situations where a human would start with common solutions first.