Local Authority Internal Audit Virtual Forum

24 November 2021

Please note:

• All Institute responses are boxed and highlighted blue
• Where the chair comments in that capacity this box is highlighted in yellow
• For confidentiality, the identities of all delegates/attendees are anonymised

Key takeaways

Stuart Clark, Head of Audit – Commercial, Digital and Innovation, Virgin Money

Innovating the audit approach to a large-scale transformation and integration programme

The scale and pace of integration meant that the existing internal audit approach was no longer fit for purpose.

The following changes were adopted:

• The three lines of the organisation joined together as a team to change the approach to risk assessment and assurance mapping, using a common language and assessment to understand risk.
• Moving to a dynamic ‘3+9’ month approach to planning, committing to the next three months with a planned backlog.
• Anchoring the internal audit plan with an opinion on three key questions:
  • Are decisions evidence-based?
  • Would the organisation be ready for key events?
  • Post-implementation, had the organisation delivered planned benefits?
• Accelerating the agile auditing process by:
  • Focusing on the minimum viable product for any audit
  • Moving away from objectives, risks and scope terminology with stakeholders and towards a plain English ‘exam question’ approach focused on what opinion each audit will ultimately provide
  • Bringing stakeholders closer to planning, execution and reporting stages
  • Moving to more real time reporting through verbal updates and memos, to allow concerns to be fixed before impacting further on delivery of the programme
• A quarterly opinion being provided across the three lines, more focused on insights than work delivered
• Data analytics to develop a suite of KPIs and KRIs to inform how well project management had been delivered across the organisation e.g. how often milestones had been shifted or RAID logs been updated. This tool was then handed back to the first line control framework
• Delivering a culture assessment on the integration programme with focus groups and surveys across 1200+ members of the delivery team, using an organisational psychologist resource on questions like would people feel safe raising concerns, were people sacrificing plan delivery for disruptive activities, and had the organisation kept the customer at the heart of its activity

There is never going to be a better time to change what you do than during a major change programme. Moving at pace, being collaborative, and driving towards a successful outcome gave internal audit a strong mandate and being seen to respond has stood them in good stead. People who deliver change are used to change so the mindset of try something new, fail fast sat well with them, and excited the internal audit team. None of this was revolutionary, but it came at the right time with the mandate and the right mindset.

Iain McGregor, Director of Innovation and Development, Government Internal Audit Agency

Innovation hub: GIAA set up an Innovation hub with agreed topics such as data analytics and AI to allow people to come together to experiment, work with others across government and other sectors, and learn from failure, in order to learn how to succeed.

Structure: A core data analytics team was set up within the hub, and sat inside a ‘Community of Practice’ of auditors and counter-fraud specialists using data analytics regularly. All members of the wider agency are encouraged to be data-confident so that they can identify and discuss opportunities for using data analytics with stakeholders and bring them back to the ‘Community of Practice’ or core team.

Example: Existing work had been requested in counter fraud to analyse timesheets of those claiming TOIL (time off in lieu) in a UK region. The process was manually and took three days. The same work could be completed in five seconds using scripts that automatically generate an interactive report with graphs and tables. This was then translated into a national script that took 50 seconds to run.

Insights engine: GIAA produce 1500 reports per year and wanted to review practice across them but couldn’t do this manually. Our insights engine uses AI to facilitate reading hundreds of reports in a few hours. Natural Language Processing identifies common phrases and sentiment analysis captures usage in positive and negative contexts for examples of positive and negative practice. For example, one person was able to analyse 102 Annual Opinions in a few days.

Learning from other sectors: Social media uses analysis of networks of people. GIAA translated this concept to documents, looking for trends amongst high volumes of documents. For example, network analysis on 20 documents from gov.uk recognised sub-categories, identified links between documents and highlighted the key document.

Lessons:

• Put people at the centre of what you do – choice of technology and how you do it
• Innovate: Learn from other sectors
• Be ambitious: Learn from failure – change mindset from ‘this might not work, should we do it?’ to ‘this might work, why wouldn’t we do it?’.

Slides from Iain’s presentation are available here. Please contact him at correspondence@giaa.gov.uk if you wish to discuss.

Q&A

Q: Have you been able to maintain this approach with business as usual audits? Or did it relate purely to the transformational work?

A: We have maintained a lot. We handed back the assurance map to the first line and it hasn’t been maintained but we will be having a conversation about that. There is still much focus on the change agenda from the board and regulators. The 3+9 planning and quarterly change opinion have been maintained. Analytics products have been handed back to the first line. We are still using memos and standalone reports a bit. The broader change that has endured through the learning of joining up of lines is the common way to talk about planned work across three lines (through SharePoint) using lenses such as business unit, risk classification etc. to present to the audit committee.

Q: Could you give an example of the "exam questions" you used? Are you now using exam questions rather than terms of reference for all audit work?

A: Looking at readiness to implement a new solution and looking at the controls in place, the exam question was ‘is there sufficient evidence to support a go-live decision?’. Everyone understands the opinion internal audit is looking to give, rather than the detailed scope. We still do use Terms of Reference and risks for those who do not engage with the exam questions approach, but we find more people engage with the exam question.

Q: The documents used for the COVID-19 work in the GIAA insights engine - were these all electronic documents or did it include paper documents as well? If so, how did you enable the engine to read these?

A: It doesn’t matter. With paper documents, you can scan them then use technology called Optical Character Recognition to convert to computer recognised text.

Q: What would you suggest could be a starting point for a small internal audit team who want to benefit from digital / AI technology?

A: I would focus on data analytics first and you can quickly move into AI. Find someone passionate within your team and give them space. Tools are free. Take some past audits and revisit using tools. Lots of free training. Don’t rush, take your time and learn from people e.g. the Institute’s DA working group.

Q: Could you include some examples of the free software?

A: Look for Python and also R. Python is an interpreted high-level general-purpose programming language; R is a programming language and free software environment for statistical computing and graphics. Both are free programming languages.