How to gather and evaluate information
Internal auditors spend their days buried in information. Gathering and evaluating information is the basic activity that allows you to evaluate the effectiveness of governance, risk management and control and to provide assurance.
This guidance will help you to think critically about the information you need to deliver a successful engagement. We highlight a number of techniques and tools for gathering and analysing this information.
What is information?
What is information important to internal auditors?
What information do internal auditors need?
What quality of information is important?
How do internal auditors gather and analyse information?
Information is simply the facts or knowledge provided or learned. It can be tacit, in people's heads, or explicit, in documents - electronic or manual.
A wide range of facts and knowledge may interest an internal auditor. They may concern:
- The organisation's business model and the industry it works in
- The culture and morale of the organisation
- Risks it faces and the responses to them
- Transactions it is undertaking
- Customers to whom goods and services are supplied, its assets and liabilities
- The way those transactions, assets and liabilities are recorded and valued financially
- Activities people undertake or how they perform them
In previous versions of the Standards and in other contexts, people talk about 'evidence', rather than 'information' (this reference is made within the Code of Ethics). In particular, external auditors and criminal investigators use the term evidence as 'something that furnishes proof'.
There are many complicated legal distinctions about the nature of 'evidence'. The reason why the International Standards now use the term 'information' is to avoid some of the 'baggage' that the word 'evidence' carries with it.
The role of internal audit is to evaluate how effective the organisation is at governance and at managing risks. In order to evaluate the effectiveness, the internal auditor must gather facts about the particular subject matter and how it is being managed and compare those facts to the criteria that describe what 'effective' looks like. Therefore, facts are an essential tool for internal auditors to use.
The rest of this guidance covers how internal auditors decide in general what information they need for an engagement. It will outline the factors they take into account to decide the type of information they will collect and the amount and characteristics of the information they want. It will then consider techniques for gathering and analysing the information.
To decide the information they need for an engagement, internal auditors need to identify the general subject matter and the criteria that describe effectiveness and then to consider the amount and characteristics of information that they require.
The objectives and scope of the engagement establish the subject matter and will, therefore, guide the internal auditors to the general area or type of information that they require for the work. Internal auditors should seek clarification of any doubt or uncertainty about the objectives and scope to avoid missing something important or setting off in the wrong direction.
Part of the planning of the engagement will have been to identify the criteria to use to evaluate the subject matter, either the existing ones that managers use or ones that the internal auditor and manager develop together.
These criteria will enable the internal auditor to home in on specific information needs. Thinking about, planning and gathering information about the achievement of the criteria will provide the systematic, disciplined approach referred to in the definition of internal auditing.
Taking the subject matter and the criteria, the internal auditor then needs to decide how much and what type of information to gather and where to find it.
A unique bit of information that identifies an object, person, place, or date. (Information in this area is likely to come from internal sources).
- Personal data
- A business record such as an order, an invoice, an accounting entry
- A record of an event such as committee minutes, progress report.
A basic idea or rule that explains or controls how something should happen or work, including standards of behaviour. (Information in this area is likely to be a combination of external and internal sources).
- Legislative requirements
- Rules set down by regulatory bodies
- Policies and ethical standards.
- Standing orders and authority limits
- Technical standards and best practice
A category of ideas or items that share common features. (Information in this area is likely to be a combination of external and internal sources).
- Mission statements and organisational policies
- Strategic and financial plans.
- Risk identification and evaluation.
- Academic papers and surveys'
A description of how something works or operates - the actions that are taken to achieve a result. (Information in this area is likely to be a combination of external and internal sources).
- High-level systems descriptions.
- Flow charts and process flows
- Technical guides and user manuals
- Project plans and schedules
The International Standards say that information should be sufficient, reliable, relevant and useful.
What does this mean?
It is important to gather enough information about a process or activity to gain an unbiased and dependable reflection of how things are working. For example, the amount of information gathered about a process that has many thousands of transactions may be greater than the information needed for an activity that has relatively few. Also, the likelihood and impact of a risk may influence how much information needs to be gathered and analysed in order to provide assurance.
This links to deciding the sampling strategy and choosing a sampling procedure appropriate to the audit objectives. Options include random, judgemental, stop and go, variable, discovery and attribute sampling.
In simple terms, the more confidence and precision the internal auditor requires, the larger the sample size will be. However, as most processes are now computerised there are tools that enable internal auditors to access and analyse the whole population.
Deciding how much information is necessary is therefore a matter of judgement given the complexity of the subject matter, the significance of risks, and performance criteria.
The auditor will need to consider whether the information gathered can be relied upon as being accurate. This relates to how the information is gathered or produced. If the information is generated by the internal auditor, is it clear what calculation or analysis has been undertaken to generate it? If part of a sample, is the auditor certain that the sample has been selected randomly and independently?
This is a model or concept often used to consider the level of reliance that can be placed upon information. While all forms of information have a place, it would not necessarily be appropriate to come to an opinion on the adequacy and effectiveness of the controls in a particular area based solely upon discussions with the people who work in that area without first understanding the culture and risk maturity of the organisation.
The internal audit should continually reassess the pertinence of the information collected to the overall objectives of the engagement - in terms of activities covered or time period of the audit engagement and to the risk responses that are being reviewed, including control testing?
This means keeping in mind that the information collected must establish a sound base to support conclusions and opinions.
For example, if the objectives are to provide assurance on the management of the risk of incomplete reporting, relevant information is about ensuring each item is reported, not that it is reported accurately, which is a separate issue.
This links to reliability and relevance of information. The internal auditor should consider if the information is recent enough to still be useful? The scope of the internal audit work may also be time bound (for example if a new process was implemented, the scope of the internal audit engagement may cover only those risk responses in place since that new process was implemented).
Internal auditors use many different techniques to gather information. Here are some of the more usual ones.
Being part of the organisation
To begin with it is important to recognise that internal auditors develop knowledge and understanding of the way the organisation works by being a part of it (informal meetings, relationship management, networking). This knowledge, along with the results of previous internal audit engagements is a valuable source of information and is part of being an 'internal' auditor.
'Walk-through' - experiencing how transactions happen
Internal auditors can add to their knowledge by observing processes in operation. In addition, internal auditors can use "walk-throughs" to gain experience of how a process or system actually works. The internal auditor selects one or two transactions and follows them through the system comparing how it works in practice to the existing documentation - procedure manuals, instructions, flowcharts - or to the results of discussions with the people involved. This helps the internal auditor to devise a testing based on key risks. In fact risks may need to be reassessed where practice has deviated from the theory.
Internal auditors can use a survey as a simple and quick technique to gather specific facts from individuals, a group of people or an organisation. For example, internet survey solutions, some of which are free, can provide a cost effective approach where the organisation covers a wide geographical area,
However, problems can arise with surveys:
- If people receive a lot of questionnaires to complete, they may be reluctant to fill out another one for internal audit
- If the internal auditor makes them too long or complicated, people may lose interest before the end
- A low number of completed questionnaires, known as a low response rate, can mean that the information gathered does not fully represent the population.
Internal auditors, therefore, need to find ways to improve response rates:
- Keep questionnaires as short and as simple as possible
- Think about the demands on people's time and try to pick the best possible moment to distribute the survey
- Consider approaching recipients directly and asking them to fill in the questionnaire
Surveys should not be confused with control risk self assessment, which is a process designed, owned and implemented by management.
A focus group provides less coverage than a survey but puts more emphasis on the discussion of specific issues in an informal setting and possibly on a task-based or team basis.
Focus groups are therefore likely to explore attitudes, beliefs, perceptions and problems, or to search for causes of problems and their solution. Discussions can be difficult to manage either because people are reluctant to talk, or more likely, their enthusiasm for the subject takes the discussion in different directions.
To avoid these problems internal auditors need to use their facilitation skills to focus attention on the subject in hand. To get the best results emphasis should be placed upon the development of practical solutions and recommendations.
One-to-one interviews with managers and staff provide a more direct route to specific information, enabling internal auditors to follow a particular thread or theme as the discussion develops, and are used in most internal audit situations to gather information. Internal auditors can conduct interviews by telephone or via video conferencing as well as face-to-face.
'Internal audit testing'
The established and most well known method for gathering information is through internal audit testing - substantive and compliance tests. The internal auditor needs to understand the purpose of the activity or process, gain an appreciation of the risks and know how the activity actually operates in practice.
By providing information on how the systems and processes actually work, the testing programme helps the internal auditor to confirm or otherwise their opinion of the effectiveness of the area under review.
Analysis is something that internal auditors do to learn about the subject area or activity they are reviewing. It is different from gathering information as it involves looking at relationships within the information and comparing the information to criteria.
There are two main techniques for analysing information: trend analysis and ratio analysis. Trend analysis compares data over periods of time. For example, the amount of bad debt outstanding this month compared to the amount outstanding each month for a 6-month period. Ratio analysis compares the size or amount of one thing to another such as financial or accounting ratios and proportional analysis, for example: overtime as a % of total pay.
Computer-assisted audit techniques and tools (CAATTs) have the ability to improve the range and quality of internal audit analysis. They enable large volumes of data from different sources to be compared and organised. This may mean that the internal auditor can test a whole population, rather than just a sample.
Some examples of their usage include the ability to access and extract information from client databases:
- Total, summarise, sort, compare and select from large volumes of data
- in accordance with specified criteria.
- Tabulate, check and perform calculations on the data.
- Perform sampling, statistical processing and analysis.
- Provide reports designed to meet particular audit needs.
When accessing and gathering information, internal auditors should also consider the legal requirements relating to the protection of personal data, such as the UK Data Protection Act, and the Irish Data Protection Act both of which were published in 1988.
This guidance has focussed on internal audit engagements and their direct needs in terms of information. Some internal auditors may, in addition, undertake other types of work: investigations that lead to disciplinary action or prosecutions. These have different requirements for gathering and for documenting information which are beyond the scope of this guidance.