All computers need software – apps or systems, or programs – in order to be of use. Some are hidden and work in the background (for example to print a document), others we call and use as we need them (e.g. payroll packages, games or social media apps). Some allow for tailoring by the user, or enable us to write our own ‘software’ in the form of databases or spreadsheets, for example.
Whatever the source and type, we place a lot of trust in their ability to do what we want – to provide reliable results for our lives and business activities. But how do we know we can rely on them? How do we know they are not doing anything devious or unreliable in addition? I’m not talking about a science fiction based world where they are deviously trying to replace mankind – but more about the day to day processing of transactions.
There are two main ways that programs are changed:
1. By projects to replace or make major changes to software.
2. By change control processes to make more minor ongoing changes.
In this chapter we will consider the risks, controls and audit of the development of new applications and changes, in particular:
• Change management.
For each of the above we will consider their risks and controls, and provide some case studies and some guidance on documenting and assessing controls.
Project lifecycle overview
Let’s start with some definitions:
Project: “A task requiring considerable or concerted effort.” Collins English Dictionary
“At its most fundamental, project management is about people getting things done.” Dr Martin Barnes, APM President 2003-2012
There are a number of ways and tools for representing an IT project. A typical project lifecycle for a system development project could be summarised as follows:
Source: Agile Governance and Audit
Most businesses have more suggestions and ideas for changes than they can handle, with the resources they have available. They therefore need to challenge these and see which are worthwhile. Before commencing any work on a potential project, there are four basic questions that need to be considered:
1. What is this project trying to do? This should be represented clearly in some form of business case, demonstrating the benefits/changes/improvements the project delivers.
2. Who is this project doing it for? The beneficiaries should be clearly identified, ideally showing their level of commitment to the change and how they will benefit.
3. How are stakeholders engaged? The representative senior stakeholders should interact fully with the project through the whole lifecycle.
4. How does it fit in with the rest of the organisation’s systems and processes? It should be making better use of existing tools wherever possible.
If the answers to these four questions are not clear – then the likelihood is that the project is doomed to failure from the outset, irrespective of the skills and disciplines of project management because (even if there is a good business case) there will not be the desire/drive in the business to change, or insurmountable technical difficulties will emerge. These four questions are only an indicator, other factors may need to be considered but they will alert the project reviewer to an increased likelihood of project risks that we will discuss later.
In the vision/needs and risks phase (aka business case), the business identifies the need for the development and expresses this in terms of business benefits. Within any organisation there may be lots of different competing ideas for development. Not all will be achievable with the resources available. The organisation therefore needs to prioritise ideas into a portfolio – ranked according to the need and business benefit. This also enables management to align the demand to their own strategic plans and to identify any common themes or potential for consolidated solutions (e.g. the implementation of a single ERP solution could combine the requirements for a new finance and a new sales system).
Having decided to proceed to the next stage, the organisation will define the requirements. This will consist of creating a design document to show the existing (‘As Is’) process and to agree an end design (‘To Be process’). It should take account of all major stakeholders and be focused on the identified business requirement and potential benefits. The extent of the design can very – in some organisations it may be a highly detailed design document, in others it may be much simpler. Whichever approach is used at the end of the design phase, there should be a clear view as to the specific requirements.
The next phase is to build the solution. This will start with the decision of what is to be built. Based on the design and requirements, the organisation will consider different options – whether to extend systems it already has, with new modules or functionality, or to choose a new solution. The solution will be the best fit to the requirements but further modification will be required to meet the organisation’s full requirements. The extent of the development work will determine the timing and manner of the final implementation. It will often involve the use of external developers. Increasingly, organisations are looking for out of the box solutions, which although they can tailor the data tables, management reports, process work-flows and the user interface for their own use, will not require any major changes to the actual program code. This makes it easier to implement new patches and releases of the underlying software, as they are provided by the supplier.
Once the development is complete the software will be tested – first to ensure that it works, and then to ensure that it meets the user requirements and can process the transaction load expected, and that it will not adversely impact any other systems in use.
Other than providing early stage support and maintenance (sometimes referred to as the warranty period), the IT portion of the project is then complete. However, for the system to be successfully deployed, the project needs to also consider the area of business readiness. This will involve developing and implementing the supporting processes, agreeing a deployment plan for all locations/divisions (training users, ensuring controls still operate, etc.) and change management (to ensure that the new processes are used in the way that they were intended).
A project can be small and of a short duration, or can be long and expensive (the smallest I have been involved with was a few hundred pounds sterling – the largest was $2bn and lasted over ten years in total). All projects require appropriate governance to ensure they achieve their business benefits (see risks below). The usual mechanisms include:
• A steering group or other regular meeting of senior stakeholders to monitor progress, identify any major risks and issues impacting the project, and ensure that the required outcomes can be achieved.
• ‘Business process owner’ or other representation from the business that will monitor what is being delivered, answer any queries on requirements or process design, and ultimately accept the solution on the business’s behalf. If Agile is being used this will be the product owner.
• A project management function or office (PMO) – to plan the project, co-ordinate the different elements of the project and report to the stakeholder’s/steering group. The PMO will also provide support to the project, for example in recruiting and on-boarding new staff, co-ordinating common activities, such as team meetings or testing, and in helping with liaison both within and outside of the project. The PMO will also be responsible for ensuring that the organisation’s agreed project management approach and methodology (such as PRINCE2 or PMBOK®) is applied effectively.
• Quality or stage gate assessments – these are approval points during the project to assess progress and agree that the project can move on to the next stage or phase. The ultimate stage gate will be to place the software into the production environment so that it can be used by the business. There is usually a checklist of issues to be completed and approved during the stage gate assessment. In the case of the ‘Go/No Go’, approval to proceed may include provisional items which the project team have agreed to resolve in the short term.
• Data quality – reviewing the quality of data before and after it has been migrated to the new system.
• Post implementation reviews – a review to assess whether and how the project achieved its objectives.
In a traditional waterfall project approach, each of the phases illustrated in the above project lifecycle may be quite long, lasting several months and involving a number of sub-steps and approval gateways. In this approach each phase is completed before moving to the next – so for example development will not commence until all of the functionally has been designed. For large projects lasting many years, the requirements and business environment may change significantly and so the end product may be out of date before it has even been implemented. Costs and timelines can therefore be greater than anticipated. Also, there is a risk that the organisation may be missing business opportunities because it cannot respond to them quickly enough. To overcome this, organisations are adopting Agile project techniques, such as lean or scrum. These tend to focus on the outcomes of the project rather than the project process, and break the deliverable down into smaller iterations which are delivered at regular intervals (generally every 30 days). These approaches are typically less formal – the risks are minimised because of the small potential impact of each iteration. The same principles described above still apply, however, the documentation available for audit and review will be reduced. For further information see Agile Governance and Audit (ITGP 2014) by the same author.
When projects fail, the consequences for the business (and often personally for the project team) can be extremely significant. Just a quick Google of project failure will identify many instances of large and significant project failures – particularly in the public sector (e.g. see www.nao.org.uk/). There have been a number of surveys conducted to identify the reasons for project failure
The main risks associated with projects are that they fail to deliver:
1. On time
2. On budget
3. The identified required business benefits
4. Systems of sufficient quality or reliability, for example by introducing processing errors.
The main causes usually identified in the various surveys are:
• Lack of senior level leadership, sponsorship or accountability for the project. Without their drive and commitment, the project is unlikely to succeed.
• Poor clarity and consensus of vision as to what the project is trying to achieve.
• Underestimating the complexity of the project, not only from the IT perspective but also from the perspective of the impact on existing business processes. This could include cultural differences if the project is amalgamating business processes from different locations or entities.
• Scope creep – new requirements being added as the project progresses.
• Poor project management techniques, including transparency, monitoring of actual costs, progress and risks. This may include challenging the information provided. One project manager told me he regarded all green (or satisfactory) statuses as potential watermelon–they may be hard and green on the outside but are red and slushy when you cut them open! Unless there is the challenge of reporting, there is a risk that reporting will be optimistic and that there will be little, or no warning of imminent failures.
Project lifecycle controls
The aim of controls over the project lifecycle is to reduce the risks of not achieving the required business benefits, excessive costs and delays.
To reduce the likelihood and impact of the risks listed above we would expect to see:
In addition to the above, some organisations now have standards for application security to be included in the design and build of new systems (e.g. OWASP).
This will be increasingly important with current trends to make more use of standard (often Cloud-based) solutions so that what appears to be one system to the user is in fact many applications seamlessly joined together. In this type of design common application security is essential.
Project lifecycle case study examples
Travelling to nowhere
I was part of a review team looking at the implementation of a website and related online market places for a tour operator. I asked seven directors, all committed to the project, what its objective and expected outcome was. I got seven completely different answers ranging from to reduce costs, obtain new markets, keep up with competitors, etc.
Unless there is a clear understanding of the objectives and requirements for the project, how can you ever know whether or not it is successful? There was a real danger in this case that some of the directors were going to be disappointed with the outcome – even if it delivered the requirements of the others.
A tale of two projects
I was involved in a very important project for an airline, providing ongoing project assurance. I was very concerned about the lack of focus of the project team and their ability to deliver. So I asked the project sponsor (the acting CEO) to attend the meeting and set the context of the significance of the project to the business. An hour and a half into the meeting and he still had not appeared. He then stuck his head round the door and said – there’s a party downstairs, why are you not all there? The meeting abruptly broke up – not the outcome I wanted.
On another project with another organisation, at the start of a two-day business awareness workshop the CEO for the whole of the US attended and clearly stated the importance of the project and why it was important to the business. This focused the workshop and ensured we had a successful outcome.
Project lifecycle documenting, assessing and testing controls
There are two main aspects to system development audit:
1. Audit of project/programme management.
2. Audit of the project/programme management of a project or programme.
There is a good deal of debate as to the best timing for a review of application development. The concern is that if the auditor becomes too involved in the project, they become part of the project and therefore lose their independence. My personal view is that it depends on the level of the risk for a project. A high-risk project should have regular assurance reviews, either from internal audit or from some other independent assurer. The review should include all of the controls listed above – based on:
1. enquiry of project management, team members and stakeholder representatives.
Documentation and monitoring of project/programme
Risk assessment and resolution
Programme/project governance frameworks.
It is important for the auditor to maintain a healthy professional scepticism when gathering evidence. Consider the following two examples:
1. Control is that there are weekly project meetings involving all workstream leads, chaired by the PMO, and actions recorded. This looks good and at first look the 20-page progress pack with ten general slides, six workstream pages and list of actions is great. But when you look at a series of these you can start to see issues that have no progress that may indicate the project is not dealing with the critical design problem or interface that will cause it to fail.
2. Control is that the PMO monitors weekly test results and assesses progress to completion. It doesn’t take long for an auditor to check the rate of testing, the rate of defect resolution and come up with a reality check on whether testing is going complete on time. So often this exposes the magical ‘hockey stick’ where somehow all the open tests and defects will resolve themselves in the last week!
Change management overview and risks
“To manage all changes that could impact on IT’s ability to deliver services through a formal, centralised process of approval, scheduling and control to ensure that the IT infrastructure stays aligned to business requirements.”
I would make two minor changes to this – firstly to add ‘or software’ after IT infrastructure and secondly to add ‘and operates effectively and efficiently’ at the end of the last sentence.
The main risk is that if unauthorised or untested changes, both scheduled and emergency, are made to the production environment, the change may:
• make the system/IT infrastructure unstable, or allow unauthorised access.
• damage controls or other integrity of the application/IT infrastructure.
• not work effectively in correcting the problem.
• cause version control issues. For example, if two people try to change the same software programme at the same time, there is a risk that the changes made by one may overwrite the changes made by the other.
• not be an effective use of resources, as a cheaper and simpler way of making the same change may have been available.
We would expect to see the following controls
COBIT 5 Best Practice ref
Authorisation included in the process to evaluate, prioritise and agree change requests.
BAI06-BP1 Evaluate, prioritise and authorise change requests
Arrangements in place for emergency changes including, restrictions on definition of what can be classed as an emergency change access control to production system, review and post-approval of change.
BAI06-BP2 manage emergency changes
Changes are tracked to ensure they are completed and tested. Also, reasons for rejection of a change are logged for reference if same change is re-submitted. This also enables root cause analysis to identify reasons behind common issues.
BAI06-BP3 Track and report change status
Formal closure of change requests when completed to confirm have met user requirement.
BAI06-BP4 Close and document changes
Bigger debt than expected
A senior member of IT staff left an organisation owing money for a car loan. The organisation agreed that he could work on Saturdays to work off the loan – even though he had left. The changes he made were conflicting with those made by other consultants on the new hardware during the week and so were causing additional re-work. It was cheaper and simpler to write-off his debt.
All change (NOT)
An organisation made non-standard changes to the core software of its ERP system. This meant that not only could it not take standard updates provided by the supplier but also it could not update the underlying operating system. This increased the risk of unauthorised access as security patches could not be applied, also other application systems became unstable as they were having to use old versions of the operating system.
Documenting, assessing and testing controls
An audit of change management is likely to include:
• Review of the agreed process and supporting documentation
• Obtaining a log of changes and review for approval
• Review of sample of changes to ensure documented, approved and tested prior to promotion to production environment.
Change, including the implementation of new systems and processes, is inevitable. In order to reduce the risk of losing integrity of IT infrastructure and systems there is a need to ensure that appropriate controls are built in.