Chapter 7: System Development and Change Control – Fundamentals of Information Risk Management Auditing: An introduction for managers and auditors

CHAPTER 7: SYSTEM DEVELOPMENT AND CHANGE CONTROL

Introduction

All computers need software – apps or systems, or programs – in order to be of use. Some are hidden and work in the background (for example to print a document), others we call and use as we need them (e.g. payroll packages, games or social media apps). Some allow for tailoring by the user, or enable us to write our own ‘software’ in the form of databases or spreadsheets, for example.

Whatever the source and type, we place a lot of trust in their ability to do what we want – to provide reliable results for our lives and business activities. But how do we know we can rely on them? How do we know they are not doing anything devious or unreliable in addition? I’m not talking about a science fiction based world where they are deviously trying to replace mankind – but more about the day to day processing of transactions.

There are two main ways that programs are changed:

1.  By projects to replace or make major changes to software.

2.  By change control processes to make more minor ongoing changes.

In this chapter we will consider the risks, controls and audit of the development of new applications and changes, in particular:

•   The traditional processes, and more modern, development lifecycles – from adoption of an idea to fully functioning software.

•   Change management.

For each of the above we will consider their risks and controls, and provide some case studies and some guidance on documenting and assessing controls.

Project lifecycle overview

Let’s start with some definitions:

 

Project: “A task requiring considerable or concerted effort.” Collins English Dictionary

“At its most fundamental, project management is about people getting things done.” Dr Martin Barnes, APM President 2003-2012

There are a number of ways and tools for representing an IT project. A typical project lifecycle for a system development project could be summarised as follows:

Figure 3: Summarised project lifecycle

Source: Agile Governance and Audit

Most businesses have more suggestions and ideas for changes than they can handle, with the resources they have available. They therefore need to challenge these and see which are worthwhile. Before commencing any work on a potential project, there are four basic questions that need to be considered:

1.  What is this project trying to do? This should be represented clearly in some form of business case, demonstrating the benefits/changes/improvements the project delivers.

2.  Who is this project doing it for? The beneficiaries should be clearly identified, ideally showing their level of commitment to the change and how they will benefit.

3.  How are stakeholders engaged? The representative senior stakeholders should interact fully with the project through the whole lifecycle.

4.  How does it fit in with the rest of the organisation’s systems and processes? It should be making better use of existing tools wherever possible.

If the answers to these four questions are not clear – then the likelihood is that the project is doomed to failure from the outset, irrespective of the skills and disciplines of project management because (even if there is a good business case) there will not be the desire/drive in the business to change, or insurmountable technical difficulties will emerge. These four questions are only an indicator, other factors may need to be considered but they will alert the project reviewer to an increased likelihood of project risks that we will discuss later.

In the vision/needs and risks phase (aka business case), the business identifies the need for the development and expresses this in terms of business benefits. Within any organisation there may be lots of different competing ideas for development. Not all will be achievable with the resources available. The organisation therefore needs to prioritise ideas into a portfolio – ranked according to the need and business benefit. This also enables management to align the demand to their own strategic plans and to identify any common themes or potential for consolidated solutions (e.g. the implementation of a single ERP solution could combine the requirements for a new finance and a new sales system).

Having decided to proceed to the next stage, the organisation will define the requirements. This will consist of creating a design document to show the existing (‘As Is’) process and to agree an end design (‘To Be process’). It should take account of all major stakeholders and be focused on the identified business requirement and potential benefits. The extent of the design can very – in some organisations it may be a highly detailed design document, in others it may be much simpler. Whichever approach is used at the end of the design phase, there should be a clear view as to the specific requirements.

The next phase is to build the solution. This will start with the decision of what is to be built. Based on the design and requirements, the organisation will consider different options – whether to extend systems it already has, with new modules or functionality, or to choose a new solution. The solution will be the best fit to the requirements but further modification will be required to meet the organisation’s full requirements. The extent of the development work will determine the timing and manner of the final implementation. It will often involve the use of external developers. Increasingly, organisations are looking for out of the box solutions, which although they can tailor the data tables, management reports, process work-flows and the user interface for their own use, will not require any major changes to the actual program code. This makes it easier to implement new patches and releases of the underlying software, as they are provided by the supplier.

Once the development is complete the software will be tested – first to ensure that it works, and then to ensure that it meets the user requirements and can process the transaction load expected, and that it will not adversely impact any other systems in use.

Other than providing early stage support and maintenance (sometimes referred to as the warranty period), the IT portion of the project is then complete. However, for the system to be successfully deployed, the project needs to also consider the area of business readiness. This will involve developing and implementing the supporting processes, agreeing a deployment plan for all locations/divisions (training users, ensuring controls still operate, etc.) and change management (to ensure that the new processes are used in the way that they were intended).

A project can be small and of a short duration, or can be long and expensive (the smallest I have been involved with was a few hundred pounds sterling – the largest was $2bn and lasted over ten years in total). All projects require appropriate governance to ensure they achieve their business benefits (see risks below). The usual mechanisms include:

•   A steering group or other regular meeting of senior stakeholders to monitor progress, identify any major risks and issues impacting the project, and ensure that the required outcomes can be achieved.

•   ‘Business process owner’ or other representation from the business that will monitor what is being delivered, answer any queries on requirements or process design, and ultimately accept the solution on the business’s behalf. If Agile is being used this will be the product owner.

•   A project management function or office (PMO) – to plan the project, co-ordinate the different elements of the project and report to the stakeholder’s/steering group. The PMO will also provide support to the project, for example in recruiting and on-boarding new staff, co-ordinating common activities, such as team meetings or testing, and in helping with liaison both within and outside of the project. The PMO will also be responsible for ensuring that the organisation’s agreed project management approach and methodology (such as PRINCE2 or PMBOK®) is applied effectively.

•   Quality or stage gate assessments – these are approval points during the project to assess progress and agree that the project can move on to the next stage or phase. The ultimate stage gate will be to place the software into the production environment so that it can be used by the business. There is usually a checklist of issues to be completed and approved during the stage gate assessment. In the case of the ‘Go/No Go’, approval to proceed may include provisional items which the project team have agreed to resolve in the short term.

•   Data quality – reviewing the quality of data before and after it has been migrated to the new system.

•   Post implementation reviews – a review to assess whether and how the project achieved its objectives.

In a traditional waterfall project approach, each of the phases illustrated in the above project lifecycle may be quite long, lasting several months and involving a number of sub-steps and approval gateways. In this approach each phase is completed before moving to the next – so for example development will not commence until all of the functionally has been designed. For large projects lasting many years, the requirements and business environment may change significantly and so the end product may be out of date before it has even been implemented. Costs and timelines can therefore be greater than anticipated. Also, there is a risk that the organisation may be missing business opportunities because it cannot respond to them quickly enough. To overcome this, organisations are adopting Agile project techniques, such as lean or scrum. These tend to focus on the outcomes of the project rather than the project process, and break the deliverable down into smaller iterations which are delivered at regular intervals (generally every 30 days). These approaches are typically less formal – the risks are minimised because of the small potential impact of each iteration. The same principles described above still apply, however, the documentation available for audit and review will be reduced. For further information see Agile Governance and Audit (ITGP 2014) by the same author.

Project lifecycle risks

When projects fail, the consequences for the business (and often personally for the project team) can be extremely significant. Just a quick Google of project failure will identify many instances of large and significant project failures – particularly in the public sector (e.g. see www.nao.org.uk/). There have been a number of surveys conducted to identify the reasons for project failure

The main risks associated with projects are that they fail to deliver:

1.  On time

2.  On budget

3.  The identified required business benefits

4.  Systems of sufficient quality or reliability, for example by introducing processing errors.

The main causes usually identified in the various surveys are:

•   Lack of senior level leadership, sponsorship or accountability for the project. Without their drive and commitment, the project is unlikely to succeed.

•   Poor clarity and consensus of vision as to what the project is trying to achieve.

•   Underestimating the complexity of the project, not only from the IT perspective but also from the perspective of the impact on existing business processes. This could include cultural differences if the project is amalgamating business processes from different locations or entities.

•   Scope creep – new requirements being added as the project progresses.

•   Poor project management techniques, including transparency, monitoring of actual costs, progress and risks. This may include challenging the information provided. One project manager told me he regarded all green (or satisfactory) statuses as potential watermelon–they may be hard and green on the outside but are red and slushy when you cut them open! Unless there is the challenge of reporting, there is a risk that reporting will be optimistic and that there will be little, or no warning of imminent failures.

Project lifecycle controls

The aim of controls over the project lifecycle is to reduce the risks of not achieving the required business benefits, excessive costs and delays.

To reduce the likelihood and impact of the risks listed above we would expect to see:

Table 7: Project lifecycle controls

Expected controls

COBIT 5 Best Practice

The organisation should have a standard methodology for project and programme management, including governance requirements, to ensure consistency and benefit from lessons learnt on previous projects.

BAI01-BP1 Maintain a standard approach for programme and project management

If there are a number of related and interdependent projects, a formal programme should be created to confirm expected benefits and ensure interdependencies are understood and not ignored by individual projects.

BAI01-BP2 Initiate a programme

There should be a plan to identify stakeholders and agree their needs and requirements from the activity. This includes agreeing their communication needs.

BAI01-BP3 Manage stakeholder management

The business case, scope and delivery of the programme should be planned and documented. This should be regularly reviewed and updated.

BAI01-BP4 Develop and maintain plan

The programme should be launched formally to increase awareness of the activity and its expected outcomes. This may include definition and planning of key stage gates.

BAI01-BP5 Launch and execute

Progress of the programme, and its constituent projects should be monitored so that any potential issues can be identified and rectified quickly before they have significant impact.

BAI01-BP6 Monitor, control and report on outcomes

Each project should be defined and scoped with formal approval by sponsors.

BAI01-BP7 Start and initiate projects within the programme

Activities within a project should be planned, showing tasks, responsibilities, key delivery dates, cost and resource requirements.

BAI01-BP8 Plan projects

A quality management plan should be used to manage the quality of the deliverable and should be reviewed and updated regularly.

BAI01-BP9 Manage project and programme quality

A risk framework should be in place to record risks as they are identified and track their resolution. This should include an assessment of the likely impact and clearly defined timelines and responsibilities for resolutions.

BAI01-BP10 Manage programme and project risk

The performance of the project should be monitored and reported.

BAI01-BP11/12 Monitor and control projects/manage project resources and work packages

The expected outcomes of the project should be clearly documented and a formal close conducted to demonstrate that they have been achieved. This should also include a lessons learnt review so that any strengths can be repeated on future projects and any weaknesses avoided.

BAI01-BP13 Close project

As for projects, the expected outcomes for a programme should be identified and after all projects are completed, checked before a formal close (including lessons learnt).

BAI01-BP14 Close a programme

In addition to the above, some organisations now have standards for application security to be included in the design and build of new systems (e.g. OWASP).

This will be increasingly important with current trends to make more use of standard (often Cloud-based) solutions so that what appears to be one system to the user is in fact many applications seamlessly joined together. In this type of design common application security is essential.

 

Project lifecycle case study examples

Travelling to nowhere

I was part of a review team looking at the implementation of a website and related online market places for a tour operator. I asked seven directors, all committed to the project, what its objective and expected outcome was. I got seven completely different answers ranging from to reduce costs, obtain new markets, keep up with competitors, etc.

Unless there is a clear understanding of the objectives and requirements for the project, how can you ever know whether or not it is successful? There was a real danger in this case that some of the directors were going to be disappointed with the outcome – even if it delivered the requirements of the others.

A tale of two projects

I was involved in a very important project for an airline, providing ongoing project assurance. I was very concerned about the lack of focus of the project team and their ability to deliver. So I asked the project sponsor (the acting CEO) to attend the meeting and set the context of the significance of the project to the business. An hour and a half into the meeting and he still had not appeared. He then stuck his head round the door and said – there’s a party downstairs, why are you not all there? The meeting abruptly broke up – not the outcome I wanted.

On another project with another organisation, at the start of a two-day business awareness workshop the CEO for the whole of the US attended and clearly stated the importance of the project and why it was important to the business. This focused the workshop and ensured we had a successful outcome.

Project lifecycle documenting, assessing and testing controls

There are two main aspects to system development audit:

1.  Audit of project/programme management.

2.  Audit of the project/programme management of a project or programme.

There is a good deal of debate as to the best timing for a review of application development. The concern is that if the auditor becomes too involved in the project, they become part of the project and therefore lose their independence. My personal view is that it depends on the level of the risk for a project. A high-risk project should have regular assurance reviews, either from internal audit or from some other independent assurer. The review should include all of the controls listed above – based on:

1.  enquiry of project management, team members and stakeholder representatives.

2. obtaining evidence to confirm controls are operating effectively, particularly:

   Documentation and monitoring of project/programme

   Risk assessment and resolution

   Programme/project governance frameworks.

It is important for the auditor to maintain a healthy professional scepticism when gathering evidence. Consider the following two examples:

1.  Control is that there are weekly project meetings involving all workstream leads, chaired by the PMO, and actions recorded. This looks good and at first look the 20-page progress pack with ten general slides, six workstream pages and list of actions is great. But when you look at a series of these you can start to see issues that have no progress that may indicate the project is not dealing with the critical design problem or interface that will cause it to fail.

2.  Control is that the PMO monitors weekly test results and assesses progress to completion. It doesn’t take long for an auditor to check the rate of testing, the rate of defect resolution and come up with a reality check on whether testing is going complete on time. So often this exposes the magical ‘hockey stick’ where somehow all the open tests and defects will resolve themselves in the last week!

Change management overview and risks

Within IT infrastructure and applications there will be frequent requests to make modifications and changes. The ITSMF (ITIL) definition of the mission of change management is:

 

“To manage all changes that could impact on IT’s ability to deliver services through a formal, centralised process of approval, scheduling and control to ensure that the IT infrastructure stays aligned to business requirements.”

I would make two minor changes to this – firstly to add ‘or software’ after IT infrastructure and secondly to add ‘and operates effectively and efficiently’ at the end of the last sentence.

The main risk is that if unauthorised or untested changes, both scheduled and emergency, are made to the production environment, the change may:

•   make the system/IT infrastructure unstable, or allow unauthorised access.

•   damage controls or other integrity of the application/IT infrastructure.

•   not work effectively in correcting the problem.

•   cause version control issues. For example, if two people try to change the same software programme at the same time, there is a risk that the changes made by one may overwrite the changes made by the other.

•   not be an effective use of resources, as a cheaper and simpler way of making the same change may have been available.

Change management controls

We would expect to see the following controls

Table 8: Change management controls

Control

COBIT 5 Best Practice ref

Authorisation included in the process to evaluate, prioritise and agree change requests.

BAI06-BP1 Evaluate, prioritise and authorise change requests

Arrangements in place for emergency changes including, restrictions on definition of what can be classed as an emergency change access control to production system, review and post-approval of change.

BAI06-BP2 manage emergency changes

Changes are tracked to ensure they are completed and tested. Also, reasons for rejection of a change are logged for reference if same change is re-submitted. This also enables root cause analysis to identify reasons behind common issues.

BAI06-BP3 Track and report change status

Formal closure of change requests when completed to confirm have met user requirement.

BAI06-BP4 Close and document changes

Change management case study examples

Bigger debt than expected

A senior member of IT staff left an organisation owing money for a car loan. The organisation agreed that he could work on Saturdays to work off the loan – even though he had left. The changes he made were conflicting with those made by other consultants on the new hardware during the week and so were causing additional re-work. It was cheaper and simpler to write-off his debt.

All change (NOT)

An organisation made non-standard changes to the core software of its ERP system. This meant that not only could it not take standard updates provided by the supplier but also it could not update the underlying operating system. This increased the risk of unauthorised access as security patches could not be applied, also other application systems became unstable as they were having to use old versions of the operating system.

Documenting, assessing and testing controls

An audit of change management is likely to include:

•   Review of the agreed process and supporting documentation

•   Enquiry and confirmation that the change process is applied effectively and consistently – including identifying any known exceptions, such as emergency changes

•   Obtaining a log of changes and review for approval

•   Review of sample of changes to ensure documented, approved and tested prior to promotion to production environment.

Summary

Change, including the implementation of new systems and processes, is inevitable. In order to reduce the risk of losing integrity of IT infrastructure and systems there is a need to ensure that appropriate controls are built in.