Chapter 13 Software Quality Assurance Plan – Software Quality Assurance

Chapter 13
Software Quality Assurance Plan

After reading this chapter you will be able to:

  • use the information provided in each chapter to develop a complete SQA plan for a project;
  • understand the SQA requirements presented in the IEEE 730 standard;
  • refer to the detailed explanations in the appropriate chapter and section of the book.

13.1 Introduction

This chapter is devoted to the use of concepts and practices presented in this book to implement a software quality assurance plan (SQAP). Figure 13.1, adapted from Daniel Galin's house of software quality, connects all of the book's concepts as components that must come together to achieve software quality.

Figure 13.1 The house of quality for software projects.

Source: Adapted from Galin (2017) [GAL 17].

Before diving into the development of a SQAP, it is worth reviewing the definition of SQA as described in the following text box.

IEEE 730 states that the term “SQA function” is not to be interpreted as a particular person, tool, document, job title, or a specific group dedicated to SQA, regardless of how that function is staffed, organized, or executed. The SQA function's responsibility is to produce and collect evidence that forms the basis for giving a justified statement of confidence that the software product conforms to its established requirements [IEE 14].

Two clauses of IEEE 730 are dedicated to the planning and execution of the SQAP: clause 5.3.3 entitled “Document SQA planning,” and clause 5.3.4 entitled “Execute the SQA Plan.” In the first part of this chapter, we present the SQA planning, and in the second part, we will present the execution of the SQAP.

Thirteen mandatory tasks that are part of clause 5.3.3 describe what has to be done by the SQA function during the software planning stage of a project. IEEE 730 states that SQA activities are planned and executed in a manner that is commensurate with product risk—the higher the product risk, the greater the breadth and depth of SQA activities.

As can be seen in the following text box, the IEEE 730 standard [IEE 14] describes the normative outline of a SQAP.

One task included in clause 5.3.3, task 12, is quite clear about the content and the development of the SQAP [IEE 14]: address all of the topics listed in the normative SQA plan outline. Every section in the plan is to be included. The topics of the SQAP are normative but the section names and order are informative. If a section in the outline is not applicable to a given project, a placeholder for that section may be included along with a justification for why the topic is not applicable. This clause must be respected if an organization wants to claim conformance to IEEE 730. If an organization does not have to conform to IEEE 730, it may use this chapter as a set of guidelines in the development of a SQAP commensurate with the risks of the product to be developed.

Preparing and obtaining sign off on a SQAP is often the project manager's or the quality assurance manager's responsibility. SQA can help by offering a standardized template, examples, and explanations that can assist with this task. Organizations with an existing quality assurance function should have developed a SQAP template adapted to their internal methodology. A quality plan, which describes the quality activities and tasks to be executed during a software project, should be created from this template, approved by stakeholders, and kept up to date throughout the project. Exemplary SQAPs (i.e., from previous projects) can be made available in the life cycle process library to facilitate the development of a project-specific SQAP and to show newcomers examples of past projects in order to help them understand the typical content required.

13.2 SQA Planning

The following sections present, in more detail, the contents of each section of the SQAP according to Annex C of the IEEE 730 standard, and refer to specific chapters and sections of this book where detailed information has already been provided on the topic. Annex C also provides “suggested inputs” to help with the preparation and execution of a SQAP.

13.2.1 Purpose and Scope

This first section of a SQAP should describe the purpose, objectives, and scope of quality assurance activities that will be undertaken, including waivers obtained if present. This section should also explicitly identify processes and software life cycle products covered by quality assurance activities. It is a good idea to summarize which business model is associated with the specific project. You can find information about the different IT business models and their influence on quality assurance practices in section 1.6 of this book.

Here are questions that the SQA function can ask during the project planning phase to help refine the purpose and scope of SQA for a new software project (IEEE 730, Table C.3):

  • Is the project scope clearly defined and well understood?
  • Is the SQA role on this project understood by the acquirer, the organization, the project team and the SQA team?
  • Are potential product risks known and well documented?
  • Are potential product risks understood so that SQA activities can be planned in a manner commensurate with product risk?

13.2.2 Definitions and Acronyms

This section of the SQAP ensures that abbreviations and acronyms are explicitly stated. Refer to Chapter 1 to review information about the terminology that should be used consistently by software engineers when discussing quality. This is important because using recognized and standardized terminology that is fully aligned with approved software engineering standards will ensure that you can use your body of knowledge when a problem occurs. If a debate occurs, during final testing and delivery, about the meaning of a deliverable or a responsibility, you will always be able to fall back on your standards and body of knowledge to explain the intended meanings and solve these issues rapidly. It also ensures definitions, acronyms, and abbreviations are clarified for the project participants.

13.2.3 Reference Documents

This section of the SQAP identifies all applicable standards, industry-specific regulations and compliance clauses of contracts, other documents referenced by the SQAP, and any relevant supporting documents. Supporting documents may include applicable professional, industry, government, corporate, organizational, and project-specific references. Here are the questions that should be answered during the project-planning phase (IEEE 730, Table C.4):

  • What government regulations are applicable to this project?
  • What specific standards are applicable to this project?
  • What organizational reference documents (such as standard operating procedures, coding standards, and document templates) are applicable to this project?
  • What project-specific reference documents are applicable to this project?
  • Is SQA expected to assess compliance with applicable regulations, standards, organizational documents, and project reference documents?
  • What reference documents are appropriate to include in the SQAP?

The configuration management chapter of this book has presented how to refer, safeguard and manage successive versions of key project documents and life cycle artifacts. Documents referred to may include industry, legislative, or contractual documents pertaining to the project. The SQAP should also identify the origin of the project's requirements such as contracts, specifications documents, or deliverables list. Key project documentation and mandatory deliverables should be described with more detail, such as:

  • the mandatory project deliverables that will be monitored, reviewed, and authorized during this project;
  • where these documents/deliverables can be accessed;
  • a reference to existing templates and examples, when available, that will help participants better understand the expected scope and content to avoid misunderstandings;
  • identification of the role or individual who will be responsible for creating the deliverables and authorizing changes once official versions are published and finalized.

The benefit of this section of the SQAP is that by trying to produce Table 13.1 at the beginning of a project, the project manager will be forced to: (1) identify mandatory deliverables and (2) see if all the resources are available for the task ahead.

Table 13.1 Example of Key Documents and Mandatory Deliverables Checklist for the Project

Document name File name Location Reference template used Authors Approbation
Project plan Project_143_ plan.doc C :Project_ 143-\Plan Yes M. Smith A.Anderson
Functional specifications Project_143_ specs.doc C :Project_ 143-\Specs Yes D. Connor B.Thomas, P.Rodriguez

It becomes a useful checklist during project planning. Table 13.1 shows an example of the description of two mandatory software project deliverables. Organizations must decide on their minimum list of mandatory deliverables to be produced during a software project to ensure that the software meets the internal requirements of their internal software development methodology.

When reviewing the SQAP and during project reviews, the following questions should be asked:

  • Is this table complete? Are all the mandatory documents/deliverables listed (i.e., refer to the mandatory deliverables mentioned in the organization's methodology) for the project?
  • Are the documents/deliverable templates identified and used by the project?
  • Is there a SQA activity in the project schedule to review the quality of documents/deliverables?
  • Can I easily find and access the content of the documents/deliverables?
  • Who has access to the documents/deliverables? Is it the authorized and most recent version?
  • Are the project documents/mandatory deliverables under version control?

13.2.4 SQAP Overview—Organization and Independence

Software project organization and project management, as a whole, are topics that have not been covered by this book. They are important topics that impact software quality.

This section of the SQAP aims at clarifying the parties responsible for performing SQA, for a project, and seeks to present the interrelationship with the project management team. A functional organization chart often represents these relationships. If subcontractors are involved, the relationships and information flow between SQA and each subcontractor should also be shown. By clarifying relationships in this section of the SQAP it will be possible to see if the required roles and responsibilities are present. Here are the questions that SQA can ask during the project-planning phase concerning this topic (IEEE 730, Table C.5):

  • Have deficiencies in the organization's SQA policy been identified and documented?
  • Has the project manager established a method for monitoring the execution of SQA activities, tasks, and outcomes along with a method for providing feedback to the independent SQA function (when present)?
  • Has the project manager established an effective and appropriate policy defining and governing SQA roles and responsibilities during the project?
  • Has the organization established an independent SQA function with sufficient influence over software processes, including an effective reporting line independent of the software development project?
  • Has the organization established responsibility for supervising a project's SQA function by an individual independent of both the project manager and the software development manager?
  • Has the organization established a method to enable projects to learn from the experiences of previous projects, if the SQA function is being established for more than one project?
  • Does the independent SQA review exist independently of SQA processes established in individual projects?
  • Have adequate resources, including sufficient numbers of suitably skilled and trained people as well as sufficiently capable tools and equipment, been identified for the project?
  • Has the degree of independence (i.e., technical, managerial, and financial) been defined? (see Figure 13.2)
  • Is the defined degree of independence appropriate, given the potential product risk and the requirements when the project uses external suppliers?

Figure 13.2 Example of a software project functional organization chart for a large organization.

It is therefore suggested that the organizational structure of the software project be described and that the role of each participant is identified in this section of the plan. Figure 13.2 shows individual part of the management committee, project committee, and those supporting the project that will be asked to play a role.

Creating a visual representation of a project organization will quickly highlight if the numbers of individuals that have been assigned to the project are sufficient, if they are located at the right place and if they are qualified to do the work.

A RACI chart can also be used to describe each individual role. Rarely does everyone on a team have the same understanding of who is responsible for what. A RACI chart, also known as a RACI matrix, clarifies roles and responsibilities, making sure that nothing is forgotten. RACI charts also eliminate redundant activities and confusion by assigning clear ownership for each task or decision. It is important to identify early if a person is playing too many roles or, conversely, if a role is currently vacant/unassigned. Table 13.2 is another simple way of documenting roles and responsibilities of individuals in a software project.

Table 13.2 Specifying the Names of Individuals Who are Involved in a Project

Role Assigned to Responsibilities
Project committee A. Lopez (Pilot), G. Wright (IT Responsible), M. Thomas (SQA), P. Smith (IT expert) Follow the progress, address issues, set project directions and priorities, authorize budgets and changes.
Key end-user P. Clark, H. Johnson Validate requirements, functional experts, conduct functional acceptance of system.

This section of the SQAP should also specify the degree of independence between the organization performing the SQA function and the project team members. There are three parameters that can be used to define independence: (1) technical independence, (2) managerial independence, and (3) financial independence. Technical independence requires that SQA utilize personnel who are not involved in the development of the system or its elements. SQA forms its own assessments of all project activities. Technical independence is an important method to detect subtle errors overlooked by those too close to the solution. Managerial independence requires that responsibility for SQA be vested in an organization separate from the software development and program management organizations. Managerial independence also means that SQA independently selects segments of software to analyze and test, chooses techniques, defines the schedule of SQA activities, and selects the specific technical issues and problems to act upon. The SQA effort provides its findings in a timely fashion simultaneously to both the software development and program management organizations. Financial independence requires that control of the SQAP budget be vested in an organization independent of the software development organization. This independence prevents situations where SQA cannot complete its activities because funds have been diverted or adverse financial pressures or influences have been exerted.

13.2.5 SQAP Overview—Software Product Risk

The chapter on risk management has presented how risk relates to SQA. Software product risk refers to the inherent risks associated with the use of the software product (e.g., safety risk, financial risk, security risk). Software product risk is different from project management risk, which is addressed later in this plan. Specific V&V techniques may be required to address software product risk. Table 13.3 lists the questions and suggested inputs, such as a RMP, that SQA can discuss during the project-planning phase (IEEE 730, Table C.7).

Table 13.3 Questions and Suggested Inputs Related to Software Product Risk to Consider Asking During Project Planning Phase [IEE 14]

Questions Suggested inputs
  • Are potential product risks known and well documented?
  • Are potential product risks understood so that SQAP activities can be planned in a manner commensurate with product risk?
  • Has the scope of product risk management to be performed been determined?
  • Have appropriate product risk management strategies been defined and implemented?
  • Is a criticality analysis planned?
  • Does the project team have adequate training in product risk management techniques?
  • Is the project team planning to adjust their activities and tasks in a manner commensurate with product risk?
  • Are the breadth and depth of planned SQAP activities commensurate with product risk?
  • Acquisition plan
  • Contract
  • Concept of operations
  • Risk management plan

To explain the extent of risk management activities, this section of the SQAP should clearly state what level of integrity the software being worked on is at. Criticality analysis of software components, which includes four levels, was presented in the chapter on V&V. The SQAP must ensure that the tasks and activities identified are carried out in a way that is commensurate with the level of criticality of the assigned software. Depending on the software criticality, projects apply more or less strict engineering and SQA requirements. This important information will allow the reader to assess the number of compensating provisions required to mitigate the consequences of software failure.

13.2.6 SQAP Overview—Tools

This section of the SQAP describes tools to be used by SQA to perform specific tasks. These may include a variety of software tools to be used as part of the SQA process. Appropriate acquisition, documentation, training, support, validation, and qualification information for each tool is included in the SQAP. Here are the questions that SQA can ask during the project-planning phase:

  • Have adequate resources, including sufficiently capable tools and equipment, been identified for the project as well as for other projects if the SQA function is being established for multiple projects?
  • Are all tools planned to be used by SQA on this project completely identified, including supplier, version or release number, system platform requirements, tool description, and numbers of concurrent users?
  • Based on product risk, do these tools require validation before they can be used on this project?
  • Is training in the effective use of SQA tools required and if so, is training planned?

Regarding the tools, do not forget to specify the version of the tools used for a given project. It is also in this section of the plan that you should describe the tools and techniques that will be identified by the team for this project.

13.2.7 SQAP Overview—Standards, Practices, and Conventions

Chapter 4 of the book described software engineering standards and how they relate to SQA. This section of the SQAP identifies standards, practices, and conventions to be used in performing activities and tasks and for creating outcomes:

  • Have all laws, regulations, standards, practices, conventions, and rules invoked in the contract been identified?
  • Have specific criteria and standards against which all project plans are to be evaluated been identified and shared within the project team?
  • Have specific criteria and standards against which software life cycle processes (supply, development, operation, maintenance, and support processes including quality assurance) are to be evaluated been identified and shared with the project team?

Describe the techniques and methods to be used for the project. Chapter 9 discussed how policies, processes, and procedures are used by a project. For software projects involving external suppliers, it is important to clarify which methodology (e.g., Oracle Unified Method, IBM Rational Unified Process, Scrum) will be used and also to clarify the obligations to produce certain mandatory deliverables (see Table 13.2).

This section also lists which standards, practices, and conventions apply to this project. It will specify the stage of the life cycle, the list of intermediate deliverables as well the standards, guides, or conventions used to ensure that quality will be met during reviews, inspections, and final acceptance steps of the project. We have discussed many standards and models in Chapter 4.

Table 13.4 provides an example of a simple presentation of these concepts.

Table 13.4 Reference Table for Standards, Practices, and Conventions

Step of the life cycle Intermediary deliverable Standard, practice, orconvention
Planning Project plan PMI - PMBOK® Guide
Planning SQAP IEEE 730
Programming Source code Java programming rules revised April, 20 1999
Transition to production Technical documentation A local production criteria and checklist

By specifying these items in advance, we facilitate the task of reviewing the quality plan and project reviews for the project manager and SQA specialist.

13.2.8 SQAP Overview—Effort, Resources, and Schedule

This section includes estimates of the effort required to complete the activities, tasks, and outcomes as defined in the SQAP. This section identifies appropriately qualified SQA personnel and defines their specific responsibilities and authority within the context of the project.

This section also identifies additional SQA resources, including facilities, lab space, and special procedural requirements (e.g., security access rights and documentation control) that are required to perform SQA activities. This section should also include a list of critical SQA project milestones and a schedule of planned SQA activities, tasks, and outcomes. Here are the questions that SQA can ask during the project planning phase:

  • Can estimated effort and schedules be based on past projects?
  • Can resource requirements for this project be determined based on past projects?
  • What resources (lab spaces, servers, software, databases, operating systems, security rights, document control, etc.) are required for this project?
  • Is effort based on factual information rather than “gut feel”?
  • What estimating and scheduling techniques can be used for this project?

Typically, the estimates describe the software activities pretty well (i.e., feasibility, requirements, design, construction, testing, and delivery). But, have all the efforts associated with the cost of quality been considered in these estimates and the schedule (prevention costs, appraisal costs, failure costs)? Refer to section 2.2 on cost of quality for details.

For the assessment of the project schedule, this section of the SQAP should describe the project milestones and the planned SQA in the project schedule. Specifically, it indicates the deliverables that will be produced by the SQA activities required by the project manager and the project committee. A good schedule reflects all project activities and shows the individual assignment of personnel. To verify whether a scheduled assignment is realistic and reviewed, an assignment email could be sent to individuals during planning. A letter of assignment (also called a work assignment email) is to formally contact a project resource asking:

  • if a review of the planned activities represented in the proposed schedule has been done;
  • to confirm the durations;
  • if this planned assignment of work is correct and to confirm with a reply.

This simple approach validates the information described by the schedule. It works especially well with team members that do not report directly to the project manager. Finally, this section of the SQAP is also used to describe the qualifications of staff who will perform SQA activities during the project and to clarify their responsibilities, their degree of independence, and their authority to perform reviews, inspections, and ask for corrections. The plan should also identify additional resources, including facilities, tools, test labs and other requirements (e.g., security permissions and code/document repository access rights).

Every project and its plan is conceived and developed based on a set of hypotheses, scenarios, or assumptions. Assumptions analysis explores the validity of assumptions as they apply to the project. It identifies risks to the project from inaccuracy, instability, inconsistency, or incompleteness of assumptions [PMI 13]. The PMBOK® Guide recommends that assumptions made in developing the activity duration estimate, such as skill levels and availability, as well as a basis of estimates for durations, be documented in project documents. When developing a SQAP, assumptions should be documented and analyzed when estimating effort, resources, and schedules.

13.2.9 Activities, Outcomes, and Tasks—Product Assurance

Product assurance activities provide confidence that software products are developed in conformance to established product requirements, project plans, and contractual requirements. An important aspect of SQA is the establishment of confidence in the quality of the software products produced by the project. These products include not only the software and related documentation but also the plans associated with the development, operation, support, maintenance, and retirement of the software. A product may also be a software service provided to the acquirer. Product assurance activities may include SQA personnel participating in project technical reviews, software development document reviews, and software validation testing.

The outcome of the product assurance activities provides evidence that the software services, products, and any related documentation are identified in and comply with the contract and any non-conformances are identified and addressed. Product assurance is comprised of three activities:

  • Evaluate plans for conformance: This section of the SQAP should include activities and tasks for evaluating the degree to which all plans required by contract have been prepared and are consistent with the contract and with each other. Use reviews as described in an earlier chapter to do this evaluation;
  • Evaluate product for conformance: This section of the SQAP should identify activities and tasks related to the evaluation of the degree to which the software product and related documentation conform to established requirements, plans, and agreement. Refer to Chapter 7 for V&V and Chapter 5 on reviews for the required outcomes for this activity.
  • Evaluate product for acceptability: This section of the SQAP should identify activities and tasks for evaluating the level of confidence that the software products and related documentation will be acceptable to the acquirer prior to delivery. Refer to Chapter 7 for V&V and Chapter 12 on supplier and contract management.

Product assurance requires that some measurement be performed on software products. This section of the SQAP should also identify activities and tasks for evaluating whether the measurements objectively demonstrate the quality of the products in accordance with established standards and processes. Section 3.3 of the book describes a process to formally define software quality requirements in a project so that the measurement can be used during acceptance. Chapter 10 also presents measurement topics.

13.2.10 Activities, Outcomes, and Tasks—Process Assurance

Similar to the product assurance, process assurance aims to ensure that the processes used by the project are appropriate, depending on the level of integrity, and are followed:

  • Evaluate life cycle processes for conformance: Ensuring that the correct life cycle has been selected confirms that adaptation, if any, of the life cycle activities and deliverables are appropriate. This section of the SQAP should identify activities and tasks for determining the degree to which project life cycle processes and plans conform to the contract and the degree to which the execution of project activities conforms to project plans (e.g., life cycles selected and adapted, procedures, deliverables templates, gating, project plan, and responsibilities alignment with project/contract requirements). Refer to Chapter 9 on policies, processes and procedures as well as Chapter 12 on contract reviews.
  • Evaluate environments for conformance: This section of the SQAP should identify activities and tasks for evaluating whether software development environments and test environments conform to project plans (e.g., suitability of development and test platforms, i.e., planning, sizing, and security), tools validation, configuration management toolset, project documentation server/folders). Refer to Chapter 8 for guidance on configuration management;
  • Evaluate supplier's processes for conformance: This section of the SQAP should identify activities and tasks for evaluating whether supplier software processes conform to requirements passed down from the acquirer (e.g., contract reviews, supplier quality audit, supplier responsibilities review, subcontract work traced and managed by the system integrator correctly). Reviews, audits as well as supplier management have already been presented in earlier chapters.

This section also requires the use of reviews and audits in the project. The reviews were presented in Chapter 5 and audits in Chapter 6. Table 13.4 provides an example of a simple presentation of reviews and audits for the project. Configuration management concepts were also discussed in Chapter 8.

Process assurance requires that some measurement be done on software processes. This section of the SQAP should identify activities and tasks for evaluating whether the measurements support effective management of the processes in accordance with established standards and processes (e.g., appropriate set of measures identified and planned, a data collection and analysis process is selected, measurement is consistent with product risk and overall quality goals). Measurement was discussed in Chapter 10.

Finally, this section of the plan should assess staff knowledge and skill. It is necessary that the plan outlines whether the staff assigned to the project has the qualifications and training necessary. If a training plan is to be prepared and executed it typically has two specific audiences: (1) technical specialists who will need specific training and (2) user representatives who will perform the final acceptance as well as future end-users.

In this part of the SQA plan, describe the scope of training needs and the scope covered by the project. Here are the questions that SQA can ask during the project planning phase:

  • Have adequate resources, including sufficient numbers of suitably skilled and trained people, as well as sufficiently capable tools and equipment, been identified for the project as well as for other projects if the SQA function is being established for multiple projects?
  • Have required project skills been identified?
  • Have staff and subcontractor training records been reviewed against required skills?

13.2.11 Additional Considerations

This section of the SQAP identifies any additional considerations, such as SQA processes that support both Project Management and Organizational Quality Management, that are not described elsewhere. The following topics are included in this section of the SQAP:

  • Contract review process: This section of the SQAP identifies or references the contract review process and describes SQA roles and responsibilities with regard to contract reviews. In the case where there are suppliers, subcontractors, consultants, or integrators, this section will specify the most important clauses of this relationship. Chapter 5 review techniques can be used here. Supplier and contract management concepts were discussed in Chapter 12 such as:
  • describe the software acquisition process for the project;
  • identify the reviews that will be conducted by the supplier during the project;
  • identify the appropriate contract type;
  • specify which contract template will be used and the planned contract reviews.
  • Quality measurements: This section of the SQAP identifies quality measurements that are appropriate for the project, specific data collection requirements associated with intended quality measurements activities as well as responsibilities for data collection, measurement, and reporting. It should identify software quality objectives, measurement processes, and tools that are appropriate for this project. Before quality can be measured, it needs to be specified and accepted as part of the project requirements. This section of the SQAP should refer to the functional and non-functional quality requirements. Recommendations on how to define software quality requirements have also been presented in section 3.3 and show how to:
  • select and describe the quality characteristics and sub-characteristics to be evaluated;
  • specify how this will be measured and clarify the software attributes to be measured;
  • set a quality objective target for the project;
  • describe, with an example, how the calculation is done;
  • explain how and what tools will be used to evaluate this measure at the acceptance stage.
  • In addition to setting specific targets for quality measures, the plan should identify the measures that will be used to report:
  • project progress for each of the five project dimensions of a software project (e.g., schedule, staff, features, cost and quality) as presented in section 2.4 of Chapter 2 and referred to by the PSM in Chapter 11;
  • defects density and reviews/audits of non-conformances correction status.
  • Finally, this section of the SQAP should identify responsibilities for collection, validation of this data, and reporting. Table 13.5 outlines the planned reviews for a project.
  • Waivers and deviations: This section of the SQAP defines or references the criteria used to review and approve waivers and deviations to the contract and project management controls. The SQAP describes SQA roles and responsibilities with regard to reviewing and approving waivers and deviations. The following text box provides a text that could be added to a SQAP.

Table 13.5 Overview of Planned Reviews of a Project

Review name Objective Life cycle phase Intermediary product Type ofreview
Requirements Ensure the completeness/testability of requirements Requirements stage Functional specifications Document reviews
Architecture and design Ensure the maintainability and traceability of design to requirements Architecture and design stage Design documents Document reviews
Source code Ensure the conformity to local programming Programming stage Source code & unit test plans and results Code and documents inspection
Quality audit just before system tests Progress and readiness of products Integration tests stage Overall project Quality audit
  • Task repetition: This section of the SQAP defines or references the criteria used to determine when and under what conditions, SQA tasks previously completed need to be repeated. This section will describe, if present, the iteration tasks policy recommended after the identification of a defect (e.g., the rework needed to correct a defect). For example, we know that each functional unit of the software is subject to acceptance testing. At this time, the customer is making reasonable efforts to continuously accept and validate this basic functionality for each functional unit. The successful execution of the acceptance test on each functional unit is the means that constitute validation of the basic functionality of software. Should a defect or failure occur, it is necessary to specify in this section of the SQAP, the process that will be initiated to document the defect, assign a severity, assess the effort to correct the defect and retest this functional unit. Refer to rework in section 2.2 of Chapter 2.
  • Risks to performing SQA: This section of the SQAP identifies potential projects risks that could prevent SQA from accomplishing its defined purposes, activities, and tasks. Examples include inadequate staffing levels, insufficient resources, and lack of training. Also included in this section are actions taken to mitigate identified project risks. Chapter 11 presents the risk activities as they relate to SQA.
  • Communications strategy: This section of the SQAP defines the strategies for communicating SQA activities, tasks, and outcomes to the project team, management, and organizational quality management.
  • Document acceptance process: Often, and especially when suppliers are involved, the project manager will have to decide on a process to review/accept intermediary deliverables (i.e., requirements document, design documents, test plans, and many other deliverables that are in the form of documents). At the review meeting, a moderator may be assigned to record the minutes, which detail all corrections and improvements agreed to during the meeting (refer to Chapter 5 on reviews). Based on the severity of the required changes, a unanimous decision must be reached to:
  • accept the document as is (acceptance);
  • accept the document with minor changes (conditional acceptance);
  • repeat the review when the significant changes agreed to in the review meeting have been incorporated.
  • The review minutes are then signed by all participants. If the document has been conditionally accepted, an update of the document is prepared and the moderator reviews the minor changes to ensure all items have been addressed. The moderator then declares acceptance for this deliverable.
  • Non-conformance process: This section of the SQAP defines activities and tasks related to the process for reporting non-conformances for the project. Non-conformances can be reported by any project member but, at the acceptance stage, will be controlled by a configuration control board (CCB), as presented in section 8.7.2.
  • We have learned by experience that it is often necessary to clarify the software Verification and Validation (V&V) terminology, processes, and techniques (including testing details) that are planned for a project, especially if it involves third-party suppliers. Contracts and third-party project plans notoriously omit details about this important quality activity. The technical details about V&V should be presented in the quality and test plans of the project. Use the SQAP to provide a checklist that will identify all of the V&V activities. Chapters 5, 6, and 7 have presented reviews, audit, and V&V topics. What is important for SQA is to ensure that the project team chooses the appropriate techniques depending on the required level of integrity and quality that is expected from this project.
  • An initial clarification that needs to be documented in the SQAP is how the project will evaluate the quality of an intermediate deliverable (e.g., a document) during an approval meeting where a review is conducted. It is also important to clarify how a defect will be classified (see example in Table 13.6). Defect severity is often used in contracts with third parties to assess if a milestone has been achieved or not.
  • In third-party contracts, it is also necessary to specify the testing levels as it often causes confusion. You will probably need to create a process map of the project acceptance processes to ensure everyone understands how this will be done. For example, Figure 13.3 presents a simple activity sequence diagram that helps with an overall view of how this software will be finally accepted into production and will meet each test level. In this example, three different organizations are involved: the customer for functionality, maintainers for its maintenance and support quality and, finally by the infrastructure team who is concerned with the production readiness and reliability and many other operational aspects (using production criteria).
  • Acceptance process: Here is another example of system and integration testing (Figure 13.4). The system and integration test acceptance process consists of the following steps: the supplier provides a system and integration testing plan with an acceptance checklist at least one week in advance of the test. The project team reviews the plan and the checklist within one week (see the document acceptance process). Within 1 week of the supplier notifying the project team of the availability of the deliverable for the system and integration test, the project team attends a demonstration of the deliverable at which the test plan is executed and the acceptance checklist is completed. Should any problems be encountered, they are recorded as incidents (using a category from Table 13.6). The supplier and the project team jointly sign the incident report summary for this test. Based on the incident report summary for this deliverable, acceptance is achieved when no unresolved category 1 and 2 defects (incidents) are open. When any outstanding incidents are repaired, associated tests must be re-run.
  • Specify the acceptance criteria for the software. For example: the project will be accepted by the customer if:
  • 100% of the functionality approved in the requirements document has been delivered and there is no level 1 or level 2 incident report pending a fix;
  • The infrastructure and maintenance/support groups have no pending level 1 or 2 incident reports. If there is a level 1 or level 2 incident report, they will have to be addressed before final acceptance of the software.

Table 13.6 Defect Severity Classification Example

Category Description
1 A defect that prevents the project from testing further as this defect needs to be corrected before going ahead. A solution must be found immediately and corrected before continuing to test this functionality.
2 An important defect that prevents the project from testing parts of a functionality but other parts can continue to be tested. A solution needs to be found in a reasonable delay.
3 A defect that does not cause a major problem for testing to continue. A solution is required but can be placed in a priority list to be treated after category 1 and 2 are done.
4 A minor defect to be placed in the list of things to do.

Figure 13.3 Example of a workflow explaining the acceptance steps of a project.

Figure 13.4 Example of an acceptance process for a software project.

13.2.12 SQA Records

This section of the SQAP includes activities and tasks for analysis, identification, collection, filing, maintenance, and disposition of quality records. Quality records document that activities were performed in accordance with project plans and the contract. These records enable information sharing and support analysis to identify problems, causes, and eventually result in product and process improvements. The project manager and/or the SQA manager should ask the following questions related to the analysis, identification, collection, filing, maintaining, and disposition of SQA records during the project planning phase:

  • What set of records is the project required to produce?
  • What set of records is SQA required to produce?
  • Is the information required to be included on each record defined?
  • What mechanisms will be used to collect, file, maintain, and eventually dispose of quality records?
  • Who is responsible for collecting, filing, maintaining, and disposing of records?
  • Who is responsible for sharing records?
  • What records are to be shared with stakeholders?
  • What protections are required to be established in order to share these records, maintain the integrity of the records, and prevent their modification or inadvertent release?
  • What records are required from subcontractors?

13.3 Executing the SQAP

Once a SQAP has been developed and approved, the project has to execute it. The purpose of clause 5.3.4 of IEEE 730 is to “Execute the SQA Plan” in coordination with the project manager, the project team, and organizational quality management. The tasks linked to the execution of the SQAP are quite explicit. To accomplish this activity, the SQA function shall perform the following tasks [IEE 14]:

  1. execute the activities and tasks defined in the SQAP, based on project schedules;
  2. create the outcomes identified in the SQAP;
  3. revise the SQAP in response to project changes;
  4. raise non-conformances when actual outcomes do not agree with expectations.

Annex C of IEEE 730 provides, for each section of the SQAP, a list of questions when executing SQA project activities described in the SQAP. Table 13.7 presents an example of such a table for the questions and suggested inputs related to software product risk to consider asking during the project executing phase.

Table 13.7 Questions and Suggested Inputs Related to Software Product Risk to Consider Asking During Project Executing Phase [IEE 14])

Questions Suggested inputs
  • Are risks identified and analyzed as they develop?
  • Has the priority in which to apply resources to treatment of these risks been determined?
  • Are risk measures appropriately defined, applied, and assessed to determine changes in the status of risk and the progress of the treatment activities?
  • Has appropriate treatment been taken to correct or avoid the impact of risk based on its priority, probability, and consequence or other defined risk threshold?
  • Has a software integrity level scheme been defined for the project?
  • Has the software integrity level scheme been reviewed and determined to be appropriate?
  • Has a software integrity level been established, if appropriate?
  • Has a set of assurance cases been prepared?
  • Have the assurance cases been reviewed and determined to be appropriate and complete?
  • Has an appropriate risk assessment been performed and documented?
  • Risk management plan
  • Improvement plan
  • Monitoring and control report
  • Risk action request

Task 3 of clause 5.3.4 states that the SQA function shall revise the SQAP in response to project changes. A project is executed in an organization where processes, such as configuration management, have been documented and implemented. The SQAP must follow the organizational configuration management process when revising and updating the SQAP. Usually, a SQAP will have a table, called a revision or history table, listing the approval and revision information. The following text box briefly describes the tasks when producing an updated version of a SQAP.

13.4 Conclusion

The SQAP is the cornerstone of any software project aiming at producing a quality product for external or internal customers. The SQAP pulls together all quality concerns of an organization, its policies, its people, its processes, and tools in producing quality products while building a competitive software capability.

While supporting the development of quality products, a robust SQAP should minimize avoidable rework. In Chapter 2, we presented the concept of cost of quality, and in many chapters we have illustrated how SQA could help in reducing the cost of rework. Hopefully, the following statement made by Dr. Robert Charette will no longer be true for organizations that have implemented the SQA practices presented in this book.

13.5 Further Reading

  1. GALIN D. Software Quality: Concepts and Practice. Wiley-IEEE Computer Society Press, Hoboken, New Jersey, 2017, 726 p.
  2. SCHULMEYER G. G. (Ed.). Handbook of Software Quality Assurance. 4th edition. Artec House, Norwood, MA, 2008.

13.6 Exercises

  1. Name five topics to be addressed in a SQAP.

  2. Describe the review and acceptance of documents approach proposed to ensure quality.

  3. Describe the process of acceptance of an acceptance test and how to use the categories of defects to ensure quality of delivery.

  4. How important is the description of the criticality of the software at the beginning of a SQA plan?

  5. Develop a checklist that will allow you to ensure that the SQA plan conforms to IEEE 730.

  6. What is the difference between process assurance and product assurance?

  7. What could be the consequences of not having a SQA plan for your organization?

  8. What could be the consequences of not requiring a SQA plan from your supplier?