Kitabı oku: «Software Testing Foundations», sayfa 2
1 Introduction
Software is everywhere! Nowadays there are virtually no devices, machines, or systems that are not partially or entirely controlled by software. Important functionality in cars—such as engine or gear control—have long been software-based, and these are now being complemented by increasingly smart software-based driver assist systems, anti-lock brake systems, parking aids, lane departure systems and, perhaps most importantly, autonomous driving systems. Software and software quality therefore not only govern how large parts of our lives function, they are also increasingly important factors in our everyday safety and wellbeing.
Equally, the smooth running of countless companies today relies largely on the reliability of the software systems that control major processes or individual activities. Software therefore determines future competitiveness. For example, the speed at which an insurance company can introduce a new product, or even just a new tariff, depends on the speed at which the corresponding IT systems can be adapted or expanded.
High dependency on reliable software
Quality has therefore become a crucial factor for the success of products and companies in the fields of both technical and commercial software.
Most companies have recognized their dependence on software, whether relying on the functionality of existing systems or the introduction of new and better ones. Companies therefore constantly invest in their own development skills and improved system quality. One way to achieve these objectives is to introduce systematic software evaluation and testing procedures. Some companies already have comprehensive and strict testing procedures in place, but many projects still suffer from a lack of basic knowledge regarding the capacity and usefulness of software testing procedures.
Grass-roots knowledge of structured evaluation and testing
This book aims to provide the basic knowledge necessary to set up structured, systematic software evaluation and testing techniques that will help you improve overall software quality.
This book does not presume previous knowledge of software quality assurance. It is designed for reference but can also be used for self-study. The text includes a single, continuous case study that provides explanations and practical solutions for each of the topics covered.
This book is aimed at all software testers in all types of companies who want to develop a solid foundation for their work. It is also for programmers and developers who have taken over (or are about to take over) existing test scenarios, and it is also aimed at project managers who are responsible for budgeting and overall procedural improvement. Additionally, it offers support for career changers in IT-related fields and people involved in application approval, implementation, and development.
Especially in IT, lifelong learning is essential, and software testing courses are offered by a broad range of companies and individuals. Universities, too, are increasingly offering testing courses, and this book is aimed at teachers and students alike.
Certification program for software testers
The ISTQB® Certified Tester program is today seen as the worldwide standard for software testing and quality assurance training. The ISTQB® (International Software Testing Qualifications Board) [URL: ISTQB] coordinates qualification activities in individual countries and ensures the global consistency and comparability of the syllabi and exam papers. National Testing Boards are responsible for publishing and maintaining local content as well as the organization and supervision of exams. They also approve courses and offer accreditation for training providers. Testing Boards therefore guarantee that courses are of a consistently high standard and that participants end up with an internationally recognized certificate. Members of the Testing Boards include training providers, testing experts from industrial and consulting firms, and university lecturers. They also include representatives from trade associations.
Three-stage training scheme
The Certified Tester training scheme is made up of units with three levels of qualification. For more details, see the ISTQB® [URL: ISTQB] website. The basics of software testing are described in the Foundation Level syllabus. You can then move on to take the Advanced Level exam, which offers a deeper understanding of evaluation and testing skills. The Expert Level certificate is aimed at experienced software testing professionals, and consists of a set of modules that cover various advanced topics (see also section 6.1.2). In addition, there are syllabi for agile software development (foundation and advanced level) as well as special topics from the testing area (for example, Security Tester, Model-Based Tester, Automotive Software Tester).
This book covers the contents of the Foundation Level syllabus. You can use the book for self-study or in conjunction with an approved course.
Chapter overview
The topics covered in this book and the basic content of the Foundation Certificate course are as follows:
Software testing basics
Chapter 2 discusses the basics of software testing. Alongside the concepts of when to test, the objectives to aim for, and the required testing thoroughness, it also addresses the basic concepts of testing processes. We also talk about the psychological difficulties that can arise when you are looking for errors in your own work.
Lifecycle testing
Chapter 3 introduces common development lifecycle models (sequential, iterative, incremental, agile) and explains the role that testing plays in each. The various test types and test levels are explained, and we investigate the difference between functional and non-functional testing. We also look at regression testing.
Static testing
Static testing (i.e., tests during which the test object is not executed) are introduced in Chapter 4. Reviews and static tests are used successfully by many organizations, and we go into detail on the various approaches you can take.
Dynamic testing
Chapter 5 addresses testing in a stricter sense and discusses “black-box” and “white-box” dynamic testing techniques. Various test techniques and methods are explained in detail for both. We wrap up this chapter by looking at when it makes sense to augment common testing techniques using experience-based or intuitive testing techniques.
Test management
Chapter 6 discusses the organizational skills and tasks that you need to consider when managing test processes. We also look at the requirements for defect and configuration management, and wind up with a look at the economics of testing.
Test tools
Testing software without the use of dedicated tools is time-consuming and extremely costly. Chapter 7 introduces various types of testing tools and discusses how to choose and implement the right tools for the job you are doing.
Most of the processes described in this book are illustrated using a case study based on the following scenario:
Case Study: Virtual-ShowRoom VSR-II
A car manufacturer has been running an electronic sales system called VirtualShowRoom (VSR) for over a decade. The system runs at all the company’s dealers worldwide:
Customers can configure their own vehicle (model, color, extras, and so on) on a computer, either alone or assisted by a salesperson. The system displays the available options and immediately calculates the corresponding price. This functionality is performed by the DreamCar module.
Once the customer has selected a configuration, he can then select optimal financing using the EasyFinance module, order the vehicle using the JustInTime module, and select appropriate insurance using the NoRisk module. The FactBook module manages all customer and contract data.
The manufacturer’s sales and marketing department has decided to update the system and has defined the following objectives:
VSR is a traditional client-server system. The new VSR-II system is to be web-based and needs to be accessible via a browser window on any type of device (desktop, tablet, or smartphone).
The DreamCar, EasyFinance, FactBook, JustInTime, and NoRisk modules will be ported to the new technology base and, during the process, will be expanded to varying degrees.
The new ConnectedCar module is to be integrated into the system. This module collects and manages status data for all vehicles sold, and communicates data relating to scheduled maintenance and repairs to the driver as well as to the dealership and/or service partner. It also provides the driver with various additional bookable services, such as a helpdesk and emergency services. Vehicle software can be updated and activated “over the air”.
Each of the five existing modules will be ported and developed by a dedicated team. An additional team will develop the new ConnectedCar module. The project employs a total of 60 developers and other specialists from internal company departments as well as a number of external software companies.
The teams will work using the Scrum principles of agile development. This agile approach requires each module to be tested during each iteration. The system is to be delivered incrementally.
In order to avoid complex repeat data comparisons between the old and new systems, VSR-II will only go live once it is able to duplicate the functionality provided by the original VSR system.
Within the scope of the project and the agile approach, most project participants will be confronted or entrusted with test tasks to varying degrees. This book provides the basic knowledge of the test techniques and processes required to perform these tasks. Figure 1-1 shows an overview of the planned VSR-II system.
Fig. 1-1 VSR-II overview
Certified Tester syllabus and exam
The appendices at the end of the book include references to the syllabus and Certified Tester exam, a glossary, and a bibliography. Sections of the text that go beyond the scope of the syllabus are marked as side notes.
The book’s website
The book’s website [URL: Softwaretest Knowledge] includes sample exam questions relating to each chapter, updates and addenda to the text, and references to other books by authors whose work supports the Certified Tester training scheme.
Web-based Training System vsr.testbench.com
We have put a free implementation of VSR-II as a test object online for training purposes1. It reproduces the VSR-II examples included in the book on a realistic, executable system, so you can “test” live to find the software bugs hidden in VSR-II by applying the test techniques presented in the book. It takes just a few mouse clicks to get started:
1 Open your browser and load vsr.testbench.com
2 Create your personal VSR-II training workspace
3 Log into your VSR-II workspace and start
Fig. 1-2 VSR-II Training System Login-Screen
Also included in your registration for a VSR-II training workspace is a free basic license for the test management system TestBench CS, which includes the VSR-II test specification as a demo project and several of the VSR-II test cases presented in the book.
You can use TestBench CS not only for learning and training, but also for efficient testing of your own “real” software. A description of all features can be found at [URL: TestBench CS].
Many thanks to our colleagues at imbus Academy, imbus JumpStart and imbus TestBench CS Development Team for this awesome implementation of the VSR-II Case Study as a web-based training system.
2 Software Testing Basics
This introductory chapter explains the basic concepts of software testing that are applied in the chapters that follow. Important concepts included in the syllabus are illustrated throughout the book using our practical VSR-II case study. The seven fundamental principles of software testing are introduced, and the bulk of the chapter is dedicated to explaining the details of the testing process and the various activities it involves. To conclude, we will discuss the psychological issues involved in testing, and how to avoid or work around them.
2.1 Concepts and Motivations
Quality requirements
Industrially manufactured products are usually spot-checked to make sure they fulfill the planned requirements and perform the required task. Different products have varying quality requirements and, if the final product is flawed or faulty, the production process or the design has to be modified to remedy this.
Software is intangible
What is generally true for industrial production processes is also true for the development of software. However, checking parts of the product or the finished product can be tricky because the product itself isn’t actually tangible, making “hands-on” testing impossible. Visual checks are limited and can only be performed by careful scrutiny of the development documentation.
Faulty software is a serious problem
Software that is unreliable or that simply doesn’t perform the required task can be highly problematic. Bad software costs time and money and can ruin a company’s reputation. It can even endanger human life—for example, when the “autopilot” software in a partially autonomous vehicle reacts erroneously or too late.
Testing helps to assess software quality
It is therefore extremely important to check the quality of a software product to minimize the risk of failures or crashes. Testing monitors software quality and reduces risk by revealing faults at the development stage. Software testing is therefore an essential but also highly complex task.
Case Study: The risks of using faulty software
Every release of the VSR-II system has to be suitably tested before it is delivered and rolled out. This aims to identify and remedy faults before they can do any damage. For example, if the system executes an order in a faulty way, this can cause serious financial problems for the customer, the dealer and the manufacturer, as well as damaging the manufacturer’s image. Undiscovered faults like this increase the risk involved in running the software.
Testing involves taking a spot-check approach
Testing is often understood as spot-check execution1 of the software in question (the test object) on a computer. The test object is fed with test data covering various test cases and is then executed. The evaluation that follows checks whether the test object fulfills its planned requirements.2
Testing involves more than just executing tests on a computer
However, testing involves much more than just performing a series of test cases. The test process involves a range of separate activities, and performing tests and checking the results are just two of these. Other testing activities include test planning, test analysis, and the design and implementation of test cases. Additional activities include writing reports on test progress and results, and risk analysis. Test activities are organized differently depending on the stage in a product’s lifecycle. Test activities and documentation are often contractually regulated between the customer and the supplier, or are based on the company’s own internal guidelines. Detailed descriptions of the individual activities involved in software testing are included in sections 2.3 and 6.3.
Static and dynamic testing
Alongside the dynamic tests that are performed on a computer (see Chapter 5), documents such as requirement specifications, user stories, and source code also need to be tested as early as possible in the development process. These are known as static tests (see Chapter 4). The sooner faults in the documentation are discovered and remedied, the better it is for the future development process, as you will no longer be working with flawed source material.
Verification and validation
Testing isn’t just about checking that a system fulfills its requirements, user stories, or other specifications; it is also about ensuring that the product fulfills the wishes and expectations of its users in a real-world environment. In other words, checking whether it is possible to use the system as intended and making sure it fulfills its planned purpose. Testing therefore also involves validation (see Principle #7 in section 2.1.6—“Absence-of-errors is a fallacy”).
No large system is fault-free
There is currently no such thing as a fault-free software system, and this situation is unlikely to change for systems above a given degree of complexity or those with a large number of lines of code. Many faults are caused by a failure to identify or test for exceptions during code development—things like failing to account for leap years, or not considering constraints when it comes to timing or resource allocation. It is therefore common—and sometimes unavoidable—that software systems go live, even though faults still occur for certain combinations of input data. However, other systems work perfectly day in day out in all manner of industries.
Freedom from faults cannot be achieved through testing
With the exception of very small programs, even if every test you perform returns zero defects, you cannot be sure that additional tests won’t reveal previously undiscovered faults. It is impossible to prove complete freedom from faults by testing.
2.1.1 Defect and Fault Terminology
The test basis as a starting point for testing
A situation can only be classed as faulty if you define in advance what exactly is supposed to happen in that situation. In order to make such a definition, you need to know the requirements made of the (sub)system you are testing as well as other additional information. In this context, we talk about the test basis against which tests are performed and that serves as the basis for deciding whether a specific function is faulty.
What counts as a defect?
A defect is therefore defined as a failure to fulfill a predefined requirement, or a discrepancy between the actual behavior (at run time or while testing) and the expected behavior (as defined in the specifications, the requirements, or the user stories). In other words, when does the system’s behavior fail to conform to its actual requirements?
Unlike physical systems, software systems don’t fail due to age or wear. Every defect that occurs is present from the moment the software is coded, but only becomes apparent when the system is running.
Faults cause failures
System failures result from faults and only become apparent to the tester or the user during testing or at run-time. For example, when the system produces erroneous output or crashes.
We need to distinguish between the effects of a fault and its causes. A system failure is caused by a fault in the software, and the resulting condition is considered to be a defect. The word “bug” is also used to describe defects that result from coding errors, such as an incorrectly programmed or forgotten instruction in the code.
Defect masking
It is possible that a fault can be offset by one or more other faults in other parts of the program. Under these circumstances, the fault in question only becomes apparent when the others have been remedied. In other words, correcting a fault in one place can lead to unexpected side effects in others.
Not all faults cause system failures, and some failures occur never, once, or constantly for all users. Some failures occur a long way from where they are caused.
A fault is always the result of an error or a mistake made by a person— for example, due to a programming error at the development stage.
People make errors
Errors occur for various reasons. Some typical (root) causes are:
All humans make errors!
Time pressure is often present in software projects and is a regular source of errors.
The complexity of the task at hand, the system architecture, the system design, or the source code.
Misunderstandings between participants in the project—often in the form of differing interpretations of the requirements or other documents.
Misunderstandings relating to system interaction via internal and external interfaces. Large systems often have a huge number of interfaces.
The complexity of the technology in use, or of new technologies previously unknown to project participants that are introduced during the project.
Project participants are not sufficiently experienced or do not have appropriate training.
A human error causes a fault in part of the code, which then causes some kind of visible system failure that, ideally, is revealed during testing (see figure 2-1: Debugging, see below). Static tests (see Chapter 4) can directly detect faults in the source code.
System failures can also be caused by environmental issues such as radiation and magnetism, or by physical pollution that causes hardware and firmware failures. We will not be addressing these types of failures here.
Fig. 2-1 The relationships between, errors, faults, and failures
False positive and false negative results
Not every unexpected test result equates to a failure. Often, a test will indicate a failure even though the underlying fault (or its cause) isn’t part of the test object. Such a result is known as a “false positive”. The opposite effect can also occur—i.e., a failure doesn’t occur even though testing should reveal its presence. This type of result is known as a “false negative”. You have to bear both of these situations in mind when evaluating your test results. Your result can also be a “correct positive” (failure revealed by testing) or a “correct negative” (expected behavior confirmed by testing). For more detail on these situations, see section 6.4.1.
Learning from your mistakes
If faults and the errors or mistakes that cause them are revealed by testing it is worth taking a closer look at the causes in order to learn how to avoid making the same (or similar) errors or mistakes in future. The knowledge you gain this way can help you optimize your processes and reduce or prevent the occurrence of additional faults.
Case Study: Vague requirements as a cause of software faults
Customers can use the VSR EasyFinance module to calculate various vehicle- financing options. The interest rate the system uses is stored in a table, although the purchase of vehicles involved in promotions and special offers can be subject to differing interest rates.
VSR-II is to include the following additional requirement:
REQ: If the customer agrees to and passes an online credit check, the EasyFinance module applies an interest rate from a special bonus interest rate table.
The author of this requirement unfortunately forgot to clarify that a reduction in the interest rate is not permissible for vehicles sold as part of a special offer. This resulted in this special case not being tested in the first release. In turn, this meant that customers were offered low interest rates online but were charged higher rates when billed, resulting in complaints.
