Software is no longer just a hidden component within products; in many ways it has become the component that differentiates product in today's crowded marketplace. In today’s world which is increasingly run by software, failure caused by bugs have never seen more visible or high profile. Additionally consumers are more empowered in today's information age. Which means that product shipped with defective or buggy software damages the brand and reputation of a company which might have been cultivated over the years.
A 2013 study from Jude Business School of the University of Cambridge found that the global cost of debugging software has risen to $312 billion annually. This number tells us that software industry has a serious quality problem and companies are spending incredible amount of money to try fixing this issue. One of the major causes of software bugs is inefficient and incomplete testing. Below image shows the cost of fixing bugs in software development life cycle.
From the above image we can deduce that the cost of fixing bugs is very low at initial stages of software development, where as cost of fixing bugs increases exponentially in later stage of development and deployment. So we do know that there is a serious software quality problem and fixing these in earlier stages is most cost effective, yet there are serious gaps when we look at individual team level. For example unit testing is not given required attention and lot of organization delegate white-box/unit testing to some third party or engineers who have very little to no knowledge of software under test.
What Is White-box Testing?
Before jumping in to the importance of unit testing and how to write efficient unit test, lets try to understand what is white box testing.
White-box testing also known as structural testing or code based testing is a methodology which ensures and validates a software applications mechanism, internal framework and components. This method of testing not only verifies a code as per the design specifications but also uncovers an application’s vulnerabilities.
During the white box testing phase, a code is run with pre-selected input values to validate the per-selected output values. If a mismatch is found, it implies that the software application is marred by a bug, either due to wrong or incomplete requirement specification, or due to implementation of software by engineer. This process also involves writing software code stubs and drivers in manual testing approach. Unit and Integration testing is type of white-box testing approach.
Why Unit Testing?
In order to make sure the software function as intended and continue to work as intended, it needs to be tested. Identifying and fixing bugs at unit testing phase is much easier and economical. Unit tests also play very crucial role in refactoring stage as it makes easier to identify cause of regression and failure in code base as an when engineers develop new features or modifies existing code.
Fundamental Principals Of Unit Testing
In order to do efficient unit testing we need to understand few fundamental concepts which are listed below:
Start Verifying Which Is Known
When starting unit testing start with things which we know. Try testing with known and fixed set of input as it will produce know and fixed set of output. If we are testing function which calculates square root of number then it would be better to start with know values like 100 whose square root is 10 and we know this for sure, instead trying of trying with some random number like 19. Starting with known input allow us to easily verify whether output is correct or not. If selected input data does not give known output data then find input data which does and use that data to build your unit test first. For inputs values whose output in not known then we can write characterization test. For example if we don’t know what is the correct output for foo(17) then we should write testcase with ASSERT_GET_VALUE(foo) or use expected/out value, according to your testing framework, which will produce correct output. In cases where output values have tolerance then using range of values for expected output is recommended approach.
Write Your Test First
Writing your test first before writing actual code is known as Test Driven Development or TDD. Writing test before writing model code allows developers to think in terms of end user, who will be using our library or application. TDD allows developers to think in terms of interfaces, without worrying about model code implementation. In order to write test before implementation requires developers to go through specs. Which means, tests become tangible asset and will remain constant through out the development lifecycle. Test first approach helps in creating better API as developer think in terms of user. Another advantage of test first approach is that it hides implementation and avoid exposing internal details, which avoid brittle and tightly coupled tests.
Unit Also Mean Independent
While performing unit testing we isolate file/unit under test from rest of the project dependencies, to verify its implementation. Its important to understand importance of independent nature of unit test, as tests can run in any order and they can also run in parallel in multiple thread. Hence unit tests should not interfere with each other as it tends to create race and dependencies while using in test runner.
Tests And Thread Safety
While writing unit test one should avoid using synchronization, semaphores or special data structures, as it can results in test failure which are very difficult to debug. Hence it is important to maintain independent nature of unit tests and avoid sharing test vector or data between tests.
Unit Test Should Run Fast
As a developers we write lot of tests and hence it is very important that framework which we use to perform unit test should allow faster test execution. For simple single test, it should be less than few seconds and for complete test suite couple of minutes. Although test execution in couple of second may not be feasible for many of the embedded software development due to inherent delay caused by flash erase and write cycles, but we should strive to reduce total execution time. If embedded target allows execution from RAM then using this option we can speed up total execution time significantly.
In case where we’ve large tests we should segregate them into additional suites. This allows us to fail fast as we can schedule test execution of slow tests at the end. For example integration test usually take long time and hence it better to put such test in the end of the list.
Suppress Unnecessary Logs
Sometime additional data can create unnecessary confusion. Hence a passing test should not produce any output. There should never be any confusion as whether a test has passed or not. It should be either green or red. Whereas a failing test should produce clear output. A failing test should give clear and unambiguous error message which help understanding exact root cause of failure. Hence it is also important to understand that we should not rely on same set of test data and we should rotate test data. For example, avoid setting all integers of test to 5 instead use 5, 23, 834, -46 etc, as this makes much easier to understand why test is failing.
Few Good Practices
Avoid adding tons of asserts into your test. Add one assert per test method. If we add ten asserts in a testcase and it fails then it becomes difficult to find source of assertion.
Avoid using single testcase to verify multiple requirements. For example do not feel compelled to stuff all your test for foo into foo_test. Each test that need slightly different initialization or setup should go into separate test.
Avoid using time or network dependencies as it can create issue if another developer tires same test who is in another time zone or network dependencies changed.
While debugging bugs try to understand root cause first and then create testcase which fails to reproducer bug in clean and predictable manner. Failing test will make sure we've understood correctly what is the root cause of bug.
Avoid source of flakiness in unit test which arises due to time dependence, network availability, explicit randomness and multi threading.
When refactoring code make sure you’ve existing tests and they should fail if you break the code base. If tests are not available then first make sure you add test as it will uncover bugs in existing code base and save days of debugging time if you directly do unsafe refactoring. Also capture details of code coverage before doing refactoring.
Comments