Have you ever encountered non-functioning infotainment system in aircraft? You are not alone. Today we are going to learn about such systems which are known as embedded system and how software gone inside is being tested.
What is embedded system?
Before jumping to our main topic i.e. embedded software testing lets dive into what we mean by embedded system. The old school definition of embedded system is: embedded system is some combination of computer hardware and software, either fixed in capabilities or programmable, that is designed for a specific function or for specific functions within a larger system. Industrial machines, agricultural and process industry devices, auto-mobiles, medical equipment, cameras, household appliances, air-planes, vending machines and toys as well as mobile devices are all possible locations for an embedded system. Some are simple and others are complex in nature.
Due to advancement of technological progress in IC(Integrated Circuit) the current micro-controllers/processors used in embedded systems are much more powerful than a decade ago. Hence line between general purpose computer (PC) and embedded system/computing unit are becoming more ambiguous day by day. For example current smart phones are much more computationally efficient and powerful a decade ago. Also current smart phones do not have fix capability (i.e. voice call and message) but it can perform almost all the task what general purpose computer can do like image processing, web browsing, running office application etc. In fact now a day standard smart phone comes with more than 4 GB of ram which cost couple of hundred dollars.
As days are progressing we are more and more becoming dependent on computers/embedded systems be it ABS of car or house hold air conditioning system. Even chances are if anything runs on electricity it must have small computer (aka embedded system) controlling some functionality. Please refer below image 2 which shows number of lines of software in different product. Depending upon area of application these embedded system may perform very critical functionality. And depending upon the criticality of application or system, special care has to be taken while developing such system.
Importance of embedded software testing!
Now we've got some idea about what is embedded systems, lets have a look at the importance embedded software testing and how it differ from traditional software testing (e.g. web technology). If we consider traditional software testing of our IT industry which mostly involves performing black box testing(of course this is an over simplification). Primary areas of testing are GUI checks, functionality, validation and some level of database testing. Since embedded system is combination of hardware and software, traditional software testing is not sufficient for delivering the quality product. Embedded software testing focus not only on the functionality of the software but also on under lying hardware, sensor, actuators etc. Depending upon the area of application product manufacturer have to follow strict guidelines while developing and testing embedded system in order to qualify for safe usage. For example DO-178B/C standard govern avionic industry for EE device development whereas ISO26262 standard is followed by automotive industry. (Note: There are other standard which need to be satisfied like for hardware DO-254 etc.)
Since we are heavily relying on embedded devices to control safety critical and business critical task, failure in embedded system be it due to hardware or software can lead to severe consequence. We've already seen lots of cases where failure in embedded software has caused lots of financial loss as well as loss of life. For example
1. Therac-25: overdose of radiation by Therac-25 (a machine for administering radiation therapy) due to arithmetic overflow.
2. Patriot Missile Bug: During gulf war of 1991 Patriot Missile was unable to shoot Iraqi scud missiles which were due to clock drift in patriot missiles on board computers. Drift was track down to 24 bit data type(floating point) was not able store time data accurately.
3. Unintended Acceleration: Due software bug in Toyota Lexus ES 350 sedan around 89 people have reportedly died and around 6,200 unintended acceleration cases have been raised. It's estimated that Toyota have paid $1.2 Billion as legal fees alone.
How embedded testing differ from other traditional SW testing?
Due to complex nature of these embedded system, software written to control over all system need special attention while developing and testing. Most widely used language for writing software are C, C++ and Ada(avionics only). Not all the language specifications are used while using these languages in embedded system software design. Depending upon the industry standards, different guidelines are specified. One of the most widely used coding guideline is MISRA(Motor Industry Safety Reliability Association) which initially developed for Automotive but later on adopted for other segments of industry which deals with safety and business critical system. Usually organization working on safety critical/business critical application/system follow traditional V model for product development (Of course this is an over simplification). There are organizations that are also using Test/Model Driven Development or Agile due to continue demand of new features and improvement. For example in automotive V model is extensively used. (See below image 3).
After design specification, written code needs to be tested. Basic code checks are achieved by static analysis tools which flags dangerous code construct and frequently occurring error like divide by zero etc. These analysis are done on host machine and do not require actual embedded system since static analysis only analyze the code. Unit testing ensures low level logic is tested well before integrating with other module/sub systems of the product. Also cost of fixing bugs is lowest at this stage. Depending upon industry and end product, organizations need to perform unit test on actual target or on simulator/emulators to run tests. It’s always recommended to test the embedded software to as close to end product/system (more on this later). Once individual units/building blocks of applications are tested it is necessary to test out the interaction between different building blocks aka subsystems/module using integration testing. And at the end complete system is integrated and tested for designed requirement to make sure end product satisfy all requirements and minimize the defects.
This being said you may observe above work-flow may be different depending upon product or industry. For example in automotive industry model based design/development is used widely. This allows users to design models in tools like Matlab-Simulink which simulate the control algorithm and system to be controlled. Once desired results are achieved at simulation level, tool can generate source code for specified target automatically. This approach is very useful as in automotive as embedded software engineer may not have access for ECU/subsystem (due to higher cost or hardware being under design) using model based approach engineers can easily mimic the system and create required embedded software. Hence in MBD engineers perform model in loop testing where plat (end system which needs to be controlled) and controller (algorithm to control plant) are tested. Here higher level tool language is used to verify underlying behavior. Once engineers are satisfied they can go ahead with software in loop testing where controller is replaced with equivalent c/c++ code. In SIL testing plant (i.e. simulated end target like vehicle with 4 speed transmissions) is still an model (as having actual transmission system hardware for each engineer will not efficient or cost effective). Once engineer is satisfied with the behavior they can go ahead with processor in loop testing as till now everything was tested on host machine(x86 architecture) which have different properties as compared to actual target e.g. ARM, PPC or TriCore architecture etc. In PIL testing controller (Simulink model) is replaced with actual embedded hardware which runs auto-generated code after MIL stage. PIL testing may uncover additional configuration/optimizations and then engineers can move to HIL testing where plant as well as controller is replaced with actual hardware target.. Apart from above mentioned different testing method, industry specific standard have additional metrics to ensure final product is as intended. One those metric is code coverage which is a metric for test completeness i.e how thorough you've tested your code. This ensures there is no unreachable code in you final application which can be somehow can trigger at run time causing unintended behavior. Depending upon safety integrity level (SIL) of system, an organization needs to test their embedded software. Refer below table/image for reference.
There are predefined method to decide whether a product/feature fall into specific SIL level or not. Depending upon the failure of embedded system it can cause catastrophic/hazardous/major/minor/no effect to environment/user. Depending upon the fallout of severity different standards has different nomenclature to define SIL. Depending upon the SIL level organizations need to take required precautions while designing and testing the system. For example for avionics SIL level can be summarize by below image 5.
Similarly for automotive (ISO26262) ASIL D is most stringent of all where a ASIL A is lowest. Hence if end system is of ASIL D then developer/tester need to make sure written embedded piece of code executes all the statement, all the decision/branch point and all the sub conditions of decision/branch has been tested. Once such example of ASIL D system is air bags unit of vehicles. In case of fatal accident, nobody want their car’s air bag unit fails to inflate. To make sure system is tested well car manufacturer needs to make sure in case of such events, system perform its intended function. To ensure this system need to be tested thoroughly and this is ensured via traceable metrics like code coverage for software part. Code coverage gives visibility to certifying authority that piece of software is tested well and all the statement, decision points are executed well before integrating with complete system. As far as hardware failures are concerned, for high SIL application to mitigate the effect of hardware failure manufacturers use redundancy mechanism. This simply means having backup hardware in case primary hardware fails its intended function. Basically depending upon the SIL, written embedded code needs to be tested to make sure written code perform desired behavior and code coverage provide one such metric (more on this later). Now we can understand why aircraft manufacturers do not give that much importance to functioning of infotainment system as failure of LCD or touch panel will have no effect of functioning of aircraft.
Hence we can see current embedded system are becoming more complex and it became more difficult to test & diagnose all the issues before the releasing the product. There has been constant effort from the industry to improve software quality, especially for safety critical application. Using combination of different workflow and automation tool we can reach towards better and safer product.
Comments