Home > Real-time Mantra > Embedded Design > System Stress and Load Testing
System Stress and Load Testing

System stress testing is run after integration testing to shake out bugs before they become critical field problems. The main objective is to "burn-in" the system in the lab environment. If the system is subject to conditions that are harsher than the field, there is a good chance that all the show stopper bugs would be caught before deploying the system in the field.

System stress testing can be divided into the following steps :

• Feature Interference Tests
• Interference Load Tests
• Stress Load Tests

Feature Interference Tests

End to end testing of features is generally tested fairly well during integration testing. However, interactions between features are generally left out in this stage of testing. This hole is filled in by feature interference testing. Feature interference testing involves testing each feature offered by the system in presence of every other feature. 

The best way to develop feature interference tests is to produce a feature interference matrix. The feature interference matrix is discussed below.

Feature Interference Matrix

Feature interference matrix is produced by making a cross of each feature offered by the system with every other feature. A table is drawn by having the list of all the system features as the first horizontal row and also as the first vertical column. Then each box in the table corresponds to the cross of two features depending on its position horizontally and vertically. A test should be developed for each such box. This table then corresponds to the feature interference test matrix.

The feature interference matrix can be easily populated by considering the row and column features for each box. A test case is identified by assuming that the row feature executed first and the column feature started execution while the row feature was in progress. Here is an example of a feature interference matrix that should clarify the population of the matrix. The matrix has been produced by taking a sub-list of the features offered by the Xenon Switching System. For simplicity, only a sub-list of features offered by Xenon have been considered.

Interference Test Procedures

Once you have developed the feature interference matrix, define detailed test procedures from the matrix. The test procedures can be divided into two categories:

• Simple tests involving initiating a row feature followed by a column feature
• Load tests involving multiple instances of row and column features. We will discuss this in the next section.

Interference Load Tests

Interference load tests are identified from the feature interference matrix. Basically you run simultaneous load for different features. Here load does not only mean handling subscriber load. Load could also mean repeatedly executing operator commands via a script , rebooting boards periodically. 

Interference load tests are best explained by examples from the above matrix:

• Run subscriber load (originating to terminating calls) and operator command load overnight.
• Run subscriber load with periodic CAS and XEN processor failures.
• Run subscriber load and periodically inject faults in subscriber ports.

Stress Load Tests

Stress load tests are the final step of system stress testing. Here the system is subjected to field like conditions. Actually the conditions for these tests are harder than what the system would have to handle after deployment.

Stress Load Testing Guidelines

• Overload the system. During stress load conditions, the system should be subjected to harsher conditions than the field environment. By doing this you can make sure that the system will run stably for extended periods of time. Thus a weekend stress test might give you confidence that the system would survive a month of regular system operation.
• Load test the system with field type traffic mix. Run a traffic mix that is close to the expected load in the field conditions. Many times field traffic mix data can be obtained from studies and papers on that subject.
• Load test the system with traffic that is varying with time. When the system is deployed in the field, it will be subjected to huge fluctuations in traffic. Simulate such fluctuations in the lab. Keep in mind that there might be bugs in the system which show up only with fluctuating traffic. Most system have bugs in handling of high load as well as low load conditions.
• Load test the system with events that have random inter-arrival time. Run load such that the inter-arrival traffic distribution is random. This will exercise several legs of your code that you could not even imagine. To make the random tests reproducible, seed the random number generator with a known value before the load test. This way you would be able to recreate the exact test conditions by feeding the same random number seed.
• Load test the system with events that have random service time. Do not run load with a fixed call/session duration. Use random session durations during your load tests. For best results you should use a load generator that works with Poisson inter-arrival and service times. 
• Load test everything. Do not restrict your load testing to just subscriber load. Load testing fault conditions and operator commands will make sure that these features do not display memory leaks and other slow build up faults.
• Measure load performance. Load runs are not just for verifying the stability of the system. Always measure and plot the performance of the system during the entire load test. The best way to do this is to use tools that plot graphs showing system performance.