placeholder
placeholder
Testing
Testing gives you user-friendly software that works to spec, both now and as future changes are made.
The only way to ensure that your software always works precisely to specifications and that old bugs don't creep back in as new features are added is to employ automated testing.
The only way to ensure that automated testing happens when it should is to integrate it into a Continuous Quality System that makes automated testing a part of the deployment process.
Looks are always deceiving...
...because you're never looking at all of it. Who knows what's happening between the screen you're interacting with, the screens other users are interacting with, and the aggregated results presented to management?
No one does. Only properly engineered tests know what really goes on inside those black boxes that run your business. And the results of these tests completely determine the actual value of your company's software assets.
There are many different types of testing, which you can read about here. The most critical forms of automated testing you need to implement are:
- Unit Testing, which tests the internal workings of an application;
- Functional Testing, which mimics a user comprehensively testing the application; and
- Performance Testing, which makes sure your application can withstand a heavy workload
Implement these three and you'll be most of the way there. When implemented correctly, this trio actually provides additional forms of testing.
For example, because functional testing works through the user interface exactly like a human user, it exercises the entire application including all external resources it integrates with, like database servers, web services, etc. So integration testing is naturally exercised as a byproduct of functional testing.
Another example is when performance testing is distributed among multiple machines running the test simultaneously, and the tests communicate and cooperate with each other to attempt database deadlocks, resource exhaustion, and other nasties brought on by a large number of concurrent users. Performance testing engineered this way also provides concurrency testing.
And finally, when all the automated testing of all types are run every time the software is deployed, this constitutes regression testing, because all the previous functionality is tested exactly as it has always been, even after new features and the tests that cover them are added.
What about usability testing?
Usability testing cannot be automated because it involves human participants who attempt to accomplish targeted tasks while their screen activities, facial expressions, and voices are recorded. Test engineers review these recordings, discover problem patterns, and report their findings and recommendations to the application's stakeholders. These reports typically include video highlights of consistent problems and the reactions they elicit from the test participants.
Because usability testing is the only mechanism that can truly verify the successes and failures of an application in the hands of its users, it most often begins during the very early stages of application design — often while the user interface is still just a series of sketches on whiteboards — and continues with decreasing frequency throughout the application's lifecycle.
Usability testing shows engineers how an application should operate; automated unit, functional, and performance testing shows engineers how well an application operates.
Every type of application can (and should) be tested:
- Websites
- Desktop applications
- Flash and Silverlight-based applications
- Complicated intranet applications
- Web services
- Mobile apps
Testing is a huge topic, and we already run the risk of boring you to death with all this page content, so let's leave any further talk to a sit-down meeting where we can listen to you and see if we can help. We can either embed our team at your office or you can outsource to us your testing setup and team training.
With either option your takeaway will be a development team with the tools and skills they need to ensure that every application is fully tested before a new build is deployed.
If you're really psyched about testing and want to know more...
Okay, this is a very summary-level 50,000-foot view for managers, so for the purpose of explanation I'm showing code that desperately needs work. Bear with me.
Say you have a very simple commission calculation method that multiplies the first number by the second number and returns the result. What could possibly go wrong with this?
public decimal CalculateCommission(int salesVolume, decimal rate)
{
return salesVolume * rate;
}
Everything.
What if either rate or salesVolume is negative? What if rate is 9,999,999,999,999,999,999.99? What if rate is zero? What if both rate and salesVolume are the largest possible values for their data types? What if the salesperson sold more than $2,147,483,647 last month?
That's one line of code! But that's not all. Read on.
One simple expression like the line of code above defines exactly one possible path through the method. That means whatever arguments you pass to the method, the calculation will always follow the same single path through it.
Now take a slightly more complicated method:
public decimal CalculateCommission(int salesVolume, decimal rate, int goal, decimal bonus)
{
decimal commission;
if(salesVolume >= goal )
commission = salesVolume * rate+ bonus;
else if(salesVolume > goal * 0.75m)
commission = salesVolume * rate;
else
commission = salesVolume * rate /2;
return commission;
}
Sure, there are twice as many things that can go wrong just because there are now four rather than two arguments used by the method, but let's put that aside for the moment and just concentrate on the complexity.
Not counting error paths for simplicity's sake, there are now three possible paths through the method: one that is taken when salesVolume is greater than or equal to the goal, a second when the salesVolume is more than three-quarters of the goal, and a third when neither of those conditions are met.
The more possible paths that can be taken, the more complex the method. In fact, the term used to describe it is cyclomatic complexity. In the former example, the cyclomatic complexity was 1. In the latter example, the cyclomatic complexity is 3.
When tests are written for a method, the collection of tests must thoroughly exercise each possible path through the method, otherwise some paths will never be tested. Within each untested path could reside one or more bugs from Day One, and you might never find them. Even if bugs don't live there in the beginning, they can creep in later on when changes are made to the software.
So every possible path through a method must be covered by a collection of tests. This is where the term code coverage comes from. 100% code coverage is the goal.
As you can imagine, the higher a method's cyclomatic complexity, the harder it is to cover 100% of the method with tests. That's one big reason why skilled developers refactor their code to reduce cyclomatic complexity.
But it doesn't end there. Now your tests must consider how each path is exercised.
When we asked earlier "What if the rate is zero," we were suggesting what is called an edge case, which tests one of the arguments with a value at the extreme "edge" of its limit. For example, if the commission calculation used the rate anywhere as a divisor, the method would throw a Divide By Zero exception. And if the method checks that only non-zero commission values are returned, that might throw another kind of exception.
Similar to edge cases are boundary cases, which test values on both sides of an edge. For example, testing rate = 0.00001 and rate = -0.00001 are two boundary cases.
When we asked earlier "What if both rate and salesVolume are the largest possible values for their data types," we were suggesting what is called a corner case, which tests two or more arguments with values at the extreme edges of their limits.
And, of course, you will also need to test a number of good old-fashioned normal cases that test target methods and their arguments within normal limits of operation.
Now consider the software that runs your business. How many possible paths do you think there are throughout the entire application? And how many of them are thoroughly covered by tests?