Carrie Puterbaugh's presentation at the Twin Cities Quality Assurance Association on using Hexawise to improve software testing at her organization (a large bank).

In one example she discusses in the video Carrie's team used Hexawise to create an optimized test suite and provided that to the software vendor to have them run it prior to delivering the software to her bank. In this example historically they vendor was finding 67% of the defects and Carrie's bank was finding 33%. Now that the vendor is using the Hexawise test suite the vendor is finding 98.5% of the defects and fixing them prior to delivering the software to Carrie's bank.

Based on these results her team was able to move staff off of testing this application and onto other testing needs of the organization. They are saving 90% of what they used to spend on QA on this project.

Another project she talked about was a high priority and high risk release that they used Hexawise on and achieved the highest quality software release they have ever had.

It was great... We were able to go to management and say "we reduced the amount of test cases we ran and we got a better quality application.

Carrie discussed how valuable Hexawise was to improving the regression testing they must do on their large number of applications. She also mentioned how much value Hexawise added by presenting software testing plan information in an easy to visualize way that greatly improved the discussions between her (as the software testing manager) and the business executives she interacts with.

Related: Large Benefits = Happy Hexawise Clients and Happy Colleagues - Kathleen Poulsen of Fidelity Investments on Using Hexawise to Improve Software Testing Results - How to Pack More Coverage Into Fewer Software Tests

By: John Hunter on Oct 3, 2017

Categories: Business Case, Customer Success, Hexawise, Pairwise Testing, ROI, Software Testing Presentations, Testing Case Studies

Kathleen Poulsen of Fidelity Investments gave a presentation at STAREAST 2017 sharing her experience using Hexawise to improve their software testing performance. Watch a 10 minute video with highlights of that talk:

We didn't really have what I'd call a scientific methodology to approaching the tests...

Our regression test suites were continuously expanding... We found there was a repition of tests.

We had 3 different projects that I will talk about that I feel like combinatorial or pairwise testing was the key to answering all of those problems.

Hexawise allows you to harness the power of combinatorial software testing with test plans designed to provide thourogh testing of interaction impacts on the software being testing. Hexawise provides more coverage with fewer tests.

All the teams that are using Hexawise can use that same file, they can talk to each other. [Another] thing I liked about Hexawise was the coverage chart... I go back to my business partners and say I am not running these tests. If they are important to you I add them back in with the click of a button. I love that... it was a game changer for me.

Using the Hexawise exporting options

the tests that we produced were converted into the given then when type scenarios automatically and when they are exported into excel you can use them to drive the Sellenium test automation framework. No additional work from us involved.

Using Hexawise's ability to create highly optimized test plans Fidelity was able to greatly reduce the number of tests while also greatly improving coverage.

We were able to reduce from 12,000 tests down to 600.

This type of result sounds amazing, and it is. But it is also what we find consisently from clients over and over. There are certain things people just cannot do well and designing test plans to cover incredible large numbers of interactions between test values and conditions is one of those things. Using highly optimized alogorithms to create test plans to cover these interactions in order to reliably create software customers will love is key. This also frees people to do what they do best.

Kathleen also discussed the significant improvement in communication within Fidelity that was brought about by using Hexawise.

The common language has become the test plan that comes out of Hexawise today.

Improving communiction is an area many organization see as important but finding concrete ways to achieve better communication is often difficult. We have designed Hexwise to aid the communication between stakeholders, including: software developers, software testers, product owners, help desk support staff and senior management.

The simplicity of this tool along with the way you can enter your parameters using the mind map tool, getting that coverage chart automatically out of it, having it export your data into a pretty commonly usable format - those are things that were teribly important to me. They gave me real value... I love that.

I can accomodate many differnt types of testing. We are testing at the class method level, at the services interface level, at the UI level...

Related: 84% of Software Defects Found in Production Could Have Been Found Using Pairwise Testing - Create a Risk-based Testing Plan With Extra Coverage on Higher Priority Areas - 2 Minute Introduction to Hexawise Software Testing Solution

By: John Hunter on Jun 9, 2017

Categories: Combinatorial Software Testing, Combinatorial Testing, Efficiency, Hexawise, Hexawise test case generating tool, Multi-variate Testing, Pairwise Software Testing, Pairwise Testing, Recommended Tool, Software Testing, Software Testing Presentations, Software Testing Efficiency, Testing Case Studies, User Experience

Hexawise - More Coverage Fewer Tests

Testers who use Hexawise consistently pack significantly more coverage into fewer software tests. It might seem counterintuitive to people unfamiliar with pairwise and orthogonal array-based testing that more thorough coverage can be achieved while using fewer tests, but this is a clear and well-established fact. This article explains how Hexawise consistently achieve superior coverage even while using fewer tests.

Time savings and thoroughness improvements achieved by testers using Hexawise at one of our insurance clients recently are typical. Let’s quickly address the first two benefits before diving deeper into a detailed explanation of how testers achieved the thoroughness improvements described in the third benefit.

Hexawise - Benefits Findings

The time savings in the test selection and documentation phase (shown in the top box) are easy enough to understand. Testers using Hexawise save time by automating selection and documentation steps that they would otherwise have to complete manually.

Similarly, the time savings in the test execution phase (shown in the middle box) are equally straightforward. Hexawise can generate fewer test scenarios compared to what testers would create on their own. Time savings in test execution come about simply because it takes less time to execute fewer tests.

So far so good. But how exactly do testers using Hexawise consistently achieve superior coverage while using fewer software tests? By helping testers generate optimized test sets without wasteful redundancies minimized, with the maximum amount of variation between scenarios, and with systematic coverage potential defects that could be caused by interactions.

Hexawise - Specific Benefits

Hexawise Minimizes Wasteful Repetition The powerful test generation algorithm inside of Hexawise systematically eliminates all wasteful repetition from every test scenario. If a given combination of test conditions has already appeared together in a test, other combinations of values will be found by the test generation algorithm and used instead of the wastefully repetitive combination. Even it if means that Hexawise’s blazingly-fast optimization algorithm needs to explore thousands of combinations of candidate values to achieve this goal. With wasteful repetition eliminated, Hexawise test sets require fewer tests to achieve thorough testing.

Hexawise Minimizes Wasteful Repetition

Hexawise Maximizes Variation Between Tests. If you take a close look at any Hexawise-generated set of tests, you will notice that variation is maximized as much as scientifically possible from one test to the next. This is the beneficial flip side of the repetition-minimization coin. Useful variation from test to test is the thoroughness-improving outcome whenever wasteful repetition is eliminated. When testers start to execute tests that explore new combinations of values and new paths through applications, they find more defects.

Hexawise Maximizes Variation Between Tests

Superior, Systematic Coverage. Interactions between different test inputs are a major source of defects. As a result, interactions between inputs are important to test thoroughly and systematically. Testers using Hexawise use a “coverage dial” in Hexawise to determine the coverage strength they would like for a given set of tests. From there, Hexawise’s test optimization algorithms systematically detect all potential interactions that are in scope to be tested for, and Hexawise ensures tests are carefully constructed to close every such potential coverage gap. Doing this kind of analysis by hand, even using tools like Excel is time-consuming, impractical, and error-prone. There are simply too many interactions for a tester to keep track of on their own. As a result, manually-selected test sets almost always fail to test for a rather large number of potentially important interactions. In contrast, the Hexawise test optimization algorithm systematically eliminates gaps in testing coverage that manually-selected test sets routinely fail to cover. Compare the coverage achieved by the project’s 37 manually-selected “business as usual” tests (above) to the more compact, efficient, and thorough set of 25 Hexawise-generated set of tests (below).

Gaps in Coverage Hexawise Optimized Coverage

In short, when testers select scenarios by hand, the outcome is typically too many tests that took too long to select and document, contain too much wasteful redundancy, and have an unknown number of potentially-serious gaps in coverage. In contrast, when testers use Hexawise to generate optimized sets of tests, they quickly generate unusually thorough sets of highly-varied tests at the push of a button that systematically achieve user-specified thoroughness goals, And testers can communicate coverage achieved by their test sets with stakeholders more clearly than ever before.

By: Justin Hunter on Dec 15, 2015

Categories: Pairwise Software Testing, Pairwise Testing, Combinatorial Testing, Business Case

Few things make us happier at Hexawise than hearing reports from clients about how much Hexawise is helping them improve their software testing efforts.

 

Happy

 

A recent conversation with our newest international banking client is a case in point. Carrie, a senior testing manager who is leading adoption efforts at the bank, reported the following impressive benefits:

 

Project 1

Without Hexawise: estimated testing effort for the project = 8.5 man months

With Hexawise: estimated testing effort for the project = 1.5 man months

Savings = > 80%

Hexawise Case Studies 2016 01 23

 

Project 2

Without Hexawise (e.g. immediately prior release): defects found during testing = 67%; defects found in UAT = 33%

With Hexawise (most recent release): defects found during testing = 98.5%; defects found in UAT = 1.5%

Defect Removal Effectiveness Improvement = Stunning

Hexawise Case Studies 2016 01 23 2

 

A few interesting things are worth pointing out as context behind these results.

 

First, let's be clear: these are significantly higher than normal Hexawise-generated benefits. We're not suggesting that every project will see benefits this large. They won't. "Your milage may vary." The 80% reduction in testing time is not unheard of but definitely larger than most teams tend to see. Similarly, the massive improvement to defect removal efficiency is larger than typically occurs. These are more typical benefits from case studies using Hexawise's pairwise testing methods and/or combinatorial testing methods and/or Orthogonal Array OATS testing methods.

 

Second, the test designers involved in these two projects are significantly more talented and skilled than "average" software testers working at banks. The test designers' skill has a lot to do with the unusually large successes they achieved in these projects. We know about how talented they are because we worked closely with these test designers during a 4-day onsite instructor-led test design training program we led and we have a good sense of the "average" test design skills possessed by software testers because we regularly conduct software test design training sessions around the world at hundreds of companies. During the hands-on interactive test design exercises in our face-to-face training sessions with the bank's software testers, several of their test designers demonstrated exceptional analytical thinking and problem solving skills.

 

Third, as we try to do with all of our new clients, our test design experts have actively kept in touch with the bank's test designers since the initial onsite training took place. We have been answering their test design questions when they have arisen, offering to review their draft test tests, and jumping on ad hoc screen sharing sessions to explain/demonstrate how to use Hexawise test design features. This helps our clients maximize the value they obtain from using Hexawise and helps us stay closely attuned to real-world testing challenges so that we can continuously improve our tool and fine-tune our software test design training messages.

 

If you're using Hexawise and have experiences to share with us, whether good or bad, we would love to hear about them. We're here to help. As corny as it sounds, helping clients succeed is a huge part of what motivates us at Hexawise. Please contact us at: support@hexawise.com

By: Justin Hunter on Sep 10, 2015

Categories: User Experience, Pairwise Testing, Combinatorial Testing, Business Case

What is the Coverage Matrix?

Some of the most challenging questions testing teams are asked include:

  • Are we testing enough?
  • Are we testing too much?
  • What is the level of testing coverage these tests achieve?
  • What if we get extremely pressed for time… What level of coverage could we achieve in half as many tests as we have planned?

Hexawise now allows you to visualize testing coverage more precisely than ever. The precise pairwise interactions covered by Hexawise-generated tests are displayed in the Coverage Matrix. This is different than our Coverage Graph, which allows users to see the additional coverage each test in their test set provides but does so without providing as much granular detail.

The Hexawise Coverage Matrix is a grid made up of all the pairs in the test plan being analyzed. With each value listed down the side and also across the top, you can find an intersection and see if that specific pair is covered at any point in your set of Hexawise-generated tests. Read on for the specific features of this reporting function.

 

The Coverage Matrix in action

Before we dive into the specifics of the Coverage Matrix, let’s see it in action.

If we have a test plan and have created some tests, we need to go the ‘Analyze Tests’ screen and click on Coverage Matrix. I recommend using a larger test plan (5+ parameters with a handful of values each) to truly see the power this visualization has.

 

analyze tests matrix arrow

‘OoooOOOooo’ snazzy new layout. That’s right, we’re taking it up a notch!

Once you click on the Coverage Matrix, the chart goes through an animation showing the coverage each tests achieves.

 

Things to look for in the animation

  • Just beneath the slider over the chart, you can see what percentage of interactions is covered (just like the Coverage Graph)
  • Also underneath the slider, you can see the precise test that is being displayed
  • The Coverage Matrix starts fully red (0 tests equals 0% coverage)
  • As each test is added, the coverage increases turning each pair covered into a green square
  • The slider at the top can be used after the animation is complete

 

Matrix Chart trimmed

 

How to read the Coverage Matrix

Now let’s get into the specifics of this feature.

 

What the squares mean

The Coverage Matrix has 3 primary indicators for coverage:

  1. Red Squares: A pair not currently covered
  2. Green Squares: A pair that has been covered
  3. Black Squares: A pair that will never be covered

 

coverage matrix

Red Squares: The Coverage Chart starts off all red, since at 0 tests, 0 pairs are covered. The number of red squares will decrease with each subsequent test as coverage increases with each test. At any given test, the amount of red squares is the equivalent to the amount of pairwise gaps.

 

Green Squares: As each test is added, more pairs are covered. The squares convert from red to green. If you were to stop executing Hexawise-generated tests before the final test, you would leave gaps in your pairwise coverage. The green squares show you what you will have covered if you stop early.

 

Black Squares: If you add invalid or married pairs to your test plan, you are intentionally removing pairs from ever appearing in your test set. Usually that will be because such combinations are impossible or impractical to test together (such as O/S = Mac OSX with Browser = Internet Explorer). The pairs that are removed are shown as black squares in the Coverage Matrix.

 

How the Coverage Matrix is laid out

Your parameter values are listed down the side and across the top. The intersection of the parameter values is the coverage status for a particular pair of values.

This is a Coverage Matrix

 

coverage matrix

Your parameter values

In order to make for a nice display, the parameters are listed down the side from the first to the second to last, and then across the top from second parameter to the last parameter.

Example: If the parameters in the plan are ‘A,’ ‘B,’ ‘C,’ ‘D,’ and ‘E,’ you will have a Coverage Matrix that looks like this one:

 

generic matrix

Using the Coverage Matrix

Since each Hexawise-generated test includes many pairs (or sometimes just a few) the test designer is sometimes unaware of when each pair is covered. The Coverage Matrix allows the tester to visualize and communicate when a precise pair is first covered in their test set.

 

Let’s take a detailed look at exactly how the Coverage Matrix accomplishes this.

 

For this example, we have a Mortgage Application system being tested. We achieve 81% pairwise coverage after just 8 tests. What kind of pairs do we see being covered by our 8th test.

 

matrix indicators explained

We also see there are sporadic red squares, since we are only viewing 80% coverage of all possible pairs. So we review this chart with our stakeholders, and they like it. But then they see towards the bottom that Region 2 is not tested with Investment Property. They tell you that situation needs to be covered, and ask how we’ll cover it.

By opening Hexawise, we can drag the slider to see when that red square turns green.

 

test 9

In this case, it’s the very next test: test 9. So we alert the stakeholders that we can cover that scenario by executing 9 tests, which bumps our overall coverage to 86% pairwise coverage.

 

In other similar situations, a given “high-priority” combination might not appear until significantly later in a plan. If a stakeholder-requested combination did not appear until near the end in this example, we might be best off executing the first 8 Hexawise tests and then addressing that stakeholder-requested combination in a one-off test outside of Hexawise. (There are “fancier” options for achieving that goal (such as using Hexawise’s “freeze” function), but that’s a different topic for a different blog post. This one is lengthy as it is.

 

Related Considerations

Anyone still reading at this point deserves a couple bits of extra insight:

First, it is important to keep in mind that Hexawise orders your tests in an optimal way so that if you stop testing after any given test, you will have covered as much as possible within the amount of time you have had to test. You can see this by comparing the (much larger) number of squares that turn green in your first test to the (much smaller) number of squares that turn green in your final test.

Second, if you are discussing the Coverage Matrix with a stakeholder who is new to Hexawise’s optimized coverage approach, they might need a brief explanation about why pairwise testing coverage is such an effective starting point for test prioritization. Introductory articles such as “Why do most software tests suck?” and “Does pairwise testing really work?” might be helpful. Additionally, it is equally important to remember that high priority scenarios that need to consist of more than 2 values should also be included in your test sets. If a stakeholder suggests including an involved scenario in your tests, keep in mind these two things. You can easily force such scenarios to be included in your test sets using Hexawise’s Requirements Feature. Those seeded high-priority scenarios will appear in the very first tests in your Hexawise test set; you do not need the new Matrix Chart to confirm that they are being included in your test set.

 

A Few Final Thoughts

This is one of our favorite Hexawise features we have ever released. We’re excited to make it available it and truly hope you find it to be useful. We are constantly seeking new ways to make it easier for software testers to not only create unusually powerful, thorough, efficient, and effective test sets but also new ways to help testers and stakeholders communicate clearly about the software testing coverage that their tests are achieving. We hope that the new Coverage Matrix feature will make it easier for you to communicate the superior testing coverage you’re achieving when you discuss your Hexawise-generated tests with stakeholders.

By: Jordan Weck on Jan 29, 2015

Categories: Combinatorial Software Testing, Pairwise Software Testing, Combinatorial Testing, Pairwise Testing, Hexawise test case generating tool

Michael Bolton is one of the software testing industry's deep thinkers. He has an impressive ability to logically analyze testing problems and clearly explain complex issues.

I like how Michael summarizes what people often really mean when they say "it works"*

It works really means...

There's a lot of truth in those words, isn't there? I've shared these words from Michael in test design trainings I've done recently and found that they immediately resonate with quite a few testers. It seems that anyone who has been in testing for more than a while has seen teams of testers test a feature or function a bit, declare that "it works," only to discover later that the feature/function works in some situations but does not work in other situations.

What's a tester to do? We recommend testers use two deliberate strategies: use a rich oracle and cover critical interactions.

First, use a "rich oracle" to enage your brain more actively and train your eye to better recognize potential issues. Imagine the following scenario. 3 people are in a room. The first person, a guy plucked from the street outside at random, is given a set of 10 written test scripts to execute and told to follow the test scripts, step-by-step in return for a six pack of beer. Being a fan of beer, and endowed with the dual-abilities of being able to both read instructions and follow instructions, he performs what is asked of him dutifully.

In the room are two testers who are allowed to observe the tests being executed but who are not allowed to communicate.

  • The first tester has a list of ten numbers, each with three boxes for checkmarks. He operates in a world of black and white where if the documented "Expected Result" is consistent with what they observe, they write a green check mark. If the "Expected Result" is inconsistent, they write a green "X."
  • The second tester operates differently. She goes beyond. She goes deeper. She notices subtle things along the way that look unexpected, or not quite right. She makes notes of those things. In doing so, she thinks of new test ideas that have not been executed yet. She documents those test ideas to explore further later, provided there is time.

I think you see where I'm going with this. My point is that the more curious approach adopted by the second tester is a far more valuable one to people who care about software quality. Why is this? Here too, Michael Bolton has words of wisdom to share that resonate well:

If you insist you need written requirements to find bugs

Second, testers should adopt test case design approaches in order to avoid the "under some conditions... once" risk. One of the most important benefits of using our Hexawise test case design tool is that, even with very basic pairwise test sets, every feature or function you test will automatically be tested multiple times. And under as many different conditions as possible in the time you have available.

After close to 10 years of introducing new groups of software testers to these types of test design approaches, people have a hard time believing how big efficiency and thoroughness improvements in this area are. That's why we always strongly encourage teams using our optimized test case selection approach to do apples-to-apples comparisons of coverage and defect-finding effectiveness. We work with teams to compare the coverage gaps of their existing "business as usual" test sets to how thorough they are when Hexawise is used to generate an optimized set of tests. Images of one recent coverage analysis is shown below. Data on defect-finding effectiveness and defect finding efficiency improvements resulting from optimized test case selection can also be found here.

Testing Coverage Analysis Hexawise Pairwise Combinatorial Testing OATS

 

*With thanks to Jon Bach for sharing this on his blog.

By: Justin Hunter on Sep 25, 2014

Categories: Pairwise Software Testing, Pairwise Testing, Combinatorial Testing, Software Testing

Hexawise helps team achieve the following qualitative benefits:

Hexawise Benefits MM

Not all of the benefits above can easily be quantified. So what should you do if you are tasked with creating a business case to support adoption of Hexawise? Simple:

  • Measure what happens when you create ~50 tests with your business as usual process; compare that to how long it takes to generate a set of ~25 tests with Hexawise.
  • Measure how long it takes to execute each sets of tests.
  • Measure how many unique defects you find executing each set of tests.
  • For typical findings, see these empirical benefit measurement studies.
  • Then put together a business case summary like the one below.
    • The summary focuses primarily on the objectively measurable efficiency savings you measured.
    • The summary also breaks out valuable qualitative benefits into separate line items (to prevent those valuable - but difficult to quantify - benefits from being forgotten about).

Hexawise - Finanacial Benefit Model 2013.10.xlsx at 2.23.05 PM

By: Justin Hunter on Aug 11, 2014

Categories: Business Case, Hexawise, ROI, Combinatorial Testing, Pairwise Testing

We have created a new site to highlight Hexawise videos on combinatorial, pairwise + orthogonal array software testing. We have posted videos on a variety of software testing topics including: selecting appropriate test inputs for pairwise and combinatorial software test design, how to select the proper inputs to create a pairwise test plan, using value expansions for values in the same equivalence classes.

Here is a video with an introduction to Hexawise:

 

 

Subscribe to the Hexawise TV blog. And if you haven't subscribed to the RSS feed for the main Hexawise blog, do so now.

By: John Hunter on Nov 20, 2013

Categories: Combinatorial Testing, Hexawise test case generating tool, Multi-variate Testing, Pairwise Testing, Software Testing Presentations, Testing Case Studies, Testing Strategies, Training, Hexawise tips

I'll be talking at QAI's 12th Annual International Software Testing Conference on Dec 6th in Bangalore, India.

Topic: Conquering the Single Largest Challenge Facing Testers Today

"There's too much to test and not enough time to test it all." According to a recent survey conducted by Robert Sabourin, this is the single largest challenge facing test managers today. And this challenge clearly won't go away any time soon. Software is becoming increasingly complex and time pressures put on testing teams are becoming ever more extreme.

To survive and thrive as testers, we need to find ways to learn more in the limited time we have. This talk addresses:

  • Proven test design methods to learn as much as possible about a System Under Test as quickly as possible

  • How these methods were originally developed and refined in other (non-IT) industries over the last 80 years

    • How the recent Apple Maps disaster could have been easily avoided by implementing these methods
  • Real world case studies: these methods sound nice on paper, but do they actually work?

  • Reasons why these methods are being used at more than 100 Fortune 500 firms today

    • What does the future hold?

 

Attendees will learn about valuable testing strategies that are being used today by more than 100 Fortune 500 firms. In particular, attendees will hear about:

  • Practical test design approaches that they can begin implementing after the conference at their firms to:

    • Reduce the amount of time spent selecting and documenting test scripts
    • Reduce the amount of tests needed for execution by creating unusually powerful tests
    • Increase the thoroughness of software test suites

 

Related: Efficient and Effective Test Design - A Fun Presentation on a Powerful Software Test Design Approach - Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed

By: Justin Hunter on Nov 22, 2012

Categories: Combinatorial Testing, Efficiency, Pairwise Testing, Software Testing Efficiency, Software Testing Presentations

It's common to have a test plan where the possible values of one parameter depend on the value of another parameter. There are many options for how you can represent this scenario in Hexawise, some options that involve using value expansions (when there is equivalence) and other options that do not use value expansions (when there is not equivalence).

Using Value Expansions in Hexawise

The general rule of thumb for value expansions is that they are for setting up equivalence classes. The key there being the equivalence. The expectations of the system should be the same for every value listed in that particular value expansion.

Let's consider a real world example involving a classification parameter with a value that is dependent on the value of a role parameter:

Inputs
Role: Student, Staff
Classification: Freshman, Sophomore, Junior, Senior, Adjunct, Assistant, Professor, Administrator

So if the Role parameter has a value of Student, then the Classification parameter must have a value of Freshman, Sophomore, Junior or Senior, but if the Role parameter has a value of Staff, then the Classification parameter must have a value of Adjunct, Assistant, Professor or Administrator.

Using value expansions in this case might be a good option. You could setup your inputs, value expansions and value pairs this way:

Inputs
Role: Student, Staff
Classification: Student Classification, Staff Classification

Value Expansion
Student Classification: Freshman, Sophomore, Junior, Senior
Staff Classification: Adjunct, Assistant, Professor, Administrator

Value Pairs
When Role=Student Always Classification=Student Classification
When Role=Staff Always Classification=Staff Classification

You would use this approach if there were no important differences in the business logic or expected behavior of the system when the different expansions of the value were used. If Freshman versus Sophomore is an important label for the users to be able to enter and see, but the system under test doesn't change its behavior based on which value is selected, then those expansions of the value are equivalent and don't need to be tested individually for how they might interact with other parts of the system and create bugs. If this equivalence scenario is true, then you will greatly simplify things for yourself and create fewer tests that are just as powerful by using value expansions.

In the scenario that would support using value expansions, the system might have different behavior for a Junior versus an Adjunct Professor, but not for a Freshman versus a Senior. A Freshman and a Senior are always equivalent in the system, so they can be combined in a value expansion.

However, if the expectations are not the same, then a value expansion should not be used. For example, let's suppose this hypothetical system has business logic giving priority class scheduling to Seniors and only last available scheduling priority to Administrators. In this case, using value expansions as described above would probably be a mistake. Why? Because a Sophomore and a Senior aren't treated the same way by the system, yet Hexawise considers all the expansions of the Student Classification value as equivalent. As long as you've got a test that has paired a value expansion of the Student Classification value with the Overbooked value of the Class Status parameter, then Hexawise won't insist on pairing all the other value expansions for the Student Classification value with Class Status = Overbooked in other tests. You could therefore miss a bug that only occurs when a Senior signs up for an overbooked class.

"One to many" or "multi-valued" married pair model

If the system under test does not consider the values to be equivalent and has requirements and business logic to behave differently, then by using value expansions to signal equivalency to Hexawise when there isn't equivalency is probably a mistake.

So what would you do in that case?

We've decided that it might be nice to be able to set up your inputs and value pairs like this:

Inputs
Role: Student, Staff
Classification: Freshman, Sophomore, Junior, Senior, Adjunct, Assistant, Professor, Administrator

Value Pairs
When Role=Student Always Classification=Freshman, Sophomore, Junior, or Senior When Role=Staff Always Classification=Adjunct, Assistant, Professor, or Administrator

Unfortunately, this kind of a "one to many" or "multi-valued" value pair is something we've only recently realized would be very helpful, and is something we have on the drawing board for Hexawise in the intermediate future, but is not a feature of Hexawise today. In the meantime, you could model it with three parameters:

Inputs
Role: Student, Staff
Student Classification: Freshman, Sophomore, Junior, Senior, N/A Staff Classification: Adjunct, Assistant, Professor, Administrator, N/A

Value Pairs
When Role=Student Always Staff Classification=N/A
When Role=Staff Always Student Classification=N/A

Another modeling option to consider, if there is only special logic for Administrator and for Seniors, but the rest of the values we've been discussing are equivalent, is to use value expansions for just the equivalent values:

Inputs
Role: Student, Staff
Classification: Underclassman, Senior, Professor, Administrator

Value Expansions
Underclassman: Freshman, Sophomore, Junior Professor: Adjunct, Assistant, Full

Value Pairs
When Role=Student Never Classification=Professor When Role=Student Never Classification=Administrator When Role=Staff Never Classification=Underclassman When Role=Staff Never Classification=Senior

I hope this helps you understand the role of value expansions in Hexawise, when to use them (in cases of equivalency), and when to avoid them, and how value pairs and value expansions can be used together to handle cases of dependent parameter values. Value Expansions are a powerful tool to help you decrease the number of tests you need to execute, so take advantage of them, and if you have any questions, just let us know!

By: John Hunter on Apr 26, 2012

Categories: Combinatorial Software Testing, Combinatorial Testing, Hexawise tips, Pairwise Testing, Software Testing, Testing Strategies

84 percent coverage in 20 tests

Hexawise test coverage graph showing 83.9% coverage in just 20 tests

 

Among the many benefits Hexawise provides is creating a test plan that maximizes test coverage with each new scenario tested. The graph above shows that after just 20 test 83.9% of the test combinations have been tested. Read more about this in our case study of a mortgage application software test plan. Just 48 test combinations are needed to test for every valid pair (3.7 million possible tests combinations exist in this case). If you are lost now, this video may help.

The coverage achieved by the first few tests in the plan will be quite high (and the graph line will point up sharply) then the slope will decrease in the middle of the plan (because each new test will tend to test fewer net new pairs of values for the first time) and then at the end of the plan the line will flatten out quite a lot (because by the end, relatively few pairs of values will be tested together for the first time).

One of the benefits Hexawise provides is making that slope as steep as possible. The steeper the slope the more efficient your test plan is. If you repeat the same tests of pairs and triples and... while not taking advantage of the chance to test, untested pairs and triples you will have to create and run far more test than if you intelligently create a test plan. With many interactions to test it is far too complex to manually derive an intelligent test plan. A combinatorial testing tool, like Hexawise, that maximizes test plan efficiency is needed.

For any set of test inputs, there is a finite number of pairs of values that could be tested together (that can be quite a large number). The coverage chart answers, after each tests, what percentage of the total number of pairs (or triples, etc.) that could be tested together have been tested together so far?

The Hexawise algorithms achieve the following objectives that help testers find as many defects as possible in as few tests as possible. In each and every step of each and every test case, the algorithm chooses a test condition that will maximize the number of pairs that can be covered for the first time in the test case. (Or, the maximum number of triplets or quadruplets, etc. based on the thoroughness setting defined by the user). Allpairs (AKA pairwise) is a well known and easy to understand test design strategy. Hexawise lets users create pairwise sets of tests that will test not only every pair but it also allows test designers to generate far more thorough sets of tests (3-way to 6-way coverage). This allows users to "turn up the coverage dial" and generate tests that cover every single possible triplet of test inputs together at least once (or every 4-way combination or 5-way combination or 6-way combination).

Note that the coverage ratio Hexawise shows is based on the factors entered as items to be tested: not a code coverage percentage. Hexawise sorts the test plan to front load the coverage of the tuple pairs, not the coverage of the code paths. Coverage of code paths ultimately depends on how good a job the test designer did at extracting the relevant parameters and values of the system under test. You would expect there to be some loose correlation between coverage of identified tuple pairs and coverage of code paths in most typical systems.

If you want to learn more about these concepts, I would recommend Scott's Scott Sehlhorst articles on pairwise and combinatorial test design. They are some of the clearest introductory articles about pairwise and combinatorial testing that I have seen. They also contain some interesting data points related to the correlation between 2-way / allpairs / pairwise / n-way coverage (in Hexawise) and the white box metrics of branch coverage, block coverage and code coverage (not measurable by Hexawise).

In Software testing series: Pairwise testing, for example, Scott includes these data points:

 

  • We measured the coverage of combinatorial design test sets for 10 Unix commands: basename, cb, comm, crypt, sleep, sort, touch, tty, uniq, and wc... The pairwise tests gave over 90 percent block coverage.

 

  • Our initial trial of this was on a subset Nortel’s internal e-mail system where we able cover 97% of branches with less than 100 valid and invalid testcases, as opposed to 27 trillion exhaustive testcases.

 

  • A set of 29 pair-wise... tests gave 90% block coverage for the UNIX sort command. We also compared pair-wise testing with random input testing and found that pair-wise testing gave better coverage.

 

Related: Why isn't Software Testing Performed as Efficiently and Effecively as it could be? - Video Highlight Reel of Hexawise – a pairwise testing tool and combinatorial testing tool - Combinatorial Testing, The Quadrant of Massive Efficiency Gains

Specific guidance on how to view the percentage of coverage graph for the test plan in Hexawise:

 

When working on your test plan in Hexawise, to get the checklist to be visible, click on the two downward arrow keys located shown in the image:

How-To Progress Checklists-2 inline

Then you'll want to open up the "Advanced" list. So you might need to click here:

Advanced How-To Progress Checklist inline

Then the detailed explanation will begin when you click on "Analyze Tests"

Decreasing Marginal Returns inline

 

This post is adapted (and some new content added) from comments posted by Justin Hunter and Sean Johnson.

By: John Hunter on Feb 3, 2012

Categories: Combinatorial Software Testing, Combinatorial Testing, Efficiency, Multi-variate Testing, Pairwise Software Testing, Pairwise Testing, Scripted Software Testing, Software Testing, Software Testing Efficiency

Combinatorial Software Test Design - Beyond Pairwise Testing

 

I put this together to explain combinatorial software test design methods in an accessible manner. I hope you enjoy it and that, if you do, that you'll consider trying to create test cases for your next testing project (whether you choose our Hexawise test case generator or some other test design tool).

 

Where I'm Coming From

As those of you know who read my posts, read my articles, and/or have attended my testing conference presentations, I am a passionate proponent of these approaches to software test design that maximize variation from test case to test case and minimize repetition. It's not much of an exaggeration to say I hardly write or talk publicly about any other software testing-related topics. My own consistent experiences and formal studies indicate that pairwise, orthogonal array-based, and combinatorial test design approaches often lead to a doubling of tester productivity (as measured in defects found per tester hour) as compared to the far more prevalent practice in the software testing industry of selecting and documenting test cases by hand. How is it possible that this approach generates such a dramatic increase in productivity? What is so different between the manually-selected test cases and the pair-wise or combinatorial testing cases? Why isn't this test design technique far more broadly adopted than it is?

 

A Common Challenge to Understanding: Complicated, Wonky Explanation

My suspicion is that a significant reason that combinatorial software testing methods are not much more widely adopted is that many of the articles describing it are simply too complex and/or too abstract for many testers to understand and apply. Such articles say things like:

A. Mathematical Model

 

A pairwise test suite is a t-way interaction test suite where t = 2. A t-way interaction test suite is a mathematical structure, called a covering array.

Definition 1 A covering array, CA(N; t, k, |v|), is an N × k array from a set, v, of values (symbols) such that every N × t subarray contains all tuples of size t (t-tuples) from the |v| values at least once [8].

The strength of a covering array is t, which defines, for example, 2-way (pairwise) or 3-way interaction test suite. The k columns of this array are called factors, where each factor has |v| values. In general, most software systems do not have the same number of values for each factor. A more general structure can be defined that allows variability of |v|.

Definition 2 A mixed level covering array, MCA (N; t, k, (|v1|,|v2|,..., |vk|)), is an N × k array on |v| values, where

| v |␣ ␣k | vi | , with the following properties: (1) Each i␣1

column i (1 ␣ i ␣ k) contains only elements from a set Si of size |vi|. (2) The rows of each N × t subarray cover all t-tuples of values from the t columns at least once.

  • "Construct Pairwise Test Suites Based on the Bak-Sneppen Model of Biological Evolution" World Academy of Science, Engineering and Technology 59 2009 - Jianjun Yuan, Changjun Jiang

 

If you're a typical software tester, even one motivated to try new methods to improve your skills, you could be forgiven for not mustering up the enthusiasm to read such articles. The relevancy, the power, and the applicability of combinatorial testing - not to mention that this test design method can often double your software testing efficiency and increase the thoroughness of your software testing - all tend to get lost in the abstract, academic, wonky explanations that are typically used to describe combinatorial testing. Unfortunately for pragmatic, action-oriented software testing practitioners, many of the readily accessible articles on pairwise testing and combinatorial testing tend to be on the wonky end of the spectrum; an exception to that general rule are the good, practitioner-oriented introductory articles available at combinatorialtesting.com.

 

A Different Approach to Explaining Combinatorial Testing and Pairwise Testing

In the photograph-rich, numbers-light, presentation embedded above, I've tried to explain what combinatorial testing is all about without the wonky-ness. The benefits from structured variation and from using combinatorial test design is, in my view, wildly under-appreciated. It has the following extremely important benefits:

  • Less repetition from test case to test case

    • In the context of discussing testing's "pesticide paradox" James Bach, I believe, used the analogy that following in someone's footsteps is a very good way to survive traversing through a mine field but a generally lousy way to find software defects efficiently.
    • Maximizing variation from test case to test case, as a general rule, is an absolutely spectacular way to find defects quickly.
    • There are thousands, if not trillions of relevant combinations to select from when identifying test cases to execute; computer algorithms will be able to solve the problem of "how can maximum variation be achieved?" far better than human brains can.
  • More coverage of combinations of test inputs

    • Most of the time, since awareness of pairwise and combinatorial testing methods remain low in the software testing community, combining all possible pairs of values in at least one test case is not even a conscious goal of testers.
    • Even if this were a goal of their test design strategy, testers would have a tremendous challenge in trying to achieve such a goal: with hundreds, thousands or tens of thousands of targeted combinations to cover, losing track of a significant number of them and/or forgetting to include them in software tests is virtually a foregone conclusion unless a test case generator is used.
    • More thorough coverage leads to more defects being found.
  • Efficiency (Testers can "turn the coverage dial" to achieve maximum efficiency with a minimal number of tests)

    • The efficiency and effectiveness benefits of pairwise testing have been demonstrated in testing projects every major industry.
    • I wanted to prominently include the message that testers using test case generators have the option to dramatically increase the testing thoroughness levels of the tests they generate because it is a topic that often gets ignored in introductions to pairwise testing case studies and introductions
  • Thoroughness - (Testers can also "turn the coverage dial" to achieve maximum thoroughness if that is their goal)

    • Too often, tester's view pairwise as a technique that focuses on a very small number of curiously strong tests; that is only part of the story.
    • This can lead to the /false/ impression that combinatorial testing methods are inappropriate where high levels of testing thoroughness are required.
    • You can create very different sets of tests that are as thorough as possible (given your understanding of what you are testing) no matter whether you have 1 hour to execute tests or one month to test.

 

Other Recommended Sources of Information on Pairwise and Combinatorial Testing:

By: Justin Hunter on Oct 7, 2010

Categories: Combinatorial Software Testing, Combinatorial Testing, Design of Experiments, Hexawise test case generating tool, Multi-variate Testing, Pairwise Software Testing, Pairwise Testing, Recommended Tool, Testing Strategies, Uncategorized