This interview with Janet Gregory is part of our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Janet Gregory is an agile testing coach and process consultant with DragonFire Inc. Her peers voted as the Most Influential Agile Testing Professional Person in 2015.

She is the co-author with Lisa Crispin of Agile Testing: A Practical Guide for Testers and Agile Teams, and More Agile Testing: Learning Journeys for the Whole Team. She is also a contributor to other software development books.

photo of
Janet Gregory

Personal Background

Hexawise: If you could write a letter and send it back in time to yourself when you were first getting into software testing, what advice would you include in it?

Janet: Dear Janet, Talk with the programmers instead of writing bug reports. You will save time.

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

Janet: I think it might be with an organization whose QA Director had the foresight to get the testers trained and understanding their new roles before the programmers did. By the agile teams were created and set up, the testers were ready and started at the same time. They didn’t have the normal issues of mini-waterfalls that most teams I see experience when they first transition.

Views on Software Testing

Hexawise: What testing practice(s) do you most wish the software testing community would embrace?

Janet: Learning to work through examples with their teams before coding happens so that the whole team has a shared understanding of what they are building. It makes it so much easier to test a stories and features. Of course, along with that, they need to be able to explore the system to find out what they didn’t think about.

Hexawise: In What’s a Tester without a QA Team? you and Lisa Crispin discuss the value of software testers working together with the entire software development team in an agile environment. What suggestions do you have for softer testers moving to such an environment from one that had the software testers separated from software developers and product owners.

Janet: I had to go back and reread the article you mentioned and it is still valid. My suggestions would be very much like they were then, perhaps with a few wording changes and maybe some updated links such as the SoftwareTestingClub is now The Ministry of Testing or (MoT).

I have a whole keynote at Star Canada on “Key Skills and Attributes for everyone who tests software.” That the video will be available after the conference.

Industry Observations / Industry Trends

Hexawise: How have you seen the practice of agile software testing evolve over the last 5 years?

Janet: I think the biggest change I see (not including technology changes), is the inclusion of operations – DevOps. With Katrina Clokie’s new book “A Practical Guide to Testing in DevOps”, I expect to see testers getting more involved in the whole development pipeline.

Hexawise: Large companies often discount the importance of thoughtful software testing. What advice do you have for software testers to help their organizations understand the importance of expecting more from the software testing efforts in the organization?

Janet: That’s a really big topic, but a couple of ideas I do have are:

  • testers need to understand what is important to the business, and be able to articulate the power of testing in words that they understand. For example, using metrics like the last release we put out cost the company xx,xxx.xx dollars in rework because we built the wrong thing, or $xx,xxx and xxx time fixing defects we missed because we rushed the release.
  • also, teams and testers need to be able to have the quality conversation and understand what it means to them in relationship to the rest of the organization.

Talk with the programmers instead of writing bug reports. You will save time.

Staying Current / Learning

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf? What blogs would you recommend should be included in a software tester's RSS feed reader?

Janet: I already mentioned Katrina’s book. Another good one is Elisabeth Hendrickson’s Explore It!. As far as blogs go, there are so many that it is hard to pick a couple. I often look at Testing Curator to see a nice consolidation of articles and blog posts.

Hexawise: How do you stay current on improvements in software testing practices; or how would you suggest testers stay current?

Janet: The first half hour (or hour) of most mornings is spent looking at new blog posts / articles, sometimes bookmarking them to read later in the day. I may see a new book mentioned on twitter that I want to read and order that. I also love going to conferences to see what people are talking about. I could probably spend all my time learning if I didn’t have to do anything else.

Hexawise: A great deal of making agile testing successful is a factor of how the organization practices agile within the organization, by which I mean things that are not exclusively related to software testing (so things like providing customer value, working together instead of in isolated departments, flexibility). What favorite sources (books, articles, blogs, people) on practicing agile methods (even if they are not focused on software testing)?

Janet: Two that come to mind are: Matt Wynn’s writings on example mapping, and Ellen Gottesdiener and Mary Gorman’s book “Discover to Deliver” for thinking about requirements elicitation.

testers need to understand what is important to the business, and be able to articulate the power of testing in words that they understand.

Profile

Janet Gregory is an agile testing coach and process consultant with DragonFire Inc. She is the co-author with Lisa Crispin of Agile Testing: A Practical Guide for Testers and Agile Teams, and More Agile Testing: Learning Journeys for the Whole Team. She is also a contributor to other software development books. Janet specializes in showing agile teams how testers can add value in areas beyond testing the software after it is built.

She works with teams to transition to agile development, and teaches agile testing courses worldwide. She contributes articles to publications and enjoys sharing her experiences at conferences and user group meetings around the world. Her peers voted as the Most Influential Agile Testing Professional Person in 2015.

Blog: Janet Gregory

Blog with Lisa Crispin: Agile Tester.ca

Twitter: @janetgregoryca

Related: Testing Smarter with Mike Bland - Testing Smarter with Angie Jones - Testing Smarter with Matt Heusser

By: John Hunter on Oct 11, 2017

Categories: Agile, Interview, Testing Smarter with...

This interview with Santhosh Tuppad is part of our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Santhosh Tuppad fell in love with computers when he was 12 and since then his love for computers has increased exponentially. He founded his first startup in 2010 and was part of growing the company to nearly 80 people.

In short, he is a passionate software tester, security researcher, entrepreneur and badass in following his heart come what may!


Santhosh Tuppad

This post includes highlights from our full interview with Santhosh Tuppad. The full interview is long and packed with great thoughts.

Personal Background

Hexawise: What drew you into a career in software testing?

Santhosh: I have loved computers since I was 12. My father enrolled me into a computer course and I got to experience Disk Operating System for the first time where I used computer using command-line terminal and also played Prince Of Persia game. And I was addicted to gaming during this phase.

After my gaming stint, I was introduced to the internet and picked up an addiction for IRC (Internet Relay Chat). Here, I met various hackers and used to communicate with them on various channels which were heavily moderated and were invite only. I had to demonstrate my interest in hacking to these folks to invite me to their channel. My first hack was to hack the dial-up network credentials and use them at my home when the internet shop used to close at night. We used to have Internet Packs at those times in India and I had to pay money to buy those: and I did not have money during my teenage years.

Without much ado, let’s skip to software testing part. After my graduation, I did not know what should I be doing (one thing I knew for sure was, anything that I do has to be with computers as I was passionate). I understood that, I cannot settle for anything which doesn’t synchronize with my heart. I was on the journey of finding which becomes part of me. And finally, I enrolled for the software testing course. And during the course days, I could connect my hacking skills (security testing) to software testing. This part of my life is what I call finding bliss.

And the story continued and I started growing in the industry as a tester, international speaker, participant in conferences across the globe, entrepreneur in software testing, keynote speaker, blogger, author and what not.

Hexawise: If you could write a letter and send it back in time to yourself when you were first getting into software testing, what advice would you include in it?

Santhosh:

Oh my dear soul,

I see that you have found yourself in a country where everyone is pressurized to become something else than they want to be. You identified something crucial and beautiful about yourself, that is you follow your heart with patience and kindness and don’t settle for something that doesn’t make you come alive. Like I know, passion is a variable and it may get boring at times; but being bored is just a temporary phase and an emotion which doesn’t mean your passion is dead. So, be rational and decide for yourself while you are kind to others. Accept yourself and forgive everyone.

You are stepping into what you love and I know you are confident about your journey and you believe in it. That’s beautiful.

It may be easy to fall into routine and get into monotony of things in your career. Nevertheless, you know how to sail through things and get out of them to start fresh or continue in a different path. You can swiftly shift based on your visceral.

Grow by following your visceral feelings and have no regrets. Be good at connecting the dots and growing out of them. The beauty of software testing has not been known by the world so well as of today, so work on your skills and demonstrate them to the world and educate professionals and students about the greatness of software testing. It’s not about you or me or anyone, it’s about next generation testers who could help their next generation and their generation to enjoy the fruit of invention which includes software. Let software make the life beautiful and not buggy.

I know that you know about your journey, but I am just saying.

With love, Your other self


Wedding picture of Gina Enache and Santhosh Tuppad in the temple at Bengaluru, India

Hexawise: What kinds of activities do you enjoy when you’re not at work?

Santhosh: I love meditation forms; talking deeply with a friend sitting on my balcony of my apartment; watching documentaries of various types; and conversations about psychology, life and many other topics with my wife Gina Enache.

I feel that educating customers is the key and it takes more leaders to spread the greatness of exploratory testing style to the world through demonstration.

Views on Software Testing

Hexawise: What do you wish more developers, business analysts, and project managers understood about software testing?

Santhosh: I wish that developers, business analysts and project managers understood that it is not low-skilled job which anyone can do. And also wish more of them learned to collaborate across the teams in order achieve the common goal.

At the same time, I also feel that testers should upskill and demonstrate the value they provide in order to gain credibility from those on other teams. I also wish to see them spending time together instead of just seeing their role as limited. Last, but not least; manage conflicts and work as a team.

Hexawise: What challenges and advantages are there to managing an exploratory based, thinking software tester organization (as you designed Test Insane to be) compared to the still common "checking" style software testing organization.​

Santhosh:

Challenges

  • Not many customers understand how exploratory testing can be valuable. And it’s hard to educate them as well because most of them do not want to hear.
  • Hiring is a bigger problem. In my experience, I have trained new testers or made some testers to unlearn their testing way and I have been successful, but it’s hard to scale in my view in the current world.
  • Pricing is something that customers choose over the skills. It’s sad, but true. Most customers appear to be happy with “checking” style organization because their pricing is good for customers. Value based testing still needs to be understood by customers. However, I have been trying my best to talk about good testing (exploratory skilled testing/technical testing) to business owners at conferences I participate in or speak at.
  • Most of the testers have half-baked knowledge about exploratory testing and yet they call themselves exploratory testing experts. This makes it hard for context-driven leaders to see a scaleable model for exploratory testing. Thanks to Ministry of Testing community which is really spreading a great message to the testing world. I appreciate Rosie Sherry, Richard Bradshaw and every CDT member who are working on scaling it up and spreading the right message to the world.

I feel that educating customers is the key and it takes more leaders to spread the greatness of exploratory testing style to the world through demonstration.

Advantages

  • Starting TestInsane (Exploratory Testing and Check Automation Services Company) has also enabled me to bring in a change and demonstrate to the world the worth of good testing and value-based testing that can be done through the exploratory testing style.
  • Experienced testers who joined TestInsane unlearned the checking style and learned exploratory style testing and they are leaders who spread their knowledge and also are happy with their profession.
  • Customers are happy when they see test coverage and have acknowledged that it helps them to make better informed decisions about shipping or not.
  • Recurring business from customers who saw the value
  • The sense of freedom with responsibilities that my team members have. And this is because they enjoy exploratory testing and they perform amazingly. Freedom has always been great, but it comes with challenges. And one of the challenge is constantly learning and adapting based on the context.

I recommend that organizations hire security specialists because you don’t want to just rely on checklist based testers unless they have mastered hacking and have practiced enough to create a mindset of hacker.

Hexawise: Do you believe security testing for software requires testers that specialize in security testing? Certainly some security testing can be incorporated by most software testers, but does the complexity and constantly evolving nature of software security mean that only specialists can provide sufficient security testing?

Santhosh: This is very context specific question. And I am glad that you mention “Certainly some security testing can be incorporated by most software testers” which is true. Most of the software testers can be “Survival Mode” security testers who follow the checklist or guidelines (The Script Kiddie I mean).

However, what the organization needs for better coverage and deeper security testing is a tester who can be an explorer and find security vulnerabilities like a black-hat hacker. I recommend organizations hire security specialists because you don’t want to just rely on checklist based testers unless they have mastered hacking and have practiced enough to create a mindset of hacker.

I believe strongly that we need better security testers who are not just certified by EC-Council (nowadays, anyone can get this certification), but are known for skills and can show it via demonstration. Even in today’s world, we need security specialists if we are serious about software security. Period.

My articles on various topics of security testing provide additional reading on security testing of software.

Industry Observations / Industry Trends

Hexawise: India is a worldwide center for software testing. What risks do you see to that business going forward? What can testers (or testing companies) in India do to protect their market and gain customers going forward?

Santhosh: In my opinion, I don’t see the risk at all in India for these reasons:

  • Overseas companies who outsource testing are happy with bad testing
  • Customers think automation solves testing problems just because they are blind to good testing and they think - "good testing is automation" - which is incorrect. Like I say, automation is a myth. Automation is just a Ferrari (faster), it doesn't solves testing problems by itself.
  • India has more manpower in terms of engineers. Now, this can be a boon or bane for individuals who were pressurised by society or parents to study engineering. However, India has more engineers and that means more manpower.
  • There is nothing that testers need to do until the customers understand the value of good testing which is value-based instead of running the N number of test cases and showcasing some decorated spreadsheets which speak about good/bad testing.
  • Companies are moving towards automation and artificial intelligence thinking it will solve their problems of testing. A big no. I believe that ideas are driven by the beautiful brain. And people believing the myth of AI and automation is not a risk as long as customers are loving them. In short, customers pay for this and people love to make money without educating the customer.
  • There can be a risk if and only if there is any other country which will gain the traction compared to India and maybe show what is good testing in a bigger proportion. And only then there may be a risk for Indian based companies.

Here are the risks for a tester anywhere around the globe if they fall into any categories mentioned below (not just India):

  • Falling into the phase of monotony and routine where there is no new learning.
  • Believing that, “If I stick to this company for long time, then I will have job security” (We do not know when things change in this rapidly changing and evolving industry).
  • Not getting to the depth of a problem and also not practicing thinking skills like lateral thinking, critical thinking, cognitive thinking etcetera.
  • Not spending money and time on credible conferences and workshops
  • Not adapting to the new learning and also being rigid by saying I cannot adapt.
  • Lack of passion. There is only survival with lack of passion. If a tester wants their work to be great and satisfying, passion is must. Or else they can only survive and not enjoy what they do. The solution to lack of passion problem could be, creating a passion for the profession by learning OR identifying a passion even if it’s any other profession (This is a context-based advice).

Hexawise: Have you seen a particularly effective process where the software testing team was integrated into the feedback from a deployed software application (getting feedback from users on problems, exploring issues the software noted as possible bugs...)? What was so effective about that instance?

Santhosh: The answer to this is available in this interview in the “Staying Current / Learning” section of the full interview.

The effective thing about that was, both developers and testers got access to the bugs that really matter. And once the fixes started rolling based on the feedback analyzer tool where feedback from users were being used in order to test better, there was improvement in terms of page views, time spent on page and also orders were checked out smoothly and quickly. The company started getting more orders (eCommerce platform) while they had great positive feedback and the when measured monthly feedback statistics, the negative feedback eventually reduced which spoke about “the effectiveness” of using the feedback from users and accommodating in the testing practice for better.

Working closely with programmers/developers is one of the beauties of an effective team. And Agile to me just means human values and these values have to be incorporated in the team. I believe there has to be great training in the companies/teams about conflict management, communication, motivation, solving problems etcetera in order to power up the teams to perform better and deliver better products to the world thereby helping the business move forward.

Staying Current / Learning

Hexawise: What do you look for when hiring software testers? What suggestions do you have for those looking to advance in their in software testing career?

Santhosh: In my experience, I have hired testers based on their attitude only. And some times, I have hired them only for their skills. I have had my own lessons and I have some checklist or guidelines that I follow in order to good testers with mixed ingredients of attitude. Well, the attitude is a tricky part because unlike WYSIWYG (what you see is what you get) editors, humans are not really WYSIWYG. It’s the perception during the interview that one carries about attitude. And attitude during the interview maybe based on best interest of the candidate to get hired. Most of the times it’s manipulation of the attitude which will fade away in months or days or years. I repeat, hiring is really a tricky situation and one can learn only through experiences and eventually grow with good hiring methods.

What do I look for based on my learning experiences in hiring?

  • Technical testing skills - My highest priority is for this. For example: If I am hiring for web application testing, I interview them concepts like web browser rendering engine, developer tools usage, tampering POST requests via Network tab, about HTTP headers and why are they important, depth knowledge about cookies, session management, their unique ideas to test web application and providing them with my own custom buggy web application that I have developed in order to analyse their skills hands-on and finally understand if a candidate can be a better fit for my team. (Note: This example is for a fresh candidate or someone who is 1 to 2 year experienced in web application testing). I would like to say that, I knew more about web browsers, sessions, cookies, tampering, hacking (thanks to my love for hacking when I was 16 and it’s been more than a decade being a security tester. Now, you know how passion is important if you want to do something great and well.
  • Attitude - This has been tricky for me and I am still learning how to hire based on attitude. Based on my experiments, I love to have discussions with the candidate and have transparency and also in the journey, speak like friends because those are the times the candidate opens up and feels comfortable.
  • Knowing the short-term plan of a candidate - Now, this is a checklist to see if I can have a good hire. Nevertheless, I need to see how a candidate performs in real environment once hired. In my experience, both these environments differ very well based on the context. It’s just like a web application or a software performance after being deployed to production/live environment.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf? What blogs would you recommend should be included in a software tester's RSS feed reader?

Santhosh: The first book I shall recommend is “Lessons learned in software testing” by James Bach, Bret Pettichord and Cem Kaner.

Here is the list of books that I love in testing,

  • Testing Computer Software by Cem Kaner
  • The Black Swan - By Nasim Taleb
  • General Systems Thinking by Jerry Weinberg
  • AmIaBug.com - Online Book by Robert Sabourin
  • Showing Up - Book by Olaf Lewitz and Christine Neidhardt
  • The Psychology of Software Testing - By John Stevenson
  • The Web Application Hacker's Handbook: Finding and Exploiting Security Flaws - By Dafydd Stuttard and Marcus Pinto (for security testing aspirants)
  • The design of everyday things - By Don Norman
  • And many more books. (follow me on twitter if you need any specific book suggestion as I cannot flood this post with so many books)

Blogs that I follow and recommend for a tester:

Hexawise: Have you incorporated a new testing idea into your testing practices in the last year? Will you continue using it? Why? / Why not?

Santhosh:

The problem statement: When I was working at Tesco on a testing engagement, I happened to see that Tesco website had a feedback form with rating system, checkbox options and radio buttons which is used to collect feedback from its users. As part of my testing activity, I love to speak to cross-functional teams in an organization and extract the information that can help me test better.

So, looking at the feedback forms I wanted to know how is this feedback processed by the test team in order to improvise their testing by learning from users feedback. I approached the Test Manager and asked him, “Hey! Are we looking into the feedback from users so that we can improve our testing practices?” to which his response was, “Santhosh, that’s a very good question I hear for the first time and sadly we do not use it because there are thousands of feedback responses and we are confused on what to focus on. Only our customer support looks into it to address the issues and we don’t really use the feedback system to learn and better our testing”.

The Solution: In a week’s time, I along with my friend developed a feedback analysis system (a web based application) which could consume the feedback in a *.txt format and then reveal the feedback in organized and intelligent way. Basically, the application we developed sorted the information in a readable format and categorized the feedback.

The surprising factor was, developers also started to use the tool as they cared about the quality of their code. This was an amazing success of how cross-functional teams can work together and develop something to achieve a desired common goal.

Since then, I personally work on developing such tools along with my programmer friends in order to do better testing. This phase I call as, “Success by collaboration and being creative”.

See the full interview for screenshots of the tool they developed and more information.

Profile


Gina Enache (my wife) and I in Germany

Santhosh Tuppad fell in love with computers when he was 12 and since then his love for computers has increased exponentially. After his graduation (Santhosh puts it this way, “Somehow, I graduated” J), he worked as software tester in one of the organization in India and he quit because he was bored with the work he was doing. After that, he started his first startup in 2010 and was part of growing the company to nearly 80 people. Alas! He got bored again in his first startup and also he was not happy. He made a choice to quit and started his second startup. He is going to start his next startup soon. He says, “Getting bored is a sign of something new to be started and it excites me”.

In short, he is a passionate software tester, security researcher (Started as unethical hacker and transformed to ethical hacker for good), entrepreneur and badass in following his heart / visceral come what may!

Social Media Contacts: Twitter: @santhoshst

LinkedIn: Santhosh Tuppad

Facebook: santhosh.tuppad

Skype: santhosh.s.tuppad

Related: Testing Smarter with James Bach - Testing Smarter with Ajay Balamurugadas - Testing Smarter with Alan Page

By: John Hunter on Sep 25, 2017

Categories: Career, Exploratory Testing, Interview, Testing Smarter with...

This interview with Katrina Clokie is part of the Hexawise “Testing Smarter with…” software testing interview series. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

photo of Katrina Clokie

Katrina Clokie


Katrina Clokie leads a team of around 100 testers as a Test Practice Manager in Wellington, New Zealand. Katrina is an active contributor to the international testing community as the founder and editor of Testing Trapeze magazine, a co-founder of the WeTest New Zealand testing community, a mentor with Speak Easy, an international conference speaker, frequent blogger and tweeter.

Personal Background

Hexawise: Do your friends and relatives understand what you do? How do you explain it to them?

Katrina: Honestly, I don't think they understand what I do. But I think that's common for most people who work in IT. Fortunately my husband is a software developer, so I can have work-related conversations with him and he understands. But otherwise, I usually avoid talking about the details of my work with my friends and relatives.

Hexawise: Related post by Katrina: How I explain software testing to people who don't work in IT.

Hexawise: Failures can often lead to interesting lessons learned. Do you have any noteworthy failure stories that you’d be willing to share?

Katrina: In the first role where I was involved in testing, before I had the word 'tester' in my job title, I was part of team that released a piece of software in a mobile phone network that had a serious bug in it. I had completed the installation and pre-release testing of the network in a South American country. After returning to New Zealand, a member of the public found a loophole that allowed multiple account top-ups with a single prepaid voucher code. It quickly went viral, and my team came under a lot of pressure to find and fix the problem as within 36 hours the mobile network operator was losing significant amounts of revenue.

The root cause of the issue turned out to be a race condition that we fixed by changing the order of our voucher recharge workflow. The experience was an eye-opener for me. It made me a lot more wary as a tester, and more willing to think of the ways that a system could be misused. It was a hard way to learn the dangers of being too confirmatory by only sticking to known error scenarios. On future projects I tried to be more creative in my testing.

Views on Software Testing

Hexawise: In How do you hire a junior tester you state "I am looking to create testing teams with complementary individual strengths that mean we are collectively strong."

I appreciate your focus on the importance of building a team that works rather than focusing on making every person have the same skills. Why do you think organizations so rarely focus on creating a strong team?

Katrina: I think that organisations who use agile principles for software development often focus on creating strong delivery teams. Recruitment activities are for product-oriented teams rather than discipline-oriented teams.

From my experience, the role I occupy is unusual in that I have influence in hiring of testers across the entire test competency of my organisation. I am fortunate that the people who are managing testers day-to-day are willing to take my input in their hiring decisions, and that I can drive the broader vision for testing. Without this oversight, particularly in agile organisations where there may only be one or two testers in a cross-functional team, it can be difficult to create meaningful diversity within a discipline.

The experience was an eye-opener for me. It made me a lot more wary as a tester, and more willing to think of the ways that a system could be misused. It was a hard way to learn the dangers of being too confirmatory by only sticking to known error scenarios. On future projects I tried to be more creative in my testing.

Hexawise: Please describe a view or opinion about software testing that you have changed your mind about in the last few years? What caused you to change your mind?

Katrina: I regularly change my mind. If you work in an industry that is constantly evolving, then I think you have to have that flexibility. Here are a couple of examples of people who challenged my way of thinking in the past 12 months. Both are sharing details of the evolution of their role.

Jesse Alford works at Pivotal in the US. He spoke at CAST2016 on the topic "Against a Harmful Divide: Testing as the lifeblood of development" sharing a really interesting perspective on the "we don't need testers" phenomenon.

Sally Goble works at The Guardian in the UK. She spoke at Pipeline Conf 2016 on the topic "So what do you do if you don't do testing?" which, again, is a real experience report on a significant change in the role of the tester.

Industry Observations / Industry Trends

Hexawise: Your post, Test Manager vs. Test Coach, echoes Dee Hock's quote: "If you don't understand that you work for your mislabeled 'subordinates,' then you know nothing of leadership. You know only tyranny." Do you have hope the testing community will gain more leaders, managers and coaches that focus on helping the team instead of directing the team?

Katrina: Yes. But I don't think that helping a team and directing a team are mutually exclusive. In the post that you reference where I contrast the manager and coach roles, I wanted to create a polarity to emphasis the differences. As evidenced in the comments section, the reality is a lot murkier.

I am often surprised by the rich conversation that happens when you take the time to ask people for their opinions. I always learn things that I would have liked to have known sooner. If you're a manager you may think that you have an open door policy, but there is more to learn when you seek out information rather than wait for it to come to you.

Hexawise: There has been a rapid increase in workers telecommuting in the last 10 years. And software testers do this even more than most other professions. In your post, Finding the vibe of a dispersed team, you discuss ideas on how to succeed at managing disperse teams. What other advice can you share on creating successful teams spread across different locations?

Katrina: I found it extremely difficult to work remotely as a coach. So much of my role is in face-to-face interaction, which the tools for remote working didn't support to a level that I was happy with. For testers though, it seems to be more of an option. If someone were to ask me for advice, I would refer them to Alister Scott or Neil Studd, who I think are both currently working as testers in distributed teams.

Staying Current / Learning

Hexawise: What blogs would you recommend should be included in a software tester's RSS feed reader?

Katrina: It's worth checking out the Testing Bits that Matt Hutchison compiles each week on the Testing Curator Blog. It's a good way to keep up-to-date with what's happening in the community and discover new voices. Similarly you can subscribe to the Ministry of Testing Feed via RSS, but I find the content a little more variable as there's no active moderation.

A prolific blogger who I really admire is Maaret Pyhäjärvi of Finland. Her blog, A Seasoned Tester's Crystal Ball, is full of practical advice and insights. A few others that I read whenever I spot a new post are:

Hexawise: Organizations often have exit interviews to learn from those leaving the organization. In your post, Stay Interviews for Testers, you suggest interviewing those testers staying in your organization. What have been some surprising ideas you have learned from using stay interviews?

Katrina: I'm unwilling to give specific examples here, but I will say that I am often surprised by the rich conversation that happens when you take the time to ask people for their opinions. I always learn things that I would have liked to have known sooner. If you're a manager you may think that you have an open door policy, but there is more to learn when you seek out information rather than wait for it to come to you.

Profile

Katrina Clokie leads a team of around 100 testers as a Test Practice Manager in Wellington, New Zealand. Katrina is an active contributor to the international testing community as the founder and editor of Testing Trapeze magazine, a co-founder of the WeTest New Zealand testing community, a mentor with Speak Easy, an international conference speaker, frequent blogger and tweeter. Her complete professional profile is available on LinkedIn.

Blog: Katrina the Tester

Twitter: @katrina_tester

Book: A Practical Guide to Testing in DevOps

Related interviews: Testing Smarter with Mike Bland - Testing Smarter with Ajay Balamurugadas - Testing Smarter with Dorothy Graham

By: John Hunter on Jul 5, 2017

Categories: Testing Smarter with..., Interview

This interview with Angie Jones is part of the Hexawise “Testing Smarter with…” software testing interview series. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

photo of Angie Jones
Angie Jones

Angie Jones is a Senior Software Engineer in Test at Twitter who has developed automation strategies and frameworks for countless software products. Angie shares her wealth of knowledge by speaking and teaching at software conferences all over the world and leading tech workshops for young girls through Black Girls Code.

Personal Background

Hexawise: Wow! You’re everywhere. Conference presentations around the globe at a break-neck pace. How’d that happen, exactly?

Angie: Ha ha. I’ve been doing test automation for quite some time. In early 2016, I attended a testing conference and didn’t learn much of anything new. It wasn’t a knock on the conference, but it was a wake up call for me.

I also interviewed people for automation roles quite often and found it was extremely hard to find people that were on the same level as my team. I realized that I have something to contribute and should be working to advance the industry in any small way that I can.

Diversity is also something that’s extremely important to me, so I thought being a black female on the stages of white male dominated conferences could also be my way of shaking up the game a little bit.

Actually, seeing a black woman announced as a speaker at a tech conference on Twitter was something that stopped me in my tracks. It wasn’t something I’d seen before. The conference was Write/Speak/Code, an event that empowers women to become thought leaders by writing blogs, speaking at conferences, and contributing to open source software! It was exactly what I needed. I attended and it absolutely changed my life. In less than 6 months after that conference, I’d become an international speaker and was being invited to keynote across the world. It all happened so fast...it was crazy!

Hexawise: You have helped Black Girls Code and TechGirlz work to provide girls the opportunity to learn to code. Do those efforts leave you with hope that the future is in good hands?

Angie: You know, I set out to inspire young girls, but every time I work with them I’m the one who leaves totally inspired and renewed. The girls are so smart and innovative! I give them a little push and they end up creating things that totally blow me away. The idea is to plant the seed. I wasn’t exposed to technology much as a young girl and had no idea that computer programming was even a career option. I almost missed my calling because of that. I don’t want our industry to miss out the next Tech Rock Star because she didn’t know about us.

The greatest businesses are ones that observe how their customers are misusing their products/services and adapt accordingly to make it easier for them to do what it is that they want to do, and still gain the benefit of the product/service.

Hexawise: Recently you moved from the Raleigh-Durham area (home to Hexawise, among other technology companies) to San Francisco and took a new role at Twitter. How did that come about? What is your new job?

Angie: Yes, I’m so excited about this new opportunity at Twitter! I told myself that my next role was going to be something fun and cutting edge. Twitter is just that! I love the platform, I love the innovative culture, and I love the possibilities for growth. I’m working on critical automation tasks and helping to drive automation efforts related to revenue...so ads and live video.

Hexawise: Have you gained a new insight into some aspect of software testing from your work at Twitter that you can share? You haven't been at Twitter long but sometimes in a new environment people are especially alert and gain insights that others may not notice.

Angie: When I have the privilege of choosing the work I'll be doing, I lean towards companies who understand the need for both a Tester and an Automation Engineer. Twitter gets that, which played a big role in my decision to accept this position. I, along with a few other industry leaders, have been preaching for a while now that we don't necessarily need all testers writing automation, but with the complexity of today's applications, testers do need to be technical in order to do a thorough job.

At Twitter, I've now seen just how true that is. I'm working with top notch testers who aren't programmers but they understand the intricate plumbing of our systems and are capable of digging into the guts of the application to ensure all pieces are working as they should. This requires quite a bit of technical acumen yet very little coding. I'll further explore this in my keynotes this Fall at Targeting Quality (Canada), and Agile Testing Days (Germany).

Views on Software Testing

Hexawise: Your article BDD Without the Three Amigos: Maybe Talking To Yourself Isn't So Bad is really thought provoking. In it, you acknowledge that Behavior-Driven Development (BDD) “done right” includes the 3 amigos, but you go on to explore what happens in situations when BA’s and Developers aren’t wiling to “play ball”. You suggest that you have seen testers gain significant value on their own by using BDD. Testers, for example, can create Gherkin feature files with clear “Given / When / Then” instructions that enable rapid creation of executable test scripts.

Angie: Yeah, we often make these hard fast rules and try to force everyone to follow them. Whenever someone “misuses” a practice, all hell breaks loose on the internet. I’ve consulted with a lot of teams and came to realize that these teams are not naive. They realize that they are using the approach differently than it was intended to be used. However, they’ve made it work for them. The greatest businesses are ones that observe how their customers are misusing their products/services and adapt accordingly to make it easier for them to do what it is that they want to do, and still gain the benefit of the product/service. To that point, Matt Wynne, co-founder of Cucumber BDD, actually read this piece and realized that there was more he could do in the industry to push for collaboration from the other amigos.

In a fast-paced software delivery model, automation is definitely needed, but many companies make the mistake of thinking it’s a replacement for testing... Automation should be used as a tool... a technique to enhance testing efforts.

Hexawise: What testing practice(s) do you most wish the software testing community would embrace?

Angie: That’s an easy one...Automation. But there’s definitely some education needed around this, which is why I write and speak on topics in this space. In a fast-paced software delivery model, automation is definitely needed, but many companies make the mistake of thinking it’s a replacement for testing. Anytime I’ve seen companies take this approach, the quality of their product has greatly suffered. Automation should be used as a tool... a technique to enhance testing efforts.

Hexawise: In your presentation at the 2016 SeleniumConf UK conference, you explore "How to Get Automation Included in Your Definition of Done." In that talk you discuss the idea that automation is useful but also that not all tests should be automated. How should a test team go about determining what software tests should be automated?

Angie: Ha ha. That’s a talk in and of itself. Actually, I’m going to give that talk at STPCon in the Fall of 2017. I get asked this question all the time, and it’s such a tricky thing to nail down. That’s because it’s highly contextual to the needs of the business. To do this topic any justice, I plan to explore several case studies in the STPCon talk and demonstrate how context plays such an essential part in answering this question correctly.

Industry Observations / Industry Trends

Hexawise: What trends do you foresee impacting the software testing community in the next 5 to 10 years?

Angie: With the explosion of the Internet of Things (IoT) and artificial intelligence, the systems that we test are going to become a lot more complex. This will require an evolution of testing. Our roles will become even more technical, and yet also require a healthy balance of humanity. The scenarios realized by these smart systems will require thorough testing and solid judgment. Software testing will look a lot different in 10 years, but will be extremely exciting!

Hexawise: Do you see organizations integrating software testers into the software development process (as compared to those instances where the first time software testers are involved is when software is delivered to be tested)? Do you believe more integration of software testers throughout the software development and maintenance process would be useful as software testing evolves?

Angie: I’ve already seen a huge improvement in integrating software testers into the development process with the embracing of agile practices. Quality is no longer solely owned by testers. Developers are testing their features before check-in, and testers are present throughout the entire process, essentially offering insight as early as the requirements phase. This has been essential as teams are adopting continuous integration and deployment processes. Time is of the essence, and the earlier we can avoid/find/correct mistakes, the better!

Hexawise: For those who are not used to involving testers early in the software development process, how would you describe the benefits to the business of involving software testers early in the process?

Angie: By involving testers early in the software development process, the team is able to identify and correct assumptions. This essentially eliminates potential bugs before production even begins. Testers bring in a breadth of knowledge about the application as a whole and how the individual components work together. Testers also serve as a customer advocate, ensuring that their goals are being considered. So, while the team is discussing a potential feature, the tester is advocating for the customer and also calling out how this can affect existing features in the system.

I sat in a design meeting recently and watched the tester poke holes in the proposed design and call out omissions that would cost us millions of dollars. The developers left that meeting with a list of requirements and considerations that they had missed. That's beyond valuable.

Staying Current / Learning

Hexawise: How do you stay current on improvements in software testing practices?

Angie: I read a ton. I’m subscribed to several blogs, and I use Twitter to find new ones all the time. People are always creating new tools or approaches to solve interesting problems and I try to absorb as much of it as I can.

I don’t limit myself to testing blogs. I also read development blogs and tech news sites to stay up to date with trends in the software industry as a whole. This helps me think past my current role and prepare myself for the future as well.

I’ve already seen a huge improvement in integrating software testers into the development process with the embracing of agile practices. Quality is no longer owned by testers. Developers are testing their features before check-in, and testers are present throughout the entire process, essentially offering insight as early as the requirements phases.

Hexawise: You have presented at and attended many technology conferences. What advice do you have for people attending software conferences so that they can get more out of the experience?

Angie: It’s so funny how people tend to flock towards topics they are familiar with. They end up sitting there nodding in agreement with everything the speaker says, sharing their own experiences during Q&A, and leaving thinking that was such a great talk. However, did they learn anything new?

I try to attend talks that will address an unsolved problem that I have, or is in an area that I know very little about. These are the talks where I gain the most insight and can bring something of value back to my job.

Also, be sure to network! You won’t remember everything that people said during their talks, but leaving with a catalog of names and contact information of not just the speakers, but attendees as well, is gold! I often hit problems and I recall meeting someone a year ago at a conference who talked about this very problem during the cocktail hour. Because I networked with them, I feel comfortable reaching out and asking for help.

Hexawise: What blogs would you recommend should be included in a software tester's RSS feed reader?

Angie: Fortunately, Ministry of Testing has an aggregated feed that I use all the time. It consists of more than 500 testing blogs! It’s my morning newspaper.

Profile

Angie Jones is a Senior Software Engineer in Test at Twitter who has developed automation strategies and frameworks for countless software products. As a Master Inventor, she is known for her innovative and out-of-the-box thinking style which has resulted in more than 20 patented inventions in the US and China.

Angie shares her wealth of knowledge by speaking and teaching at software conferences all over the world and leading tech workshops for young girls through Black Girls Code.

Links

Blog: Angie Jones

Twitter: @techgirl1908

Read previous Testing Smarter with... interviews: Testing Smarter with Alan Page - Testing Smarter with Dorothy Graham - Testing Smarter with Mike Bland

Subscribe to the RSS feed for the Hexawise software testing blog.

By: John Hunter and Justin Hunter on Jun 14, 2017

Categories: Interview, Interesting People , Software Testing, Testing Smarter with...

This interview with Matt Heusser is part of the Hexawise “Testing Smarter with…” software testing interview series. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Matt Heusser
Matt Heusser

Matt Heusser is a software craftsman with a deep background in software delivery and testing.

In 2014, Matt received the Most Influential Agile-Test Professional Award at Agile Testing Days in Potsdam, Germany.

Personal Background

Hexawise: If you could write a letter and send it back in time to yourself when you were first getting into software testing, what advice would you include in it?

Matt: My advice would be to trust your instincts and experiences more than what you read in books or online. Often the advice I read in books and online seemed vapid (shallow), or simplistic, or I felt it "just wouldn't work here." Eventually I realized that a lot of it (this was the testing advice of the 1990's) wasn't working well most places.

Today we have better advice. The time from research to publish is short, and there is a lot less "loss" in the system. Still, what works for Google and Microsoft might not work for your 20 person company, and what works for that cool, 100% physically distributed, 40 person software company might not work for your 2,000 employee insurance company. Take it with a grain of salt, trust your instincts - but always keep exploring and experimenting.

Hexawise: Which person or people have had the greatest influence on your understanding and practice of software testing?

Matt: It's hard to come up with a list of influences, but Cem Kaner, James Bach, Ken Pier, Brian Marick, Johanna Rothman, Lee Copeland, Jerry Weinberg, Kent Beck, and Ron Jeffries all come to mind.

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

Matt: I remember once we wanted to get a insurance data extract out on a friday, but it had to be right. Serious students of testing will tell me you can never know that it is right - but non-serious student customers don't know that. I had about a half a day. The change was to one field, and we had the results in a database table. The table-to-file and the file transfer we had confidence in; the change, not so much. So I wrote my own computer program to loop through every current member in the insurance plan, over four hundred thousand, calculate the expected result, and compare them.

We found a small bug in the requirements; an edge case that was ambiguous. The programmer and I had interpreted the requirements in different ways that were both arguably correct. The "differ" program I wrote found the case, the customer explained that either was acceptance - and we shipped!

Views on Software Testing

Hexawise: What do you wish more developers, business analysts, and project managers understood about software testing?

Matt: A decade ago, when I went to the Google Test Automation Conference, one of the speakers said that automation was better because it was "repeatable." I almost stood up and asked aloud "so what?"

Every build of the software different. If the software is different, if the risk picture has changed, if we have some idea of what tests we ran before and how valid they might be on this build - why would we ever test the exact same way?

Software isn't an assembly line. Every build is different. The way we test it can be different, but I certainly don't see a ton of value in spending extra money to make sure we test things the exact same every time.

Turns out you don't need to remove the friction from handoffs. Often, you're better served to make communication cheaper and getting good at collaboration.

Hexawise: Can you describe a view or opinion about software testing that you have changed your mind about in the last few years? What caused you to change your mind?

Matt: It's going back a bit, but in graduate school, I wrote a paper "On the optimization of physically distributed requirements, development, test, operations, and management development groups" The body of the paper was considerably shorter than the title - the body was "You're Screwed."

At the time, the literature suggested that communication costs were high, so we needed get everything right prior to "the handoff." We needed to get every document, every bit of code, everything complete, consistent, correct, unambiguous, before "the handoff."

I knew that trick never worked, and thought the fix was co-location.

Then I learned about agile/XP, which was still focused on co-location - still, part of XP was making communication, collaboration, and change cheaper. Then I worked at Socialtext, where my co-workers were all over the world - Ingy took a skiing vacation in France, skiing during the day and working core hours at night. And it was far more productive than any other job I had ever had.

Turns out you don't need to remove the friction from handoffs. Often, you're better served to make communication cheaper and getting good at collaboration.

Hexawise: You have written about the benefits of lean thinking in software testing. What advantages do organizations gain when they adopt a lean thinking view of software testing?

Matt: You know that thing that happens, where you can't do your job because you filed a ticket and it will take the DBA's a week to add a column to a table, so you can't do your job, for a week?

Or whatever else it is? Right now I've got a contractor billing on my team with no laptop. He'll have it nine days after he started ... if we're lucky.

Typically, when a company goes to lean thinking, that kind of stuff stops happening.

Industry Observations / Industry Trends

Hexawise: Do you have specific suggestions for testers working within an organization using agile or lean software development methods?

Matt: It's hard to come up with examples without context, but generally, I'd start by looking at the delays we have in our job and the amount of multitasking. Be sure to include failure demand - things that should be reasonably expected to work the first time, but took a round of fixes. Often you'll find what should take an hour is taking you a week.

Hexawise: What do you see as the most powerful trends in the software testing field over the last 5 to 10 years? What trends do you believe will be the most powerful over the next 5 to 10 years?

Matt: The past ten years? Test-Driven Development, Continuous Integration, REST APIs that can be tested at the integration level, Virtualization, Lightweight Virtualization.

The next five? The promise of continuous delivery might just be realized. I realize that sounds like a lot of technical stuff, and I'm a process and people guy. The challenges of the next few years will be all skill, people and process.

Hexawise: There is still a widespread belief in fairly mechanistic software testing (checking) by some of those using software testing. Lean thinking, exploratory testing, etc. encourage engaging the minds of software testers. Are you optimistic about the prospects for tapping more of the potential software testers have going forward?

Matt: Oh yes. That's what I hope the next five or ten years are all about.

You know that thing that happens, where you can't do your job because you filed a ticket and it will take the DBA's a week to add a column to a table, so you can't do your job, for a week? ... Typically, when a company goes to lean thinking, that kind of stuff stops happening.

Hexawise: Have you seen a particularly effective process where the software testing team was integrated into the feedback from a deployed software application (getting feedback from users on problems, exploring issues the software noted as possible bugs...)? What was so effective about that instance?

Matt: My preference is for cross-functional delivery teams, so I get a little down on the term "test team", but yes, I have seen delivery teams where customer feedback was part of the process. One team that Justin Rhorman and I worked with managed a fortune 500 retail website; they had a process that periodically popped-up requests for feedback. The testers worked through these in a rotation, summarizing them, reporting them to management and working with management.

I think that worked well because of the rotation - no single bottleneck. If multiple testers observed the same feedback independently week over week, it was harder to dismiss.

Staying Current / Learning

Hexawise: You have spoken at many conferences. What advice do you have for people attending software conferences so that they can get more out of the experience?

Matt: Try to come up with three things to do on Monday that justify the investment. These things should be entirely within your power to do. They should not require training, a team-level change in process, particular learning time, purchase of tools or hiring of consultants. Things you can just do - that you will not ask permission for. (Your boss probably doesn't know what you do anyway.)

Then do it and tell the team about it. If you want to write a trip report, it should be a one-pager, and describe what you are doing, not what you heard.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf? What blogs would you recommend should be included in a software tester's RSS feed reader?

Matt: I don't read blogs like I used to, instead I follow twitter and click on interesting links. A few weeks ago I wrote an article on 28 testers to follow on twitter that might be a good place to start.

As for books, I'd say How To Reduce the Cost of Software Testing, Lee Copeland's A Practitioner's Guide to Software Test Design, Lessons Learned in Software Testing and Jez Humble's Continuous Delivery book (the writing is a bit tough but it's worth it). I'd also suggest some non-testing books, like Out Of The Crisis by Deming, Peter Drucker on Management, and Taleb's Black Swan book.

Hexawise: We share an interest in seeing lean thinking concepts be adopted by software testers. How would you suggest someone interested in learning more about lean thinking in software testing do so?

Matt: You could try a google search for "heusser lean" and see what it returns, seriously I've written a lot. Here are two older articles that I think cover the start of it Applying lean concepts to software testing and The secrets of successful Lean software testing.

Profile

Matt Heusser is a software craftsman with a deep background in software delivery and testing. After earning his undergraduate degree in Math with a concentration in computer science in 1997, Matt began his career as a programmer, writing code in C, perl, PL/SQL, and Visual C++. Along the way, Matt was the initial organizer of Grand Rapids Perl User's Group ("Perlmongers"), earned a master's degree in CIS from Grand Valley State University, taught IS part time at night at Calvin College, and served as the initial lead organizer of the Great Lakes Software Excellence Conference.

After leaving Priority Health, a Health Insurance Company, in 2008, Matt went on to become a member of the technical staff at Socialtext, the world's first wiki company, where he worked with Audrey Tang, Ingy DotNet, and Dan Bricklin, the creator of the Spreadsheet, to help build a web-based spreadsheet/wiki that predated google docs. The test framework Matt worked on at Socialtext, WikiQTests, is documented as a case study (chapter 16) of O'Reilly's book "Beautiful Testing."

Matt left Socialtext in 2011 to become a full-time consultant. Since that time he reviewed Robert C. Martin's books "Clean Code", "The Clean Coder" (for which he wrote the preface) and "Clean Architecture." In 2014, Matt received the Most Influential Agile-Test Professional Award at Agile Testing Days in Potsdam, Germany.

Links

Blog: Creative Chaos

Twitter: @mheusser

Previous Testing Smarter with... interviews: Testing Smarter with Rikard Edgren - Testing Smarter with James Bach - Testing Smarter with Michael Bolton

Subscribe to the RSS feed for our blog.

By: John Hunter on May 23, 2017

Categories: Testing Smarter with..., Software Testing, Lean, Agile

Hexawise is proud to be a Gold Sponsor of the STAREAST Testing Conference in Orlando, Florida this week. We are particularly excited to be co-presenting with Fidelity Investments on the benefits of combinatorial test design.

Kathleen Poulsen, Lead Software Engineer in Test will talk about how Fidelity adopted combinatorial test design with Hexawise as a standard across testing groups. Case studies from two Fidelity projects (services testing and front-end integration) will examine the measurably improved test coverage and efficiency achieved with this approach.

Update: watch the presenation online


Kathleen Poulsen, Fidelity Investments


We will also continue our Testing Smarter series, with interviews from the conference and an evening event featuring short talks by some of the conference speakers and “Testing Smarter” interviewees like Dorothy Graham and Michael Bolton.

If you plan to go to #StarEAST, please visit us at Booth #19 to talk about “Testing Smarter” with Hexawise. While you’re there, register for a chance to win a free Amazon Echo. You will also find us sponsoring the Wednesday evening reception in the Expo Hall.

If you can’t make it to Orlando (we’re sorry!) you can still check out conference keynotes and industry presentations via the Virtual Conference.

Hexawise is famously easy-to-use yet powerful software test design tool with an enthusiastic following. More than 100 of the Fortune 500 to improve software test design and reduce defects. Hope to see you at StarEAST in Orlando.

Happy Testing!

Also follow us on Twitter @Hexawise.

By: John Hunter on May 8, 2017

Categories: Hexawise, Testing Smarter with...

This interview with Ajay Balamurugadas is part of the Hexawise “Testing Smarter with…” software testing interview series. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Ajay considers being inducted into the 'Bach Brothers' Testing Legion of Merit, presenting his keynote at CAST 2015 and attending Problem Solving Leadership workshop by Jerry Weinberg and Esther Derby to be his biggest achievements to this date.

His contributions to the software testing community include co-founding Weekend Testing and Test Maniac. His short books are popular with many testers for the practical, ready to use tips.


Ajay Balamurugadas

Personal Background

Hexawise: I like the Weekend Testing concept that you helped bring into existence. What gave you the idea to do so? What were you trying to accomplish? How is it working?

Ajay: Thank you. I wanted to practice software testing and had the first online paired testing session with Parimala Hariprasad (@Curioustester). It was fun learning about a new tool. Next weekend, Sharath Byregowda (@Sharathb), Manoj Nair and myself tested for two hours. That night, we decided that this could be even more fun and enriching if more folks joined us. We then opened the forum to the public on August 15th (Indian Independence Day), 2009. Regarding the aspirations, I never knew that this would grow as big as it has grown today.

With more than 300 sessions spread over India, Americas, Europe, Australia/New Zealand, the forum seems to have connected many testers and helped them hone their skills across different approaches, techniques, tools and products. Thanks to so many volunteers and facilitators who have jumped in at various stages of this journey and helped Weekend Testing evolve over the years.

We are still having the sessions during the weekends, the frequency has reduced to one or two per month. To be honest, testers need not wait for a session to participate in a weekend testing session. They can pickup any session report, timebox their testing activity for an hour and then compare their report with other testers' reports.

Hexawise: If you could write a letter and send it back in time to yourself when you were first getting into software testing, what advice would you include in it?

Ajay:

  • Focus on the basics
  • Spend money on learning
  • Take care of health

Hexawise: What kinds of activities do you enjoy when you’re not at work?

Ajay: I love playing and watching cricket other than reading books. I and my wife (@AbiTheTester) enjoy going to the mall, watch a movie, eat ice cream and play a video game of bike racing.

I have come to the realization that agile teams are one of the ideal places for a skilled software tester. With so little time, emphasis on finding critical information early and in a crisp manner is necessary. Who, other than a skilled tester can switch contexts, interact with multiple stakeholders, think critically and add value to the teams?

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

Ajay: I am thankful to many software testing mentors who have helped me grow personally and professionally. Knowing so many testers at a professional and personal level is in itself a satisfying moment for me.

Attending Problem Solving Leadership with Jerry Weinberg, Esther Derby and so many testers from different countries, presenting a keynote at CAST 2015, receiving the "Bach Brothers Testing Legion of Merit" award, helping Fiberlink (now acquired by IBM) adopt mind maps are some of the moments that make me think that I must have done something right in my testing career.

I certainly celebrate every small moment that has taught me - for example I cherish the moment when parents of a tester called me and thanked me for helping their son get a job in software testing.

Views on Software Testing

Hexawise: Can you describe a view or opinion about software testing that you have changed your mind about in the last few years? What caused you to change your mind?

Ajay: My immaturity at the start of my career made me repel automation. I always thought that it was the skill of the testers that is more important than the tools. Later, I realized that there are many activities that would be done better if they were automated. So, I started to shift my focus on learning to automate and here I am evangelising Sahi Pro - The Tester's Web Automation Tool.

Hexawise: What testing practice(s) do you most wish the software testing community would embrace?

Ajay: These are a few practices that I would like the whole software testing community to think about, not necessarily embrace:

  • Spend more time interacting with the application under test than with documents, processes that have little value for the customer.
  • Learning to test well is a long journey. Do not accept shortcuts like 2 days workshops as __the__ solution to your learning. They accelerate your learning, highlight the probable mistakes you can do and give you a learning path.
  • Trying to get everyone on the team to learn everything is a sure recipe for killing motivation. Get people with complementary skills as part of one team and see how they create magic.

To be convincing, you might have to work on your reputation first. Work on it. People most of the times say "No" until they are convinced. Once you highlight the benefits of pairing and collaboration, people will listen.

Industry Observations / Industry Trends

Hexawise: Do you have specific suggestions for testers working within an organization using agile software development methods?

Ajay: Gel with the team members and at the same time, do not forget your core skill: Testing - Providing information about the quality of the product and project to stakeholders who matter.

I have come to the realization that agile teams are an ideal place for a skilled software tester. With so little time, emphasis on finding critical information early and in a crisp manner is necessary. Who, other than a skilled tester can switch contexts, interact with multiple stakeholders, think critically and add value to the teams?

Hexawise: Do you have suggestions for how testers can be more effective if they are isolated from the software developers (either by practice in the organization or by geography)?

Ajay: If it is by practice, talk to the managers and see why it is the case. To be convincing, you might have to work on your reputation first. Work on it. People most of the times say "No" until they are convinced. Once you highlight the benefits of pairing and collaboration, people will listen. If they still don't listen, maybe its time to change teams or company.

If the distance is because of the location, it is easy to solve. How would you collaborate if your best friend was from a different city? You will find ways to get in touch. Today, there are multiple tools that help you have that seamless experience. Make use of those tools. At the end of the day, everyone is a human being. The moment we consider everyone as a human being and not as someone who has a developer/manager/tester role, most of the problems would just disappear.

Check out my book - 50+ tips to improve tester-programmer relationship which will help your learn how you can improve the relationship.


Communication tips graphic from Ajay's book

Hexawise: "Whenever you need to test the best combinations out of all possible combinations, I recommend Hexawise. It can help you create data quickly and in a format that you can directly use. Very cool tool. Use it to see the power of Hexawise." How did you learn about Hexawise. How does it help you?

Ajay: That was quite long back and I think I might not change the statement even now, though maybe I would edit it to say:

"Whenever you need to provide effective test coverage for multiple parameter combinations, I recommend Hexawise. It can help you create test plans quickly and in a format that you can directly use. Very cool tool. Use it to see the power of Hexawise."

Many testers use tools without understanding why they have to use the tool. Someone who understands the technique of combinatorial testing, will appreciate the ease of use Hexawise provides.

I learned about Hexawise through the Twitter world of #testing . There have been multiple instances in my testing career where I have helped the teams reduce the number of test cases with the help of Hexawise.

Staying Current / Learning

Hexawise: You have blogged about your career journey in software testing several times. What tips do you have for those looking to deepen their knowledge of software testing and move forward their careers?

Ajay: Many testers don't seem to have a learning path at all and that is sad. I really appreciate those who take time to get better at their craft. Today, we don't have shortage of sources of knowledge. People just have to spend time and dedicate themselves to learning.

Realize that there is a lot to learn. Pick what you want to learn, carefully. You will face challenges and you need to be motivated throughout the journey to excel in learning the subject. Set time limits, take help of mentors, take it slow if needed.

Each person has their own style of learning - some like it hands-on, some read books, some watch videos. Know your style and keep measuring your learning quotient - are you happy learning the subject. As long as you are happy learning it, continue or else modify the plan to suit your needs.

I am focusing on 1-2 quality criteria per year and have started with Security and Automation for the last few months. I am loving the journey and wish others too good luck in their journey. At this moment, I am reminded of the different pathways Katrina Clokie has blogged about.


Mind map of his learning by Ajay


Hexawise: What advice do you have for people attending software conferences so that they can get more out of the experience?

Ajay: Make notes, connect with the speakers even after the conference, do not sit with your team members in the conference, make new friends and spend time talking to people rather than spending your whole time inside at the talks. I created a mind map for anyone attending EuroSTAR 2011. It looks like the points apply even today.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf? What blogs would you recommend should be included in a software tester's RSS feed reader?

Ajay: There are many books that can be part of a tester's bookshelf (Huib Schoots' list). Check out those books which will help you achieve your immediate goals. My books can also be found here.

For blogs, check out the Ministry of Testing's testing feed everyday. You can also follow @JorisMeerts on Twitter.

Profile

Ajay considers being inducted into the 'Bach Brothers' Testing Legion of Merit, presenting his keynote at CAST 2015 and attending Problem Solving Leadership workshop by Jerry Weinberg and Esther Derby to be his biggest achievements to this date. He is also happy at the number of testers who have been influenced positively in interactions with him.

He started his career as a software tester and he continues to be a hands-on software tester along with training new testers, presenting at conferences, conducting workshops and sharing his thoughts through his blog and tweets.

Ajay started with testing standalone desktop applications and soon moved on to web applications and mobile applications. His journey was boosted by co-founding Weekend Testing, Test Maniac. His short books are popular with many testers for the practical, ready to use tips.

photo of Abinaya and Balamurugadas Ajay
Abinaya (his wife) and Balamurugadas Ajay

Links

Website: Test With Ajay

Blog: Enjoy Testing

Twitter: @ajay184f

By: John Hunter on May 1, 2017

Categories: Interview, Testing Smarter with..., Software Testing

This interview with Hans Buwalda is part of the Hexawise “Testing Smarter with…” software testing interview series. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.


Hans Buwalda

Hans has gained experience as a developer, manager, and principal consultant for companies and organizations worldwide. His approaches to testing—action-based testing and soap opera testing—have helped a variety of customers achieve scalable and maintainable solutions for large and complex testing challenges.

Personal Background

Hexawise: What drew you into a career in software testing?

Hans: It was "accidental". I worked as a management consultant in my home country, the Netherlands and was sent on an assignment to a major customer. They had serious problems with both test design and automation. In particular it was difficult to involve the (very busy) domain experts, and the automation was virtually impossible to keep current. Using keywords turned out to solve both these problems, and I have been pioneering that ever since with the Action Based Testing Method.

Hexawise: You were an early pioneer and key contributor to the keyword-driven test automation framework which has stood the test of time and is widely adopted in the industry. What drove you and other early keyword-driven framework pioneers to create it and advocate for its broader adoption?

Hans: I believe the two main drivers for using keywords are readability for non-technical people, and long-term maintainability of the tests. I think my core message is not as much the keywords themselves, but more the importance of test design in achieving those goals. Without a good modularized organization of tests keywords will fail, and so will for example Behavior-Driven Development (BDD). To say it even stronger: the worst automation projects I have seen were keyword projects.

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

Hans: I like it when people are successful. In my world that means being able to leverage automation to achieve better quality at a faster pace. Two of my colleagues recently showed me how they were able to help a company in the oil industry reduce time needed for testing dramatically (more than twenty times).

A nice detail there was that those tests were for equipment workflows, and did not involve any User Interface (UI). It is a common misunderstanding that keyword automation is for testing via the UI, but it can work just as well for non-UI testing, like Internet of Things (IoT), services, games, telecommunication, embedded software, etc..

Views on Software Testing

Hexawise: What testing practice(s) do you most wish the software testing community would embrace?

Hans: I believe that with Agile, DevOps and service oriented architectures, we're well on our way. Agile allows for good cooperation between the Quality Assurance (QA), developers, product owners, domain experts and stakeholders. DevOps allows testing to be incorporated into the build and deploy pipelines. And service oriented architectures gives more ways to test than just through the UI.

scalability in automation is not as much a technical challenge as it is a test design challenge. This gives a bit of a paradox: test designers influence the automation success, but are not developers, and do not necessarily have affinity with engineering concepts like structured programming and refactoring, which in turn are key elements of good maintainable automation.

Hexawise: Can you describe a view or opinion about software testing that you have that many or most smart experienced software testers probably disagree with? What led you to this belief?

Hans: "Disagree" is a big word. But I do like to focus on one particular area that many others tend not to address that much: It is my observation over the years that scalability in automation is not as much a technical challenge as it is a test design challenge. This gives a bit of a paradox: test designers influence the automation success, but are not developers, and do not necessarily have affinity with engineering concepts like structured programming and refactoring, which in turn are key elements of good maintainable automation. There are some more elements that can influence automation success, like "testability" of the application under test: to what extend is an application friendly to testing and automation.

Hexawise: You have extensive experience with automated testing. What interesting anecdotes would you be able to share about some of your earliest experiences automating tests?

Hans: An early experience was the very first project where action words (my term for keywords) were used. It was for a major screen trading system, and before I came in a substantial amount of tests had been created with record and playback. When I say that you can probably guess the upcoming/impending disaster... The development team made a slight change in a lay-out of the main screen and virtually all tests stopped working because of it. It was in the early days and people weren't expecting such problems.

Industry Observations / Industry Trends

Hexawise: One of my (Justin’s) favorite software testing articles of all time is “Soap Opera Testing.” What experiences led you to write it and why do you think it struck a chord with so many testers and managers?

Hans: Thanks for the kind note. It was based on a major project in a large financial institution. The project included re-architecting of all systems, and getting ready for the year 2000. Proper testing was essential, but the tests also needed to be developed and fully automated in a very short amount of time.

To get this done I introduced an approach based on real life stories from the every day practice of real users and good test design (test modules, keywords) to support successful automation. I guess the method appeals to teams and managers because it is an efficient way to translate domain knowledge into, aggressive, test cases, and to get them automated quickly as well.

I believe the two main drivers for using keywords are readability for non-technical people, and long-term maintainability of the tests. I think my core message is not as much the keywords themselves, but more the importance of test design in achieving those goals.

Hexawise: Large companies often discount the importance of software testing. What advice do you have for software testers to help their organizations understand the importance of expecting more from the software testing efforts in the organization?

Hans: You have to be serious about your craft. Know testing techniques, engineering principles and the domain of an application you're testing. Your attitude is important too. Be aggressive to the system under test, but cooperative as member of a team. Second, always keep an eye on the business side.

You should not be testing because you have read somewhere that testing is important. Testing costs time and money and there must be business reasons to invest in it. Not testing saves money, but also introduces risks that can cost money later on. Saving money and losing money are business considerations that well managed large companies tend to take very seriously.

You, or your QA management, must be ready to explain the value of their testing, and automation.

Staying Current / Learning

Hexawise: I see that you’ll be presenting at the StarEast conference in May. What could you share with us about those topics? What gave you the idea to talk about them?

Hans: I'm doing two tutorials, one on Better Test Design for Great Test Automation. The other focuses on what makes automated testing scalable. The classes are based on real-world experiences from various projects I’ve done, across many industries. With the many recent developments like cloud and DevOps, automation is constantly evolving, so it is fun to discuss about them.

Hexawise: How do you stay current on improvements in software testing practices; or how would you suggest testers stay current?

Hans: Reading, attending conferences and webinars and hands-on practice still work well. Don’t despair when current developments can be overwhelming. Concentrate on what you need, and be curious.

Profile

Hans Buwalda has been working with information technology since his high school years. In his career, Hans has gained experience as a developer, manager, and principal consultant for companies and organizations worldwide. He was a pioneer of the keyword approach to testing and automation, now widely used throughout the industry. His approaches to testing—action-based testing and soap opera testing—have helped a variety of customers achieve scalable and maintainable solutions for large and complex testing challenges. Hans is a frequent speaker at conferences and other events and is lead author of Integrated Test Design and Automation.

Links

Web site: Happy Tester

Twitter: @hansbuwalda

Previous Testing Smarter with... interviews: Testing Smarter with Mike Bland - Testing Smarter with Dorothy Graham - James Bach

Subscribe to the RSS feed for our blog.

By: John Hunter and Justin Hunter on Apr 17, 2017

Categories: Interview, Software Testing, Testing Smarter with...

Rikard Edgren has been a humanistic and technical tester since 1998. He currently works with testing healthcare software at Nordic Medtest in Karlstad, Sweden. Rikard enjoys the dynamics between people/machines, objective/subjective and whole/details.

photo of Rikard Edgren
Rikard Edgren

Personal Background

Hexawise: What drew you into a career in software testing?

Rikard: Like most people it happened by accident; I wanted to become a programmer, but had a chance to start at a company with testing as a stepping stone.

I realized I liked testing and was good at it, and wasn't upset that I never got the chance to be "promoted" to developer, and just went with it by doing a lot of testing.

I did a couple of years as project manager (good insights for a tester), but went back to what I enjoyed most.

I think I love testing because of the dynamics between the technical and the humanistic; the details and the whole; the objective and subjective.

And of course the thrill of being first to see a nasty problem!

Hexawise: You share authorship of the thoughts from the test eye blog with a few of your former colleagues. This joint author setup is fairly rare; what advantages do you see from this arrangement?

Rikard: Henrik Emilsson and Martin Jansson tricked me into this in 2008. In the beginning it was great to have two readers!

We also commented on each others posts, which made us learn more. We also did some joint efforts, e.g. a list of generic Software Quality Characteristics.

It is a benefit that the blog keeps going also when one or two are very occupied with other things. It also gives a healthy pressure to provide to our common project. The activity is lower nowadays, but still alive.

It has kept the three of us more in touch even though we have worked at different places and locations.

So I see only positive aspects from the joint setup, and encourage writing as a learning process, with sharing as a good side-effect. Ideas you discuss, try, discuss, are typically better than ideas you work out by yourself.

Views on Software Testing

Hexawise: In your book, The Little Black Book On Test Design, you write: "You want to create tests that are testworthy, and typical examples can be those tests where you don’t know if it is important or not, but you want to make sure." How often do you see software developers providing guidance on testing focus due to concerns about the code? My background is in software development and often we know when certain aspects of the code have higher potential for bugs - often due to complexity or novelty. But in my experience this insight (that could focus software testing) is not provided as often as would be useful.

cover image with the text, Little Black Book On Test Design, on a black background

Rikard: In my experience I haven't received important information like this on direct questions. However, when I have worked together with the developers for some time, and (hopefully) have earned respect by digging up interesting information, this sort of guidance come more and more often. I hope it is because they realize that my testing can act on vague and disparate information, and that they value the findings and want more of it. It doesn't seem to be enough to say that you can test the software well, you have to actually do it first in order to gain a healthy respect that everyone benefit from.

Once the trust is there, communication is more open and better, and it is easier also for me to admit areas where the testing wasn't the best.

Respect isn't given, it must be earned, and you do that by finding information others need, that they wouldn't have found by themselves.

Hexawise: Do you have any specific suggestions on how testers can gain respect from developers and encourage them to see a tester as someone to assist them in creating great software. Too often it seems software developers can view software testers as a bother rather than a help?

Rikard: Make sure you provide valuable information. Find bugs the developers want to fix. Give feedback on things that work well. Find bugs that other stakeholders want to fix. Listen to developers input on what needs testing. Collaborate. Be nice and do good work.

Hexawise: In your book you also mention the value of combinatorial testing to discover problems that don't appear in isolation but only with interaction between components of the software and different use cases. Do you have a favorite example of a combinatorial bug? How did that bug illuminate a challenge in software testing?

Rikard: I will never forget a dialog box that crashed when you clicked Cancel, but only if the dialog first had been moved. I don't think I would have come up with the idea of testing just this, it was something I stumbled on because I like to do things differently. It is still a mystery how someone managed to create this bug, and I don't know if it was important to fix it. This illuminates the challenge in software testing on which parameters to explicitly deal with, and which parameters to implicitly deal with by serendipity-enabling variations in our testing.

Hexawise: What is one thing you believe about software testing that many smart testers disagree with?

Rikard: I think many would disagree that it often is a good idea to run tests without no particular reason; that a random test can be better that one carefully picked; that using my subjectivity will help me test the product better.

These work (according to me) because testing is a sampling business, and we should have serendipity working for us.

Also, people have phenomenal capacity when motivated, so I would rather have you perform five tests you believe in, than one that I think is better.

Industry Observations / Industry Trends

Hexawise: Do you believe testing is becoming (or will become) more integrated with the software development process? And how do you see extending the view of the scope of testing to include all the way from understanding customer needs to reviewing actual customer experience to drive the testing efforts at an organization?

Rikard: I think testing have been integrated with the software development process for a long time, and I think good testing often needs to understand customer needs and actual experience. In my 19 years with testing this is how I have worked, unless there are circumstances that make it impossible.

But I have heard similar comments before, and I rather believe that there was some years where there were loud thoughts that testing must be totally objective, both in regards with who we work with, and with the sole objective to verify the documented requirements. Nowadays more people say that they look for more things than what was defined in advance, and that they collaborate with developers as much as possible. Testers have been doing this all the time, and it is a good thing that the idea of "test documentation over insights" is retiring.

Hexawise: What do you wish more developers, business analysts, and project managers understood about software testing?

Rikard: That it is difficult, but done well software testing provides a lot of value to any software where quality is important (but not every project needs testing!)

That they can influence us: tell us about what is important, and we will find out useful stuff about this (good and bad things.)

That automation is great, and exploratory testing too.

Staying Current / Learning

Hexawise: How do you stay current on improvements in software testing practices; or how would you suggest testers stay current?

Rikard: I think each testing situation is unique, so I try to stay current by doing the most effective testing in my current situation. This might include old stuff, recent stuff, brand new stuff, or at least a combination of "stuff" that makes sense for me and the project at the moment. I try to do small experiments with new tools, and disregard most of them, but maybe learned something in the process. By going deeper into the current context, I learn a lot in that area, but maybe not about what is the latest in general.

Many new ideas come right out of the blue when I do something else, so I think my subconscious is helpful to me.

My suggestion to testers would be to use whatever methods they think are fun (and valuable if it is work time), because that will keep up the motivation, and enhance the learning process.

Hexawise: Have you incorporated a new testing idea into your testing practices in the last year? Will you continue using it? Why? / Why not?

Rikard: I don't think I have incorporated any brand new testing idea the last year. But of course there are new combinations of old ideas for my specific purposes.

Recently our team created a "TimeBlaster" in SoapUI; the results from a request is analyzed for time elements, and then many requests are sent with different start-end times (some of them randomized) whereupon the new responses are analyzed for conformance to the specification. Since these rules for time is somewhat complicated, this automation makes us (and other users of the same services) test better and faster, so we will continue using it.

My ongoing, low-frequency, private research currently deals with mental models; how I and other testers think, how we come up with new ideas, how we act on certain perspectives, but not on other. How we handle the complexity of software and human interaction and figure out what we should be testing with limited time available.

But I don't have any answers yet, at least in my head it is a multitude of (invisible) mental models that interact in mysterious, and productive, ways.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf?

Rikard: The best testing book for me is Lessons Learned in Software Testing, by Kaner, Bach, Pettichord. Another favorite is Exploring Requirements by Weinberg/Gause, which (secretly) is about "test analysis" - finding out what is important.

The most brilliant free article is Heuristic Test Strategy Model by James Bach, because it is generic for any software project, and the reader has to do all the hard work by themselves when using the structure it gives for attacking a test project.

My favorite blogger/youtuber is Alan Richardson.

There is lots of good stuff available on the net, and a lot of bad stuff as well, so a healthy dose of skepticism is needed.

And to practice "a healthy dose of skepticism" is something we need to do as testers!

Profile

Rikard Edgren has been a humanistic and technical tester since 1998. He currently works with testing healthcare software at Nordic Medtest in Karlstad, Sweden. Rikard enjoys the dynamics between people/machines, objective/subjective and whole/details.

Rikard has been as a consultant with many education companies and higher vocational studies programs. Teaching is a learning experience, with the number one testing question: What is important?

He is a regular at international conferences, with many appearances at EuroSTAR. He is co-organizer of Swedish Workshop on Exploratory Testing (SWET).

Links:

Blog: thoughts from the test eye

Book: The Little Black Book On Test Design

Presentation: ExploratoryTestDesign

Previous Testing Smarter with... interviews: Testing Smarter with Mike Bland - Testing Smarter with Alan Page - Testing Smarter with Dorothy Graham

Subscribe to the RSS feed for our blog.

By: John Hunter on Apr 10, 2017

Categories: Interview, Software Testing, Testing Smarter with...

This interview with Michael Bolton is part of our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Michael Bolton is a consulting software tester and testing teacher who helps people solve testing problems that they didn’t realize they could solve. He is the co-author (with senior author James Bach) of Rapid Software Testing, a methodology and mindset for testing software expertly and credibly in uncertain conditions and under extreme time pressure.


Michael Bolton

Personal Background

Hexawise: Do your friends and relatives understand what you do? How do you explain it to them?

Michael: [laughs] A lot of my friends and relatives don’t really care what I do! But if they’re curious, and they want an explanation, I tell them this:

"All that software you’re using every day is hard to develop. Whenever someone wants software built, development groups try to figure out what to do and how to do it. But since people never completely understand one another, there is a risk that the proposed solution won’t solve the problem, or will introduce problems of its own.

"Then, even when the ideas about how to build something are pretty clear, we’re still building it for the first time. It’s new to us, so we have to learn how to build it. As that happens, people make mistakes, and they realize new problems along the way. It’s the tester’s job to help identify problems all the way through that process.

"And then, as people are trying to build the product, testers explore it and experiment with it, looking for problems that might threaten its value. Ideally, the important problems get found and get fixed before the software comes to you.

"In other words, testers help people to try to figure out whether the product they’ve got is the product they want. I help testers and teams learn how to do that effectively, as a consultant and as a teacher. And I do that worldwide; in classes, at client sites, at conferences, and in social networks."

If friends or family want to know more, I can tell them more. But usually they’re happy with that.

Friends and family are one thing; managers and teams are another. Some testers say their managers don’t understand them. Recognize this: managers need to know about problems that threaten the on-time, successful completion of the project, where “project” means “some burst of development work”. Think about what you’re doing and the problems you’re facing in terms of how they relate to that. Talk about your work that way, and things will become a whole lot clearer.

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

Michael: That’s a tough call. On projects, I really like the detective work. In one specific case I investigated the roots of a nagging customer support problem, and found six or seven wildly different factors that, if addressed, could have eliminated that problem. That was satisfying. On another project, at a bank, I constructed a really elaborate oracle [Hexawise: "If you are not familiar with using oracles in software testing we strongly recommend following the link to learn more about this important concept."] that modeled consumer financial transactions through all of the different account flows. That involved a lot of programming, and using powerful features of Excel, and learning a ton about the business domain. That was fun. I found some cool bugs and weird behaviour, too.

From 2003 through 2008, I did pretty extensive study with Jerry Weinberg —several of the AYE Conferences, some consulting workshops, and the Problem Solving Leadership class. Those were really helpful for my own personal development.

But the most satisfying thing is teaching Rapid Software Testing, helping people develop their mindsets and skill sets, and reframing the ways they think and speak about testing. It’s rewarding to hear from the class participants about their renewed energy and focus—and to hear from their organizations about the value and power of testing, value they might not have seen clearly before.

Views on Software Testing

Hexawise: What testing concept(s) do you wish more software testers understood?

Michael: Oh dear. [laughs] That’s an awfully long list, and it varies from day to day.

I wish more testers were more articulate in describing their models—their models of the product and how to cover it; their models of oracles, and how they recognize problems; their models of testing itself.

I wish that more testers understood that test cases are not central to testing. To be an excellent tester means to explore, to experiment, to study, to model, to learn, to collaborate, to investigate. To discover. None of these activities, these performances, are captured in test cases. But testers keep talking about test cases as being central to the work, as though recipe books and written ingredient lists were the most important things about cooking.

I wish more testers understood that testing is not about “building confidence in the product”. That’s not our job at all. If you want confidence, go ask a developer or a marketer. It’s our job to find out what the product is and what it does, warts and all. To investigate. It’s our job to find out where confidence isn’t warranted; where there are problems in the product that threaten its value.

I wish more testers were more articulate in describing their models—their models of the product and how to cover it; their models of oracles, and how they recognize problems; their models of testing itself.

Of course, sometimes testers find those things hard to talk about because their intuitive models are vague and fuzzy. That’s one of the important tasks in the Rapid Software Testing space: to help people sharpen up their models so that they can sound like they know what they’re doing — while actually knowing what they’re doing. Then we can talk more precisely amongst ourselves about what we’re doing, and how we’re going to approach solving problems for our clients.

Some people are bothered by our focus on expressing things precisely. They complain “If we use jargon, the non-testers we’re talking to will get confused!” If you use jargon with people who don’t need to hear it, they may get confused, so don’t use jargon with them when there’s nothing at stake.

On the other hand, when non-testers hears “automated testing”, that affords the interpretation that testing can be automated. It can’t. That’s why, in Rapid Software Testing, we talk about automated checking: to alleviate confusion between things machines can do (checking) and things that you need clever humans to do (testing, which includes the process of developing and interpreting checks).

It’s okay for different testing and development communities to adopt and adapt their own ways of speaking. But amongst ourselves, within our communities and within our teams, we had damn well better learn to speak precisely, because imprecision is where lots of problems start, and where bugs live. Don’t jump on the newbies; help them learn and guide them along.

But I would say this to people who seek respect: getting straight on what things mean and how they matter is essential in real professions. [laughs] Doctors don’t say “virus, bacteria… who cares? The guy’s sick! He’s got… a… bad thing! And he needs… stuff to make him better! Pharmacist, give him… some... better-making-stuff and tell him to take it… whenever.” Fail to make appropriate distinctions, and the attempt to help the patient will fail. And then society gets antibiotic-resistant bacteria as a byproduct.

I wish more testers recognized that testing can be applied to anything—a running product, of course, but also to documents, diagrams, models, ideas. I often hear people saying “testers should do more than just testing,” and when I ask them what they mean by “just testing”, it turns out that their notion of testing is really impoverished.

Testing isn’t just operating a product and looking for bugs. Testing involves modeling and questioning and studying, making inferences, challenging beliefs. Generating, describing, refining, revising, abandoning, and recovering ideas, all the way through the project. Designing and performing experiments, not just by interacting with running code, but also by performing thought experiments on whatever we might have in front of us, from an idea to a document to a diagram to a program. Review is testing a story or some code. Critical thinking is testing an idea — helping people sharpen their understanding. Developing tools to help test is testing. [laughs] What’s with the “just testing” business? All that isn’t enough?

Hexawise: Can you describe a view or opinion about software testing that you have changed your mind about in the last few years? What caused you to change your mind?

Michael: Many years ago, when I first started out as a program manager, I thought it would be a good idea to write lots of things down, formally, in advance, and hand those instructions to programmers. After that, building the product and testing it would be simple.

Well, that sure sounded appealing, but it didn’t work very well most of the time. Some programmers really liked very specific instructions, and occasionally we could give them those when we had a really good idea of what we wanted.

What I was ignoring was all the work, all of the trial and error, all of the collective tacit knowledge that gets you to the point where you can make a lot of stuff explicit… but by then you don’t need most of it to be explicit, so making it explicit is a waste of time!

Excessive or premature formality is really dangerous for testing. Notice those adjectives: excessive, premature! Sometimes, in certain contexts, there are specific things that must be tested formally, in specific ways, or to check specific facts. That formality might be important because of technical risk, business risk, legal risk, risk to finances or to human health and safety.

Much of the time, though, your testing doesn’t need to be very formal. Even when you’re in a context that requires formal testing, you need plenty of excellent informal testing in order to get to excellent formal testing.

I see a lot of people investing in formality and documentation really early in the project. But I think it’s a mistake to do that before we’ve learned about the product and the problem space. That learning is a fundamentally exploratory process, one that can’t be formalized unless you want to suppress it or damage it or slow it down needlessly.

Testing isn’t just operating a product and looking for bugs. Testing involves modeling and questioning and studying, making inferences, challenging beliefs.

Hexawise: You have clearly articulated the inherent limitations of testing coverage metrics. For example, from your blog post: 100% Coverage is Possible, you state:

To claim 100% coverage is essentially the same as saying “We’ve looked for bugs everywhere!” For a skilled tester, any “100%” claim about coverage should prompt critical thinking: “How much” compared to what? 100% of what, specifically? Some model of X—which one? Whose model? How well does the “model of X” model reality? What does the model of X leave out of the universe of possible ways of thinking about X? And what non-X things should we also be considering when we’re testing?

What advice do you give to teams you work with who use quantitative coverage metrics?

Michael: [laughs] That’s a little like asking “What advice do you give to journalists who use spreadsheets?”

Start from the default premise that we don’t need them. Don’t start from the premise that we must hang a number on our coverage. Assume that we will describe our coverage. If there’s some way in which numbers will help, assume that whatever we decide to quantify will be based on some model of the software. All models focus on something and leave everything else out, and at best we’re only aware of some of that “everything else”.

After all, what might we be covering? We could decide to model our coverage in terms of how many lines of code, or branches, or conditions have been covered. We could count them. But not all lines of code (or branches, or conditions) are equally significant. And code coverage tools report on our code, but not necessarily the code in third-party libraries and frameworks, in the operating system, in the hardware platform.

Some people model coverage in terms of “requirements”. What they really mean by that is specific statements in requirements documents. Documents model ideas about the product.

So here’s one statement: “the site shall provide scheduling information for all flights on the airline’s network”. Here’s another: “when the customer is logged in, each page shall display the customer’s first and last names in the upper-right corner of the screen”. Are those statements of equivalent significance? How would we test them?

[laughs] Don’t say something silly, like you’ll fully test your product by having “one positive and one negative test case for each sub-requirement”! (The Wikipedia article on “test cases”, as of today, says exactly that.) Your testing of a product is a performance.

Sometimes a journalist will find it useful to illustrate a point with numbers, or a table, a spreadsheet. Baseball is a great example of a game where statistics help us to evaluate performances. Bill James, the guy at the centre of Moneyball, has shown how the numbers you choose can support arguments about a player’s value. But the magic of the Baseball Abstract, which was a kind of journal he wrote for many years, was that he told great stories, using stats to illustrate them. He also showed how stats don’t tell the whole story, and how they can mislead, too.

Sometimes journalists will quote figures from researchers, or use poll numbers to back a story. But the history of polls—particularly recent history—provides a great example of the ways we can be fooled by numbers.

I wrote these articles on talking about coverage a while ago, but I think they still stand up:

You’ll find useful stuff on measurement in Quality Software Management, Vol. 2: First Order Measurement (Weinberg) or the two e-books that make up that book, How to Observe Software Systems and Responding to Significant Software Events.

But if you want to quantify coverage, be skeptical. Be professionally unsure. Read "Software Engineering Metrics: What Do They Measure and How Do We Know" (Kaner and Bond). Read How Not to Be Wrong (Ellenberg). Read Proofiness (Seife). Read How to Lie with Statistics (Huff).

Industry Observations / Industry Trends

Hexawise: What software testing industry trends make you optimistic about the future? And which software testing industry trends make you concerned?

Michael: There’s been some degree of progress over the years in refining how people think about testing, but the people who are pushing for that are in a tiny minority. We’re a gaggle of hobbits facing an army with entrenched ideas that might as well have been handed down from orcs. Those ideas didn’t work 30 years ago and are crazily inefficient now. People still don’t think very critically about testing folklore and mythodology. (Yes, I said mythodology.)

Many people treat certain ideas as Grand Truths about testing. But many of the claims that people make—especially some of the testing tool vendors—are myths, or folklore, or simply invalid. People often say things that they haven’t thought about very deeply, and some of those things don’t stand up very well to critical scrutiny.

People don’t seem to notice how rickety everything is at the best of times. Have you tried to print something lately, using the new printer software that came out yesterday? Tried to buy a plane ticket? Tried to set up or top up a pay-as-you-go mobile hotspot when you’re travelling? I do all that stuff regularly, and it’s usually time-critical, and almost every time I enter a world of pain. Technology is complex, and our lives are complex, and the least little bump in the road can make the wheels fall off. Every day, as I try to use software, I feel like I’m being pecked to death by ducklings.

We want to do stuff like move money between accounts with our smart phones, securely, but the global financial system is at the mercy of institutions that are getting testing services from the lowest bidder, trying to find ways to make testing cheaper instead of more powerful and more risk focused. That life-critical medical software comes from the same larger community that brought us JavaScript and CSS. Yikes. [laughs]

I’m concerned that we’re still overly focused on testing the clockwork, the functional aspects of the product without thinking about how people will respond to it. As Harry Collins would say, machines are social prostheses. Like insulin pumps or artificial legs, they’re being plugged into human life and human purposes to do work that humans once did (or where humans with superpowers could do that work). But protheses do don’t what the real thing does, and the surrounding body has to adapt to that fact. Software doesn’t do what humans do, so humans often must adapt to it. We should all be asking: “how does the technology change us—for good and for ill?”

Staying Current / Learning

Hexawise: I see that you’ll be presenting at the StarEAST conference in May. What could you share with us about what you’ll be talking about? / What gave you the idea to talk about it?

Michael: I’ll be giving two tutorials. The first is about critical thinking for testers. We describe critical thinking like this: “thinking about thinking, with the intention of avoiding being fooled”. That’s central to our work as testers. Testers think critically about software to help our clients make informed decisions about whether the product they’ve got is the product they want.

Many people treat certain ideas as Grand Truths about testing. But many of the claims that people make—especially some of the testing tool vendors—are myths, or folklore, or simply invalid. People often say things that they haven’t thought about very deeply, and some of those things don’t stand up very well to critical scrutiny. That’s one of the reasons I started developing and teaching this class in 2008. I was dismayed that testers and other people in software development were accepting certain myths about testing unskeptically.

Not only that, but testers and teams allow themselves to be fooled by focusing on confirmation, rather than challenging the software. So, in the class, we talk about the ways in which words and models can fool us. In a safe environment, it’s okay—and even fun—to be fooled, and to figure out how not to be fooled quite so easily the next time.

On Tuesday, I present on one-day class called “A Rapid Introduction to Rapid Software Testing.” RST (the methodology) is focused on the mindset and the skill set of the individual tester. It’s about focusing testing on the mission, rather than on bureaucracy and paperwork. We sometimes joke that RST (the class) is a three day class in which we attempt to cover about nine days of material. [laughs] In “A Rapid Introduction to Rapid Software Testing”, I try to do the three-day class in one day. It is a rapid introduction, but we’ll be able to explore some of the central ideas.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf?

Michael: The two that I can recommend that are explicitly about software testing are Lessons Learned in Software Testing and Perfect Software and Other Illusions about Testing.

But you asked about testing-related books, and there’s an absurd number of those. Thinking Fast and Slow is about critical thinking and how easy it is for us to get fooled. The Shape of Actions (Collins and Kusch) is about what machines can and cannot do, and the Golem series (Collins and Pinch) is about the nature of science, technology, and medicine. Code (Petzold) is about how data is represented and processed on digital computers. Everyday Scripting in Ruby is a decent, tester-focused book on creating little tools and doing work with scripts. And the list of Jerry Weinberg books is plenty long on its own: Exploring Requirements, and An Introduction to General Systems Thinking, and Errors, and Agile Impressions.

James Bach and I have not written a book on software testing together. But you could read our blogs, which would be like reading a long-ish book of essays on testing.

Profile

Michael Bolton is a consulting software tester and testing teacher who helps people solve testing problems that they didn’t realize they could solve. He is the co-author (with senior author James Bach) of Rapid Software Testing, a methodology and mindset for testing software expertly and credibly in uncertain conditions and under extreme time pressure.

With twenty-five years of experience testing, developing, managing, and writing about software, Michael has led DevelopSense, a Toronto-based testing and development consultancy, for the past fifteen years. Previously, he was with Quarterdeck Corporation where he managed the company’s flagship products and directed project and testing teams both in-house and worldwide.

Links:

Blog: DevelopSense

Twitter: @michaelbolton

Related posts: Testing Smarter with Alan Page - Testing Smarter with Dorothy Graham

By: John Hunter and Justin Hunter on Apr 3, 2017

Categories: Interview, Software Testing, Testing Smarter with...

This interview with Mike Bland is part of our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Mike Bland aims to produce a culture of transparency, autonomy, and collaboration, in which “Instigators” are inspired and encouraged to make creative use of existing systems to drive improvement throughout an organization. The ultimate goal of such efforts is to make the right thing the easy thing. He's followed this path since 2005, when he helped drive adoption of automated testing throughout Google as part of the Testing Grouplet, the Test Mercenaries, and the Fixit Grouplet.


Mike Bland

This post includes highlights from our full interview with Mike Bland. The full interview is long and packed with great thoughts.

Personal Background

Hexawise: If you could write a letter and send it back in time to yourself when you were first getting into software testing, what advice would you include in it?

Mike: When I first started practicing automated testing and had a lot of success with it, I couldn’t understand why people on my team wouldn’t adopt it despite its “obvious” benefits. One of the biggest things that experience, reading, and reflection has afforded me is the perspective to realize now that different people adopt change differently, at different rates and for different reasons, and that you’ve got to create the space for everyone to adapt accordingly.

As I say in my most recent presentation, “The Rainbow of Death”, metrics and arguments are far from sufficient to inspire action in either the skeptical or the powerless, and the greater challenge is to create the cultural space necessary for lasting change.

Oh, and you’ve got to repeat yourself and say the same thing different ways multiple times—a lot.

Hexawise: What change management lessons did you learn while driving adoption of test automation methods at Google between 2005-2010? Which of those lessons were applicable when you were involved in the recent U.S. federal government effort to bring in talented tech people to bring new ways of working with technology into the government? Which of those lessons were not?

Mike: The top objective is to make the right thing the easy thing. Once people have the knowledge and power to do the right thing the right way, they won’t require regulation, manipulation, or coercion—doing things any other way will cease to make any sense.

Most of what I learned in terms of specific approaches to supplying the necessary knowledge and power has come from trying different things and seeing what sticks—and I’m still working to make sense of why certain things stuck, years after the fact. The most important insight, as mentioned earlier, is that different people adopt change differently, for different reasons, and as a result of different stimuli. Geoffrey A. Moore’s Crossing the Chasm was the biggest eye-opener in this regard. Then, years later, when I saw fellow ex-Googler Albert Wong present his “Framework for Helping” to describe his first experience in the U.S. Digital Service, I instantly saw it snapping in place across the chasm, describing how the Innovators and Early Adopters from Moore’s model—who I like to call “Instigators”—need to fulfill an array of functions in order to connect with and empower the Early Majority on the other side of the chasm. Of course, through the filter of my own twisted sense of humor, I thought “Rainbow of Death” might make the model stick in people’s brains a little better.

So these models helped provide context for why the specific things the Testing Grouplet did worked; and how, despite the fact that there were many scattered, parallel efforts underway, they ultimately served to reinforce one another, rather than creating confusion and chaos. Of course, Google’s open communication channels and the Testing Grouplet’s shared vision—that emerged two years into our five year run—helped keep everything aligned. The point being, don’t wait for the clear vision and perfect plan up-front—start doing things and pay attention to what’s working, and why, and develop your plans as you go. That’s just good Agile practice, isn’t it?

And did I mention that you have to reiterate things you’ve already said in different language over and over—like, a lot?

Every lesson applied, in that the real lessons were about human nature, not technology. Google disabused me of the notion that one metric, one tool, or one method of persuasion would suffice to change an entire population's behavior. In other words, there's no silver bullet.

See the full interview for useful details.

The top objective is to make the right thing the easy thing. Once people have the knowledge and power to do the right thing the right way, they won’t require regulation, manipulation, or coercion—doing things any other way will cease to make any sense.

Hexawise: Describe a testing experience you are especially proud of. What discovery did you make while testing and how did you share this information so improvements could be made to the software?

Mike: Probably like many folks, I remember my first time the most vividly. Immediately on the heels of a death march—when my team barely got a steaming pile of other people’s code to meet a critical spec by a harsh deadline and very nearly would’ve killed one another were it not for Strongbad’s Emails to keep us one hair’s breadth away from going completely insane—we got some time and freedom to try to make the program faster.

I’d gotten the idea that we needed to rewrite a particular subsystem to take advantage of data we weren’t even using, and at about the same time, I happened to read an issue of the C/C++ Users Journal that had an article on using CppUnitLite, I believe. Unit testing sounded like a neat idea, so I practiced it at the same time I started rewriting this subsystem from scratch.

In the end, my new subsystem was rock-solid and improved performance by a factor of 18x. When a couple bugs came up, I diagnosed and fixed them very, very quickly, when the norm was on the order of weeks or months. It totally transformed our relationship with our client.

See the full interview for additional useful details.

Hexawise: In watching your videos and reading your content online your ideas resonate with those of W. Edwards Deming, Russell Ackoff and Peter Senge from management, culture change and systems thinking perspectives. Who are your greatest influence in this area?

Mike: I’m a little ashamed to admit I haven’t read any of their stuff, or at least not much. Certainly what little I’ve gleaned of Deming resonates with my experience. I’ve begun reading Senge’s The Fifth Discipline, and while the introduction resonated very clearly, I’ve not yet read further. Ackoff is a new name for me (and thanks for the tip!). That said, it is gratifying when I do read an established author and find that, yep, more learned minds than mine have clearly articulated widely-accepted concepts that I’ve only figured out due to trial, error, and intuition.

In fact, one of the things I’m trying to do moving forward is to go back through the literature and connect it to the experiences I’ve had—not just for my own validation, but to reassure my audience and clients that the things I’ve done and the things I recommend aren’t all crazy talk. I’ve got Geoffrey A. Moore’s Crossing the Chasm model combined with Albert Wong’s model to form the Rainbow of Death, which comprises the core of my narrative now; and I’ve also recently added a very high-level view of Kurt Lewin’s theory of social change, which someone only recently suggested to me.

Views on Software Testing

Hexawise: In your online presentation, Making the Right Thing the Easy Thing, you note: [Use] "amplifying feedback loops to make sure knowledge is shared where needed as quickly and clearly as possible." How do you suggest this idea be applied by those involved with software testing?

Mike: Heh, that’s a paraphrase of the Second Way of Devops (out of Three), originally articulated by Gene Kim. Clearly the extent to which you can automate testable cases, to make it easy and fast to do, the better. People need to know that there are different kinds of automated tests for different levels of the software—they need to learn how to do the right thing the right way. Once developers in particular have gotten some traction with writing automated tests, then folks performing manual or system-level automated testing won’t waste their time catching (and re-catching!) bugs the developers could’ve easily caught, and can focus on truly pushing the limits of the software—reporting not just on whether it meets functional and nonfunctional requirements, but on the overall quality of the product.

In other words, a healthy balance of automated testing and manual testing plays to the strengths of all the humans and machines involved. When you’ve got optimal resource utilization happening, you eliminate a lot of both physical (in terms of slowness) and human friction, and a feeling of true partnership can take hold. Testers aren’t just the people reminding you that your code isn’t perfect—you’ve already reminded yourself of that through your own automated tests!—they’re the ones helping you make it even better.

it’s not about defects; it’s about feedback and collaboration. If you arrange incentives to produce an adversarial relationship between team members, e.g. if developers are incentivized to minimize defects and testers are incentivized to report defects, then that’s a house divided against itself.

Hexawise: What do you wish more developers, business analysts, and project managers understood about software testing?

Mike: Oh my. For one, it’s not about defects; it’s about feedback and collaboration. If you arrange incentives to produce an adversarial relationship between team members, e.g. if developers are incentivized to minimize defects and testers are incentivized to report defects, then that’s a house divided against itself. Some people think a degree of competition and/or adversarialism is a good thing, but when it comes to producing a product as a team—i.e. achieving a mission—you should keep it to a minimum in favor of fostering a spirit of collaboration.

Collaboration doesn’t mean blind consensus; it means communicating honestly in an environment in which we feel safe to do so, in which we share criticism in a spirit of mutual self-interest, not cutthroat competition.

One test type does not fit all. First, in terms of automated tests, unit testing can find a truly large number of errors, very quickly and cheaply, and tends to encourage better code quality (i.e. readability, maintainability, extensibility) overall. Integration tests can shake out errors and ambiguities between component contracts. High-level, developer-written system tests (as opposed to more extensive system tests developed by a dedicated tester) can quickly affirm that the entire product is in a buildable, runnable state. All of this “white box” testing by the developers is essential to giving the testers as high-quality a product as possible, so they can apply their “black box” techniques to push the product to its limits, rather than waste time alerting developers to defects they could’ve much more quickly, easily, and cheaply discovered themselves.

To this last point, I like to point to the examples of goto fail and Heartbleed. So many Internet “experts” threw up their hands and claimed that bugs like these were “too hard to test”. In both cases, after 2.5 years out of the industry (another story), I spent an evening diving into code I’d never seen before and wrote a test to reproduce each bug and validate its fix. After that, some liked to say, “Oh well, lots of other tools and techniques could’ve found these bugs.”

My claim isn’t that automated testing would’ve been the only way; my claim is that the discipline of automated testing likely would’ve prevented these bugs from ever existing even before writing a single test. With goto fail, the offending block of code was copied and pasted throughout the file six times! It was just that one of the six contained the errant “goto fail” line. But as I demonstrated with my version of the “fix”, extracting a common function and testing that six ways from Sunday likely would’ve avoided the problem entirely. In the case of Heartbleed, it was a failure to validate that an input buffer was actually as long as the user-supplied length indicated. Testing that kind of corner condition is unit testing 101, and the kind of thing you become more sensitive to every time you write a line of code once you’re in the habit of testing.

Hence, as difficult as it would’ve been for manual testing to discover these errors, and as long as it took for them to get shaken out months or years after their widespread deployment—Heartbleed via third-party fuzz testing, goto fail who knows how—both very, very likely could’ve been stopped dead in their tracks (or never would’ve existed!) if the developers were in the everyday habit of unit testing their code.

Hexawise: Our CTO, Sean Johnson, shared your memorably-named "Rainbow of Death" presentation with our management team. We absolutely loved it. In your presentation, you describe a series of concrete, practical steps you and your colleagues at Google took over the course of 5+ years to overcome resistance to change, educate teams, and successfully achieve broad adoption of automated testing efforts at Google across many teams, including lots of teams that were initially very change resistant. Can you please describe for our readers 2 or 3 noteworthy aspects of that change management journey?

Mike: What I hope the Rainbow of Death model, in combination with Geoffrey A. Moore’s Crossing the Chasm model, make apparent is that different people adopt change differently. There are many needs that need to be met by and for many different people, and the chances of figuring out the perfect plan to execute before taking any action are practically zero. After all, don’t the Agile and DevOps models that are all the rage comprise tools and practices for adapting to change, for performing experiments and adjusting course based on feedback? Organizational change is no different, yet many people remain conditioned to expect waterfall-like solutions to their social problems.

image showing items supporting change in the organization


Also, I mention in the talk that “The problem you want to solve may not be the problem you have to solve first.” In our case, we wanted to solve the problem of developers not writing enough automated tests. But first, we had to solve two other problems: People back then had very little exposure to or experience with automated testing, leading to the “My code is too hard to test” excuse, because they had no idea how to test it, or to write testable code to begin with.

The second problem was that the tools at the time couldn’t keep up with the growth of the company, its products, and its code base. It was growing ever more painful to write any code to begin with, yet delivery pressure was intense and Imposter Syndrome was rampant—on top of the fear of admitting your code might contain flaws, how could you make any time to learn how to write automated tests to begin with? Hence the “I don’t have time to test” excuse.

So we couldn’t just say “Testing is good! Yay testing! Please write moar testz!”

I think this mix of perspective, empathy, creativity, collaboration, tenacity, and patience is crucial to changing not only tech organizations, but society at large. I hope to put this notion, and the Rainbow of Death model in particular, to the test continuously throughout the remainder of my career.

my advice to both developers and testers is to identify the priorities, the social structures and dynamics at play in the organization. How can you work with these structures and dynamics instead of against them—or do you need to create a culture of open communication and collaboration in parallel with (or even before) communicating the testing message?

Industry Observations / Industry Trends

Hexawise: Large companies often discount the importance of software testing. What advice do you have for software testers to help their organizations understand the importance of expecting more from the software testing efforts in the organization?

Mike: Sadly, there’s no one message that works for every company, every culture, everywhere. It’s up to the Instigators in each environment to take the timeless principles I believe are essential—that testing is about feedback and collaboration, that different types of tests all catch different and important bugs, that developers and testers have different and mutually-reinforcing roles to play—and find the right cultural hooks to hang those messages on. In the case of Google, it took the Testing Grouplet five years to figure out and successfully implement, and it took an array of parallel efforts across multiple groups to saturate the culture with the message, not just one magical tool or technique or team to bind them all.

So my advice to both developers and testers is to identify the priorities, the social structures and dynamics at play in the organization. How can you work with these structures and dynamics instead of against them—or do you need to create a culture of open communication and collaboration in parallel with (or even before) communicating the testing message? This is the punchline of my Rainbow of Death presentation: The problem you want to solve may not be the problem you have to solve first, and the Standard Narrative from which all the problems emerged will not produce any solutions—though it may provide the keys necessary to unlock effective solutions.

At Google, it was the Testing Grouplet’s Test Certified program and all the other education and tooling efforts supporting it that provided the right hook—after two years of experimentation and reflection! But don’t focus on what Test Certified was comprised of: focus on why that approach worked for us, and see if that reflection inspires an approach that will work for your company.

In addition to that, it probably wouldn’t hurt to remind anyone who’ll listen of goto fail and Heartbleed, and how basic unit testing practices and the coding habits they encourage could’ve prevented these potentially catastrophic defects from even being written in the first place.

Hexawise: The story of your team's journey is fantastic. We highly recommend it to IT organizations embarking on any large improvement effort. Thank you for sharing it. We've recently started using elements of your approach to help our clients successfully adopt test optimization approaches at scale in their organizations.

Mike: That’s incredibly gratifying to hear! Please keep me in the loop of how well it’s working for you and your clients. Just as the impact of Testing Grouplet’s efforts were far greater than the sum of any individual part, and as my Rainbow of Death presentation benefited enormously from the input of trusted fellow Instigators to help illustrate it, I’m sure there are many more insights and improvements waiting to emerge from the model once more people have applied it farther and wider than I ever could on my own!

Hexawise: Do you believe the DevOps movement is resulting in better software testing within organizations? Do you see any other trends that software testers could leverage to promote improved application of software testing practices?

See full interview for Mike’s response.

Staying Current / Learning

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf?

Mike: Sadly, I’m not current enough to make any solid recommendations. In my career, I’ve moved more into the culture change space than being a 100% in-the-trenches practitioner. That said, I certainly did years of quality time with my old “library” of programming and algorithms books, and was fortunate to be part of a culture that itself generated a broad swath of automated testing knowledge. Though I don’t keep up with the details of the latest developments, that time spent internalizing the core principles has served me very well throughout my career.

That said, I’m sure that great books exist, and people dedicated to the craft would do themselves a great service by discovering them and spending years plumbing their depths, rather than trying to read every book on the subject forevermore. That’s the model that worked for me, at least; but perhaps a more voracious reading regimen suits you better. Everybody’s different.

See the full interview for more questions and answers.

Profile

Mike aims to produce a culture of transparency, autonomy, and collaboration, in which “Instigators” are inspired and encouraged to make creative use of existing systems to drive improvement throughout an organization. The ultimate goal of such efforts is to make the right thing the easy thing. He's followed this path since 2005, when he helped drive adoption of automated testing throughout Google as part of the Testing Grouplet, the Test Mercenaries, and the Fixit Grouplet. He was instrumental in the execution of Test Certified and Testing on the Toilet, and the four company-wide Fixits he organized led to the development and rollout of the Test Automation Platform. His account of Google’s automated testing adoption also appears as a case study in The DevOps Handbook by Gene Kim, et. al.

He also served as a member of the Websearch Infrastructure team, which practiced DevOps before he was aware it had a name. Frequently working in concert with other indexing infrastructure teams, he also worked closely with Release Engineers and Site Reliability Engineers to package, release, deploy, and monitor multiple indexing services.

Most recently he served as Practice Director at 18F, a technology team within the U.S. General Services Administration, where he personally launched and drove several initiatives to increase 18F’s capability as a learning organization, including the Pages platform, the Guides series, and the Handbook.

Links:

This post includes highlights from our full interview with Mike Bland. The full interview is long and packed with great thoughts.

By: John Hunter on Mar 27, 2017

Categories: Interview, Software Testing, Testing Smarter with..., Testing Strategies

This interview with Alan Page is part of our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Alan Page has been a software tester (among other roles) since 1993 and is currently the Director of Quality for Services at Unity. Alan spent over twenty years at Microsoft working on a variety of operating systems and applications in nearly every Microsoft division.


Alan Page

Personal Background

Hexawise: Looking back on over 20 years in software testing at Microsoft can you describe a testing experience you are especially proud of?

Alan: There are a lot of great experiences to choose from - but I'm probably most proud of my experiences on the Xbox One team. My role was as much about building a community of testers across the the Xbox ecosystem (console, services, game development) as it was about testing and test strategy for the console. We had a great team of testers, including a handful of us with a lot of testing experience under our belts. We worked really well together, leveraged each others strengths, and delivered a product that has had nearly zero quality issues since the day of release.

Hexawise: What did you enjoy about working in such a large software company, Microsoft, for so long?

Alan: The only way I survived at Microsoft so long was that I could change jobs within the company and get new experiences and challenges whenever I felt I needed them. I love to take on new challenges and find things that are hard to do - and I always had that opportunity.

The biggest thing I look for in testers is a passion and ability to learn. I've interviewed hundreds of testers... The testers who really impress me are those who love to learn - not just about testing, but about many different things. Critical thinking and problem solving are also quite important.

Hexawise: What new challenges are you looking forward to tackling in your new role at Unity?

Alan: I'm looking forward to any and all challenges I can find. Specifically, I want to build a services testing community at Unity. Building community is something I feel really strongly about, and from what I've seen so far, I think there are a lot of opportunities for Unity testers to learn a lot from each other while we play our part in the coming growth of Unity services.

Views on Software Testing

Hexawise: What thoughts do you have in involving testers in A/B and multivariate testing? That stretches the bounds of how many people categorize testers but it could be a good use of the skills and knowledge some testers process.

Alan: My approach to experimentation (A/B testing) is that there are three roles needed. Someone needs to design the experiment. A product owner / program manager often plays this role and will attempt to figure out the variation (or variations) for the experiment as well as how to measure the business or customer value of both the control (original implementation) and treatment / experiment.

The second role is the implementer - this is typically a developer or designer depending on the nature of the experiment. Implementing an experiment is really no different than implementing any other product functionality.

The final role is the analyst role - someone needs to look at the data from the experiment, as well as related data and attempt to prove whether the experiment is a success (i.e. the new idea / treatment is an improvement) or not. I've seen a lot of testers be successful in this third role, and statistics and data science in general, are great skills for testers to learn as they expand their skill set.

image of book cover for How We Test Software at Microsoft

Hexawise: During your 20 years at Microsoft you have been involved in hiring many software testers. What do you look for when choosing software testers? What suggestions do you have for those looking to advance in their in software testing career?

Alan: The biggest thing I look for in testers is a passion and ability to learn. I've interviewed hundreds of testers, including many who came from top universities with advanced degrees who just weren't excited about learning. For them, maybe learning was a means to an end, but not something they were passionate about.

The testers who really impress me are those who love to learn - not just about testing, but about many different things. Critical thinking and problem solving are also quite important.

As far as suggestions go, keep building your tool box. As long as you're willing to try new things, you'll always be able to find challenging, fun work. As soon as you think you know it all, you will be stuck in your career.

Combinatorial testing is actually pretty useful in game testing. For example, consider a role-playing game with six races, ten character classes, four different factions, plus a choice for gender. That's 480 unique combinations to test! Fortunately, this has been proven to be an area where isolating pairs (or triples) of variations makes testing possible, while still finding critical bugs.

Hexawise: Do you have a favorite example of a combinatorial bug and how it illuminated a challenge with software testing?

Alan: I was only indirectly involved in discovering this one, but the ridiculously complex font-picker dialog from the Office apps was a mess to test - but it was tested extensively (at least according the person in charge of testing it). A colleague of mine showed them the all pairs technique, and they used it to massively decrease their test suite - and found a few bugs that had existed for years.

Hexawise: It seems to me that testing games would have significant challenges not found in testing fairly straightforward business applications. Could you share some strategies for coping with those challenges?

Alan: Combinatorial testing is actually pretty useful in game testing. For example, consider a role-playing game with six races, ten character classes, four different factions, plus a choice for gender. That's 480 unique combinations to test! Fortunately, this has been proven to be an area where isolating pairs (or triples) of variations makes testing possible, while still finding critical bugs.

Beyond that, testing games requires a lot of human eyeballs and critical thinking to ensure gameplay makes sense, objects are in the right places, etc. I've never seen a case where automating gameplay, for example, has been successful. I have, however, seen some really innovative tools written by testers to help make game testing much easier, and much more effective.

Hexawise: That sounds fascinating, could you describe one or more of those tools?

Alan: The games test organization at Microsoft wrote a few pretty remarkable tools. One linked the bug tracking system to a "teleportation" system in the game under test, including a protocol to communicate between a windows PC and an xbox console.

This enabled a cool two-way system, where a tester could log a bug directly from the game, and the world coordinates of where the bug occurred were stored automatically in the bug report. Then, during triage or debugging, a developer / designer could click on a link in the bug report, and automatically transport to the exact place on the map where the issue was occurring.

Hexawise: What type of efforts to automate gameplay came close to providing useful feedback? What makes automating gameplay for testing purposes ineffective?

Alan: I would never automate gameplay. There are too many human elements needed for a game to be successful - it has to be fun to play - ideally right at that point between too challenging, and too easy. If the game has a story line, a tester needs to experience that story line, evaluate it, and use it as a reference when evaluating gameplay.

Tools to evaluate cpu load or framerate during gameplay are usually a much better investment than trying to simulate gameplay via automation.

That said, there are some "what if" scenarios in games that may provide interesting bugs. For example simulating a user action (e.g. adding and removing an item) thousands of time may reveal memory leaks in the application. It really comes down to test design and being smart about choosing what makes sense to automate (and what doesn't make sense).

Hexawise: What is one thing you believe about software testing that many smart testers disagree with?

Alan: I don't believe there's any value from distinguishing "checks" from tests. I know a lot of people really like the distinction, but I don't see the value. Of course, I recognize that some testing is purely binary validation, but this is a(nother) example of where choosing more exact words in order to discern meaning leads to more confusion and weird looks than it benefits the craft of testing.

Industry Observations / Industry Trends

Hexawise: Do you believe testing is becoming (or will become) more integrated with the software development process? And how do you see extending the view of the scope of testing to include all the way from understanding customer needs to reviewing actual customer experience to drive the testing efforts at an organization.

Alan: I believe testing has become more integrated into software development. In fact, I believe that testing must be integrated into software development. Long ship cycles are over for most organizations, and test-last approaches, or approaches that throw code to test to find-all-the-bugs are horribly inefficient. The role of a tester is to provide testing expertise to the rest of the team and accelerate the entire team's ability to ship high-quality software. Full integration is mandatory for this to occur.

It's also important for testers to use data from customer usage to help them develop new tests and prioritize existing bugs. We've all had conversations before about whether or not the really cool bug we found was "anything a customer would ever do" - but with sufficient diagnostic data in our applications or services, we can use data to prove exactly how many customers could (or would) hit the issue. We can also use data to discover unexpected usage patterns from customers, and use that knowledge to explore new tests and create new test ideas.

There's an important shift in product development that many companies have recognized. They've moved from "let's make something we think is awesome and that you'll love" - i.e. we-make-it-you-take-it to "we want to understand what you need so we can make you happy". This shift cannot happen without understanding (and wallowing in) customer data.

Hexawise: In your blog post, Watch out for the HiPPO, you stated: "What I’ve discovered is that no matter how strongly someone feels they “know what’s best for the customer”, without data, they’re almost always wrong." What advice do you have for testers for learning what customers actually care about? To me, this points out one of the challenges software testers (and everyone else actually) faces, which is the extent to which they are dependent on the management system they work within. Clayton Christensen's Theory of Jobs to Be Done is very relevant to this topic in my opinion. But many software testers would have difficulty achieving this level of customer understanding without a management system already very consistent with this idea.

Alan: Honestly, I don't know how a product can be successful without analysis and understanding of how customers use the product. The idea of a tester "pretending to be the customer" based purely on their tester-intuition is an incomplete solution to the software quality problem. If you hear, "No customer would ever do that", you can either argue based on your intuition, or goo look at the data and prove yourself right (or wrong).

There's an important shift in product development that many companies have recognized. They've moved from "let's make something we think is awesome and that you'll love" - i.e. we-make-it-you-take-it to "we want to understand what you need so we can make you happy". This shift cannot happen without understanding (and wallowing in) customer data. Testers may not (and probably won't) create this system, but any product team interested in remaining in business should have a system for collecting and analyzing how customers use their product.

Hexawise: I agree, this shift is extremely important. Do you have an example of how using such a deep understanding of users influenced testing or how it was used to shift the software development focus. Since software testing is meant to help make sure the company provides software that users want it certainly seems important to make sure software testers understand what users want (and don't want).

Alan: First off, I prefer to think of the role of test as accelerating the achievement of shipping quality software; which is similar to making sure the company provides software customers want, but (IMO), is a more focused goal.

As far as examples go...how many do you want? Here a few to ponder:

  • We tracked how long people ran our app. 45% ran it continuously - all the time. This is the way we ran the app internally. But another 45% ran it for 10 minutes or less. We had a lot of tests that tried to mimic a week of usage and look for memory leaks and other weirdness, but after seeing the data, we ended up doing a lot more testing (and improving) of start up and shut down scenarios.
  • We once canceled a feature after seeing that the data told us that it was barely used. In this case, we didn't add the tracking data until late (a mistake on our part). Ideally, we'd discover something like this earlier, and than redesign (rather than cancel)
  • We saw that a brand new feature was being used a lot by our early adopters. This was pretty exciting...but as testers, we always validate our findings. It turned out after a bit more investigation, that the command following usage of the brand new feature was almost always undo. Users were trying the feature... but they didn't like what it was doing.

Staying Current / Learning

Hexawise: What advice do you have for people attending software conferences so that they can get more out of the experience?

Alan: Number one bit of advice is to talk to people. In fact, I could argue that you get negative value (when balanced against the time investment) if you show up and only attend talks. If there's a talk you like in particular, get to know the speaker (even us introverts are highly approachable - especially if you bring beer). Make connections, talk about work, talk about challenges you have, and things you're proud of. Take advantage of the fact that there are a whole lot of people around you that you can learn from (or who can learn from you).

Additionally, if you're in a multi-track conference, feel free to jump from talk to talk if you're not getting what you need - or if it just happens that two talks that interest you are happening at the same time.

Hexawise: How do you stay current on improvements in software testing practices; or how would you suggest testers stay current?

Alan: I read a lot of testing blog posts (I use feedly for aggregation). I subscribe to at least fifty software development and test related blogs. I skim and discard liberally, but I usually find an idea or two every week that encourages me to dig deeper.

Biggest tip I have, however, is to know how essential learning is to your success. Learning is as important to the success of a knowledge worker as food is to human life. Anyone who thinks they're an "expert" and that learning isn't important anymore is someone on a career death spiral.

Profile

image of book cover for The A Word by Alan Page

Alan has been a software tester (among other roles) since 1993 and is currently the Director of Quality for Services at Unity. Alan spent over twenty years at Microsoft working on a variety of operating systems and applications in nearly every Microsoft division. Alan blogs at angryweasel.com, hosts a podcast (w/ Brent Jensen) at angryweasel.com/ABTesting, and on occasion, he speaks at industry testing and software engineering conferences.

Links Books: How We Test Software at Microsoft, The A Word

Blog: Tooth of the Weasel

Podcast: AB Testing

Twitter: @alanpage


Related posts: Testing Smarter with Dorothy Graham - Testing Smarter with James Bach

By: John Hunter on Mar 16, 2017

Categories: Combinatorial Software Testing, Customer Success, Software Development, Software Testing, Testing Smarter with..., Interview

This interview with Dorothy Graham is part of our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Dorothy Graham has been in software testing for over 40 years, and is co-author of 4 books: Software Inspection, Software Test Automation, Foundations of Software Testing and Experiences of Test Automation.


Dorothy Graham

Personal Background

Hexawise: Do your friends and relatives understand what you do? How do you explain it to them?

Dot: Some of them do! Sometimes I try to explain a bit, but others are just simply mystified that I get invited to speak all over the world!

Hexawise: If you could write a letter and send it back in time to yourself when you were first getting into software testing, what advice would you include in it?

Dot: I would tell myself to go for it. When I started in testing it was definitely not a popular or highly-regarded area, so I accidentally made the right career choice (or maybe Someone up there planned my career much better than I could have).

Hexawise: Which person or people have had the greatest influence on your understanding and practice of software testing?

Dot: Tom Gilb was (and is) a big influence, although in software engineering in general rather than testing. The early testing books (Glenford Myers, Bill Hetzel, Cem Kaner, Boris Beizer) were very influential in helping me to understand testing. I attended Bill Hetzel’s course when I was just beginning to specialize in testing, and he was a big influence on my view of testing back then. Many discussions about testing with friends in the Specialist Group In Software Testing (SIGIST), and colleagues at Grove, especially Mark Fewster.

Hexawise: Failures can often lead to interesting lessons learned. Do you have any noteworthy failure stories that you’d be willing to share?

Dot: I was able to give a talk about a failure story in Inspection, where a very successful Inspection initiative had been “killed off” by a management re-organisation that failed to realise what they had killed. There were a lot of lessons about communicating the value of Inspection (which also apply to testing and test automation). The most interesting presentation was when I gave it at the company where it had happened a number of years before; one (only one) person realized it was them.

One fallacy is in thinking that the automation does only what is done by manual tests. There are often many things that are impossible or infeasible in manual testing that can easily be done automatically.

Hexawise: Describe a testing experience you are especially proud of. What discovery did you make while testing and how did you share this information so improvements could be made to the software?

Dot: One of the most interesting experiences I had was when I was doing a training course in Software Inspection for General Motors in Michigan, and in the afternoon, when people were working on their own documents, several people left the room quite abruptly. I later found out that they had gone to stop the assembly line! They were Inspecting either braking systems or cruise control systems.

When I was working at Ferranti on Police Command and Control Systems, I discovered that the operating system’s timing routines were not working correctly. This was critical for our system, as we had to produce reminders in 2 minutes and 5 minutes if certain actions had not been taken on critical incidents. I wrote a test program that clearly highlighted the problems in the OS and sent it to the OS developers, wondering what their reaction would be. In fact they were delighted and found it very useful (and they did fix the bugs). It did make me wonder why they hadn’t thought to test it themselves though.

Views on Software Testing

Hexawise: In That's No Reason to Automate, you state "Testing and automation are different and distinct activities." Could you elaborate on that distinction?

Dot: By “automation”, I am referring to tools that execute tests that they have been set up to run. Execution of tests is one part of testing but testing is far more than just running tests. First what it is that needs to be tested? Then how best can that be tested - what specific examples would make good test cases? Then how are these tests constructed to be run? Then they are run, and afterwards, what was the outcome from the test compared to what was expected, i.e. did the software being tested do what it should have done? This is why tools don’t replace testers - they don’t do testing, they just run stuff.


Diagram from That's No Reason to Automate

Hexawise: Test automation allows for great efficiency. But there are limits to what can be effectively automated. How do you suggest testers determine what to automate and what to devote tester's precious time toward investigating personally?

Dot: There are two fallacies when people say “automate all the manual tests” (which is unfortunately quite common). The first one is that all manual tests should be automated - this is not true. Some tests that are now manual would be good candidates for automation, but other tests should always be manual. For example, “do these colours look nice?” or “is this what users really do?” or even “Captcha” where it should only work if a person not a machine is filling in a form. Think about it - if your automated test can do the Captcha, then the Captcha is broken.

Automating the manual tests “as is" is not the best way to automate, since manual tests are optimized for human testers. Machines are different to people, and tests optimized for automation may be quite different. And finally, a test that takes an inordinate amount of time to automate, if the test isn’t needed often, is not worth automating.

The other fallacy is in thinking that the automation does only what is done by manual tests. There are often many things that are impossible or infeasible in manual testing that can easily be done automatically. For example if you are testing manually, you see what is on the screen, but you don’t know if all the GUI (graphical user interface) objects are in the correct state - this could be checked by a tool. There are also types of testing that can only be done automatically, such as Monkey testing or High Volume Automated Testing.

Hexawise: Can you describe a view or opinion about software testing that you have that many or most smart experienced software testers probably disagree with? What led you to this belief?

Dot: One view that I feel strongly about, and many testers seem to disagree with this, is that I don’t think that all testers should be able to write code, or, what seems to be the prevailing view, that the only good testers are testers who can code. I have no objection to people who do want to do both, and I think this is great, especially in Agile teams. But what I object to is the idea that this has to apply to all testers.

There are many testers who came into testing perhaps through the business side, or who got into testing because they didn’t want to be developers. They have great skills in testing, and don’t want to learn to code. Not only would they not enjoy it, they probably wouldn’t be very good at it either! As Hans Buwalda says, “you lose a good tester and gain a poor programmer.” The ultimate effect of this attitude is that testing skills are being devalued, and I don’t think that is right or good for our industry!

The best way to have fewer bugs in the software is not to put them in in the first place. I would like to see testers become bug prevention advisors! And I think this is what happens on good teams.

Hexawise: What do you wish more developers, business analysts, and project managers understood about software testing?

Dot: The limitations of testing. Testing is only sampling, and can find useful problems (bugs) but there aren’t enough resources or time in the universe to “test everything”.

The best way to have fewer bugs in the software is not to put them in in the first place. I would like to see testers become bug prevention advisors! And I think this is what happens on good teams.

Hexawise: I agree that becoming bug prevention advisors is a very powerful concept. Software development organizations should be seeking to improve the entire system (don't create bugs, then try to catch them and then fix them - stop creating them). I recently discussed these ideas in Software Code Reviews from a Deming Perspective.

Dot: I had a quick look at the blog and like it - especially the emphasis on it being a learning experience. It seems to be viewing “inspection” in the manufacturing sense as looking at what comes off the line at the end. In our book “Software Inspection” (which is actually a review process), it is very much an up-front process. In fact applying inspection/reviews to things like requirements, contracts, architecture design etc is often more valuable than reviewing code.

I also fully agree with selecting what to look at - in our book we recommend a sampling process to identify the types of serious bug, then try to remove all other of that type - that way you can actually remove more bugs than you found.

Hexawise: In what ways would you see "becoming bug prevention advisors" happening? Done in the wrong way it can make people defensive about others advising them how to do their jobs in order to avoid creating bugs. What advice do you have for how software testers can make this happen and do it well?

Dot: I see the essential mind-set of the tester as asking “what could go wrong? or “what if it isn’t” [as it is supposed to be]. If you have a Agile team that includes a tester, then those questions are asked perhaps as part of pair working - that way the potential problems are identified and thought about before or as the code is written. The best way is close collaboration with developers - developers who respect and value the tester’s perspective.

Industry Observations / Industry Trends:

Hexawise: Your latest book, Experiences of Test Automation, includes an essay on Exploratory Test Automation. How are better tools making the ideas in this essay more applicable now than they were previously?

Dot: Actually this is now called High-Volume Automated Testing, which is a much better name for it. HiVAT uses massive amounts of test inputs, with automated partial oracles. The tests are checking that the results are reasonable, not exact results, and any that are outside those bounds are reported to humans to investigate. Harry Robinson described using this technique to test Bing - quite a large thing to test!

Hexawise: When looking to automate tests one thing people sometimes overlook is that given the new process it may well be wise to add more tests than existed before. If all test cases were manually completed that list of cases would naturally have been limited by the cost of repeatedly manually checking so many test cases in regression testing. If you automate the tests it may well be wise to expand the breadth of variations in order to catch bugs caused by the interaction of various parameter values. What are your thoughts on this idea?

Dot: Nice analogy - I like the term “grapefruit juice bugs”. Using some of the combinatorial techniques is a good way to cover more combinations in a reasonably rigorous way. Automation can help to run more tests (provided that expected results are available for them) and may be a good way to implement the combination tests, using pair-wise and/or orthogonal arrays.

Hexawise: In your experience how big of a problem is automation decay (automated tests not being maintained properly)? How do recommend companies avoid this pitfall?

Dot: This seems to be the most common way that test automation efforts get abandoned (and tools become shelfware), particularly for system-level test automation. It can be avoided by having a well-structured testware architecture in the first place, where maintenance concerns are thought about beforehand, using good programming practices for the automation code. Although fixing this later is possible, preventing it in the first place is far better.

Hexawise: Large companies often discount the importance of software testing. What advice do you have for software testers to help their organizations understand the importance of software testing efforts in the organization?

Dot: Unfortunately often the best way to get an organisation to appreciate testing is to have a small disaster, which adequate testing could have prevented. I’m not exactly suggesting that people should do this, but testing, like insurance, is best appreciated by those who suffer the consequences of not having had it. Perhaps just try to make people aware of the risks they are taking (pretend headlines in the paper about failed systems?)

Staying Current / Learning

Hexawise: I see that you’ll be presenting at the StarEast conference in May. What could you share with us about what you’ll be talking about?

Dot: I have two tutorials on test automation. One is aimed mainly at managers who want to ensure that a new (or renewed) automation effort gets off “on the right foot”. There are a number of things that should be done right at the start to give a much better chance of lasting success in automation.

My other tutorial is about Test Automation Patterns (using the wiki). Here we will look at common problems on the technical side (rather than Management issues), and how people have solved these problems - ideas that you can apply. This also a code-free tutorial - we are talking about generic technical issues and patterns, whatever test execution tool you use.

Hexawise: What advice do you have for people attending software conferences so that they can get more out of the experience?

Dot: Think about what you want to find out about before you come, and look through the programme to decide which presentations will be most relevant and helpful for you. It’s very frustrating to realise at the end of the conference that you missed the one presentation that would have given you the best value, so do your homework first!

But also realise that the conference is more than just attending sessions. Do talk to other delegates - find out what problems you might share (especially if you are in the same session). You may want to skip a session now and then to have a more thorough look around the Expo, or to have a deeper conversation with someone you have met or one of the speakers. The friends you make at the conference can be your “support group” if you keep in touch afterwards.

I don’t think that all testers should be able to write code, or, what seems to be the prevailing view, that the only good testers are testers who can code. I have no objection to people who do want to do both, and I think this is great, especially in Agile teams. But what I object to is the idea that this has to apply to all testers.

Hexawise: How do you stay current on improvements in software testing practices; or how would you suggest testers stay current?

Dot: I am fortunate to be invited to quite a few conferences and events. I find them stimulating and I always learn new things. There are also many blogs, webinars and online resources to help us all try to keep up to some extent.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf?

Mine, of course! ;-) I do hope that testers have a bookshelf! For testers, I recommend Lee Copeland’s book “A Practitioner’s Guide to Software Test Design” as a great introduction to many techniques. For a deep understanding, get Kem Caner’s book “The Domain Testing Workbook”. I also like “Lessons Learned in Software Testing” by Kaner, Bach and Pettichord, and “Perfect Software and other illusions about testing” by the great Gerry Weinberg. But there are many other books about testing too!

Hexawise: What are you working on now?

Dot: I am working on a wiki called TestAutomationPatterns.org with Seretta Gamba. This is a free resource for system level automation, with lists of different issues or problems and resolving patterns to help address them. There are four categories: Process, Management, Design and Execution. There is a “Diagnostic” to help people identify their most pressing issue, which then leads them to the patterns to help. We are looking for people to contribute their own short experiences to the issues or patterns.

Dorothy Graham has been in software testing for over 40 years, and is co-author of 4 books: Software Inspection, Software Test Automation, Foundations of Software Testing and Experiences of Test Automation, and currently helps develop TestAutomationPatterns.org.

Dot has been on the boards of conferences and publications in software testing, including programme chair for EuroStar (twice). She was a founder member of the ISEB Software Testing Board and helped develop the first ISTQB Foundation Syllabus. She has attended Star conferences since the first one in 1992. She was awarded the European Excellence Award in Software Testing in 1999 and the first ISTQB Excellence Award in 2012.

Website

Twitter: @DorothyGraham

By: John Hunter on Mar 10, 2017

Categories: Software Testing, Testing Smarter with..., Interview

This interview with James Bach is the first our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

James Bach, one of the most well-known and controversial leaders in the software testing community, challenges himself and others to continually develop their software testing approaches. James believes that excellent testing is a craft that requires many skills and ongoing practice and focus to develop and maintain those skills. The skills of testing include general systems analysis and critical thinking, but also social skills. In some sense, any child can test. But children and other amateurs cannot test systematically, nor can they provide professional self-assessment and reporting on the testing they do.

photo of James Bach
James Bach, Founder and CEO of Satisfice Inc

Personal Background

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

James Bach: In 2002, Microsoft complained to a federal judge that I hadn’t given it a power cord. Yes, an ordinary power cord of the kind you can pull out of the back of any standard desktop computer. Yes, to a federal judge. No, I am not making this up. Yes, I also thought it was bizarre-- bizarre but kind of satisfying.

This happened during the Microsoft Remedies Trial, wherein nine American states were suing Microsoft and the government because they wanted a tougher punishment for Microsoft after it lost its big antitrust case. The states hired me as an expert witness to find out if Microsoft was telling the truth when it claimed it was “technically infeasible” to remove IE and the Windows Media Player. I gathered a team and went to work. I soon discovered that it was possible to remove these things-- using only public information and Microsoft’s own helpful tech support people to set up my testing to prove it. (The tech support people did not realize the purpose of my questions, and cheerily gave me all that I needed to know.)

When I revealed my results, Microsoft demanded that I turn over all the materials necessary to reproduce my results. So I gave them one of my test systems. At the last moment, I pulled out the power cord, thinking that I was causing some low-level techie 30 seconds of annoyance. But the next day Microsoft was in court acting like I had withheld the Golden Power Cord of Truth.

It was satisfying because Microsoft never even attempted to proved me wrong on the facts. They used lawyer tricks to stop my truth bombs, instead. Way to go, Bill Gates.

Another satisfying moment was watching my son find a catastrophic bug in a life-critical piece of medical equipment. He didn’t ask for a spec or a test case specification. He used video-gamer techniques to confuse the system until it overrode its own safety features and melted itself inside the simulated patient. Yeah. Melted. That was a $3000 piece of equipment he ruined. I’ve rarely been prouder of myself for having such foresight as to create a son like him: he is such a good test tool.

people who don’t “embrace exploratory testing” are, to me, not even testers. They are fact checkers, maybe. I think that’s not good enough.

Hexawise: Failures can often lead to interesting lessons learned. Do you have any noteworthy failure stories that you’d be willing to share?

James: How about the time I tried to set up my corporate server. I got it all working. Then I moved it from the conference room to the server room. I couldn’t get it to come online after that. For 12 hours I worked on it, all through the night. At long last I relented to my brother’s suggestion-- that we move it back to the conference room. I had refused to do that because it didn’t make any sense. The conference room simply connected to the server room. How would adding an extraneous variable like that change anything. But, zoom, we were back online.

After a moment of “wha??” the solution flashed into my mind: we must have two feeds to the Internet instead of one. It turned out that the conference room was patched into the open net, but the other port I had been using in the server room was routed through a firewall which in turn connected to the net. Nobody told me this during the buildout of my office space. It was a completely missing possibility in my mind.

So what did I learn? I learned about the importance of de-focusing, which includes trying apparently silly things to solve problems. I’m more open to that now.

Here is another interesting failure. I recently wrote a report involving the calculation of percentages. A non-technical person (a lawyer I worked with) checked my math and found it to be wrong. In fact, every one of her calculations was wrong. But in the process of refuting her claims, I discovered a different error in one of my own numbers. So, isn’t that interesting? Even if a critique of your work is incorrect, it could still be a useful stimulant to help you find your own problems.

Hexawise: What kinds of activities do you enjoy when you’re not at work?

James: I run a business, so I feel like I’m always at work. But I guess I do take little bits of time off each day. What do I do? I daydream. I read science news. I solve math and logic puzzles. I try to walk each day. And I watch videos with my wife. We binge on English television series, mostly.

Hexawise: Describe a testing experience you are especially proud of. What discovery did you make while testing and how did you share this information so improvements could be made to the software?

James: Well, many of those things I now use as testing exercises for my students, so I don’t want to spoil them. But, hmm, here’s one. I was given one day to break into an invoicing system for a large pharmaceutical company. I found three ways to do it. One of the methods I used was to get one of the sales engineers to sit with me while I tested. I asked him to demo the system to me and then he hung around while I tried to break-in. The first time I broke in (using a traversal attack if you follow such things) I didn’t even know I had done it until the sales engineer said “hey you aren’t supposed to see that data.” Good thing he was there, huh? So part of testing can be charming people into helping you, and you never know what that help will bring.

Views on Software Testing

Hexawise: Some of the thought-provoking ideas you and Michael Bolton have come up with, like the important distinction between Testing vs. Checking have received a great deal of attention within the community. Other intellectual contributions to the community you have made are not as well known but are arguably equally important and insightful. One such contribution that comes to mind which really resonates with us at Hexawise is the exploratory-scripted (or formality) continuum you and your brother Jon described.

image of software testing formality continuum

Do you have one or two intellectual contributions to the community that you wish were more widely known?

James: I wish that more people understood the folly, the sheer silliness, of counting test cases and calculating pass rates.

I don’t care if you have 80 test cases or 8 million of them. That number tells me nothing about you or your testing. It tells me nothing by itself, and it tells me nothing in conjuction with other information (except in rare cases not worth talking about). It’s like telling me that you have broken your day into 27 tasks, of 1353 tasks, or whatever. Just stop. Instead of fake science smoke rings, tell me what you actually did. Here’s a simple suggestion: instead of giving me a number, give me a list: a list of test ideas, test cases, test activities, bugs, features, people… I can do something with a list. But if you give me a number I just have to say “show me the things you are counting.”

Hexawise: Can you describe a view or opinion about software testing that you have changed your mind about in the last few years? What caused you to change your mind?

James: I used to think it was useful to talk about exploratory testing. But now I think it’s more helpful to say that all testing is exploratory. To say “exploratory testing” is the same as saying “testing.” Instead, I speak of how testing can be more or less formal, but it is always informal to some degree or else it ceases to be testing.

Also, in the last few years I have concluded that term “test automation” is toxic and should be avoided. It deposits a little poison in the mind whenever it is uttered. An angel loses its wings every time someone calls himself an “automated tester.” I changed my mind on both things as the result of ongoing attempts to teach students and hearing their questions and seeing where they get confused. That, and deep conversations with Michael Bolton.

Hexawise: How do/would you test very complex systems such as genetic algorithm systems and evolutionary systems? How do you test systems when we don't understand how they work? It seems kind of like medical differential diagnosis: poke, observe, learn, hypothesize, poke again. Or is there a better way?

James: I test them using social science methods. That, after all, is how scientists attempt to test their theories about social life. That means an emphasis on qualitative analysis, but bringing in statistical methods whenever applicable.

I agree that the medical world is a good example of where statistical methods and heuristic approaches are also needed. In testing complex things, some of what you need to do includes:

  • You must use time to your advantage-- observing systems over time the way primatologists observe chimps in the wild.
  • You must use Grounded Theory, beginning with immersion and observation, until patterns begin to reveal themselves.
  • You must focus on testability. To create an environment where you can control and observe more of what is there.
  • You must pay attention to clues. Many, many clues. Stop looking for simplistic “test cases” that will “prove” that the software works.
  • You must become expert at data wrangling, since these systems usually involve huge amounts of data.
  • Let other people help you.
  • Forge partnerships with users.

If it’s a training gig then my objective is to show them what testing can be, show them a path to get there, and encourage them to walk that path. A lot of that is about removing the obstacles to moving along.

Hexawise: It is clear from your writings and frequent presentations, that you feel passionately that the software testing community would greatly benefit if far more testers embraced Exploratory Testing. It’s a deeply held conviction. What particular testing practice(s) do you most wish the software testing community would embrace?

James: Testing is exploratory. So, people who don’t “embrace exploratory testing” are, to me, not even testers. They are fact checkers, maybe. I think that’s not good enough.

I wish more testers were mathematically inclined. You must see this, too, at Hexawise-- the widespread math-phobia in our field. I want to talk about Karnaugh maps and the value of de Bruijn sequences. But I have to keep that stuff out of my classes or I will freak most people out. It’s not that I am a mathematics expert. I’m just an enthusiast who wants to be held to a higher standard. But even my dalliances in Bayesian belief nets sound like high elf incantations to most testers. Mathematical disabilities, in general, make our craft prey to quackery and fraud of all kinds.

At the same time, I want to be inclusive. Mathophobes have a lot of offer and I don’t want them to think I don’t welcome them. But must they necessarily be the majority of testers? I guess I’m saying I want a cure for mathophobia, please.

Hexawise: What do you wish more developers, business analysts, and project managers understood about software testing?

James: I wish they understood that it benefits from specialization. When software people get heart disease they don’t limit themselves to a GP, do they? If they need surgery they don’t insist on being operated on by a rotating team of generic medical people who took a three day class in “Agile medicine” do they?

I get to be a specialist tester mainly because I do it on my own time. My clients are paying me, usually, for training, not for doing testing. So I usually test in my “free time” to sharpen my skills. I recently did have a wonderful and lucrative testing-related gig (because this particular project knew that it needed the best tester and analyst it could possibly get and became convinced I was the Chosen One), but those gigs are few and far between for someone like me.

Hexawise: When individual companies hire you for consulting engagements, how would you describe what it is that you usually seek to provide to them?

James: If it’s a training gig then my objective is to show them what testing can be, show them a path to get there, and encourage them to walk that path. A lot of that is about removing the obstacles to moving along. Chief among those obstacles is lack of confidence. So I do a lot of pep talking. Another obstacle is the very primitive, mechanical way that people think about testing. I have to replace that with systems thinking.

If it’s a testing gig then my objective is usually to provide deep, exemplary testing, that is transparent to my client. I want them to feel that they see their product in a beautiful focus. The danger I am always in when I test is that I will get too deep (and therefore be too slow and expensive). But for me, deep testing is the most fun, so it’s a constant struggle to hold myself back from using my most penetrating methods and tools.

Industry Observations / Industry Trends

Hexawise: As Artificial Intelligence increases in capability (for example the strides made by Watson) do you foresee an increase in the capabilities of computer checking? I am thinking, not of an elimination of the difference between human lead testing and what can be done without people but to what extent you see the possibility for AI to do a progressively much better job of checking.

James: I foresee a collapse of critical thinking about these complex systems, followed by some sort of disaster, followed by a new realization of the risks of surrendering human judgment to a machine. I foresee that this will be an ongoing cycle. This collapse of critical thinking will lead to more shallow testing and perfunctory checking, presented as if it were deep. For an example of what I mean, see this old computer commercial:

Pay attention starting at 2:05. Oh look, the computer is assuring us that it has no errors. Everyone relax! It’s “electronic brain” can be trusted!

Hexawise: Do you have any predictions about how large an impact Artificial Intelligence and Machine Learning will have on software testing in the next 5-10 years?

James: I don’t think it will have any impact on testing as such, except inasmuch as many people (not skeptical testers, but people who might otherwise hire testers) will trust black boxes when they should be challenging them.

I suppose as machine learning become more available to the masses, someone might try to train one to recognize bad output of some kind. That’s a sort of test tool. But it would only apply to well-established kinds of badness.

If you think about it, anti-spam systems are a sort of test system. Machine learning is used in spam filters. So, I guess testing is already using machine learning in that sense. But I don’t see the average tester applying machine learning methods to testing. I don’t see a developer doing that, either. It’s too involved and complicated; too narrow in application.

I hear that people at Google are going to “put coders out of business” with a system that writes code based on people just talking. You know what that’s called? A compiler. They are inventing a high level compiler. Now the people who talk will be called developers and will have to learn to talk properly, because it will emerge that normal people can’t say what they actually want.

Hexawise: Have you seen a particularly effective process where the software testing team was integrated into the feedback from a deployed software application (getting feedback from users on problems, exploring issues the software noted as possible bugs...)? What was so effective about that instance?

James: Not really. What I see is developers ignoring feedback. It’s too overwhelming. I suspect there are people who are really good at doing that. But I haven’t run across any.

One of the things that has happened with DevOps is a de-emphasis on testing and more of an emphasis on overall risk management. That’s a valid strategy, of course, but it has interesting blind spots. Whenever I hear a developer speak about wanting feedback from users I immediately think about how abusive and incompetent most users are about reporting problems. No, my developer friends, you really don’t want to read all those Internet comments on your software. You will be demoralized. But testers? We love reading that stuff. It’s our wheelhouse. We get clues and then we can reproduce the problems and make them sensible for the devs.

My brother, at eBay, with his testing mentality, loves going over the user feedback and bringing it to the teams there. But he will tell you it’s a constant struggle to get the attention of the dev teams.

attend a conference. Don’t bother to go to the talks, though. Most of the talks are full of fluff. Instead, find people and talk to them. Compare notes, make friends. Go to the testing lab.

Hexawise: Often one of the major roadblocks to software testers is their own management. Do you think this is a fair statement? Do you have suggestions for how testers can attempt to improve the situation. My background is strongly influenced by W. Edwards Deming so I have a tendency to look at the organization as a system and see room to improve the management system. It seems to me often the biggest gains are not possible if we keep departments separate (software development, software testing, marketing, customer service...). We can make improvements in software testing even if it is largely seen as separate from the organization but in doing so we miss much greater potential improvement.

James: The collapse of the test management industry is a terrible problem. It’s getting harder to find any kind of test manager out there. Do they even exist in Silicon Valley any more or have they all been hunted down by parasitic wasps who lay “scrum master” eggs in their living carcasses?

People who seem to know little about management or testing tell me that test managers are not needed. Okay, that means a whole lot of things that test managers do will not get done. This includes: providing a protected place for testers to work, free of harassment; negotiating for testability; negotiating for resources; assuring that schedules are reasonable; assuring that testing gets the respect it requires in order to attract and keep talented people; assuring that deadlines are met; explaining testing to management; assuring that testers are properly trained. When those things aren’t happening, testers tend to become more zombie-like and reactive (I’m not speaking of those fast zombies); or they become cheerleaders for the devs, instead of critical thinkers.

Staying Current / Learning

Hexawise: How do you stay current on improvements in software testing practices?

James: I’m not convinced there are improvements in testing practices in the absence of improvements in the thinking and social systems that drive practices-- and those things don’t improve much, as I’m sure you’ve noticed. Seems to me that the current nonsense in our craft is very similar to the old nonsense. Maybe some of the buzzwords have changed, but not much else.

The landscape of testing has definitely changed. Agile and Lean have aggressively colonized a lot of the testing space. Since most testers are young people (and test management has been eviscerated) they are easy pickings for the Agile Universalists (the people who think that we don’t need testers because we can just all test whenever we feel like it).

What this means is that testing remains rather primitive wherever I go (with a few interesting exceptions, driven inevitably by a single enlightened Elrond-like or Galadriel-like manager, who always seems to disappear off into the Grey Havens within a couple of years of me meeting him or her).

How I become aware of new and interesting ideas is through my community. For instance, a student told me about Karnaugh maps the other day and now I am trying to find a use for them.

Hexawise: How would you suggest testers stay current?

James: I don’t think currency is a thing in testing, except with respect to learning about certain emerging technologies and buzzwords.

The bigger thing in testing is to push us forward, which not enough testers are trying to do. Don’t worry about currency, worry about whether you truly understand testing, and keep working on that study.

Read widely about science. Get ideas from that. And play with the ideas. For instance, I read on Hacker News about 350,000 free images being released by the Metropolitan Museum of Art. I decided to experiment with turning that into a practical resource for test data. This led to playing with data wrangling and image analysis tools.

Also, attend a conference. Don’t bother to go to the talks, though. Most of the talks are full of fluff. Instead, find people and talk to them. Compare notes, make friends. Go to the testing lab. Or host a little conference. Invite testers to a small gathering where you can share experience reports.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf?

James:

Profile

James Bach has authored two books and consulted and presented on software testing worldwide. It is difficult to put into words how unique and insightful James is. In order to get a feel, we suggest listening to his presentations yourself and reading his excellent blog.

Some Career Highlights

Links

By: John Hunter and Justin Hunter on Mar 2, 2017

Categories: Interesting People , Software Testing, Testing Strategies, Interview, Testing Smarter with...