Archive for the ‘Process’ Category

I tried a couple of ways of doing “testing days” or really focused testing sessions. The common denominator might be that testing should be done really focused under a short time period. Administration and planning are done to a minimum and bug reports is often enough as documentation once the testing is done. The expectations are usually that a lot of issues will be found that the usual, often requirement focused testing with few testing opportunities, won’t find. Let’s leave this for now… what I want to write about is that I’m starting to find a way of doing this that I really like.

In my example I will tell you how I and one other person (a project leader) did one of these testing sessions. We tested an application, available for both Android and Iphone, which I knew almost nothing about. Let’s call the application “AppX”. What I will show you below is what we did but I changed it a bit to anonymise the application.

We spent 2 hours like this:

• 15 min test planning
• 1h 25 min testing
• 20 min test reporting

The goal was to test as much as possible and to find as many critical bugs as possible. All testing should be visualized as much as possible, in test logs, notes, pictures etc.

The result was 15 new bugs and this is what it looked like:

What we did?

The notation I used:

I really like this way of working. Why don’t you try it!?!

This is my backlog:

This is my “Done Bowl” (just emptied):

Sometimes if feels quite unstructured with all these sticky notes but there are some really great benefits

1. It’s easy to sort them into different groups and priority.

2. The really really urgent tasks I can put on my screen so I won’t miss it.

3. I don’t need my computer to write new tasks.

4. When I leave them on my desk I really do leave them so they won’t stress me.

5. They are really visual and the amount of tasks is really obvious as well.

6. Thanks to the “Done bowl” even the tasks that are done are really visible so they make me feel good and happy.

7. They make my life more colorful.

Don’t underestimate the power of sticky notes!

”I have a dream where people look beyond waterfall steps and agile manifests.
I have a dream where scripted testers and exploratory testers are living together in harmony.
I have a dream where testers stand up proudly and talk about their job and progress.”

From “The Supertester – A Slightly True Story”

The last couple of years, I’ve experienced two sides within the test world being formed. One side that is for scripted testing and ISTQB and then there are those who are against ISTQB and completely into exploratory testing. On test conferences and seminars I’ve heard test guru insult “the other side”. Those who are for ISTQB are narrow-minded academics that spends way too much time on statistics and administration, which is a completely waste of time and of no use at all. What is a certification about anyway? Is it really a proof of a skilled tester? The others, the ones that are into exploratory testing, they are fuzzy, unstructured and count on gut feeling way too much and they just laugh science in the face. But, who is right?

I don’t know who’s right but I know what I think. No one! Those who’s right, according to me, are the ones not taking sides, but sees the advantages of both approaches, pick the cherries and uses a little from both sides.

Below we can see a fairly typical picture for describing scripted testing and exploratory testing. The picture shows a scale where one can do more or less of the two ways of testing.

At previous workplaces of mine and also as a consultant I have had the privilege to see different organizations and projects and it is only at one single occasion that I’ve seen pure scripted testing. There where one person who was responsible for designing and writing new test cases but also updating them. The test team consisted of a couple of testers which only task was to execute test cases and report bugs. We ran a complete function test every week. It’s the same with the other side of the scale. I only experiences 100% exploratory testing once. We worked according to Scrum and all new development was tested with exploratory testing. The regression tests, on the other hand, were a kind of scripted test (the test steps were not that limited, especially when it came to test data). At all other projects and organizations I’ve been to, testing have been a mix of techniques and approaches. The test cases are completed parallel with execution, some test cases are very detailed and others invite creativity to take full advantage of the testers’ experience. Do I then feel that the tests have been better or worse at some particular place? The organizations that succeeded to pick the best parts out of several methods and found a harmonious mix, are the ones I think have had the best quality, the best control over their work and orderliness. They also had the most satisfied test team, which was proud about their work and gladly embarked on new bug hunting adventures every day. The places I feel didn’t work well at all, are those where management had to nag about some particular method/process every week and they had to force the team to use this process or some fancy tool, but all these stuff didn’t fit the products worked on or the personality of the team members. These organizations spent a lot of energy discussing why/why not to use this particular method and this steals a lot of focus from the day to day work and no one gets any peace or inspiration to perform some really good work.

Another aspect that I have been thinking about is the testers’ personality. Not everyone is suitable as exploratory testers. I think most people agree that they don’t, but is it then right to limit the title “Tester” to solely creative and imaginative persons? No, I don’t think we should do that. We are all different and that’s something we should take as much advantage of as possible. There are people that love documentation, accuracy and that manages, well even enjoy, repetitive tasks and possesses the ability to keep the same focus and commitment the first time they run a test cases as the fiftieth time. Both personalities are worth a lot so why not use them both.

Just think of Thorkil Sonne and his “Testspecialisterne” (test consultants diagnosed with autism). Here it just doesn’t work with exploratory testing. So, don’t they do a great job then? Sure they do! In the right situation, with suitable tasks, they do hundred times better work than most of us should have done. They are needed. But everyone can’t be like them, then a lot of test scenarios would be missed, where exploratory testing might be the only way to find them. One part of testing is about predicting situations and then imagination and exploratory mindset is very valuable. Without that our tests would be limited to requirements and documentation and then we probably would miss serious bugs and issues.

No, both sides are needed! Stop fighting and do something really good together!

” All those easily charmed project managers who try all new cool methods and tend to forget everything they ever learnt in school or in previous projects. All scrum fan boys who read the agile manifest saying that no one ever more needs to document or follow a boring process! All administration nerds that documents for months and months in their  waterfall projects with all the dead fish floating down the waterfall with them and all those theoretical brain heads that memorized all the answers to the ISTQB-certification. And still no one has a clue.”

From “The Supertester – A Slightly True Story” by Linda & Anna Hoff

Is agile better than waterfall? Or is waterfall better than agile?

Well, it depends!

It depends on your organisation, your staff, your product, your previous methods and processes. If you are happy with your waterfall projects today, why change? If you already are efficient and have high quality, why change? Don’t try to squeeze Scrum into your organization just because you think it is cool and the only way of having fun at the same time as you are really efficient.

And before you dismiss Scrum and Agile be sure you really tried the true Scrum/agile. The Supertester has heard people say things like “We have adopted Scrum to our organisation by taking parts of Scrum. But Scrum is really frustrating and it doesn’t work for us testers.” How can this person know that Scrum doesn’t work when he/she haven’t tried Scrum the way it should be?!

There are pros and cons with both agile and waterfall. Please be open minded and try to see the whole picture!

The supertester couldn’t agree more:

Lisa Crispin on YouTube, Agile vs Waterfall

We all seen this picture:

… and we all know this is the truth and cruel reality. Or is it?

I’ve heard about companies that are concentrating on fixing bugs really fast when they are discovered by the end user, instead of trying to find them before production. How on earth could that save time and money?

Perhaps there are some products/system that are really easy to release. Then it might be possible. But what about the customer satisfaction? If I was the user and I found a lot of really serious bugs in my product I would be really angry, even if I got a fix just a couple of days later.

The company is relying on the end users to report the bugs and updating to a new version.

But are there any other better to test a product/system than the end users!?! They know what is really important and what is not. But still… how do you manage to keep the end users happy? I can only think of one situation where end users do the job and are still happy, and that is the game industry where there often are a lot of end users doing test on beta versions.

Is this a new trend? Will all testers be out of work? No, I don’t think so. There are a couple of products that this never will work on. For example, pacemaker. You can’t rely on the end user to report any bugs. They’ll never get the chance!