Purpose, Audience, Context and Testing - PACT

Calling Time

2009-07-06T06:34:00.000-07:00

I've decided to call it a day with the PACT blog on blogger.

My new blog address is:

http://thesocialtester.posterous.com/

Hope you can all follow me over there. As you may be able to tell from the title I'm moving more towards communication and testing through social tools and social aspects. It's an interesting arena and I'm hoping to partner up with a few other like minded testers to bring some interesting topics, stories and ideas.

I do hope you will join me

Thank you all for following and reading this blog.
Rob Lambert

It’s a social revolution

2009-07-02T09:22:00.001-07:00

It's clear that the future of software testing is changing and those not ready to embrace this new software testing evolution could find themselves out in the cold.

The revolution I'm proposing is already taking place in society and it's essential the testing community embrace this change in order to remain relevant, topical and trusted as we head fast in to the 21st Century. The social change I'm talking about is social media and collaboration.

Our society today is becoming increasingly technology savvy, are communicating in ways not envisaged even a decade ago and are collaborating on everything from writing novels to online gaming. Social media is here to stay and with the next generation entering the work force it's essential we make software testing relevant, modern and trusted.

At the dawn of the 20th century the newspaper was king. Then came radio enabling mass communication directly to people’s homes. Then came cinema with news reels and feature films. And then came the ubiquitous TV. And in just a few decades we moved from paper based communication system to electronic mass media. Advances in technology made these mediums cheaper and even more accessible opening them up to even more people. Sure, that was a whirlwind tour of mass media and no doubt I’ve missed a few out, but I’m building a picture in your mind, not a historical document.

We then created more interactive ways of obtaining information through these mediums. We invented digital TV, car stereos with news and travel searches, the ability to record and pause live TV, Teletext and satellite TV.

We also created the two most important social media technologies; the Internet and mobile phones.

But the truly revolutionary part of this communication change is the social aspect of how we now obtain our information.

In the early days of newspapers, radio and TV we had to wait for our information. We tuned in or bought our papers at set times. In between these intervals we received little to no information. With new social and mass media we can now access information anywhere and at any time, but more importantly, from almost anyone.

Technology is enabling society to move from a push system to a pull system. Or did society influence technology? We now obtain information when we want to. We pull the information to us. We no longer wait to be informed.

Phones are now smarter than ever, broadband is now commonplace in households, Wi-Fi allows for Internet access on the go and we can now watch the TV we want on demand. As society embraced the Internet and connectivity; social media was truly born.

We now have Facebook, Bebo, Twitter, blogs, RSS, YouTube and wikis. We have really smart mobile phones that have full Internet access through intuitive UIs. And we now have cloud computing, networking and conferencing tools. We can work or play from anywhere we have an Internet connection or mobile signal.

In just a couple of decades society has changed.

Where society and technology goes from now is unknown but we can be pretty sure that it will be Internet and communication based. We can also be sure that social media will play an ever increasing part in our roles, either directly or indirectly.

Here are some reasons why social media is so important to the software testing industry.

Reason 1 - Point Blank Testing

The software under test will be built on or around social media and it's principles. We will be testing applications that adopt, integrate or support social media tools. We will be interfacing to instant chat, video streaming, Facebook, Bebo, YouTube, MySpace, DiggIt, Delicious, Blogs, RSS, Yammer etc or new concepts not yet invented. We will be testing future collaboration channels and tools as well as increasing our support for browsers, mobile phone access and wireless connectivity.

Even critical systems such as financial and accountancy software is bringing departments together and aiding collaboration through social media principles and plug ins. Almost every industry on the planet is realising that improved internal and external communications are not only cost effective but essential to maintain market share, loyal customers and outstanding employees.

Every week a new social media site or concept is launched. Every week we hear of project success stories made possible through improved communication and collaboration.

The future is communication and collaboration. So we need to know how to test it and how to make it work for us and our teams.

Reason 2 - Focused Collaboration

This is split down in to 2 further categories:

2.1 - Narrow focus collaboration

The shift in social media in society also needs to be reproduced within the working environment. Our teams are increasingly becoming spread out throughout the country, maybe even the world but they are still our teams and we still need to communicate.

Working from home is now a common practice in many IT companies. We can now connect via our mobiles and laptops whilst travelling and all of this is only possible through the cultural shift towards social interaction. So we need to build internal systems of communication that cater for these changes.

Our workforces will expect to be consulted and engaged in company policies and decisions, so wikis, e-democracy and social engagement tools will become increasingly more important. Old fashioned intranets don’t work anymore.

2.2 - Wide focus collaboration

We will increasingly need to connect and collaborate with wider testing and development communities. We will need to learn and seek out new testing ideas. And where better than the large pool of testers available via social media?

We will be working on collaborative books, documents and articles and maybe even collaboratively testing software.

In times of economic uncertainty we will need to formulate new ways of engaging with our customers, generating business and maintaining market leadership. Social media is already helping teams with this.

Reason 3 - Society is changing and the digital native are on the way

If we don't embrace communications and social media, we will struggle to be relevant to the work force of tomorrow. The digital native (is a person for whom digital technologies already existed when they were born) are entering our work places this year (estimated) and they didn’t grow up with a video recorder, they use google rather than the local library (or google in the library), they watch the tv shows they want to watch and on demand, they have digital music collections. They can access the Internet on their mobile phone and ‘chat’ with someone anywhere in the world – and it is second nature to them. And these digital natives will soon be entering the work force and expecting the same technology, communication and mindsets at work as they grew up with.

But the really frightening part is that it's not just the digital native who are expecting these changes. It’s the current workforce who are embracing these changes now and are actively working on bringing software testing in line with modern thinking. (think of the emergence of exploratory testing, automated acceptance driven testing, agile testing). It's this current generation that will help to kick start the revolution in testing. They are sowing the seeds now and preparing the ground work for the testers of the future.

Reason 4 - Intelligent environments are changing..fast

The environments we find our applications being used in is changing extremely fast. The hardware being used is changing at a pace that many companies struggle to keep up with, so how do we ensure reliability, accuracy and security when we are refactoring for new hardware monthly?

Our applications are needing to support more browsers, more operating systems and more contexts than ever before, but what happens when the surrounding environment gets thrown in the mix? Cars, tables, chairs, streets, public benches, bus shelters, trainers, sunglasses, sat navs, buildings, advertising boards, household appliances, gym equipment and so forth. What happens when in a single walk down a street in any given city our applications could be interacting with 100 different devices and apps?

Will we have personal servers at home that monitor our health, our electricity, our diet, mood and behaviors? How do we test those systems? How does the user interact with these systems and what are the security implications?

But why is this revolutionary?

It’s revolutionary because as a testing community we need to embrace these social evolutions and accept that times are changing. If we don’t , we’ll fail to remain relevant when the digital native hit our work places. And here is why they are different:

This generation of workers no longer require an organisation to certify they are capable of doing the job they are already doing. They just do it.

This generation no longer need or are sold big vendor testing and process tools. They just use the right ones at the right time and if it a tool doesn't exist, they create it. And then open source it and share it. Collaboration is key. (think Selenium, Fitnesse, Ruby and Python test tools)
This generation wants heads up testing displays and efficiency tools built in to their browsers. (Xpather, Firebug etc)
This generation want virtual test environments that can be built and deployed to in seconds complete with snapshots and known states. They want continuous integration and other instant quality feedback channels.
This generation want to collaborate closely and share information with like minded team members to achieve excellent and rapid software delivery. Their customers expect this too.
This generation are flexible, dynamic, creative, intuitive, connected, open and honest (think Facebook, Twitter, Friendfeed, Bebo, blogs, YouTube etc)
This generation don't follow convoluted and incredibly detailed test cases. This generation place a preference on current and changing information, iterative cycles of learning, critical thinking and human judgment.
This generation value automation as an extension of their testing toolbox, not a replacement for manual testing.
This generation have little time for inefficiencies, archaic processes and poor communication.
They have little company loyalty and actively seek interesting, fun and rewarding companies to work for.
This generation promote people over rigid process, working software over reams of documentation and creativity over uninspiring and monotonous list ticking. (think exploratory testing, low tech dashboards, agile)
This generation collaborate online, meet with like minded testers outside of work and constantly strive to improve their understanding and knowledge of testing.
This generation expect pressure, engagement and involvement in the software process. They are no longer content with waiting months to test and then spending more time performing test case admin tasks than testing. Their programming counterparts value rapid feedback early too.
This generation want to drive out the defect in the design and implementation rather than catch it in test and gloat about high defect counts.
They no longer want to write documents that no one reads. They value context when deciding the detail of their test cases. They place the emphasis on the tester and not on the script. They encourage learning. They don't just do something because it's the norm. They create their own way of working.
This generation want to engage in and take responsibility for their companies environment and prospects. They want to have their say. They want social intranet systems.
This generations customers expect full transparency and more engagement. The element of trust in software and the development process to deliver is disappearing. Could this explain the popularity of agile?
This generation expect increased mobility and flexibility in the work place. They want to be able to work from home. They expect to be able to work whilst travelling.

And this generation are here now; in the work place and doing great things and actively dragging software testing in to the 21^st century.

They are adopting agile processes, promoting testing, standing up for quality and bringing real and beneficial results to their customers. A revolution in software testing is happening and the future is exciting.

Now the question is, are you ready for the revolution?

http://www.softwaretestingclub.com/group/testingrevolutionuksouth

Some pretty pictures, part 2

2009-07-01T01:42:00.001-07:00

I got some quite heated responses from people who apparently work in traditional methodologies and see nothing wrong with the process. That is cool.

I never said waterfall was bad. I just said in my experience it doesn't fit 'me' as a tester.

The thing is, that post was intended to raise some discussions as it clearly shows the divide in the testing world between those who run traditional and those who run agile.

Of the email responses I got from the last post I asked each respondents whether they spent a lot of time updating test cases and metrics. And they all said yes, but it wasn’t a bad thing...

I also asked whether the software was well received and what the customer wanted. Mostly they said yes, but when pushed further and asked - ‘did you have another release of the same software later with some fundamental changes made?’ and the general response was yes.

What is clear to me though is that traditional versus agile is a pointless argument.

Agile is simply common sense. It’s about getting timely and regular feedback on work done. This means you get visibility of problems, deviations and suitability early. Agile is also about team work. Not programmers versus test, but a team as a whole. A whole unified development team, delivering good quality software. It’s common sense.

The scary thing is that most of the principles of agile can be applied to traditional projects too. Regular feedback, quality and test driven teams, regular drops, CI and many other aspects. Does it mean you are agile? Who cares? If you are working well as a team, showing your customer how you are progressing and delivering a great product fit for purpose, then you are probably doing just fine, whether that be agile, traditional, RUP, yada yada etc.

Myself though, well, I prefer to be agile. And if you truly want to know what being an agile tester means then you should absolutely read Lisa Crispin and Janet Gregory's book 'Agile Testing'. It is awesome.

Some pretty pictures, part 1

2009-06-30T09:00:00.001-07:00

I've been feeling kind of annoyed with testing recently. I've been like this since I attended SIGIST this month and saw first hand just how few testers really know the world around them is changing.

It's actually a little unfair of me. In fact, a lot unfair of me to say that because I too was one of those testers a few years ago. Not through ignorance, but through simply not really having ever experienced anything different. And I reckon these testers I saw are in the same place.

I can see quite clearly when I look back what 2 things changed my outlook on testing.

1. Social Media brought me in touch with people far outside my normal circle of contacts.

2. Agile principles, when applied in context, re-ignited my passion in testing.

So I sat down late last night in the balmy summer heat and came up with the following diagram. It explains, at least to me, all of the things I've come to realise I just 'accepted' in a traditional project. I knew nothing else. I had no outside communication with other testers.

I simply thought this was the norm. And I was low, depressed and looking for a new career. And I KNOW there are other testers still stuck in that rut. I've spoken to them. And this following diagram explains to me why I was in that position. It was the testing environment. Is it the same for you?

The revolution went well

2009-06-18T14:16:00.000-07:00

We kicked off the testing revolution in style with special guest Michael Bolton at the Washington pub in Primrose Hill.

We were also surprised by Fiona Charles (another testing guru) turning up - which was great stuff. I didn't get much chance to speak to her which was a shame, but hopefully our paths will cross again some time soon.

We were joined also by Stephen Allot (from BCs), Anand Ramdeo (www.testinggeek.com) (hope I got your name spelt correctly), Mark Crowther (NMQA and speaker at SIGIST), my boss Ant Gardiner and Dorothy Graham (consultant and organiser of Eurostar). It was a real shame other members of the group couldn't make it.

The night was fairly informal to be honest. We swapped some stories and chatted about testing. As expected, the conversations were illuminating with Michael present. We then sat the pub quiz, which we didn't do too badly on. We also saw most of the cast of Hollyoaks too who were round the other side of the bar. Michael did some card tricks too which were pretty incredible followed by more stories of testing. It was a really fun night and incredibly relaxed.

There was no planned agenda so we let the conversation flow about anything we liked. What it has done is re-enforced where I believe the focus of the revolution needs to be. I'll reveal more within the next few weeks - but you can be sure it will interesting and incredibly relevant stuff.

We are hoping to have a meeting in/around Winchester within the next few weeks to kick things off with an agenda and put some milestones / roadmarking down for where we want to take these meetings. We'd also like to get to know the regular attendees of the event.

Here are some photos from this weeks meeting in London.

http://www.flickr.com/photos/39583741@N06/with/3636027973/

Hope to see you at the next meeting - keep checking LinkedIn and the Software Testing Club groups for more.

Rob..

The importance of testing

2009-06-11T07:27:00.001-07:00

I’ve been quiet on the blog and tweeting (@roblambert_PACT) front recently, but that is because I’ve been busy at work (www.imeta.co.uk) with releases, my wife has started back at work (more childcare by me), have been indulging in my hobby of social media and the weather has been very nice indeed.

One of the side effects of nice weather is that I tend to sit in the garden with a beer and read books. My current book, which is nearing the end, is ‘The Importance Of Living’ by Lin Yutang. The book was written in around 1937 and is a philosophy of life. It is not directly related to testing and I’m reading it as I have an intense fascination with Chinese literature and philosophy.

Yet I got to a chapter in the book about learning and education and certain parts of it seems to be a perfect fit for many of the discussions I’ve seen recently on forums such as The Software Testing Club, Test Republic and SQA forums. The topics I’m talking about are to do with certification (ISTQB/ISEB) and learning. From my observations, I reckon there are three camps of thinking when it comes to certification and learning.

Camp 1 tend to HATE certification and place more value on learning, experience and passion.

Camp 2 value learning, experience and passion but appreciate that certification is a necessary evil and that some certifications can be worthwhile, if only for making you more employable.

Camp 3 are people who generally believe certification will make them professional testers overnight and that without certification, they cannot call themselves a tester.

I am firmly in Camp 2. I do despise some certifications in testing, but I appreciate that many people need them to get jobs. If you need it – do it. Sure, it may be undermining the value of learning and the art that we call testing. But putting a roof over your head and maintaining/getting a job is of higher importance. There is no way I would promote a current certification in testing, but I appreciate why people want them and obtain them. I often worry that if I don’t get the latest certifications that I am falling behind. Then I listen/read/speak to some testers with every certification under the sun and realise quickly that certifications don’t always work.

In ‘The Importance of Living’ Lin Yutang is writing about the school system as he saw it and bear in mind, he was writing this in the 1930’s. What he is writing can be directly related to certification.

“We have this system (placing more value on measured, passive cramming, mechanical and uniform assessments) because we are educating people in masses, as if in a factory, and anything which happens inside a factory must go by a dead and mechanical system. In order to protect its name and standardize its products, a school must certify them with diplomas. With diplomas, then, comes the necessity of grading, and with the necessity of grading come school marks, and in order to have school marks, there must be recitations, examinations, and tests. The whole thing forms an entirely logical sequence and there is no escape from it”

So true. Although this is directly about school systems it can be applied to certification and where it is moving. Especially when you consider we can now ‘buy’ agile certification and be classed as a certified agile practitioner after just 2 days of study. (Scary when you think real world, practicing, long term agile practitioners are still finding their feet, fine tuning the process, defining concepts, altering the theory and growing the concept)

Lin then moves on to say:

“But the consequences of having mechanical examinations and tests are more fatal than we imagine. For it immediately throws the emphasis on memorization of facts rather than on the development of taste or judgment”

I was not very good at biology at school, my coursework proved that so I had to work extra hard in preparation for the exam. The problem was I didn’t enjoy learning about biology so I left my revision to the two days before the exam. I got 100% in my G.C.S.E exam. 100% – just from memorizing facts. Yet I was fundamentally poor at the subject. I just had an amazing capacity to remember things. I wish I still had that……….forgot what I was writing about now…..

“The danger of prescribing a course of compulsory studies is that it implies that a man who has gone through the prescribed course ipso facto knows all there is to know for an educated man” - This is very obvious on some of the forums where testers recite certification material when responding to a real world business problem. But you should be doing X instead of Y when in reality X is not possible due to external factors not covered by certifications.

Lin makes many other very insightful remarks in this chapter, all of which paint a very dark future for examination and education. And yet he wrote this in the early 20 century. And I lived through the same education nightmare he predicted. I crammed, sat exams, passed some, forgot the subjects and moved on to something new.

But my point is, I only ever really retained the subjects and topics I was passionate about. My history knowledge is still good but I still study and read about history. I can still speak, read and write Russian. A topic I loved at school and college. I also still remember almost everything from my communications and media degree because I have a passion and interest in these topics. Learning these subjects was a pleasure, not a chore.

Lin sums up the problem with something that will certainly appeal to those in camps 1 and 2:

“Instead of the measured, uniform and passive cramming of information, we have to place this ideal of a positive, growing individual pleasure [in learning]”

Unfortunately though, I’m intelligent enough to realise that there are companies and recruitment agencies that can’t or won’t spend time/money interviewing candidates who actually are passionate, intelligent, good at communicating and excellent testers unless their resume shows that all important certification during the interview selection process. Certification checking for testers is a quick way of whittling out CVs based on a false belief testing can be measured by certifications alone.

It’s a shame really because I think that some certification courses in testing are a good introduction to testing. They offer some guidance and information that is a good introduction to what we do and some of the techniques used (albeit nothing that cannot be gained from reading blogs, articles and forums online, for free) .

But to gauge a testers competence on *just* the certifications they hold is a dangerous act indeed.

I love it when a plan comes together

2009-05-27T12:30:00.000-07:00

I've been tied up with lots of work recently so have neglected this blog site.

I've been busy sorting out the first Test Revolutions Meeting:
http://blogs.imeta.co.uk/RLambert/archive/2009/05/19/the-testing-revolutions-first-meeting-london-tuesday-16th.aspx

It is set to be a great night. No real agenda as of now, but Michael Bolton will be joining us and I'm sure we will chat testing whilst drinking real ale.

I've also been working on the latest edition of GoodTester (I know it is late but I'm on it).

I was also involved in a forum post-war with a fellow tester in the US who was intent on telling everyone that his way of working was a best practice. The best practice in question was that of Test Plans.

To me, a test plan is a contencious artefact that most of us write and distribute at some time in our career. Most of us produce a plan of some form and may not actually be aware of it.

In my world, a test plan is just a note on how the tester plans to test. This could be a project level plan. This could be a one day plan in an email. Either way - it is a plan of some sort. A plan that is intended to communicate something, to someone.

The tester I was 'arguing' with had a template (not too dissimalr to the ISTQB/ISEB template to be honest) which he was positive was all a tester needed to use to create the perfect plan.

It was a word document. It had about 20 sections that needed to be completed. He argued that even if that section couldn't be completed the tester should leave it in there.

To me, it is this very type of documentation that kills the testers creativity, that destroys the rapid feedback and more importantly, wastes masses of time.

I've spent weeks writing test plans where I include comprehensive and detailed entry and exit criteria, where I specify training needs and resources, test environments, plans, strategies, how I'm going to manage the testing and everythinng else you can shake a stick at. I've submitted these for review - they've been approved. Then in the first iteration every single one of the statements has either been broken, bent or destroyed.

And this is not down to me not controlling the testing. These things are down to the usual project constraints that all testers are very good at complaining about. Poor quality software being dropped, software not doing what it should do, budget cuts, sickness and unplanned holiday, time delays, time being cut yada yada yada.

It happens. All the time.

So why are we writing test plans that people don't read, that mean very little three phases on in the project and that take so long to write? And why would we even remotely believe that a test plan I write can be used as a best practice for Joe Bloggs in a different IT industry with a different team and in a different part of the world????

So back to the point. Isn't it about time we started looking at Purpose, Audience and Context? And just started writing a plan that was 'enough' to get the job done? A plan that hit the mark perfectly and didn't simply follow a 'best practice' that people have been doing since testing began?

Purpose
What is the test plan aiming to do. To influence? To persuade? To inform? To describe and communicate? To guide other testers? To guide yourself?

Audience
Who is actually going to read it? What are their expectations? If no-one is going to read it, why write it?

Context
In what context are you working. Agile? Waterfall? Iterative?
There is no reason why you cannot write a short plan for each iteration. Maybe a short plan for each sprint in an agile project. Maybe a base plan for obvious items such as automation strategy etc for a waterfall, but a more specific 'iteration' / 'build' level plan to describe a more detailed plan at that time, with the information you have available?

Even in the best performing long term waterfall projects there is absolutely no way a plan written in January will still be relevant in December unless it is just the basics.

But that is the point I am trying to make. Write what you need to, when you need to, for who you need to and nothing more and nothing less. That will be the very best test plan you can produce. Writing one that covers months in the future and you will have a static, inflexible and out of date test plan.

The plan could be an email, a wiki page, a document, a tweet, a blog or simply a scrap of paper. If you think about (and get right) the purpose, audience and context then your plan will be the plan that is right for you.

Go forth and plan.

Is this the beginning of the revolution?

2009-05-18T11:07:00.000-07:00

The Testing Revolutions first meeting - London - Tuesday 16th June 2009.

In a second attempt to kick start the revolution we will be holding the first meeting in London.

The first meeting will also have a very special guest. Michael Bolton will be with us to chat about testing, to share stories and to drink real ale.

We are meeting at the The Washington, Primrose Hill (50 Englands Lane, NW3 4UE) in London at 7:30pm - 8:00pm on Tuesday 16th June 2009.

The first meeting will really be a gathering of people to chat about testing and share stories, but we will also have a look at where we take the meetings moving forward and a rough estimate of who is involved.

(Regular meetings will be held more towards Winchester in Hampshire)

Even if you don't want to join the group but do want to meet Michael and talk about testing with like minded people then please do join us. Did I mention we would be drinking real ale too?

The following day Michael will also be at SIGIST presenting. Going to be a great couple of days talking testing.

Link to the pub, here: http://www.fancyapint.com/pubs/pub1436.html
Link to Revolution Group on Software Testing Club, here:http://www.softwaretestingclub.com/group/testingrevolutionuksouth
Link to Revolution Group on LinkedIn, here: http://www.linkedin.com/groups?gid=1856424&trk=hb_side_g
Link to Michael site here: www.developsense.com
Link to SIGIST here: http://www.bcs.org/server.php?show=nav.9262

What's it all about?

2009-05-06T08:28:00.001-07:00

Ok, so I've had a few people asking me what PACT is all about. To be honest, I'm not entirely sure of an exact definition. It's a blog; a space for me to lay down some ideas and thoughts. It's also out there in the public domain for people to read, enjoy, critique, learn from (hopefully) and contribute to.

It's therapy. It let's me get stuff off my chest. Out of my head. It gives me a forum for my thoughts.

Yet I feel I need to define PACT further.

Well, let's start at the top.

I'm a communications graduate and very firmly believe that communication underpins everything we do. There's no escaping it. It's around us all of the time.

I also know that the best people I have ever worked with are people who can communicate effectively. I've worked with very competent people who lack communication skills meaning working with them is tedious, difficult and boring. I've worked with people who aren't amazingly competent but make up for that by being able to communicate well. And when push comes to shove I know who I would rather have working with me and my customers.

So what is PACT? Well PACT standards for Purpose, Audience and Context Testing. It's just a series of words. It needs some more work and it needs some more fine tuning, but for now PACT is what I have chosen.

PACT is about acknowledging the three Core elements of communication.

Purpose - why are we communicating?
Audience - who are we communicating to?
Context - under what circumstances will the communication be delivered or received?

I was watching a political debate the other day. Not something I normally watch. I find political debates dull, uninspiring and irrelevant. But in this instance the debate was getting heated and the three politicians involved were starting to feel the pressure.

I continued to watch and noted the following:

When under pressure effective communication becomes even more important
When communicating effectively you can 'pursuade' or 'sell' something with ease (you can also avoid difficult questions)
With excellent communication comes trust
Poor communication undermines the message you are sending
Good communication skills means listening as well as speaking

In politics, trust is everything. How many Prime Ministers, Presidents and other political leaders have wowed a crowd with an amazing speech which, when deconstructured, actually has little or no real content, which is full of lies of exaggerations? Only then to be elected or given power where they then consistently fail to deliver so many promises?

Live debates show communication (good and bad) at its most intense. This is the type of communication we can all learn from.

Pitching the bug report at the right level to get bugs fixed, reporting back to stakeholders on the quality of the build, liasing with other departments for information and resource, convincing our managers that we need X to get the job done. They are all sales pitches. They all require us to pursuade the right people to give us something or to do something or to acknowledge something.

Purpose, Audience and Context Testing is a blog exploring the core elements relating it to testing, looking further at how you test, what you produce and how you work with your colleagues and customers.

The software under test, the end user, your documentation, your release notes, your test cases, you bug report, your standups, your meetings, your exploratory sessions, your colleagues, your management, your specs and your test metrics are all affected by Purpose, Audience and Context. Mastering these three core elements comes from understanding them, acknowledging them, observing them and appreciating them. Only then can you be sure your focus is in the right place, at the right time and with the right intent.

"Quality is value to some person" - damn right, but who is this person? What is their purpose? What context are they working in?

I'm hoping to spend some more time working on PACT over the next few months and include some real world examples I've received from you, the readers.

It seems that when people break down what they do day to day, it boils down to communication. Yet many don't realise or don't see the underlying communication that takes place. Only when they do appreciate it and analyse it, do they see where they can improve or what they are doing right/wrong. But then why should they see it? Very few people have formal training or education in communication.

And in some way I am hoping that this blog will help testers understand the importance of communication in what we do and acknowledge that effective communication can go a long way in making us all 'better' testers.

Rob..

Rapid Software Testing - Filling Fast

2009-04-24T07:39:00.000-07:00

Michael Bolton's course is attracting lots of attendees so don't delay and miss out. It's a great chance to sit the course and meet Michael.

June 2008. Southampton. UK.

http://www.imeta.co.uk/momentum.aspx

Enjoy.

Rob..

There are bugs I like and bugs I don't like

2009-04-23T13:43:00.000-07:00

I've been pretty quiet recently on the blog and tweet front. And that is because I have been getting stuck in to some testing. Lots of it, with not much time. It's the best testing in my opinion.

Time pressured testing forces you to use your best risk based approaches and keeps you on the edge. For me personally, it brings out the exploratory (and best) side of me. Headphones on, charter in hand and I'm off.

However, one thing struck me the other day. And that is I actually hate finding some defects.

The defects I hate finding are:

* Basic Validation Errors

Should I really have to test for these? Should I have to perform boundary value analysis on a number field that should only accept 1-50 as a whole positive integer? Do I really need to test all 100 of these fields on an app? GRRR. - Where's my automation tool?

* Implementations that don't meet design or spec standards

For example a new web page that so blatantly and clearly does not match any other web page within that site. I'm talking different colour scheme, different button/link naming conventions etc.

* Defects that stop obvious elephant path tests

These are by far the worst. The elephant path is the heavy weight, standard and well trodden path through the system. Yet on your first smoke test you cannot reach the end due to a stopper defect.

* Defects that have ocurred due to not considering the wider picture

These are annoying because somewhere along the line something has been designed or decided without the right input from a key member of the team. You know the ones, "..but what happens when I plug in X?" Blank faces.

* Defects that are only reproducible with a massive data set or test setup

You have spent hours crafting the data, restoring the DBs or deploying the VMs and BANG. Error on first use. And you know that to reproduce and to retest this issue is going to be the same long and convoluted process. You also know that if on retest it still fails, you will be going postal and throwing hardware around.

For all of the defects I hate finding though, there are defects that I do enjoy finding. The defects I love finding are:

* Defects that require time, patience and effort to track down.

The ones that require all of my detective skills. I love these ones. They take every little bit of care, thought and consideration to track down. They require you to pull together every possible weapon in your arsenal to outroot that bug.

* Defects that are obvious, but only to a tester (i.e. entering valid but unexpected text to a field)

These are superb bugs for testers. You know the ones where the designer or dev turns round and says 'why would you do that?' 'why would anyone enter that?'.

* Defects that are caused by 'trekking'

Trekking is something I have called (might have read it and nicked it from somewhere but can't place it) to the process of trekking through your app. The bugs generated by these treks are superb finds as they take a bit of luck and keen wits and superb observational skills.

These types of bugs are the ones customers can sometimes come across but are never reproducible. The ones where you follow the reproduction steps and nothing happens. Most of the time, in my experience, these bugs are related to steps much earlier in the process, that are not recorded, recognised or included in the steps to reproduce.

I had one the other day: Log on > Open Window X > Log Off > Log On > Open Window X > Open Window Z - BANG. There is no way you could ever anticipate that and write a test case for it.

It's not that I don't like raising defects or finding them. It is the extra (and often unnecessary) work that goes on because of the bug that makes me hate finding them. The raising, triaging, fixing, resolving, retesting and verifying when often some careful thought and consideration early in the design or implementation or early involvement of the tester, could save a whole load of time.

At the end of the day though, whether I like finding them or not. I do find them. I do raise them and we do have to retest them.

Rob..

You suck

2009-04-17T10:53:00.000-07:00

You have a meeting where someone says something that you don't agree with. Maybe you don't believe it. Maybe you know your idea is much better. So how do you tell that person that what they have said is wrong or stupid or maybe just not quite as good as yours?

Maybe you need to explain to people that what they have been doing is wrong and that we need to make some changes. How would you go about doing that?

I once knew a test manager who would literally say "you're wrong" or "what a stupid idea" or "don't be so stupid" or "you've been doing it all wrong" or "You suck". He wasn't very popular. At all.

I also knew a manager who would never say boo to a ghost. He wasn't very popular either.

In fact, looking back through my experience and researching the topic online and reading around this subject I've realised that what my communications lecturer said all along was right (and there could be someone else to reference here too). There's only really one effective way of communicating 'bad news' (when I say bad news I mean negative news, not to be confused with real bad news that doctors and policemen sometimes deliver) and that is in the order of:

Good News > Bad News > Good News

or

Positive > Negative > Positive

Delivering what you need to say in this way isn't quite so harsh as simply delivering bad news.

Let's look at why:

The first element of good news gives confidence to the person that what they have been doing / saying is in some way valid and worthwhile.

The bad news is then the real criticism or suggestion. It's sandwiched in there so that it actually loses some of the harshness and critical direction but not effectiveness.

The final good news element adds extra confidence, makes the negative news less upsetting / controversial and also leaves the comment / conversation on a good note. If delivered well, it can also re-enforce the negative message.

Here's some examples:

[Good news] I think the sprint we have just done has delivered some great functionality and we worked very well as a team, [Bad News] however we need to change the way we write user stories as they are too large and we are not delivering what our customer expects. [Good News]This will then enable us to increase our velocity and start to deliver quality software.

[Good News] The testing you have been doing has uncovered some very important defects, [Bad News] however I believe you need to start looking at more negative tests. [Good News] This will ensure that the customer gets the best experience when they start entering invalid / confused or illogical data in to the app. This will add lots more quality.

[Good News] You've contributed to some process improvement and found some important bugs, [Bad News] however you are probably the worst tester this company has ever seen. [Good News] Saying that though, we are going to work closely together so you can see how you can improve, develop and become a good software tester.

I find delivering the bad news in this way to be amazingly effective and see this behavior in the people i work with or have worked with who I believe are great communicators, yet at the same time, great motivators.

Being able to respect peoples feelings yet still deliver criticism which inspires and instills action in people is a difficult task. Maybe this technique will go some way to helping when you need to deliver negative feedback.

Please let me know how you deliver criticism or negative feedback. Do you just go for the throat, adopt a slowly slowly catchy monkey approach or do any other technique? I'd be interested to know.

Rob..

Rapid Software Testing in the UK

2009-04-14T04:47:00.001-07:00

I’m very much looking forward to June 8th 2009. The reason being is that Michael Bolton (www.developsense.com) is here in the UK presenting the Rapid Software Testing course.

The course is going to be three days long and is being held at Solent University, Southampton, UK. A great location with excellent road, rail and airport access.

More information can be found here:

http://www.imeta.co.uk/momentum.aspx

A great chance to meet Michael and sit the legendary Rapid Software Testing course.

Enjoy.

Rob..

Quality Control - well sort of

2009-04-12T03:42:00.000-07:00

I've been thinking a lot about quality recently. Mainly inspired by reading Zen and the Art of Motorcycle maintenance, a book that could well have been written directly for the software industry.

But I’ve also been paying a lot of attention to the word 'quality' recently in other areas of life. So bear with me on this article - I feel something bubbling in my mind and am just articulating what comes from it in this blog. So it may lack direction, may lack cohesiveness and may even lack quality. But I need to get something off my chest and I'm not yet sure of what it is....

I was watching the football the other day, it was a Liverpool match and Fernando Torres scored a stunning goal. A chest down, right kick half volley at a really tight angle. My mate shouted something along the lines of "Quality goal".

In fact, he uses the word "Quality" quite a lot. For anything that is "good", "outstanding", "perfect" etc. "Quality beer", "Quality" in response to a good story or yarn. He's not the only one either. A quick scan on the Internet finds the word Quality being used for all of the following meanings:
Top of the range, cool, awesome, amazing, superior, good, grade of excellence, a characteristic of something or somebody, top choice.

Wikipedia describes quality as: "Quality in business, engineering and manufacturing has a pragmatic interpretation as the non-inferiority or superiority of something. Quality is a perceptual, conditional and somewhat subjective attribute and may be understood differently by different people. Consumers may focus on the specification quality of a product/service, or how it compares to competitors in the marketplace. Producers might measure the conformance quality, or degree to which the product/service was produced correctly."

http://en.wikipedia.org/wiki/Quality_(pragmatics)

Jurassic 5 (hip hop artists) have an album called Quality Control (a great album by the way) where they extol the virtues of being true to their art form, not necessarily about the high production quality of their sound.

I've come across bible blogs expressing the quality of the text written in the bible.

The urban dictionary has some rather insightful definitions of quality - no link here and I cannot be responsible for the content.........

The WORD magazine has a quality Friday session on the new radio streaming service Spotify.

http://www.wordmagazine.co.uk/content/spotify-its-quality-friday

Where only 'quality' tracks are played. What does that mean? What is a quality track? Good production values? High-end recording? Intellectually pleasing lyrics? Or just a good beat? It certainly doesn't mean high quality websites or user interaction, as the link to the list of tracks to be played doesn't work!

But that is why it is so difficult to pin down what is meant by the word quality. Robert Pirsig in Zen and the Art of Motorcycle Maintenance goes in to some depth (every tester should read this book by the way) but the main definition I took from the book was that people whom 'care', naturally posses and produce quality.

Isn't that so true? Look at the people who really care about what they do? They produce a quality end product or service. They tend to have a higher quality of life (not financially, but mentally and physically) Compare them to people who are out to make a quick fortune or are just doing their jobs or stuck in jobs they hate.

Our gardener who installed our patio was very expensive, but that's because he did a good job. A quality job in fact. It took him longer than his competitors but the finish was amazing. Our patio still looks amazing all these years later. Money well spent. My friend is a policeman - he cares about his job and the people he serves - he does a quality job. I have another friend who makes dolls house furniture. He cares and the products he produces are exquisite. People pay a lot of money for a 3-inch long sofa.

I care about the job I do as a tester too, therefore the service I offer is quality. I hope... I care about the customer and I care about the process. But that doesn't mean the software will be quality though. Unfortunately.

The reason being is that despite me saying I offer a quality test service, that my friends dolls furniture is quality or that my policeman mate offers a quality service - it's all subjective and in some circumstances, quality is not directly in our control.

If quality can mean cool, awesome, amazing - then it could be that extremely high reliability, stability and finish is not quality to some people. What could be quality to them is how cool the product is, how awesome the new software makes them feel, how much more productive they can be, despite the bugs.

To me, quality software is software that does what it should do, without major defects, with ease and in a way that is pleasing to the user. To some people though, it could be buggy as anything, but if it is awesome then that product can be called quality.

I've also just fallen in to the trap of saying things like 'pleasing to the user'. Hmm. Which user? Under which circumstances? Running on which platform? With which data?

My dad always said to me 'If a jobs worth doing. It's worth doing well' - something I've always believed to be true and something I abide by at all times. If I'm going to do anything worth doing, I'm going to do it well, to the best of my abilities anyway. Isn't this the same as caring. Caring about what you do, investing the right time in what you do, researching, learning, practicing, planning and being methodical?

Could this be why so many software companies and the applications they produce fail to deliver quality to the end user? Because they are not thinking of the bigger picture? They are not thinking about the long term goal? They don't care?

I've seen software companies with a whole team of people doing their job 'well' and the end product is always good 'quality'. Sure there are sometimes bugs (sometimes major defects), but as the training is 'quality', the support is 'quality', the bug fix is 'quality' and the re-release is 'quality', the customer remains happy.

I've seen teams not doing their job 'well' with 'quality' test teams in place and the end product is still poor. Poor release, poor management of defects, poor understanding of the customer, poor support, poor training - and the customers are unhappy. I worked on a product many moons ago that shipped with over 500 known issues in it. To say the customers were unhappy with the product is an understatement - but the test team were 'quality' - we raised over 1500 defects in 6 months between 4 of us. Yet, someone else’s decision to ship with lots of bugs undermined and negated the quality we had added.

So maybe, when people think of quality being the responsibility of the testers, they should ask themselves three questions:

1. What exactly does quality mean to me and my customers?
2. Will having just one piece of the puzzle offering 'quality' mean the end product is 'quality'?
3. Is this job worth doing? If so, are we ALL doing it well?

I'm at the end of the blog post now and I feel much better about quality and why I cannot *ensure* it in software. I can certainly shape it, form it, work on it, improve it, report on it, inform others about it and analyse it. But the most important thing I can do for quality is to realise this job is worth doing and to do it well.

And if everyone else I work with, live with and socialise with is doing the same - then great things will happen. I'm sure of it.

Get up, Stand up

2009-04-08T09:20:00.001-07:00

I’ve been exploring the theme of communication over the past few blogs and as you may know, I strongly believe that if we can understand more about communication, our skills as a tester will develop.

Over the next few weeks I’m going to take a look at how understanding, appreciating and knowing your Purpose, Audience and Context can help you in day to day events. I’m currently working in an agile environment, so will go through some of the most common activities I encounter during my work.

I’ll try and associate these to communication, with particular reference to Purpose, Audience and Context. (PAC).

Today’s example is the daily standup.

Peterstev on agilesoftwaredevelopment.com refers to a standup as a daily scrum (same thing)

“The Daily Scrum is simple daily routine to help the team self-organize, focus, and identify and eliminate impediments to progress”

Martin Fowler also sums up some other sources on his posting on http://martinfowler.com/articles/itsNotJustStandingUp.html

“Summarizing several papers and references ([Anderson, 2002], [Beedle et al., 2000], [Cochango, 2006], [OrgPatternsStandUp], [Rising, 2002], [Rising and Janoff, 2002], [Wells, 1999]) daily stand-ups should achieve the following goals:

share commitment
communicate daily status, progress, and plans to the team and any observers

identify obstacles so that the team can take steps to remove them

set direction and focus
build a team”

There are no doubt, hundreds of other definitions, but what is pretty much common amongst all is that the purpose is to ‘communicate’ progress and impediments.

It’s a simple premise. Rapid, consistent feedback from all team members. And when it works – it works very well. So here are some simple tips on how to make yours (and the teams) communication better. I’ve broken it down in to Purpose, Audience and Context.

Purpose:

Rapid feedback – keep it short, sharp and to the point. Don’t repeat yourself. Keep it succinct.
To inform team members – Make sure your information is aimed at your team members. Make it relevant. Make it accurate.
Consistent – Keep your feedback routine the same. Suggestions are normally ‘What I have Done’ followed by ‘What I am doing today’ followed by ‘What problems do I have’
To find out project information – Listen, carefully. Make sure you treat each person with the same respect and listen as though they are the most interesting person on the planet. Even if they aren’t.
Guidance on where the project is heading. Make notes of anything of importance. If you are listening well and others are aware of their purpose, there will be loads of information for you to gather.
To be of importance and use to all. Raise the hand of ‘off track’. If someone is heading away from the purpose and heading off track – raise your hand and bring it back on track. Force them ‘off line’ if you have to.
Informal – Try not to make it too formal though. These meetings work best if the team are relaxed, happy and confident with each other.

Audience:

Your team– Be aware of your main audience. They are your team members, so show them respect and keep your communication at a level and relevance that is suitable.
Your teams roles. Sub audiences – In any team there are sub audiences. i.e an audience with a different skills set, outlook or team objective. For example, developers and testers. Be aware of this. Keep your communication at a level and relevance that is suitable.
Location of teams – you could have off shore teams. Take your time to work out how best to communicate and what works / doesn’t work. Be aware also that not all team members may be able to see non verbal communication.

Context:

Short, sharp feedback – you are in a meeting where the context is to keep it short. If the meeting starts dragging, people need to make some changes.
Standing up – might be that one of the reasons to stand up is to force a quick meeting.
Location of teams – you could have some teams located outside the building. Make use of relevant technology to communicate. Make allowances for the fact you are communicating non directly.

I’ve been in plenty of stand ups where people are looking out of the window, checking their mobile, watching other people in the office, babbling on about technical stuff way above the heads of most of the audience and in meetings that go on and on and on. I’ve done these things also. I’ve planned my evening whilst stood in stand ups.

I’ve also paid the price by not being informed, not being in touch with what’s happening and not being on the ball. It’s often the tester that needs the full round picture of what is happening on the project so they can prioritize their work and often, the standup is the first and most important place to find that information.

Context > Fail = Disappointed Boss

2009-03-31T12:13:00.000-07:00

A few minutes ago a remark on Twitter made me think about a context failure I encountered a while ago. Well, to be honest, the failure was very much on the part of a fellow senior tester I worked along side.

I was test lead on one aspect of a decent sized project. My colleague was the test lead on another aspect of the same project. We worked very well with each other and the testing was very effective.

We normally put together a weekly update including all of the dreaded metrics and stats that managers love, coupled with an update from each test lead on how things are going etc etc. We always, but always put this together on a Monday afternoon. Every week.

However, one week an email popped in to the inbox asking for an email update by close of play that day. Strange, it was a Thursday. Why a report on a Thursday? So I went over to find out. It turns out my boss was going to meet a selection of his bosses and he needed some metrics. So I duly emailed some stats and an update.

Boss man was out of the office on the Thursday afternoon stating he would pick up the reports later that evening, he was out Friday in this all important meeting. It wasn't until Monday when he came back in that he gave my test lead colleague a real ear full.

It turns out he couldn't access his report and was made to look a bit of a fool in front of his peers and bosses. He ended up having to wing it and it didn't go down well.

So here is why my colleague failed:

He had got his audience spot on - his boss. He knew him well and knew he liked his metrics.

He had his purpose spot on also - to inform and update on the current testing effort.

He failed on context though. He didn't know his boss was off to a meeting with his bosses. He didn't know that his boss would be on a train when the report popped in to his inbox. He also didn't know that his boss had no access to his laptop and would be using his old, but semi functioning mobile phone to get the report.

So, what my colleague did was put together the most amazing set of graphs, tables, pie charts and other fantastic looking metrics in a spreadsheet, that as it turns out my boss couldn't open.

I on the other hand, knew all about the context and produced a very simple email with the key facts.

Now, I must point out that I should (and to this day wish I had) have pointed out to my colleague that boss man would be on the train, so to keep it simple. I would have been more than happy to share this knowledge but for some reason assumed he would go and ask his himself. He didn't.

I learnt two very valuable lessons.

1. Always be aware of the context of your communication. If you have to find it out, do so, but never take it for granted that it hasn't changed recently.

2. Always share any information you have about your Purpose, Audience and Context with those in a similar situation.

Rob..

Extra Extra Read All About It

2009-03-31T01:54:00.001-07:00

This month sees the launch of Good Tester. A collaborative tabloid newspaper with a hint of sarcasm and humor. The newspaper can be found here:

http://www.scribd.com/full/13674544?access_key=key-28iq3et2ldgbyuqc6rea

or you can visit the site at:

http://www.goodtester.com

The newspaper is intended to be tongue in cheek and each month a new topic is covered in a tabloid style.

We are now looking for contributors to provide about 700 words on any topic relating to software testing. The funnier the better, just make sure it comes from the heart and is relevant to testing.

We don’t tolerate name calling, name dropping or private digs at people. We want the tabloid to be a fun way of communicating testing, mainly aimed at other testers.

If you feel you have 700 words to contribute in a tabloid style then please submit them to testmonkey@goodtester.com

The editor will get back to you and we may edit the content to meet the article style. Please provide your name and any other details you want to include in the article. We will put this information at the end of the article, if you wish.

Any questions direct them to testmonkey@goodtester.com. This tabloid is a collaborative piece of work so comments should go through this email address testmonkey@goodtester.com, rather than through this blog or myself. I’m just a messenger…..

Enjoy.

Rob..

Audience > Fail = Angry Developer

2009-03-25T03:50:00.001-07:00

If we all took time to check our audience and produce communication that was suitable, focused and aimed at them then I feel we could avoid arguments, disagreements, misinformation, loss of respect and instances like the following example.

Quite a few years back I worked with a senior tester who appeared to have made the position of senior because he had worked there for so long. I was new, I looked to seniors with respect (I still do), I wanted to learn from them and I was always ready to consume knowledge.

I sat with this guy for a while and he found no defects whilst running through he test case. Nothing for over an hour. His test case was pretty uninspiring and his approach to testing even more uninspiring. But he was a Senior – so I had stuff to learn.

Time went on and I wondered what he was trying to achieve and as a junior I lacked the guts to speak up, to question his tactics or to suggest a new approach. So I sat there. And watched. And watched. And nodded off. And watched some more.

Then eventually he found a defect. Result. Bring out the trumpets. It was a minor discrepancy between the label of a button in his test case and one in the app. The label in the test case stated “Continue”. On the screen it said “Next”. I’d noted that the other windows in the wizard had all said “Next” – so I would have made a note, checked the spec, checked with the Business Analysts and probably recommended the spec be changed as the software was consistent.

He didn’t do that. Instead he raised a high priority defect. But what was more important about this example was what he wrote in the defect report description.

“…the spec states ‘Continue’ yet the screen states ‘Next’. The developer clearly has not read the spec. This is unacceptable”

I remember sitting there whilst he typed it, thinking to myself why anyone would write something so stupid, something so offensive and something so demeaning to a team member? I also remember sitting there after he had submitted it whilst a developer made his way over, red faced, blood boiling, rage on his mind.

I also remember sitting there whilst the dev laid in to the tester, berating him, making him feel small, threatening him. It was not a nice sight and something I’ve never seen since.

So why did the tester raise defects like that?

I’ll tell you why. Because he thought the defect would end up with a different developer. A developer who would see the funny side of this comment. A developer who he had worked with for the last few months and had become accustomed to these humorous, yet subtly accusational defects.

As it happened that dev had been moved from the project and all of his defects were being automatically routed to another dev, who, as it appears, didn’t have a sense of humor at all and was somewhat sensitive (and aggressive).

In this example there is an epic amount of communication failure and it actually triggered a communication and process review, which I was lucky enough to be a part of.

Here’s why this example highlights communication as being so important:

The test case was actually out of date, the spec had been changed to show the label as being Next. The tester had not been shown the new spec (or at least that was the claim), therefore had out of date tests. (The fact the latest version of the spec was available on a shared network seemed to have been ignored)
The tester had been involved in a standup meeting that morning where it was stated that his friendly dev was being moved off to another project. So he should have known another developer would get the bug. He should have therefore checked his audience and raised an appropriate defect for the new audience.
He shouldn’t have been raising defects in that personal way anyway. Defect reports are descriptive and factual, not personal and emotional. They may also be seen by other audiences and in-jokes and snide comments will never go down well. Sure, in this example the tester thought he knew the audience and if the friendly dev had been the one who got the bug report, his audience would have been perfect. But it didn’t work out that way.
The developer should have discussed this like an adult rather than going postal and losing it.

It was evident all along that audience communication had failed. People had been communicating but not to the right audience. There’s an element of context and purpose failure in there also. The audience of the spec, the audience of the standup, the audience of the defect report were all misguided and it resulted in this scene.

The end result was that the Dev got a written warning and soon afterwards left the company and the tester lost any cred he may have had.

So we need to not only improve our own understanding of Purpose, Audience and Context but we need to make sure we understand other peoples abilities, and then aim to improve these where we can.

I’ve got a backlog of examples, some funny, some tragic but my work will only improve if I get some of your stories also. I’m sure we can all recall an instance of audience communication gone bad. Let me know..

Distilling more information in User Stories

2009-03-23T04:37:00.001-07:00

I’ve recently sat an online course hosted by Elisabeth Hendrickson on Acceptance Test Driven Development where Elisabeth introduced me to the concept of distilling acceptance criteria in to user stories. (http://www.qualitytree.com/classes/atdd/)

It is something I (the team) do in an agile context anyway but probably not with as much focus as we should do and certainly with no name, it’s just part of the process. This is where most of the problems occur with agile, in my experience anyway. The acceptance tests are very rarely complete and often don’t get updated throughout the sprint. They are the contract, they are what we are delivering, they are what the customer wants, yet we spend so little time distilling them, fine tuning them, thinking about them.

This culminates in the test team not being fully in tune with the whole picture, the customer possibly expected more/less than they got and the dev team coding on stuff that isn’t in the story.

So why do we get to this point in our agile process when if we took some time up front with customer (if we have that luxury) we could iron out some of these issues through the process of distillation?

Here’s why I think minimum distillation happens:

The sprints are often short and the stories in the sprint are demanding, so time spent working through the stories is time eating in to the sprint. Spending a day working through acceptance criteria is a large percentage of the sprint used up. Even though in my mind, it is time well spent.
The development team are keen and eager to get coding, to get working, to get something together, to deliver the goods. This is a commendable attitude and maybe they don’t need to be involved in the distillation anyway, but how can they be sure they deliver what is asked, if they don’t know what that ask is?
The customer may not be able to spare the time or want to spare the time or see this as valuable.
The distillation process may not be focused or directed or time boxed or constructive adding to the resentment of having this meeting in the first place.
Team members (and customers) may not see the process as being valuable, instead deciding that details will be driven out during the sprint (again, commendable but maybe slightly misguided)
Team members may secretly know that the story and associated acceptance criteria is too large for the sprint and adding more acceptance criteria will mean it does not meet the definition of done.
Team members may believe the acceptance criteria is fully defined already. Unlikely and also unlikely it ever will be complete in advance. Being open minded about the fact acceptance criteria will change and increase/decrease could create complacency, in that there is no point in defining much now upfront – let’s wait until the tester asks the question or the customer feeds back mid sprint.
Developers may see some of the acceptance criteria as minor details that do not concern them. (Validation checks for example) Maybe they don’t concern them immediately, but they will in the long run and if TDD is being used, then they should actually concern them in advance – surely?

There are probably a thousand other reasons why we don’t spend long distilling the acceptance criteria but one thing seems clear to me. The distillation part of the sprint is just as vital as the implementation part or the test part or the mid sprint demo or the standup.

Sure, we are never going to know all of the details in advance and some of that detail is likely to change, to evolve, to move or even disappear, but whilst we have the customer and before we go implementing code, shouldn’t we all spend a little more time finding out what exactly it is we are building?

At least if we do get it wrong, we find out about it earlier, one of the good aspects of agile for sure, but time still wasted none the less.

Let me know why you think teams spend less time on distillation than they should do. Maybe your team doesn’t. If that is the case then let me know if it works for you.

Software Testing Revolution

2009-03-23T02:45:00.001-07:00

In keeping with the essence of this blog (communication) I’ve decided to take the lead of my colleague Clive and bring together like minded testers to discuss, learn and improve the focus of our testing.

The group is aimed at testers from all backgrounds and all experience levels. Here’s the blurb:

"Welcome to Testing Revolution UK South (TRUS). A group open to testers of all abilities and experiences to discuss the ever changing landscape of software testing.

Discussions will focus on what the group decide they want to focus on. Initial suggestions have been; people, communication, techniques, methodologies and training. If you are interested in sharing, contributing, learning and developing to the future of testing then please sign up?

Meetings will be held monthly on an informal basis at a variety of locations, initially in and around the Hampshire area (Berkshire, Dorset, Wiltshire, Surrey, Oxfordshire).. We encourage testers of all abilities and experiences and the focus will very much be directed by the members. It's a great chance to meet likeminded testers where we can share our experiences and learn from each other."

It’s also a good excuse to have a couple of drinks and an informal chat about testing. To join the group or find out more you can leave a comment, email me or join the group at the software testing club or LinkedIn group .

http://www.linkedin.com/groups?gid=1856424
http://www.softwaretestingclub.com/group/testingrevolutionuksouth

Rob..

WAKE on LAN – testing going green

2009-03-19T07:58:00.001-07:00

We’ve been trialing an internal tool written by one of our very clever devs. The little tool basically allows you to wake up your PC from sleep mode via the LAN. Perfect for those of us who work quite a lot from home or out of the office.

It’s not only a great way to wake your PC but what about the environmental savings of being able to let all of your PCs go to sleep. We used to leave them all running as people worked from home. They would be on all weekend also. Now they sleep after an hours none use and we can wake via a simple web page. We’ve certainly noticed the difference in temperature in the morning.

Not only does it save trees, but it’s also going to save money as well, perfect in today’s market. We’ve used it with confidence for the last few months. Thought I’d share it with any other green minded testers.

http://blogs.imeta.co.uk/nrees/archive/2008/11/26/an-asp.net-wake-on-lan-application.aspx

http://wol.codeplex.com/

Double check your audiences values

2009-03-11T13:58:00.000-07:00

I've been wondering a lot about phrases mentioned in the test world like 'quality is the value to someone who matters'. I completely subscribe to this notion that what a tester may perceive to be a high priority bug, the customer may actually not. It is a sound theory and something that I try to remember when testing. Its something a tester must appreciate so that they don't get all gnarled up over a defect they think should have been fixed.

So day one, you find out what your 'person that matters' deems to be important and not important, so you raise defects and only highlight (add extra emphasis on communicating in a rapid way) defects in areas your audience has told you have more value. You keep doing this until one day you get asked why you let a high priority defect go in the pot out without making more of a fuss. 'But you didn't see the value in fixing that bug last week...' is your response.

What's happened is that some dimension of the value has shifted and you've been left out in the cold. The 'person that matters' now perceives this once low value area to be a high value area because of new information they now have at this moment in time. You feel annoyed, frustrated and betrayed.

So when we say that quality is the value to someone that matters I feel we need to add on to that 'at that moment in time', in fact I'm pretty sure someone, somewhere will have already done that.....

I've seen this happen more times than I can remember where a defect in an area or a defect of a type (i.e. tab order) was not important at point X, but now a shift in focus has happened and it's suddenly the top priority now at point Y. Whether you get told about this shift in focus or not is irrelevant - I see it as a testers job to find out what the focus is, from the 'person that matters'. You can report defects until the cows come home but what really adds value for your 'person that matters' is when you report the defects with an emphasis on the ones of value to them.

I'm not saying don't raise every defect you encounter, far from it, but from my experience, testers have a direct influence on how well communicated these issues are to your audience. So spend the time making the ones important to your 'person that matters' read like a horror story and also make them aware of these issues faster.

So find out what is important at that time to whoever may matter and work with them to add 'quality'. Don't forget that the person who matters will have more dimensions of information at their disposal than you do. They may not care about 100 defects in module X, they may care about 1 defect in module Y due to commercial issue Z. Work with them, let them know they can trust you, let them know you are pro-active and let them know about the bugs that matter to them. But most importantly, keep checking your audiences values as they can, sorry they will, change over time.

Test Cases and Exploration

2009-03-11T07:07:00.000-07:00

Introduction

I’ve been thinking a lot about Test Cases recently and noticed quite a few posts on some of the regular testing forums and other people’s blogs. These posts generally tend to have two camps of testers, with a few exceptions. (Note – these two camps may not be exactly what these testers think, but they are certainly what they are posting and blogging about)

Camp 1 write detailed test cases with step by step instructions – normally based off a requirements list or spec or some other document.
Camp 2 write only the test cases they need for guidance. The emphasis is on exploration, using the information they have at the time.

I fall pretty firmly in Camp 2. I generally tend to mix a little bit of high level scripted test cases with exploratory testing and automated tests where applicable. But the interesting part is that I actually started out very firmly in Camp 1. In fact, I was such a supporter of detailed test cases that I would argue to the bitter end with anyone who said otherwise.

So why the change?

Simple really – I looked at the audience and purpose of what I was doing and realised I’d got it all wrong.

I naturally (as the industry kept telling me) assumed that we had to have at least one test case for each and every requirement. I also naturally assumed that the whole effort of rewriting the tests was part of my job. It was something that I had to accept as a tester. It was something we did.

The test cases I wrote against a spec were always out of date when I came to run them and I spent lots of time rewriting, tweaking, modifying, adding and deleting. It was part of my job. It was something I factored in to each of the estimates I provided. I wrote test cases because I was told to, because the industry wrote test cases, because I had to ensure anyone else could run my tests, that I needed detailed steps for future testing. I WAS WRONG. I (and the whole team – maybe even some of the testing industry) had/have got the audience and purpose wrong.

After a couple of years of going down this route I started to write higher and higher level tests. I extracted all data from the tests to a separate spreadsheet. I started using the detailed test cases as guides only. I started dropping the detailed steps. I started writing vague expected outcomes so it would generate thought in what I was testing. I also stopped including all of the validation that I would copy and paste between field tests. I started using crib sheets and I started writing guidance level test cases only.

But the most important thing I started to do, was to consider Purpose, Audience and Context in everything I did.

Why am I writing the test cases?
Who am I writing it for?
In what context would the test case be used?

PAC and test cases

So let’s look further at Purpose, Audience and Context.

After reading a very long and very opinionated thread on Test Republic the one thing that stood out for me was that some testers were under the belief that they HAD TO write test cases. We had some testers expelling the virtues of high level test cases and exploratory and we had other expelling the virtues of controlled testing through detailed test cases. There was a definitive split and what was clear was that most of those in Camp 1 (detailed test cases) were writing them because they thought they had to. (note – there were some who’s customer (audience) actually required detailed test cases as part of the contract, but the majority wrote detailed test cases even when no audience demanded them)

Why did they think they had to write test cases? Why did they believe writing detailed test cases was more controlled and accurate than writing guidance level tests? Exactly the same reasons I thought I needed to all those years ago. Everyone else was writing them like that, the industry recommended we write them like that, the test tools encouraged us to write them like that, management wanted metrics and stats on test case coverage – it was just part of the job as a tester.

The fact that this forum thread generated (at the time of writing this) 131 replies shows test cases are a contentious topic.

What was also clear was that those in Camp 2 (high level where necessary) were blatantly aware of the Purpose, Audience and Context of the testing they were doing. They were aware that the purpose was to guide, to point, to help, to be a heuristic and not a detailed robotic step by step test. They were aware that the audience was themselves or other testers who *should* have the ability to be guided to and through an area yet maintain their own skills, learning and initiative to hunt bugs. They were aware that the test cases would be used in the context of a test environment where time pressures apply, where rewriting test cases is not a valuable use of time, where misguided steps could actually detract from the visibility of a bug. They were aware of their PAC. (purpose, audience and context).

This is something that came to me after a particularly gruelling project in which the software changed daily and my tests cases were out of date the minute I had finished writing them. I could have given up, gone home, moved jobs, but instead I wrote high level test cases. Something I have since labelled Guidance Level Test Ideas. It worked. I found bugs. I reported the dreaded metrics still. I explored. I used the knowledge I had at the time to get the job done. I was free to test and not to perform admin duties and rewrites. I realised rewriting tests WAS NOT part of my job – my job was to test.

The comparison

I thought it wise to list the differences between me now and me some years ago. The numbered statements are me some years ago. The alphabetical responses are me now. You may see something similar in yourselves. You may begin to think that this is the way to go. You may completely disagree with my approach.

If I got hit by a bus someone else could run my tests. I needed to write tests so *anyone* (substitute anyone for non-test professional) could run them. Something I was taught and told to do by my manager when I first started in the job. Everyone else writes tests this way also.

If I got hit by a bus I wouldn’t want a robotic tester running uninspiring, boring tests. I’d want them to explore with the information they had at the time, using judgement and context to make an informed decision about bugs in the software. Guidance Level Test Ideas would still offer direction but the emphasis would be shifted to exploration and contextual decisions. The emphasis would be placed on the tester.
If I got hit by a bus and *anyone* could run my tests then this really devalues me and my testing skills. If I can be replaced by a non test professional then you are not taking your testing seriously enough.
I now challenge the old school ideas about scripted testing in a constructive way. I find out why writing them in detailed steps is the norm and why people believe this would bring benefit. I bring to the table my views and opinions. Testers should be challenging individuals who can break the norm and question the value of doing something that is not efficient or effective.
I didn’t want *anyone* to run the tests so I didn’t have to write them for *anyone*. If I had wanted *anyone* to run through the tests like a robot – I would have automated them.
Future testing will have different ideas and information available at that time. So guidance is all they will need. It is likely the steps I write will be out of date in the future, so guidance level test cases are actually incredibly future proof.
We used to induct junior testers by letting them run through the existing test cases to get familiar with a product. This failed on two accounts: The first being that the testers always found inconsistencies between the test case and the software – most of which were not defects, simply scripts that needed updating. The resulted in a massive list of test to update and no time to do it. The second aspect of failure was that the testers didn’t really learn anything. My generation and future generations tend to learn best when left to explore stuff. I know very few people who read the instructions or follow the guides. They just have a go and see what happens. (there are always exceptions to this rule – i.e. my wife, but then she doesn’t work in IT..)

I needed to prove to my managers I had done some testing and report on how it was going. (accountability and metrics)

I too still needed to prove to my managers I had tested something. I could do that with Guidance Level Test Ideas. My managers were not bothered (and I have yet to find a manager who is) about what the tests contained – they just wanted high level area coverage stats / metrics.
Proving you have tested does not mean you have to have detailed tests as evidence. There are many ways of proving you have tested something. A large bug list is surely one way. A happy customer is another. But Guidance Level Test Ideas are still effective records of your testing.
I want to use my experience to explore the software, yet still offer the recorded metrics managers love to see. The Guidance Level Test Ideas still offered history, repeatability (not that exact repeatability is possible) and metrics. The way I write them also allows room to document exploration and initiative.
The metrics produced by test cases and run rates are very misleading and also assume that you have considered all of the questions you want to ask the system up front. I try to avoid them where possible but reality means people do still use them. Carefully think about Purpose, Audience and Context and provide metrics with reasoning, care and detailed explanations. Metrics can be very misleading and it will make no difference whether your test cases are detailed or guidance level ideas.
With long winded and slow to maintain test cases there is slow feedback to relevant stakeholders. Guidance Level Test Ideas allow a fast response to your stakeholders as you are not spending hours checking whether the bug is in the test case or the software. You are not updating documents to reflect changes or bug fixes, you are just testing.

I had the spec in advance and had time scheduled to write test cases. What else would I be able to do with my time?

I also had the spec up front but I knew it would change so didn’t see the point in writing detailed tests.
I spent my time allocated in the schedule writing Guidance Level Test Ideas, questioning people with domain knowledge, thinking about how I would test better, jotting down test ideas, learning any new skills I needed to get the job done, preparing test environment, exploring competitors products, exploring existing internal products and working closely with the devs to help with Test Driven Development.

This is the way the industry has been testing for years. It is what we do as testers.

I don’t especially care what the industry is telling me to do or what best practice has been decided this month – if it doesn’t fit with me, the company, the product or the project team then I don’t do it. I only have a short period of time to test so I do the best I can, not the best someone else thinks is the best I can do. If I need to automate – I automate. If I need to write detailed tests I will – but only if I really really have to (i.e. substitute really really with if the contract tells me to)
We need to move testing to the 21^st century. Development moves on, project frameworks move on (agile etc) so surely testing should move on. What worked in the 1980’s might not be relevant for now. What was once seen as a poor side career to development is now just as important. Surely we are capable now of making the right decision for that moment in time, for that project, for that team? Check out James Bach, Elisabeth Hendrickson, Michael Bolton, Jonathon Kohl et al. They know testing is a career, testing is educated, testing is interesting and testing does have a future.

I wanted to be sure I covered everything I could think of and record it somewhere.

I need to document all of the areas I need to test to ensure good coverage as well, which I do with Guidance Level Test Ideas. I also know that these ideas could change, be removed, more added so I don’t want to bed myself down with lots of expensive maintenance costs. High level lists in a suitable tool are easier to maintain. I currently use either Excel or a mind mapping tool.
Guidance Level Test Ideas do not have to be a document, they do not have to have steps, they do not have to have anything other than a description and some guidance information. They are ideas. If the idea needs more detail then provide it, if not, then let the idea run its course.
My Guidance Level Test Ideas are just that – ideas. A simple list of these to replace test cases is all that is required. They do not have to be even in a document – they could be scraps of paper. The list can grow. The list can shrink. But importantly - the list must be dynamic. It must be easy to change, easy to update and easy to use. Things will change.
Techniques such as Traceability Matrix’s are also a good way of recording test coverage, but they do not often show elements such as risk and prioritisation. Find what works for you and roll with it – evolve it, fine tune it, learn from it.

I needed all of the information in front of me to say whether the software should ship or not.

I don’t need to say whether the software should ship. No tester should have that responsibility. That is not our role, job or responsibility. It was mine at one company (I even had to sign that I thought it should go, but looking back I should have stood my ground and pushed back on the project stakeholders to make that decision)
There is always someone else that is better placed to make the ship / no ship decision. Someone who knows the commercial pressures, senior management decisions and marketing needs. Someone with more information than yourself.
My job is to report the bugs. It is someone else’s to say whether they are happy to ship it given all the information I have provided.

I had to provide UAT users with test cases.

I do not supply my UAT users with test cases to run anymore. I supply them with high level areas for guidance. I ultimately try to encourage my UAT users to write their own tests and use the system as they would in situ.
If I give them a script I am simply leading them through something in a way I think they should use it. We are giving them the software in the end so why not let them loose on the system earlier? Let them find the bugs that are important to them. Let them find the bugs that they would find in LIVE. Let’s not guide them through the process, avoiding the buggy areas.
Any self respecting tester would also make sure the tests the users are running pass. So what benefit are the users bringing to the table?

I need to provide detailed information on how to test validation and forms and what data to input.

I now extract all data out of the test cases. The test data forms the basis of all of my testing – get this right and the chances of finding defects that are important are much higher. I also want to extract the data out so that I only have one place to update the data.
I use a crib sheet like Elisabeth Hendrickson’s http://testobsessed.com/wordpress/wp-content/uploads/2007/02/testheuristicscheatsheetv1.pdf to give me ideas and inspiration on what to test.
I do not specify much in the way of expected outcomes. Most defects I encounter are not expected so how can I specify it as an outcome? Sure, there may be some items to list but listing them in a vague way lets me make up my own mind about the outcome. It also keeps me sharp as it makes me observe as many outcomes as I can think of at the time rather than what is said on the test case.
I once had a tester miss a really obvious bug because the test case did not say to look for it. We need to guard against this. The tester needs to use their judgement.

Unless I have a direct audience and purpose that requires step by step tests (contractual, demanding manager) then they serve no function to me at that detailed level. I have shifted my focus from writing tests to getting stuck in and testing. I’ve learnt from my experience that maintaining test cases is time away from valuable exploration and testing. I highlight the word my because everyone is different.

If there is an audience that needs these detailed tests then I would suggest writing the tests after the testing, once it has been explored and issues ironed out. Once the UI designs are stable. Once the product is nearing release. Only then can there be a higher probability of little change taking place. This is even more evident in agile where the emphasis is on working software – not documentation.

In the end, it all comes down to Purpose, Audience and Context. Think about these elements and write what is applicable and only that.

Some examples

Old school detailed Level test case

*Step ID*	*Test Step*	*Expected Outcome*	*Pass / Fail*	*Spec ID*	*Comments*
1	Open url ............	· Page will show · User will see option A · User will see Option B · This will be available · That will be available		1.1
2	Validate the page Check this Check that	· Spelling will be correct · Grammar will be correct · Alignment will be correct · Links will be correct · Colours will be correct		1.1
3	Click on the add user	· Add user window will open · User will be shown form X on page Y of Spec Z		1.1
4	Validate the page	· Spelling will be correct · Grammar will be correct · Alignment will be correct · Links will be correct · Colours will be correct · Check this · Check that		1.1
5	Validate Name field with long name	· Can enter long name		1.1
6	Validate Name field with alphanumeric name	· Cannot enter alpha numeric · Error message X comes back · Data will not be saved		1.1
7	Validate Name field	· Etc
8	Validate DOB field	· Etc		1.1
10	Enter Value of Rob in name field	· Can enter Rob		1.1
11	Validate mandatory fields	· Etc		1.1
12	Enter 9/3/78 in DOB field	· DOB is ok		1.1

As you can see, this is very detailed level testing. They are boring, dull and robotic. They lead rather than encourage interaction.

Guidance Level Test Idea

Step ID	Step	Expected outcome	Pass / Fail / Question	Exploratory Comments
1	Open the website and add a user (based on data entry in user sheet.xls)	· Can save the user

In the example above I have put the emphasis on the tester running the test. They need to work out how to add a user; if it is not obvious then bingo – here’s your first bug.

The validation does not need to be spelt out. Not only should the devs be catching this in unit testing (in my opinion) but we should not have to list it in our test cases. The emphasis is on the tester to use a crib sheet (http://testobsessed.com/wordpress/wp-content/uploads/2007/02/testheuristicscheatsheetv1.pdf) and any relevant document at the time of running the test – it will be far more up to date than the one used when writing the test. It also means you do not have to update the tests when the spec changes.

The data input (data entered in to each field) is also taken out to a separate sheet where the inputs are specified. The reason being is that the data inputs may be used across more than one test case. The data may also change and updating it in one place is cheaper than everywhere it occurs – that’s if you can remember where everywhere is. If we have testers simply robotically running each test, then running each test with invalid/out of date data will make this scenario even worse.

The test case goes from many rows of repetition to one line encouraging exploration. It is now up to the tester running the test case to explore.

A revolution in testing

Don’t just keep writing tests in the same old way because everyone else does or has done. Use initiative and evolve. It feels we need to change and I’m a firm believer that this change needs to start with test cases. We still have too many people writing long winded, boring, uninspiring test cases in the belief that they are what we should do. This is why managers want to automate the entire testing process –they see test cases that make that seem possible. These types of test case belittle what we do; they make our job seem easy, boring and repetitive. No wonder managers think they can automate what we do.

Where’s the learning, exploration, judgement and intuition in test cases? There is none – but that needn’t be the case. We can still produce test cases for all the reasons we need to and still make them future proof, dynamic and exploratory. They can still serve as a metric and still be a record of work done, but they can place the emphasis on the tester running the test case (who has more information available at the time) than the tester who wrote it (against a document that may be out of date now).

Passing the emphasis to the tester running the test should encourage thought, learning and consideration. Combine these elements with a little bit of passion and bugs will be found, time saved on test case writing and the start of a revolution in testing. But more importantly – testing will be seen in the IT industry for what it is; a learning, exploring, intelligent and creative process that is highly skilled. It will also have the profound ability to ensure the good testers are nurtured, developed and retained in the business.

I’ve seen this revolution happen in small circles. It takes time and it needs performing with care and consideration, but when it happens you will be amazed at how receptive people are to this approach, how free you are to test the important things, how little time you spend writing tests that need rewriting and how many more important bugs you find.

Whilst drafting this blog (which took me way longer than it should have) Ainars Galvans posted an interesting and insightful blog on the software testers club:

http://www.softwaretestingclub.com/profiles/blogs/the-different-faces-of-a-test

It’s an interesting read and although I’m not a fan of the step by step cases, but you knew that already, I do like the examples he has listed below the detailed step one. These examples are very similar to the data sheets I reference in the test cases. My sheet sit separately from the case and the tester running the test can decide which data set(s) to run. An experienced tester will write only suitable data sets and run them based on priority. Junior testers may write duplicate data sets and run them in order, potentially missing a wider variety of data. It’s safe to say that the data entered could be pretty unlimited (I know it is for the stuff I test) so making sure the data is accurate is crucial. Reviewing of this data by an experienced tester would be advisable. I like Ainar’s blog posting – I think it bring to light the different approaches we all take in testing. Remember the three questions though:

Why am I writing the test case?
Who am I writing it for?
In what context would the test case be used?

Performance Testing without requirements

2009-03-05T07:32:00.001-08:00

Note: this post has been ported over from an old blog. It was written in about July 2008.

I've been organizing performance testing quite a lot over the last few years but have only recently faced a customer who claims they have no requirements and performance considerations at all, yet they want performance testing done. Does this mean that performance testing should not be conducted? Of course not.

As a professional tester I can still run performance tests and here are some ways I go about doing it when there appears to be no requirements:

Identify any requirements that may actually exist somewhere, in someone's head. I ask as many questions to as many people as possible to see if I can tease out something - no matter how small it may be. Maybe a customer has reported that it takes them a long time to do something...
I prepare a brief plan (don't spend ages on documentation at this stage). The plan takes the form of some test ideas or scenarios/routes through the software. Alongside the scenarios and basic lists of the type of metrics I aim to gather. (i.e. memory usage, CPU usage, TCP/IP connections, packets etc) – My work article on gathering basic metrics here.
I prepare a default list of performance elements per scenario. Here's what elements means to me: user base, user spikes, load levels, times, machine specs, network activity etc. So I may end up with a scenario that says “User 1 does this and then does that. At the same time there are 1000 other users performing the same activity over a period of 20 minutes, with a spike in users to 2000 at 10 minute intervals for a period of 1 minute. The machine spec will be as follows.” A simple word document with tables keeping the layout and ease of reading consistent is all that is needed in the early stages.
I prepare a very simple results sheet that is easily read by none technical audiences (be aware of your target audience) - Remember, contributing to a document is easier for your stakeholders than writing the document from scratch.
Ensure the test system is as near to LIVE as possible.

In the first instance if you get no response to questions about performance requirements do not be disheartened. It may be that your stakeholders are wanting a benchmark for reference. It may be that they want to explore how it performs before making decisions on fine tuning the process.

It could be that they have a figure / benchmark / value in mind that they are secretly finding out whether the system meets it. It may be that when you report your findings that you have done all you need to. It could then move to the people in the know to arrange a new project to increase performance, it could be good enough moving forward or it could be enough for now. It could stop the project dead in it’s tracks.

No matter what the outcome try to be flexible in what you are testing. The one thing I have experienced with all performance testing is that your focus will shift. You may start out with scenarios X and Y and then move to scenario Z as a customer reports an issue. Be flexible, be accommodating, be prepared to explore just like you would with functional tests and be prepared to change what you capture and how you report it.

I would also suggest you don't spend too long putting together masses of documentation. Just do enough to capture your ideas, your plans and your results. These too will change and evolve and the easier they are to maintain, the quicker and more efficient you will be.

No doubt there are people out there who will disagree with the following definitions, but these are what I am familiar with and always consider when asked to perform any type of non functional performance testing. As usual, be aware of your Purpose, Audience and Context.

Base lining

Every product needs a baseline to work from, otherwise how do you know whether the performance is improving or not? Your first task could be to capture this baseline

Scalability, capacity and load

Can the system size / user base be increased/decreased without affecting performance?
What is the ideal system size / user base for optimum performance? What configuration does that include?
What is the minimum system requirements? I reckon your sales/support team may need to know this. What configuration does this contain?
Do we need to upgrade the hardware to increase scalability? What affect does more memory do? What about more CPU power?
What system configuration causes the worst performance?
What is the maximum load the system can handle? Is this future proof enough? Is it realistic?

Availability of Service

When under load does data remain stable, messages/communication remain intact and the system continue to operate effectively?
What happens when lots of users start using the system? Is the system stable?
Does the system remain available during a large load? Is it unresponsive? Does it allow more tasks / user requests to be queued? Should it?
Does the system stay online for prolonged periods of time (memory leaks, dropped connections, service restarts)? With a large load?
How long does each process take to give a response to the user?
Does the user receive updates on the process? Is this accurate?

Restriction

Are any of the system areas bottlenecks? DB, SSL Certs, Servers, VMWare test environment, users?

In my experience performance testing can often rely on stubs and drivers and nifty tools to manage states and data. Plan ahead for these as much as possible. They may take time to develop, test and prove reliable.

Early detection of performance problems could save time and money in the long run. Catching design issues at the early stages of the project is even cheaper and easier. Try and get involved at the design stage, you would be surprised at how effective a well placed performance question can be. "Oh, not thought of that, we'd better use this way of retrieving and displaying data from the DB instead"

Performance issues are often quite difficult to troubleshoot, pinpoint and fix. Note - this does not mean they are hard to find - sometimes they are very easy to find. The trouble comes in trying to work out where the issue is.

I'm by no means a guru at performance testing but have been doing it successfully now for about 4 years. There are some true gurus out there and here are their details. Check them out, subscribe to their blogs, read their posts on forums. I do and every day I learn something new.

Scott Barbers blog - http://www.perftestplus.com/scott_blog.php
Scott Barbers website - http://www.perftestplus.com/
http://agiletesting.blogspot.com/2005/02/performance-vs-load-vs-stress-testing.html
http://googletesting.blogspot.com/2007/10/performance-testing.html
Acutest blog - http://performance-testing.blogspot.com/
http://testinglondon.wordpress.com/
This is my work blog posting on basic performance metrics. http://blogs.imeta.co.uk/RLambert/archive/2008/09/19/386.aspx

Turn off, tune in and get stuck in

2009-03-04T07:52:00.001-08:00

I've been experimenting recently with lots of different tools to try and find one that meets my testing needs with regard to exploratory testing. I've created some of my own in Python (not very successful), word (difficult to report on) and excel (good, but took lots of time to maintain). I've also used James and Jonathon Bach's Session Based Testing tool (SBTM)

In the end the SBTM tool suited me the best. It was also proven, written by the gurus and straight forward enough that I could get straight in to it.

I also stumbled across a great blog by a guy called Paul Carvalho. The blog and associated website has some cracking information on exploratory testing and some good pointers with regard to extra learning we should probably be indulging in. But one quote from Paul stood out the most to me "Exploratory Testing (ET) can be done well or it can be done poorly."

Such a great line – should be something I read before and after each session….

I've been doing exploratory testing since I first started out in testing. I just didn’t know it at the time. Alongside my now defunct process of scripting tests I would sporadically 'give the system a good kicking'. The term became synonymous with me. I would regularly get approached to give the product a good kicking. It was hugely successful and I often spent time on other projects just kicking it.

The approach I took was simple. I would select an area of interest (now based more on risk) and just go for it. I would try to stick to what I had planned but would often go off on a tangent. I would find a significant numbers of bugs. Enough for the other testers to want me to help out. When I explained to them what I did people thought I was some sort of superhero. “You just look at it, explore and find bugs?”. Yet, they were all superheroes waiting to be discovered. I encouraged them to schedule in a day of kicking each week for themselves. They too became superheroes and started finding a larger number of bugs. Why? Cos they were giving the software a good kicking. But more importantly, they had a passion. They wanted to find bugs. They believed they could find bugs.

I'm not sure who coined the term Exploratory Testing (some report Cem Kaner in his legendary Testing Computer Software book in 1987). Whoever it was coined a much better phrase than mine of giving the software a good kicking. I didn't really pick up the term of Exploratory Testing until a couple of years ago when I decided to research whether anyone else kicked the software and if so, how did they do it. I suspect many people use exploratory testing approaches, they either don’t realise it or don’t have a term for it.

Here's what I used to do when I gave the software a good kicking:

Dedicate the time to kicking
Turn off email, phone and any other distractions
Put MP3 player on a suitably work friendly album(I'm thinking, The Leaves, Hybrid, Jimmy Smith, Air, Aim)
Put a sign up on my desk with the words 'Giving the software a kicking please leave me a note'
Set myself a goal.
Get all my supporting documentation together (specs, crib sheets, training docs, help files, user guides, notes)
Get stuck in
Make paper notes of the actions I took and things I observed
Document all defects in my notebook
Create a list of interesting areas to look at later
Finish the kicking when I said I would (unless investigating a bug)
Write up what I had found in the test case template - check it in
Log all defects
Chat through with the test lead or test manager or PM and explain what I had done.

Now here is what I do with the SBTM tool (changes made are in italics):

Dedicate the time exploring
Turn off email, phone and any other distractions
Put MP3 player on a suitably work friendly track (I'm thinking, The Leaves, Hybrid, Jimmy Smith, Air, Aim)
Put a sign up on my desk with the words 'Giving the software a kicking please leave me a note'
Set myself a charter.
Get all my supporting documentation together (specs, crib sheets, training docs, help files, user guides, notes)
Get stuck in
Make electronic notes of the actions I took and things I observed
Document all defects in my electronic session text file
Create a list of interesting areas to look at later
Finish the session when I said I would (unless investigating a bug)
Log all defects
Chat through with the test lead or test manager or PM and explain what I had done.

I now have one less step. It’s about writing the test up - the tool now allows me to do that as I go, it has replaced my notebook. Other than that I have changed the goal to be charter. Terminology change only.

I can still give the software a good kicking, but I now have a more formal process in place. I also now have a tool which is easier to sell to management than just me kicking the software.

Am I doing it correctly?

I'd like the think so. I am building a list of areas I think are important to test (future tests). I have a record of testing done. I have a record of bugs. I have historical electronic copies of my work. I have a list of areas I have tested. More importantly though - I am still free to explore and learn, yet able to provide those all important metrics.

I've only had to change the logos and headers on the reports to my companies. I've got the results sheet hosted on a website so management, fellow testers and stakeholders can all see what i am working on.

There is some education for them in being able to read the results clearly, but that is nothing compared with trying to explain that I am kicking the software and being constructive in the process.

Is the tool all I need?

There are however, elements that the tool cannot provide which are needed to make Session Based Testing successful. The only person who can provide these elements is yourself.

They are:

Dedicated and uninterrupted time
A passion for testing
A dedication to bug hunting with no clouded thoughts or judgment
An excellent ability to communicate findings
The belief that exploratory testing is valuable

If you can provide the 5 points above, then the tool can provide the rest. Communication is key to the success of this tool. Poorly kept notes, poor bug descriptions / cross references, badly captured data such as time and date will hinder your ability to revisit, report and prove the testing you did. It will also mean you have less valuable data to reference in the future. Build your communication skills, grab a good tool like the SBTM and provide the passion and exploratory testing could be your best testing friend.

Methodology Wars

2009-02-25T08:17:00.000-08:00

Over a few beers a while ago I was chatting with some fellow testers from many different backgrounds and different testing environments. When I mentioned that we use agile techniques on some projects I was met with a very definitive split of reactions. About 50% of the testers passed air through their teeth and looked at me with sympathetic faces. The other 50% perked up and stated how they wished they had the chance to learn more or that they were currently working in an agile environment.

After a few more beers the conversation descended in to a vicious verbal battle between agilists (those who support agile) and waterfallists (those who support waterfall). Each debate spitting more vitriol than the last. Each negating the other methodology in terms I can’t repeat here. The only thing that seemed to quell the anger was the arrival of our food.

Over the meal I delved a little deeper and found that not one of the waterfallists had ever tried agile. So why did they think agile sucked? From what they had read?

This was strange, we had some people saying agile sucked and they had never even tried it. On the other hand we had some people slating waterfall because it had not worked for them. Each group thought the other ‘methodology’ was rubbish and were more than capable of providing examples of how ineffective the other one was.

After some Google searching I found out that this is what is known in the industry as Methodology Wars. http://www.kohl.ca/blog/archives/000200.html

From this heated pub argument it seemed obvious to me that neither of the methodologies is better than the other. They are just different ways of working. It’s about finding the right context to apply it. Some projects, teams and companies lend themselves better to agile techniques, some to waterfall. It’s about applying the right way of working for your customer/business/project needs. It’s all about context.

The negativity against the other methodology is a side effect the industry as a whole could do without. I’ve been lucky enough to work on both agile and waterfall projects which have been successful (vaguely on time, in budget and with good customer satisfaction) and I think both were right for that project, team and moment in time. However, as a tester who hates having to rewrite scripts, I cannot help but lean more towards agile. It encourages exploration, learning and more importantly, it sticks testers right in the mix from the start.

What was evidently clear from the discussion was that there are testers out there who will probably never embrace agile and it’s principles. If these testers are forced to work in an agile environment then it is guaranteed to fail for them, the mindset is already there. They wont let it work. Which is fine if they are never going to work on an agile project.

As a software tester though I feel it’s important to drop these preconceived ideas about any methodology. We should be able to work in any business context and still offer value, we can still add value in agile the same as we can in waterfall, so why fight the change? We may have to adapt, be flexible and change the way we are working but isn’t that a positive thing?

What are we all going to do if Kanban becomes the next big thing? Fight it or try it? I’m not sure it makes much difference what methodology we use, our test ideas are what counts.......does it really matter what methodology that’s in?

http://testobsessed.com/wordpress/wp-content/uploads/2008/08/AgileTestingOverview.pdf
http://www.kohl.ca/blog/archives/000200.html

My software was a complete success. The user was a failure.

2009-02-24T09:26:00.001-08:00

This is a take on a quote from Ashleigh Brilliant: "My play was a complete success. The audience was a failure.”

How many times have we heard this from devs, project managers, business analysts, even testers? How often do we hear that the user has made the error? We still hear it said even when large numbers of users all make the same error. Is the problem really the user?

In other words my software does what it should do, it's not my fault the users of it can't use it properly. Well, sorry but it is your problem. Our problem even. The whole teams problem.

Usability, logical flow, accessibility and fit for purpose are very much the responsibility of the team. Users are fallible. They do stupid (unexpected) things. That is partly the reason for testers having a job as we attempt to make these same stupid mistakes before the software goes live. We ask the software questions. Hopefully a large proportion of these questions would be the same as the end users in live.

In order for us to ask the right questions though, I believe we need to understand the audience better.

From my experience I often find that some team members (when designing, coding and bug triaging) don't have that same level of ability to contemplate someone doing something stupid. There is an assumed intelligence, experience and logic level of all people using the system. And these people often believe that level of competence is about the same as theirs. Why would someone do that? You've have to be stupid to do that? You deserve to see that system error if you do that? Have they not read the help or user guide?

I'm sure we've all experienced comments like that.

Even with well designed products it is still possible to be completely well wide of the mark when it comes to audience. Often this is because the audience of the software may vary, even conflict against each other. The audience may not be clearly defined, sought out or even considered at all.

Finding out who the end users are is crucial to successfully ensuring the product achieves its purpose. How do we as testers, know what level of understanding the end users will have unless we actively try and find out? The more we can find out about our audience, the more chance we have of ensuring they get the software they want and that we find the defects that are important to them. Or have I missed the point of what we do?

I'm not saying we should replace the audience or customer. Far from it. The tester can never be the audience, the customer or the end user fully - we don't know what they are thinking. There is absolutely no substitute for the real thing. But then the end user should never replace the tester. The end user is not an experienced and trained tester. But, if we find out who the audience is, who they are, what they do, why they do it, in what context they do it then we stand a much better chance of finding the bugs that will really cheese them off. If we get 'real' users involved during the early testing phases for regular feedback on the product, then that's even better.

When the product doesn't meet the needs of the customer it doesn't matter how few defects there are, how cool the code is, how many platforms it supports, how much code coverage there is and how quickly we can build and deploy. If the user hates it, can't use it, doesn't want to use it, will not recommend it and general feels they've paid for nothing useful then the product has failed. To me however, it is test who have ultimately failed.

Sure, the BA, DEV or PM should have designed it with the right audience in mind but a good tester should have asked the questions, prompted the discussions, asked to meet the end user and pushed for ensuring it is fit for its intended audience. How can we be sure something is working if we don't know who should be using it? Cos the spec says so? Shouldn't we have questioned the spec? Asked whether the spec was right?

So here's some questions to ask to see if we can find out more about the audience:

Has the audience been defined in the documentation / project goals / PID / SPEC / whatever document you use?
Is the audience clear or is it confused?
Do the intended audience conflict with each other (i.e. skills, use of product)? (these are some of the worst products for sure)
What domain level knowledge do the audience have?
What technical skills do the audience have?
In what context do the audience use this app? Can I reproduce this context?
Can I get some of the audience involved? If not, why not?
Why are the audience using this software?
What expectations do the audience have of this app / software?

As usual - let me know how you feel you get to know your audience, whether you feel you need to know the intended audience and any other feedback. I'm trying to develop PAC-Testing and feedback is the only way I can fine tune and improve my learning and ideas.

Some more reading:

http://www.infoaccelerator.net/blog/post.cfm/variations-on-shakespeare-know-thy-audience

http://www.softwaretestinghelp.com/wp-content/qa/uploads/2008/02/how-to-improve-communication-skill.ppt

http://totalcommunicator.com/vol2_4/knowaudience.html

Is bug finding emotional?

2009-02-18T09:35:00.000-08:00

Could increasing our emotional involvement increase our bug count?

Ask yourself the following:
Think about all of the tester you have worked with. Think about the best testers you have worked with. The ones you would rate as good bug finders, who could manage a project with their eyes closed, who could manage expectations expertly, who could communicate well and who you would have no qualms with working with again. (maybe you currently do work with one of these people). Now ask yourself why that person is a good tester? What personal attributes does that tester have?

I bet you have a nice long list of attributes. When I sit and do the above there is one attribute that stands out the most for me. It's not patience, tenacity, humour, calmness or willing. It is passion. Now think of the 'worst' tester you have ever worked with and no doubt they certainly lacked passion for what they did.

Passion can also be described as emotion and this is what I feel we all need spend some time developing in ourselves and our colleagues. Developing an emotional involvement in what you do drives you to improve, to create, to strive, to engage and to be involved. Think of all those hobbies you had an emotional involvement in. You loved them at first, then the emotional involvement dwindled, your passion for the hobby was lost and now you no longer involve yourself in that hobby. Surely the same thing applies to your career.

I believe all testers (in fact, all people) go through highs and lows in their careers (and daily lives) that are directly related to their emotional involvement in what they are doing. This could be an involvement in the testing industry, the company they work for, the technology or maybe their colleagues. I've been at a low point before where my emotional involvement in what I did was low, partly from the environment I worked in, partly from aspects of my personal life.

During my testing in that period of about 3 months, I found very few defects, I worked my core hours religiously and nothing more, I didn't engage with my colleagues, I was constantly looking for a new job and I wished I'd taken a different career path. I had become what James Bach has recently described as a 'test tool'. "someone who executes someone else's tests is not a tester-- he's a test tool" James Bach recent post on twitter.

I think James was referring to a tester with no power to explore or step off the path, but the same could be said for me during those three months. I had the power to explore and to find bugs I just didn't have the motivation or passion.

I simply sat there, writing none demanding test cases, running other peoples test cases and not using my initiative and exploratory skills. I was at a low point. I had no emotional involvement in what I did - at all. There were a few of us like this and the number of defects making it to live was on the increase. In the end, personal circumstances involved a home relocation at which point I took a new job and my passion was re-ignited. I wanted to become emotionally involved in what I did and I was given the opportunity to also. I cared about the company, the products and the testing I was doing. I investigated new testing products, started my blogging again, joined local groups but more importantly I became emotionally involved in my career.

As Rollo May points out in 'The Courage To Create' (http://www.amazon.co.uk/courage-create-Rollo-May/dp/055302955X/ref=sr_1_7?ie=UTF8&s=books&qid=1234801361&sr=8-7):

"....that people can more accurately observe precisely when they are emotionally involved - that is, reason works better when emotions are present; the person sees sharper and more accurately when his emotions are engaged'

Is this why we find more defects when we are passionate about finding them? Like trying to spot a particular coloured car on a long journey.

I think it is inevitable we will all have patches in our careers where our emotional involvement is low and it is at times like these when we could all do with a skill set to help us ride them out. It is equally important to recognise in fellow testers this low point and offer help and support where possible. In my experience low levels of motivation and passion over a prolonged period of time result in the gnarly, complaining and inflexible tester we have all worked with. Who loves a good moan and refuses to adopt any new ways of working.....maybe that's just my experience....

Re-igniting your passion for testing is one of the most important steps you can make to keep your testing skills sharp and your work life interesting. I've now seen the light and am determined my career is not going to go the same way as my extreme knitting pastime did.

Eliciting requirements

2009-02-13T06:41:00.000-08:00

I was reading through one of my old blogs looking at which content I could move in to this blog and found one posting about firming up / gathering requirements. It was posted in response to a forum post on SQAforums in which a fellow tester (and many others it seemed) claimed they could not test without requirements. They claimed there were NO requirements at all for a system they were working on. Hmm. I disagreed. Strongly.

Someone, somewhere must have some form of requirements otherwise why is anyone working on anything at all? It seemed to me they had not searched hard enough for the right information.

Part of being an excellent tester, in my opinion, is to find out who the key players are in each project you work on.

Who is the dev guy who appreciates testing and has the time and inclination to help out with queries? They will know what they are building.
Who is working out in which direction the product should go?

You need to know who to go to in order to get the information for you to get the testing done. You have a testing job to do and you need to know who will help you. The same goes for the opposite - who will hinder you? Stay away from them if you can!

So...there has to be some requirements somewhere. Someone must know something. Ignoring the fact that even without direction and requirements, we can still do an effective job of testing, it was obvious to me these testers had not made use of their communication skills.

Had they not asked the questions?
Listened to the responses?
Built a rapport with dev?
Questioned any of the communication they had received?

There simply is no such thing as a project without any requirements. Sure, the requirements may not be documented, they may be confused, contradictory to each other, existing in lots of people’s heads - but they do exist. If no-one else is gathering them in one place and making sure everyone knows about them, then take the initiative and do it yourself.

On a project a few years ago, neither the devs nor testers really knew what they were doing. So I and one of the proactive devs called a mutiny and demanded we get the requirements sorted. Amazingly, the team of about 30 all felt the same and all had no real direction on what was happening (including the Project Manager (PM) and the Product Owner (PO)). Yet, they were all happy to continue blindly. After a few days of grilling the PM and PO we had a definitive set of requirements and a direction to head in. We got the product finished (albeit late) and we learned some extremely valuable lessons.

So, you have read this far and are wondering what this ramble has to do with the film The Goonies. Well, here it goes:

My university degree was based around me eliciting, researching or learning complicated ideas and theories and then presenting them in the best medium possible to any audience. An example piece of work would be to learn all we could about semi conductors and then write an article to appear in a British Tabloid newspaper. We then had to write another article on the same subject for readers of popular scientific magazines. The subject matter was the same but the audience were different.

One of the communication modules during that course was taught by an amazing communications expert called Noel Williams. During that module I worked on how to extract pertinent information from people whilst researching topics. Two of the techniques I learnt to use were paraphrasing and reiteration of what was said. I used the following Goonies quote as a base for my project.

Stef: Data where are you going?

Data: I'm setting booty traps.

Stef: You mean booby traps?

Data: THATS WHAT I SAID! BOOBY TRAPS!

It is a simple quote but tells so much about why it is important to question, to paraphrase and to re-iterate what has been said. Taking what people say on face value can be misleading. In this example it is obvious that what was said is wrong. It's not always that obvious in software development.

A tester with no questioning ability or judgement is no better than an automated test. They will only ever find what they are told to find. If you never question anything then how can you learn? How can you improve? How can you find out more information? How can you do your job?

So here are the original blog ideas for eliciting more requirements:

Editing a document is FAR EASIER than creating a document – thanks Chief Quality Officer (http://www.chiefqualityofficer.com)

If no requirements exist, simply sit down and create a simple spreadsheet of Requirement ID, title, description and priority. Then circulate this doc to project management, management, sales, engineering, dev, release, support, documentation, and test management; anyone who may be involved
You will find people contribute to the document if it already exists.
Great article on this technique here: http://www.chiefqualityofficer.com/2007/05/08/starting-from-scratch-requirements-gathering

When discussing requirements with someone firm up the vagueness by paraphrasing with MUST, SHOULD, DEFINATELY HAS TO

Billy says "that the system should perform X before Y"
You paraphrase and say "So, Billy, the system MUST/SHOULD/DEFINATELY HAS TO perform X before Y?" Billy changes tack and says "ahh with the exception of..."
This paraphrasing is very powerful as it makes the person making the statement double check how confident they are about the statement.

Flip the statement on its head to question the logic.

Again Billy says "that the system should perform X before Y"
Flip it to ask "Can it perform Y before X?"
This kind of questioning drives out more logic and often gets people to think about things they had not considered before

Diagrams can be FAR MORE POWERFUL than words

Draw out any flow diagrams when discussing concepts to get a clear visual image of what is happening. This often draws out scenarios not considered.
Draw/design small use cases using 'real' people to perform tasks. This will often draw out much more detail

Storyboarding is a kind of further development of a use case and can be very powerful for gathering requirements.

Storyboarding should not really be done with no other communication and can be time consuming but offers a much richer way of communicating to the customer and the development team what exactly the user is doing.
More on storyboards can be found here (you may need to register): Techtarget article

Read any existing information

Read any existing documentation on the product to find holes or gaps or potential new features
The existing user manuals can be a good source
The existing help text is often very useful for questioning new features
Support documentation, known issue lists, customer defects etc all add a lot more information in to the pot
Previous test cases and test plans
Test reports and reports

Look at any existing products

Work through any existing products or the current software you are expected to test
Check out competitors products. See how they have done things.
Read around the product and market the product is being released in to.

Get in touch with your customers (if you’re in a position to)

Speak to everyone who is involved in the project no matter how senior/junior they are
Everyone on the project will have an idea of what they are responsible for and where they gained the information to make that decision. Find out where and you may find your requirements.

There really is no excuse for not testing something because you were not told to or you had no information GIVEN to you. You MUST ask the questions. Am I expected to test that? Does a requirement exist for that? If not, who should I tell/ask? Should that work that way?

You must do the leg work upfront. Be professional, use your communication skills, be pro-active and in turn you will be respected for your enthusiasm and abilities.

Software Testing Club

2009-02-11T05:36:00.000-08:00

It feels like I've been spreading myself too thin at the moment since I've found the Software Testing Club. I've suddenly felt that re-igniting of passion for the two things I do best, testing and writing. And I'm involved in so much stuff that I feel I could burst. But I am enjoying myself though.

A few years ago I wondered what kind of career I could have in software testing and was ready to jack it all in and do something different. The job I was in was tedious, boring, uninspiring and limited. I'd moved through the testing ranks too far to move off elsewhere for the same pay. I felt trapped and stuck in a dead end job. I wished I had retrained earlier and done something different. I'd been testing for years and had lost my motivation.

I then realised it was up to me make the most of my career. Software testing wasn't just a job, it was a career and it was up to me to shape it. Opportunities weren't going to land on my plate. So I moved jobs and ended up working for a forward thinking and leading edge company where my passion was re-ignited.

I was positively encouraged to make my own way, to write blogs, to learn anything I wanted. It was a breath of fresh air. I also realised there was more to software testing than your job. There is a whole community out there with interesting side projects. Sure, they mean investing some time out of hours and a little hard work, but believe me, it is worth it.

I'm feeling renewed enthusian to write about topics in testing. I've stepped up my long term project of communication in testing and have stepped up my own interest and enthusiasm in writing fiction.

If you start feeling bored and have pushed the boundaries of what is possible in your current role then don't feel down. Join the software testing club, join community testing projects and realise there is a career to be had in testing. If nothing else succeeds, then quit your job and try another company.

Anyway - better get back to some testing.

Bruce Lee, Jackie Chan and Agile

2009-02-10T07:57:00.001-08:00

Despite this being a communication and testing blog – this posting has little to do with communication. It’s more a blog posting that a fellow tester Phil Kirkham inspired. On a forum recently we discussed Tai Chi and how it aids learning and both agreed we had yet to find a link / analogy to testing. Well – it got me thinking. And here is the result.

This blog was originally going to be about Agile and Waterfall and draw comparisons to Jackie Chan and Bruce Lee. Bruce Lee being the waterfall guy who dispatched his enemies in an order he has already defined, with moves he has already planned. Jackie Chan being the agile guy who uses whatever is to hand to dispatch his enemies; chairs, tables, step ladders, pens, anything really. Agile through and through.

It seemed to me that it was obvious. Bruce Lee was waterfall and Jackie Chan was agile. I was going to ask 'Which of the two is better?' The answer would be neither, they are both exceptional in their own circumstances. The blog idea was genius. I was sure of it. It made perfect sense and I'd even managed to form a witty remark about Jean Claude Van Damme being wooden. It was going to be a crowd pleaser for sure. Maybe even win a prize. Neither development process being better than the other, both valid in difference circumstances. Genius.

But then I started researching Bruce Lee and found his philosophy, in fact the martial arts philosophy as a whole, was extremely agile. It has to be. They don't know what their opponent is going to do next. It therefore ruined my original blog idea but it did give me some great quotes and titles for up coming blogs on agile.

Look out for these blogs coming soon.

Quote
Be like water making its way through cracks.

Quote
The highest technique is to have no technique. My technique is a result of your technique; my movement is a result of your movement.

Quote
A good [Martial arts] man does not oppose force or give way completely. He is pliable as a spring; he is the complement and not the opposition to his opponent’s strength. He has no technique; he makes his opponent's technique his technique.

Quote
One should not respond to circumstance with artificial and "wooden" prearrangement.

Quote
A martial artist who drills exclusively to a set pattern of combat is losing his freedom. He is actually becoming a slave to a choice pattern and feels that the pattern is the real thing. It leads to stagnation because the way of combat is never based on personal choice and fancies, but constantly changes from moment to moment, and the disappointed combatant will soon find out that his 'choice routine' lacks pliability.

Quote
Being wise in Jeet Kune-Do (a style of Kung Fu) doesn't mean adding more; it means to minimize, in other words to hack away the unessential.

Quote
For something that is static, fixed, dead, there can be a way or a definite path; but not for anything that is moving and living

Of course, no posting on Kung Fu with the mention of Jackie Chan and Jean Claude Van Damme would be complete without a quote from each of them.

Jackie Chan
Don't try to be like Jackie. There is only one Jackie. Study computers instead.

Thanks Jackie - I did.

Jean-Claude Van Damme
I am the Fred Astaire of karate.

Hmm.

Why listening is the best weapon for testers.

2009-02-05T03:44:00.000-08:00

Why listening is the best weapon for testers.

Listening is one of the most important aspects of communication. It's a lost art and few of us actually have formal training / learning in it. Many people believe that they are good listeners. I personally believe many of us aren't very good at listening. As a tester you need to be sure to listen to everyone, to assimilate as much information as possible, from as many people as possible. Being able to listen effectively is a key part of your overall communication skills. After all about 45% of all your communication is listening (http://www.ericdigests.org/pre-928/listening.htm).

So what is listening?

Listening is not hearing. Just because you heard someone say something, it doesn't mean you listened. Listening is about receiving the information, assimilating it (processing it) and remembering it. Listening is about understanding what has been said and reacting to it in a considered fashion.

So what can affect your listening skills?

Background noise and general distractions (quite common in an open plan office environment, hunger, distracting colleagues)
Your mental state and mind set at the time. (Your thoughts may be distracted or blurred. You might not want to be there)
You interrupting (Constantly interrupting people whilst they speak is a big hindrance to communication)
Formulating a response mentally before they have finished speaking
Mentally arguing or getting impatient with the speaker
Trying to take notes or working on something at the same time as listening
Being more distracted with the delivery of the communication rather than what is being said (non-verbal aspects such as appearance, voice, mannerisms)

Why is it important to listen?

It shows others you are assimilating and respecting their views and opinions (whether you agree or not)
It builds strong and effective relationships
It empowers you by giving you understanding and knowledge
It can give you many dimensions of a problem or conversation by taking in to account other people’s views
"Effective leaders need good data to make decisions, solve problems and draft strategies. Poor listening will result in patchy data." - http://www.fullcirclefeedback.com/resources/leadership-development/listening/

So why is listening important to testers?

Listening gives you knowledge. With more knowledge you are able to question things in more detail. Questioning is what testers are here for.
It gives you the respect of other team members - essential.
It gives you an insight in to how and what others are thinking.
Other people do not listen - so you have an advantage in the work place immediately

Are there any real world examples of poor listening?

Example 1

Daily stand-up:

One of the developers is giving an update and states that a section of the application is finished now and that he has a problem updating the 'task management' tool. It is ready to test.

The tester during this has been so bored by the previous dev that she is thinking about what to have for tea that evening. She didn't hear any of it.

She then spends the whole day doing anything other than testing the app available. As a competent tester though, she does chat to the dev to see what the delay is. The dev says he'd mentioned it in the meeting in the morning. She looks stupid in the eyes of the dev and respect and appreciation is dented.

This may happen many times and the trust is lost along with a lot of time.

Example 2

Project Design Meeting

The tech lead and project manager are drawing diagrams on the board and talking through the design. The tester is sat not listening. The tech leads mentions how this little feature will work. It actually goes completely against another feature discussed earlier in the day. No one picks this up. The tester knows this will clash but didn't listen properly.

The designs get drawn up and sent out for review. The tester sends feedback stating it will not work because of the functional clash. Time has been lost. Time that could have been better spent of looking at how to solve it. A new design session is now required to redesign and reproduce the design doc. Time wasted.

So how can we improve listening?

We all need to improve our communication, using the following list as guidance is a good place to start. In my opinion though, researching more and more about communication will help you to understand and learn about listening. Practice is also good. Spend time practicing your listening, looking at where your weakness is and constantly improving it.

Submerge yourself in the communication taking place. Tell yourself that this person is the most interesting speaker in the world. Tell yourself that they are communicating the most interesting topic in the world. You will be surprised at how much more you listen to people when you convince yourself they are interesting.
Repeat the words being said in your own mind. Spend as much time as you have in the conversation considering each word and working out what has been said.
Wait for the other person to finish speaking before responding. Interrupting is hugely negative.
Think about what you are going to say before you say it. This way you will have considered what has been said (listened) and then formulated an appropriate response.
Respond with verbal and non verbal cues to the speaker. Head nods, summarize what has been said, "uh huh" etc
Pay attention to non-verbal communication, the things that are not being said - these are often more important than what is being said (separate post on this coming soon)
"The way to become a better listener is to practice “active listening”. This is where you make a conscious effort to hear not only the words that another person is saying but, more importantly, to try and understand the total message being sent." - http://www.mindtools.com/CommSkll/ActiveListening.htm
Remain calm and centred even if you do not understand what is being said
Ask appropriate questions. If you do not understand what has been said then ask for clarification using paraphrasing and simple questions.
Try to understand the speakers point of view. Even if you disagree with what is being said it is important that you try to understand why the speaker thinks or feels this way. This is a great attribute that most tester possess anyway; the ability to think about something from a different point of view.

Links:

http://www.au.af.mil/au/awc/awcgate/kline-listen/b10toc.htm

http://www.mindtools.com/CommSkll/ActiveListening.htm

http://www.fullcirclefeedback.com/resources/leadership-development/listening/

Why written communication is essential

2009-02-04T03:12:00.001-08:00

In previous postings I've looked at how Purpose, Audience and Context are essentially to making you a good tester. All of these together are 'Communication', so being able to improve your written communication is key to success as a tester.

If you look at how many times you write something as a tester you will begin to appreciate how important written communication is. Here are just a few of the things I write as a tester:

1. Bug Reports
2. Test Cases / Scripts
3. Test Plans
4. Emails
5. Blogs
6. Summaries and Reports
7. Notes and ideas

There are probably a load more as well. So being able to write these things in a way that meets the big three (Purpose, Audience and Context) will ensure you are a super uber tester.

So let's take the most common of the 7 written communications above and look at the big 3.

1. Bug reports

These are essential in our business. The real power of a tester lies in whether they can convince someone to fix a bug. Part of that comes from how well the bug report is written.

>> Purpose

The purpose of a bug report can vary but in reality the main purpose is to inform. You are wanting to inform the audience regarding what has gone wrong. You want to explain what the issue is, how you created it, how you reproduced it, when it happened and how to reproduce (including environment etc).

If someone reads your bug report and has no idea what is wrong then you have failed. You have failed to meet your purpose.

>> Audience

The audience of a bug report could be anyone. This depends on your organisation but ultimately the bug report audience is the developer. This is key really as you need to tailor the bug report for how your company works. If the bugs are all triaged by a product / project management team then you will need to ensure you write the bug report for that audience. If the audience is the documentation team then it will require a different approach again.

Making sure you know what the audience is can mean the difference between a bug getting fixed or being rejected.

Different audiences expect different pieces of information regarding the bug, so consider this. My advice is to include ALL of that required information in the bug, that way no matter who ends up with the bug report - they will be able to understand it.

>> Context

Are these bugs being raised at the start of the project when there is a lot of time? In the final week when time is tight? Who is reviewing the bug reports? When are they reviewing the bugs?

Context is the tricky one to consider when raising a bug report but again can alter the way the communication is received.

Summary

Improving your written communication is essentially for you to succeed in testing. Other teams, colleagues and customers rely on timely, accurate information. If, as a tester, you can provide this information with the right purpose, audience and context then you have succeeded.

The trick is not to get down if you fail though - learn from it and adapt. In today’s market you need to evolve your skills and find what is working at that moment.

Being aware of Purpose, Audience and Context in all written communication will help you stay ahead of the crowd.

Best Practice and Argumentative Testers

2009-01-23T06:14:00.001-08:00

Cem and James et al have written loads on this subject so I am not going to repeat it here, but be aware of any blog or testing resource that tell you how to run a project. They can advise, guide and give you the theory but there is no such thing as a best practice. A best practice is a label someone has applied to something that worked for one project. If it works on your project then you are lucky. For it to work on two projects in the same company would be another lucky step. People come and go, time contrainst and differing customers all mean that any best practice is instantly no longer a best practice when these contexts change.

The worst testers I have ever worked with are the ones still trying to force out dated testing methodologies and theories to projects that need something more dynamic. You have to keep evolving and working within the constraints you have, driving for quality all the time. Testers who are argumentative, restrictive in their outlook and unable to accept change are the ones who developers like to complain about. They are the testers who bring a bad reputation to our industry, the ones who no-one wants to work with. One of the most important things you can do as a tester is to make sure the devs and project managers respect and trust you. The only way to do that is to be an effective communicator, both in interpreting communication and communicating yourself.

Why a tester should be aware of PACT.

2009-01-23T06:10:00.001-08:00

You will be able to analyse designs and documents with more clarity
You will be able to assess the software under test more accurately, raising potential issues earlier
You will be able to source the correct audience to gain information from or to use for user acceptance testing
You will be able to direct your questions, queries, concerns, defects etc to the right person
You will be able to pursuade people to fix your defects easier
You will be able to clearly explain often complicated ideas, theories or issues
You will be able to diverge and broaden your skills across roles and offer a wider ranging service(i.e. project management, product management, customer support, sales)
You will be more trusted and respected for your invaluable insight
You will be able to contribute to designs and improvements that offer real improvements to the customer/audience
You will be respected for your succinct, accurate and timely communications
You will become less frustrated with trying to enforce methodologies and theories
You will become more adept at driving in quality to the process and software

In essense, you will become one of those testers that appears to have a natural gift of testing, when in reality, you will be working very hard at what you do. Nothing comes easy in software testing but the more you can adapt, learn and target the people and purposes that matter, the less frustrated you will get with your work. The more time you will have to concentrate on what is important to you and the business.

What is Context?

2009-01-19T07:59:00.000-08:00

Context is under what circumstance the communication will be received, interpreted or consumed.

It is essential the context is understood and accounted for.
For example: You have a remit to produce a project update for your test manager. So you go about putting together a reasonably detailed excel spreadsheet with some graphs. Nice one. All done. You then send it to you boss.
In scenario 1, he/she (the boss) is at their desk. They have access to their PC/Laptop and compliment you on a great document with lots of detail.
In scenario 2 however, your boss is on the train on their way to an important senior management meeting and they only have access to their phone. They berate you for producing a report with far too much information which they had trouble accessing on their phone.

The above scenario highlights that although the audience and the purpose is the same (audience=boss, purpose=inform of project update) the context in which they were received and consumed effected the communication you should have used. You would also probably have needed to know how technologically advanced their phone was. (i.e. blackberry phones)

If you had known they would be on the train then you could have simply put together a high level summary in an email with the key facts.

It is really important that context is taken in to account with all communication as it fundamentally has the effect of altering how you communicate.

Received, interpreted and consumed are mentioned in the first sentence of this section. The reason being is that the context could change between the audience receiving the communication and actually then interpreting it.

This is essentially the basis for context driven testing. The reasoning behind it being that no two projects will be the same, just like no two companies are the same. Therefore the theories of best practices and recommended ways of working are thrown out of the window. What works on Project A may fail miserably on Project B. The context will change. The team may change, the customer may change, the management may change, the software supporting testing (servers, desktops etc) may change and even the frame of mind of the team may change.

A tester can often work in many contexts and apply the same processes and methodologies. A good tester however will adapt to meet the context changing and evolving the testing approach to add quality and value but more importantly; making sure that testing is respected and trusted.

When you are communicating you need to assess and be aware of the context to make sure your audience and purpose are met. The same audience and purpose will be directly affected by the context. Do not assume you can bypass, override or change the context. You need to adapt and work with the contexts you identify.

Being a tester is all very well but to be a good tester you need to be able to analyze communication and communicate effectively yourself. This will set you apart from the crowd.

This content is very much developing and growing as I expand my thinking and ideas. Any help on these theories and concepts from you guys out there would be very welcome. As a communication specialist I can often be over analyzing things too far and missing obvious ways to improve communication.

What is Audience?

2009-01-19T07:58:00.002-08:00

Audience is who the communication is targeted at. The recipient.

Every piece of communication has an audience. The audience is the person or persons who are to receive the communication and therefore interpret the purpose.

Here are some examples of audience:

Project manager

Senior manager

Developer

Tester

Test Manager

Friend

Partner

The list could continue. There may be more than one audience. These multiple audiences may be compatible, they may be at odds with each other. It can often be the case that the audience is confused or contradictory to the other; perfect material for a keen software tester. What is the audience of that function? Module? Feature? Button? Document?

If you know or can work out the audience then you are in a good place to make better decisions about the communication. You are also much better placed to find the real audience and find out more about them.

When you are communicating yourself, you also need to be wholly aware and clear of what the audience of the communication is. This will mean your communication is direct, powerful and reaches it's target audience in the first instance.

What is Purpose?

2009-01-19T07:58:00.001-08:00

Purpose is the aim of the communication.It's end goal.

Every piece of communication has a purpose. Sometimes the purpose may be obvious, sometimes not so. It is your job as a tester to find out what that purpose is, maybe even define it but definitely to always be aware of it.

Here are some examples of purposes:

Persuade

Inform

Educate

Impress

Disgust

Save something to the database

Make the process easy for the user

Make the process difficult for the user

Again, the list could continue for ever. You will notice in the list above that some of the purposes are more abstract (persuade) and others are more object based (make the process easy). This is deliberate as it will depend on the communication you are looking at as to how you apply purpose.

Each piece of communication could have several purposes or maybe just one. Sometimes these purposes can become confused, sometimes they can be at opposites to each other. These are the kinds of things you, as a good tester, need to be looking out for. They are what will set you out from someone who just tests. They will make you a more powerful analyzer, someone other teams can rely on to whittle out those defects early in the process.

When you are communicating yourself, you also need to be wholly aware and clear of what the purpose of the communication is. This will mean your communication is direct, powerful and fulfils it's purpose.

What is Communication?

2009-01-19T07:57:00.001-08:00

I know this is not one of the words in the acronym but it is important to define here what I mean by communication. Communication is the actual piece you are analysing, the actual communication being transmitted. Here are some examples of pieces of communication:

Email
Letter
Specification
Hair style
Clothes someone wears
General visual appearance
Conversation
Verbal presentation
Film
Music

The list could go on. Basically, when I mention communication I am meaning the piece of communication being analysed.

What is PACT?

2009-01-19T07:55:00.001-08:00

Simply put PACT stands for:

P = Purpose
A = Audience
C = Context
T = Testing

It's a way of looking at your work, your communication at work and the way people communicate with you.

Introduction to Communication

2009-01-19T07:49:00.001-08:00

Communication is the key to the success of all software projects in all companies. Without effective communication no one single project would ever succeed.

Communication underpins everything we do both at work and in our personal lives. The clothes we wear, the car we drive and the way we speak are all forms of communication. So why do we spend so long perfecting our hair, our cars and our general appearance yet we spend little time looking at how we communicate at work?

The topic of communication is vast and I'm not going to even consider discussing it here. But what I am aiming to do is give all you testers (and any other audience that cares to read on) a basic introduction to how communication will make you a better tester. Not only will it make you think about communication when testing software, but it will also help you fine tune your own communication meaning you will be more concise, professional and ultimately more respected for what you do.

No matter how you look at it, every day of your week you communicate. When you are reviewing specs, raising defects, writing test cases, developing test plans, providing status updates or even simply sitting quietly in a meeting; you are communicating. Controlling, managing and improving this communication will make you a better tester.

There are countless sites out there that will tell you to test in this way, test in that way, do this, do that, follow this methodology. I'm not going to tell you to do any of these. Part of your journey through your test career will give you experience and insight in to these methodologies anyway and there are already too many sites and blogs offering that kind of advice. You'll know when to use structured tests, when to automate, when to carry out performance testing and when to raise a defect from working in that environment. No one practice is better than another anyway, it is all about context.

What your career will not teach you though, is how to actively improve your communication to enable you to be more effective at what you do. My own test approach is very much based on James Bach's and Cem Kaner et al's, context driven testing. There is no best practice, there is no set way, everything is based on context. I 100% agree with this and this site aims to help you appreciate communication as a part of that context.

PACT:
P - Purpose
A - Audience
C - Context
T - Testing

It is simple but effective. Consider the purpose, audience and context in your daily work and you will be amazed at how clear your thinking can become. You'll also be amazed at how much more respected you become, how much clearer you can lay down your thoughts and how many more angles you consider when testing software.

Hello and Welcome

2009-01-19T07:27:00.000-08:00

Hi all,

This new blog is dedicated to software testing.

The aim of this blog is explore and build upon my strong belief that software testing is in fact driven by communication and the social aspects of life. It is not just about knowing a few SQL commands, being able to sit and pass an ISEB foundation test and being able to explain boundary value analysis. Almost anyone can do that.

What testing is about is more human than that and its a trate that Cem Kaner is exploring in his wonderful presentations here. It is also a topic covered by Michael Bolton in his exceptional article about the future of software testing, found here.

What I'm looking at doing in this blog is basing my work around communication. Communication is in everything we do, no matter where we are and what we are doing. There is no escaping it and what I am hoping to do is explore how we can all improve our communication to make us good testers.

It will also be a journey of self discovery regarding my communication and how I apply it.

To ensure I do not take on too much here (as communication is vast and all encompassing) I've narrowed my theory down to three aspects of communication (still pretty vast):

Purpose
Audience
Context

These three form the word PAC - stick testing on the end and you have my PACT theory. It's not groundbreaking but it is quite interesting (to me anyway!).