TLDR; Any time you see a list of pros and cons on the internet and ‘believe’ the pro or con, ask yourself “But What if…?”
I see a lot of pro and con posts and questions on the internet. They lack the context behind the pro and con identification, so I find it hard to use them directly.
Pro/Con Articles are easy to write and plentiful
Pro/Con articles are plentiful. And some companies make a business out of selling you reports filled with pros and cons or approaches and tools to help you identify the ‘best’ one, without undue effort on your part.
pros:
- you might see some benefits that you didn’t know about
- you might be more interested in a tool/approach/technique that you had ignored
cons
The pros and cons tend to be based on unstated assumption about the environment that may not apply to you.
The pros and cons are based on someone else’s experience, and potentially an environment.
- You might not have that experience.
- You might not work in that environment.
As such they might be misleading.
Question: do you read them with this critical view in mind?
An Exercise I Conducted - What are the pros and cons of Test Automation?
As an exercise, I executed a web search for “pros and cons of test automation” and evaluated the results. I’m not mentioning which search engine I used and which sites I paraphrased the pro and con below from.
Pro: Test Automation is useful for repetitive tasks that are boring or are susceptible to human error
Great.
But…
Assuming What?
If I ask myself what are the assumptions?
- Assumption: repetitive tasks are boring
- Assumption: repetitive tasks are susceptible to human error
And then challenge the assumptions.
But What If?
But what if it wasn’t boring?
- Perhaps you should make the task non-boring first before leaping to automating?
- what makes the task boring?
- Are you repeating the task with exactly the same data? Perhaps you should vary the data?
- What is the output of the task: data or information? Does it always show the same result? or do you have to work to interpret the output?
- Do you care about the output? If not, perhaps you should stop?
- Does it always provide the same output? e.g. if it is a ‘test’, has it ever failed? Perhaps the risk it is testing for now has a very low likelihood?
But what if it wasn’t susceptible to human error?
- Does that mean automation isn’t useful?
- How will automation make it less suitable to human error, won’t that make things harder to automate?
But what makes it susceptible to human error?
- Perhaps you should make it not susceptible to human error and identify what makes it error prone.
- Perhaps you could automate some of the task, rather than all of the task? e.g. data setup.
- Is it the task that is error prone, or is it the variation in the task that exposes an error in the system, that when it is automated and executed consistently the error goes away?
- e.g. people type at different speeds, when a slow typist uses this, the system times out, this makes it susceptible to human error. Automating it removes the human error and hides the fact that the system is unusable to slow typists.
- perhaps the system needs a fix?
- Perhaps you should train your staff to avoid errors?
- Perhaps the system is hard to use?
- Perhaps you need to rotate staff to different teams so that things are not ‘boring’ but are ‘new’?
There are environmental nuances that the pro does not elaborate on. If you experience the symptoms that are part of those nuances, perhaps you should fix the cause.
And I could go on:
- identify ‘but what if?’ scenarios that invalidate, challenge or questions, the pro or con
- find more pros and cons
But I won’t because then this will be too long and then this would become a post about “Pros and Cons of Test Automation” and I intended to explore the “Pros and Cons of Pros and Cons”.
I think it is a useful exercise.
Critical Evaluation of Pros/Cons lists
Points I want to draw out:
- their environment likely differs from yours, but their pros and cons may not help you identify the differences
- you might not have the skills, approach, environment to make their ‘pros’ work
- you might be able to find a better solution than theirs that better matches your environment
- treat any list of pros and cons as ‘possibilities’ rather than ‘absolutes’
- ask questions of the person creating the pros and cons to see if the underlying environment matches yours
Exercise: perform a google search for “pros and cons of test automation” and evaluate the content of the results.
Or choose your own search term. I don’t mind. I’ve done the exercise.
Does it need to be “But.. What if?”
I don’t just have to ask “But… what if?”.
That would be overly prescriptive and doesn’t take into account the actual wording or context of the pros and cons.
The point is to challenge them.
e.g.
- When would this pro/con be valid/invalid?
- What needs to be true for this to be valid/invalid?
- What assumptions are encoded here?
- Is there an agenda behind these pros and cons?
- Am I seeing a description of the only experience this person has?
- i.e. are they recommending a tool because it is the tool they use?
- Did they actually try other tools and approaches?
- Did they have the skills and experience to make those other tools/approaches work?
- etc.
PACES - Pro And Con Evaluation System
I might summarise this as:
- identify assumptions
- identify bias
- challenge the assumptions
- challenge the statements made
I might refer to this as “Pro And Con Evaluation System” i.e. PACES.
But I won’t. And neither should you.
Or should you? Challenge that assumption.