š¹ This post is a transcript from our wtfuzz video series. š¹
So, Iām starting to realize all of the testing vocabulary that I hear people say - or that I even use - but that I donāt actually know how to define.
For example, take my last video about the testing pyramid. Here's something that I said...
"The testing pyramid is defined as a framework for designing a test suite that optimizes for speed and confidence."
Watching it againā¦ I donāt actually know what the āconfidenceā part really means. I know that when my tests pass, I feel confident - but thereās probably more to it than that.
So I investigated.
And for this episode of wtfuzz, weāre going to define the term confidence in context of software testing.
According to the dictionary, yes like the word book, confidence is defined as āthe feeling or belief that one can rely on someone or something.ā
Moving over to math, like numbers and symbols and things, they call it the āconfidence level intervalā - and itās the percentage of time that a statistical result would be correct if you took a bunch of random samples. Essentially, the percentage says how āsureā you are that something will happen.
With both definitions, thereās that factor of reliability - and thatās true for testing as well. The main difference though is that with software - there isnāt just one definition of confidence.
Some, like Stefano Magni, form it into a question: "How can you be sure that the application you're working on works if the tests pass?"
Others, like Brad Thompson, look at it in terms of how much testing we feel we need to execute before we can sign off on anything.
But no matter how you approach it, it comes down to this: Confidence is a measurement of trust. Particularly, the trust that you have in your tests to prove that the product is working as expected.
So ok, so thatās cool - but how do we measure this trust and feeling of confidence?
Well, in my exploration I found out that there is a mathematical equation for calculating confidence levels.
Confidence Level = 1 ā Ī± (alpha)
It looks smart, butā¦ I donāt really know what to do with this. Fortunately, though, many software engineers who came before found systems for measuring confidence. Much like the definition, how you measure confidence can greatly vary. But here are a few ways people are doing it.
Number of Bugs
There are teams that count bugs because they believe that the more bugs found after a feature is shipped, the lower the confidence.
They ask questions likeā¦
- How bad are the bugs?
- How many bugs found were fixed? Reopened? Closed? Or deferred?
Quality of Test Cases
This looks at exactly what youāre testing for and the total number of test cases you have.
Higher quality test cases help us know that areas with the most business value (like signups or payment transactions) are working correctly - giving us more confidence in our product.
Stability of Test Results
This asks how sure we are that the tests are reliable.
With this approach, we can askā¦
- Whatās the current pass rate like?
- How many test cases failed or are blocked?
- How stable are these results?
Code Coverage
Code coverage might be most popular way to measure confidence. Itās the idea that the more tests you have that expand to different parts of your system, the higher the confidence.
How much code coverage an application should have is up for debate, but many resources point to 80% as a target to aim for.
And finally, my personal favorite...
A Combination of Them All
Blogger Mike MacDonagh claims that you can have 100% confidence "if all of the in scope requirements have test coverage and all of those tests have passed for the last few test runs."
While I donāt think we can ever reach 100%, I do like that itās multi-dimensional and looks at a variety of metrics to determine overall confidence.
But in the end...
Itās your decision how you measure confidence. The most important thing is that youāre measuring it at all.
Now Iāll turn it over to you, dear viewer: Are there any software testing terms that you arenāt sure about?
As I mentioned earlier, thereās a lot of terms I donāt know - so this āDefining ___ā type video might just become recurring in this wtfuzz series.
Some potential ones from my list:
- Smoke test
- Scalability
- Blackbox testing
- Functional requirement
And if there are any you want to know about, please comment and let me know.
But for now, thank you so much for watching!
You can check out the Gist of resources that I used for this post and please subscribe, šš¦ or follow if you found it interesting or helpful at all.
Andā¦ yeah. See you next time!