On Hacker News I found a very interesting article in which the author suggests using the Petersen-Lincoln method, which is used by biologists to count the population of birds, monkeys and other animals, to *drumroll* count bugs in an application.
A Bug in the Wild – Bigfoot Sighting by Derek Hatfield
The method is very simple, we take two ornithologists, they find birds of a certain species, their task is to determine the population size of these birds. The found birds are marked by both ornithologists, then the number of common ones is calculated, substituted into the Lincoln index formula and we get the approximate population size.
Now for applications – the method is also very simple, we take two QA and they find bugs in the application. Let’s say one tester found 10 bugs (E1), and the second 20 bugs (E2), now we take the number of common bugs – 3 (S), then according to the formula we get the Lincoln index:
This is the forecast for the number of bugs in the entire application, in the given example ~66 bugs.
Swift Example
I have implemented a test stand to check the method, you can see it here:
https://paiza.io/projects/AY_9T3oaN9a-xICAx_H4qw?language=swift
Parameters that can be changed:
let aliceErrorFindProbability = 20 – percentage of bugs found by QA Alice (20%)
let bobErrorFindProbability = 60 – percentage of bugs found by QA Bob (60%)
let actualBugsCount = 200 – how many bugs the app actually has
In the last run I got the following data:
Estimation bugs count: 213
Actual bugs count: 200
That is, there are 200 bugs in the application, the Lincoln index gives a forecast of -213:
“Alice found 36 bugs”
“Bob found 89 bugs”
“Common bugs count: 15”
—
Estimation bugs count: 213
Actual bugs count: 200
Weaknesses
This method can be used to estimate the number of errors in an application, at all stages of development, ideally, the number of bugs should decrease. I can attribute the human factor to the weak points of the method, since the number of bugs found by two testers should be different and different bugs should be found, however common should also be found, otherwise the method will not work (zero common bugs – division by zero)
Also, such a concept as common bugs requires the presence of an expert to understand their commonality.
Sources
How many errors are left to find? – John D. Cook, PhD, President
The thrill of the chase – Brian Hayes
Source code
https://paiza.io/projects/AY_9T3oaN9a-xICAx_H4qw ?language=swift
https://gitlab.com/demensdeum/statistics/tree/master/1_BugsCountEstimation/src
