Switzerland
gds said:Well, in an ideal world you'd be totally right. But software is never really error-free, so it's just a matter of defining metrics to say what works (and to what extent it works) and what doesn't work. To make myself clearer, think about Microsoft releasing Windows XP: when they released it, it wasn't certainly bug-free (a couple of Service Packs and some more updates prove I'm not wrong...) but they let it out of the beta stage when most of the show-stopper bugs were solved for most of the possible HW configurations. It was not perfect but under most conditions it worked. If they had waited for the "perfect" XP to let it come out, it would never come out...
I think it would help the developers to know what always works, what works most of the time and what doesn't work at all for most of the users, so that they can prioritize bug-hunting, working first and foremost to solve show-stopper bugs and leave the least annoying one waiting for their spare time.
And, by they way, I agree that having non binary test results can lead the test results to be conditioned by human judgement, but here comes statistics to help us figure the real priorities.
Generally when a tester, test a feature it can only check it on one computer. So on his computer the feature will work or not. If on someone else's computer it fails then we'll see that it's not stable under some conditions / hardware configuration. This keep the things easy so even non computer scientist can help debugging.
gds said:
Ok your point of view is clearer now. But I think the test can be :
Open MyTV, record now, record the entire show, push ! and look at the dropped frame counter. Does it increase ? : yes / no
There is no way for subjectivity.
