reply to A Lurker
Re: CIPPIC is watching DSLReports
said by A Lurker:Yes, but false positive as well as false negatives are just a microscopic part of it all. People have the tendency to get hunged-up on software testing, and actually, software testing is about 20 to 30 % of an entire ITQA process. Remember, it is a *system* that is composed by many, many parts that has to work OK for the result to be credible. That system involves (typically), network componenets, server components, OSs, apps, AND people. ALL of them must work OK, otherwise it is garbage. That's precisely why it is SO easy to punch holes if you audit an IT system. Most people focus on software only, if even that...
I don't know enough about ITQA (my background is mfg QA/QS/QMS), however, the collection software should have some type of verification testing done. ie. someone should have specifically tried to fool the system with a spoofed IP. When building a piece of manufacturing and/or assembly equipment it's not unusual to try and make a bad part on purpose. This allows you to prove to everyone involved that you can't pass a discrepant part (or to make adjustments to get there).
said by A Lurker:That's precisely why an "evidentiary hearing" is necessary. The lawyer is basically questioning the quality of Canipre's evidence and is asking for Canipre to "ponny-up". But, even before that, it is possible to request access to such evidence for review.
The problem here is that Canipre wouldn't necessarily want to show holes in their system. In a good manufacturing system 'problems are good' since if you don't define the problem you can't fix it. It is however unlikely that they would want to do any testing that showed less than 100% accuracy. If it gets to the court case level someone would have to verify the accuracy of the system.
In ITQA that's what User Requirement Specification and System REquirement Specification documents are for. One tests against them where all the functions are defined. So, you end up with only three possibilities:
1 - All the functions were tested and are 100% OK (we loose)
2 - Not all the functions were tested. We can see this by comparing the Testing against the URS and SRS (we win) since there is no evidence to show that Canipre's systems perform as expected.
3 - All the functions were tested and are not 100% OK (and Canipre went ahead anyways) then we probably win, but it depends of the criticality of the function.
Even in the likely case that Canipre does not have a sound QA system in place , could they simply lie to the Judge? Sure. However, I think it would be pretty much suicidal. Why? Because the other party will request the documentation to prove so. And believe it to me, you *cannot* fake that amount of info in such a short period of time.