WRITTEN ON May 26th, 2005 BY William Heath AND STORED IN Uncategorized
I’ve now found and read the Atos Origin report on the biometric enrolment trials. The evidence says that depending on biometrics to work with the whole population is wildly impractical. It would be Hutton-like to draw any other conclusion.
To set it in context – if this “gold standard” service is to exclude – say – fewer than 4200 out of 42m people, the success rates for enrolment and verification together would have to be 99.99% (and failure rate less than 0.01%).
Just for enrolment, the success rate is
Facial – “nearly 100%” (but 98% for disabled people)
Iris – 90% (61% for the disabled)
Fingerprinting “nearly 100%” (96% ditto)
For verification the success rates are
Facial – 69% (48%)
Iris – 96% (91%)
Fingerprinting – 81% (80%)
I was kidding earlier about tossing a coin, but these results suggest that for disabled people tossing a coin would actually give statistically better verification results than electronic facial recognition.
The report’s resolutely positive tone is at odds with the facts it conveys. We need discussion, but with a bit of chiaroscuro. Under the heading “Facial verification success” with a figure of 69% you have to consider the implications of a failure rate of 31%.
Even where it says “almost 100%”, implying almost total success, this is far too imprecise in this context. “Almost 100%” of 42m people might exclude any number from one individual to 210,000 people (which is 0.5%). Bear that in mind when it says for example that 0.62% of the disabled group could not enrol under any of the three biometrics.
It reports a time of 90 seconds to do a “one to many” search. That, one assumes, is on the database of 10,000. What would that take then on a database of 42m? Is it 42,000 times harder, leaving you waiting four days while they check your ID to hire you a video? Set me straight on this.
There are major variations in enrolment success which should worry disabled black pensioners. They have threefold good cause to argue the process seems to be discriminatory.
The report is a summary only, hinting at further detail in a full report which may or may not be made available. It points out that its findings on customer attitude among the survey volunteers may not be representative of the UK population. This may prove to be an understatement. Compare testing on volunteers with the dragging a margin of hardcore protestors kicking and screaming through the process.
The report makes 10 recommendations (education, better equipment, try harder etc). Surely the first recommendation is that we relinquish the notion that a national biometric database can provide anything close to comprehensive service. It may be able to do something, and surely has a role to play for those who wish to use it, but it can’t provide a single standard.
If that’s the case, can someone explain how this ID system affects the many inside the considerable margins where the biometric technology isn’t up to the job?
2 Responses to “Lessons of the biometric trials summary”
Bori Toth comments –
> face recognition … Coin tossing
The FRVT (Face Recognition Vendor Test) reports are an excellent source to gain an understanding how poorly face recognition really works. Mathematically, 2D face recognition is impossible (you are missing a dimension!). Additionally, recognition is knowledge-based. Much of what
we “see” is actually what our brains interpret into the signal but a machine has only the signal … E.g. it will calculate the distances
between certain features etc. but these are not stable over time and they are affected by many external influences. My thesis has a section
on face recognition which also incorporates the main findings of the last FRVT results. I would recommend you read it in case you are
interested. (see her thesis attached above)
> It reports a time of 90 seconds to do a “one to many” search. That, one assumes, is on the database of 10,000. The report states that the system has been pre-loaded with 118,000 iris
images and 1 million fingerprints. The report, however, does not state which technology has been used for the one-to-many search (to screen out
multiple identities). The time taken to perform the search indicates that they used fingerprinting: it is a public figure that the FBI
fingerprint DB contains 45 million entries and it takes 68 minutes to run an exhaustive search on it. At the same time, with optimised DB
architecture, 1 million IrisCodes can be searched in 1 second. Also, it is native to the nature of the iris recognition algorithms that error
rates do not accumulate with DB size – the algorithms make automatic adjustment to the decision threshold so that performance stays optimal. Again, this is native to iris recognition; error rates will accumulate for other technologies – I can give you the technical background on why that is if you are interested.
It’s also seriously bad for some racial groups: “They all look alike…” to the designer. Apparently this applies not just to facial features but also to finger print.
I will not allow anyone to shine lights into my dicey eyes.
(While in Africa, I found that all Europeans and other northerners looked alike to local people. It depends on the points of comparison.)