In the business of biometric testing, data is at the heart of all operations. Without bona fide biometric data (that is, unaltered data from genuine people), the results of testing simply cannot be reflective of real-world scenarios, or some modes of testing simply can’t be done.
For example, using a dataset of AI-generated face images (an example of synthetic data) to test the performance of an algorithm can provide some insights, but only about how the algorithm performs on synthetic data; in short, for testing to provide insights on real people, the data needs to include real people. By testing the presentation attack detection subcomponent of a system with only attack instruments and without gauging the performance of real users, any results will lack the context and substance that makes them meaningful.
As such, a crucial component of operations that often seems to surprise outside observers in both its complexity and necessity, is the planning, recruitment, and management of test crews.
When planning a test crew, there’s a few high-level questions to ask:
Who should be included in this test crew? Who is therefore not included?
Test crews are unable to include coverage of every single demographic. There are many reasons for this including: clients’ budgets, required test crew size, the technology under test’s deployment environment, access to participants, and general practical considerations.
However, it is not automatically problematic to not include every demographic group in every test crew, as long as it is done in a considered and intentional manner. A test crew should either be representative of the general populace who will be using the technology under test in the real world, or it should be tailored to achieve the evaluation’s specific aims.
For example, a test crew for a technology being deployed in Australia should include Aboriginal and Torres Strait Islander participants; on the other hand, a test crew for technology only being used in the United States should instead include American First Nations participants. A banking app may want to target its test crew to cover mostly people over the age of 60, to gauge whether they experience higher dropout rates compared to other age groups; an age verification technology vendor will need to test across a large portion of the age spectrum to assess overall accuracy.
That being said, it is important to always consider the consequences of including and not including different demographic groups in any particular test crew. The decision of who is and is not tested may mean the difference between discovering a demographic bias issue before versus after a technology is deployed to the public. It might mean the difference between foreseeing a large group of customers experiencing more friction or rejection versus discovering after the technology has been deployed.
Once the desired makeup of the test crew has been established, the next question can be discussed.
What do we need from our test crew?
Once any testing process starts to involve human subjects, it instantly becomes more complicated and rightly so. By agreeing to participate, members of the test crew are placing significant trust in whoever is conducting the testing- ultimately, the whole testing process and the relationship with test participants rests on their consent to participate.
In the world of biometrics and digital identity, participants may be agreeing to anything from providing copies of their genuine passport to interacting with novel technology and sharing their biometric data. It’s also why the test crew process is so complicated- factors such as accreditation compliance, ethical concerns, client and test participant concerns, testing requirements, practical requirements, and more, must all be constantly considered and balanced against each other throughout the lifecycle of test crew engagement.
The consent form is therefore the ground truth for all intended collections and use of the data as well as any communications, instructions, and documentation; if an activity is outside the consent form, then it cannot be done (at least not without an additional consent process). The best practice is to only collect data and consent as necessary, rather than for the sake of collection. As such, it’s important to establish early in any project what exact data needs to be collected o complete the evaluation. For some types of testing, a large dataset of neutral-expression selfie images will be required; other tests could need the test crew to visit the laboratory and provide their fingerprints repeatedly to the technology being tested.
Each consent form is therefore carefully tailored to cover the requirements of the project while also informing the participant on planned data use and their rights. The overall consent collection process also requires careful management to ensure that:
- Informed consent has been established and all collected forms are validly filled out
- The completed forms are correctly filed and stored
- The information extracted from the form is accurate and properly stored
- Personally identifiable information has been deidentified
- The overall participant breakdown is still in alignment with the desired test crew makeup
- The participant has received all relevant instructions and documentation
How do we get that data?
Once the test crew has been decided and the preparation has been completed, the final test crew step is actually getting the data.
For some of BixeLab’s evaluations, the process is straightforward: for example, technology-style biometric performance evaluations generally use existing datasets, so the upfront test crew management does not need to be repeated for every single use. However, for the majority of evaluations, at least some biometric data needs to be collected live. This then creates a range of operational questions, concerns, and requirements.
Can the biometric data be collected remotely?
Sometimes, the technology under test can be securely and appropriately accessed by test participants using their own devices. However, concerns about confidentiality, usefulness of the collected data, and the ability of participants to interact with the technology without direct supervision then arise. There are measures to counteract and mitigate each of these issues, but they all require operational management.
Do the test participants have to provide their data in-person?
If the technology under test hasn’t been set up for remote access or has specialised capture technology, then the test crew will need to actually visit BixeLab premises to participate in testing. This also has operational consequences for organising test participants and managing the facilities to accommodate them.
How do you find the actual people to be in the test crew?
Underpinning all these questions and concerns regarding test crews, is the need to find and recruit the people. The private nature of biometric data and the ignorance around biometric technology means that people are naturally sceptical about participating in testing. Combined with the set requirements and demographics, this means that the pool of people able and willing to join test crews is often small.
This pool can be further shrunk by operational concerns and testing requirements. For example, a technology being tested may need test participants who hold a particular type of passport and for the passport to be tested in-person. This is sometimes easier said than done; finding 50 Australian passport holders for a laboratory based in Australia requires much less work than locating 50 Colombian passport holders.
Stay Connnected
If your organisation is progressing toward certification, regulatory alignment, or independent performance validation, we would welcome the opportunity to continue the discussion. Please feel free to contact us for further discussion.