A smart environment is one that is able to identify people, interpret their actions, and react appropriately. Thus, one of the most important building blocks of smart environments is a person identification system. Face recognition devices are ideal for such systems, since they have recently become fast, cheap, unobtrusive, and, when combined with voice-recognition, are very robust against changes in the environment. Moreover, since humans primarily recognize each other by their faces and voices, they feel comfortable interacting with an environment that does the same.

Facial recognition systems are built on computer programs that analyze images of human faces for the purpose of identifying them. The programs take a facial image, measure characteristics such as the distance between the eyes, the length of the nose, and the angle of the jaw, and create a unique file called a "template." Using templates, the software then compares that image with another image and produces a score that measures how similar the images are to each other. Typical sources of images for use in facial recognition include video camera signals and pre-existing photos such as those in driver's license databases.

Unlike other biometric systems, facial recognition can be used for general surveillance, usually in combination with public video cameras. There have been three such uses of face-recognition in the U.S. so far. The first is in airports, where they have been proposed - and in a few cases adopted - in the wake of the terrorist attacks of September 11. Airports that have announced adoption of the technology include Logan Airport in Boston, T.F. Green Airport in Providence, R.I., and San Francisco International Airport and the Fresno Airport in California.

A second use of the technology was at the 2001 Super Bowl in Tampa, where pictures were taken of every attendee as they entered the stadium through the turnstiles and compared against a database of some undisclosed kind. The authorities would not say who was in that database, but the software did flag 19 individuals. The police indicated that some of those were false alarms, and no one flagged by the system was anything more than a petty criminal such as a ticket scalper. Press reports indicate that NewOrleans authorities are considering using it again at the 2002 Super Bowl.

The technology has also been deployed by a part of Tampa, Ybor City, which has trained cameras on busy public sidewalks in the hopes of spotting criminals. As with the Super Bowl, it is unclear what criteria were used for including photos in the database. The operators have not yet caught any criminals. In addition, in England, where public, police-operated video cameras are widespread, the town of Newham has also experimented with the technology.

Computers can do increasingly amazing things, but they are not magic. If human beings often can't identify the subject of a photograph, why should computers be able to do it any more reliably? The human brain is highly adapted for recognizing faces - infants, for example, remember faces better than other patterns, and prefer to look at them over other patterns. The human brain is also far better than computers at compensating for changes in lighting and angle. The fact is that faces are highly complex patterns that often differ in only subtle ways, and that it can be impossible for man or machine to match images when there are differences in lighting, camera, or camera angle, let alone changes in the appearance of the face itself.

Not surprisingly, government studies of face-recognition software have found high rates of both "false positives" (wrongly matching innocent people with photos in the database) and "false negatives" (not catching people even when their photo is in the database). One problem is that unlike our fingerprints or irises, our faces do not stay the same over time. These systems are easily tripped up by changes in hairstyle, facial hair, or body weight, by simple disguises, and by the effects of aging.

A study by the government's National Institute of Standards and Technology (NIST), for example, found false-negative rates for face-recognition verification of 43 percent using photos of subjects taken just 18 months earlier, for example. And those photos were taken in perfect conditions, significant because facial recognition software is terrible at handling changes in lighting or camera angle or images with busy backgrounds. The NIST study also found that a change of 45 degrees in the camera angle rendered the software useless. The technology works best under tightly controlled conditions, when the subject is starting directly into the camera under bright lights - although another study by the Department of Defense found high error rates even in those ideal conditions. Grainy, dated video surveillance photographs of the type likely to be on file for suspected terrorists would be of very little use.

In addition, questions have been raised about how well the software works on dark-skinned people, whose features may not appear clearly on lenses optimized for light-skinned people.

Samir Nanavati of the International Biometric Group, a consulting firm, sums it up: "You could expect a surveillance system using biometrics to capture a very, very small percentage of known criminals in a given database."

Several government agencies have abandoned facial-recognition systems after finding they did not work as advertised, including the Immigration and Naturalization Service, which experimented with using the technology to identify people in cars at the Mexico-U.S. border.

However, the government also has possession of a huge, ready-made facial image database - driver's license photos - and is looking into how they can be used. By law, the government can't sell those photos to private companies, but there are no prohibitions on their use for surveillance purposes by the government itself. The Federal government has begun to fund pilot projects on expanding the use of driver's license photos to facial recognition databases.

It makes no sense to use face-recognition in airports. To begin with, there is no photo database of terrorists. Only two of the 19 hijackers on September 11 were known to the CIA and FBI - and surviving terrorists aren't exactly lining up to have their photo taken by the U.S. government. In addition, the technology simply isn't reliable enough for such an important security application. It would work especially poorly in the frenetic environment of an airport, where fast-moving crowds and busy background images would further reduce its already limited effectiveness. The evidence suggests that these systems would miss a high proportion of suspects included in the photo database, and flag huge numbers of innocent people - lessening vigilance, wasting precious manpower resources, and creating a false sense of security.

If facial recognition is unjustified in airports and at public events such as the Super Bowl, its use for general surveillance is even more inappropriate. The security threat on a public street is far lower than in airports, and sociological studies of closed-circuit television monitoring of public places in Britain have shown that it has not reduced crime. The balance between the risks and benefits of facial recognition is even more unfavorable in such locations than in airports.

One threat is the fact that facial recognition, in combination with wider use of video surveillance, would be likely to grow increasingly invasive over time. Once installed, this kind of a surveillance system rarely remains confined to its original purpose. New ways of using it suggest themselves, the authorities or operators find them to be an irresistible expansion of their power, and citizens' privacy suffers another blow. Ultimately, the threat is that widespread surveillance will change the character, feel, and quality of American life.

Another problem is the threat of abuse. The use of facial recognition in public places like airports depends on widespread video monitoring, an intrusive form of surveillance that can record in graphic detail personal and private behavior. And experience tells us that video monitoring will be misused. Video camera systems are operated by humans, after all, who bring to the job all their existing prejudices and biases. In Great Britain, for example, which has experimented with the widespread installation of closed circuit video cameras in public places, camera operators have been found to focus disproportionately on people of color, and the mostly male operators frequently focus voyeuristically on women.

While video surveillance by the police isn't as widespread in the U.S., an investigation by the Detroit Free Press (and followup) shows the kind of abuses that can happen. Looking at how a database available to Michigan law enforcement was used, the newspaper found that officers had used it to help their friends or themselves stalk women, threaten motorists, track estranged spouses - even to intimidate political opponents. The unavoidable truth is that the more people who have access to a database, the more likely that there will be abuse.

Facial recognition is especially subject to abuse because it can be used in a passive way that doesn't require the knowledge, consent, or participation of the subject. It's possible to put a camera up anywhere and train it on people; modern cameras can easily view faces from over 100 yards away. People act differently when they are being watched, and have the right to know if their movements and identities are being captured.

Facial recognition - or any security technology - should not be deployed until two questions are answered. First, is the technology effective? Does it significantly increase our safety and security? If the answer is no, then further discussion is beside the point. If the answer is yes, then it must be asked whether the technology violates the appropriate balance between security and liberty. In fact, facial recognition fails on both counts: because it doesn't work reliably, it won't significantly protect our security - but it would pose a significant threat to our privacy.

Given the requirement for determining people's identity, the obvious question is what technology is best suited to supply this information? There are many different identification technologies available, many of which have been in wide-spread commercial use for years. The most common person verification and identification methods today are Password/PIN (Personal Identification Number) systems, and Token systems (such as your driver's license). Because such systems have trouble with forgery, theft, and lapses in users' memory, there has developed considerable interest in biometric identification systems, which use pattern recognition techniques to identify people using their physiological characteristics. Fingerprints are a classic example of a biometric; newer technologies include retina and iris recognition.

While appropriate for bank transactions and entry into secure areas, such technologies have the disadvantage that they are intrusive both physically and socially. They require the user to position their body relative to the sensor, and then pause for a second to `declare' themselves. This `pause and declare' interaction is unlikely to change because of the fine-grain spatial sensing required. Moreover, there is a `oracle-like' aspect to the interaction: since people can't recognize other people using this sort of data, these types of identification do not have a place in normal human interactions and social structures.

While the `pause and present' interaction and the oracle-like perception are useful in high-security applications (they make the systems look more accurate), they are exactly the opposite of what is required when building a store that recognizes its best customers, or an information kiosk that remembers you, or a house that knows the people who live there.

Face recognition from video and voice recognition have a natural place in these next-generation smart environments -- they are unobtrusive (able to recognize at a distance without requiring a `pause and present' interaction), are usually passive (do not require generating special electro-magnetic illumination), do not restrict user movement, and are now both low-power and inexpensive. Perhaps most important, however, is that humans identify other people by their face and voice, therefore are likely to be comfortable with systems that use face and voice recognition.

Facial recognition utilizes distinctive features of the face - including the upper outlines of the eye sockets, the areas surrounding the cheekbones, the sides of the mouth, and the location of the nose and eyes - to perform verification and identification. Most technologies are somewhat resistant to moderate changes in hairstyle, as they do not utilize areas of the face located near the hairline. When used in identification mode, facial recognition technology generally returns candidate lists of close matches as opposed to returning a single definitive match (as do fingerprint and iris-scan technologies).

The performance of facial recognition technology is very closely tied to the quality of the facial image. Low-quality images are much more likely to result in enrollment and matching errors than high-quality images. For example, many photograph databases associated with drivers' licenses or passports contain photographs of marginal quality, such that importing these files and executing matches may lead to reduced accuracy. Similarly well-known problems exist with surveillance deployments. If facial images for enrollment and matching can be acquired from live subjects with high-quality equipment, system performance increases substantially. For facial recognition at slightly greater-than-normal distances, there is a strong correlation between camera quality and system capabilities.

As with all biometrics, 4 steps - sample capture, feature extraction, template comparison, and matching - define the process flow of facial scan technology. Enrollment generally consists of a 20-30 second enrollment process whereby several pictures are taken of one's face. Ideally, the series of pictures will incorporate slightly different angles and facial expressions, to allow for more accurate matching. After enrollment, distinctive features are extracted (or global reference images are generated), resulting in the creation of a template. The template is much smaller than the image from which it is derived: facial images can require 15-30kb, templates range from 84 bytes to 3000 bytes. The smaller templates are normally used for 1:N matching.

Verification and identification follow the same steps. Assuming your audience is a cooperative audience (as opposed to uncooperative or non-cooperative), the user 'claims' an identity through a login name or a token, stands or sits in front of the camera for a few seconds, and is either matched or not matched. This comparison is based on the similarity of the newly created match template against the reference template or templates on file. The point at which two templates are similar enough to match, known as the threshold, can be adjusted for different personnel, PC's, time of day, and other factors.

System design for facial scan verification vs. identification differ in a number of ways. The primary difference is that identification does not utilize a claimed identity. Instead of employing a PIN or user name, then delivering confirmation or denial of the claim, identification systems attempt to answer the question "Who am I?" If there are only a handful of enrollees in the database, this requirement is not demanding; as databases grow very large, into the tens and hundreds of thousands, this task becomes much more difficult. The system may only be able to narrow the database to a number of likely candidates. Human intervention may then be required at the final verification stages.

A second variable in identification is the dynamic between the target subjects and capture device. In verification, one assumes a cooperative audience, one comprised of subjects who are motivated to use the system correctly. Facial scan systems, depending on the exact type of implementation, may also have to be optimized for non-cooperative and uncooperative subjects. Non-cooperative subjects are unaware that a biometric system is in place, or do not care, and make no effort to either be recognized or to avoid recognition. Uncooperative subjects actively avoid recognition, and may use disguises or take evasive measures. Facial scan technologies are much more capable of identifying cooperative subjects, and are almost entirely incapable of identifying uncooperative subjects.

The four primary methods employed by facial recognition vendors to identify and verify subjects include eigenfaces, feature analysis, neural network, and automatic face processing. Some types of facial scan technology are more suitable than others for applications such as forensics, network access, and surveillance.

"Eigenface," roughly translated as "one's own face," is a technology patented at MIT which utilizes two dimensional, global grayscale images representing distinctive characteristics of a facial image. Variations of eigenface are frequently used as the basis of other face recognition methods.



As suggested by the graphic, distinctive characteristics of the entire face are highlighted for use in future authentication. The vast majority of faces can be reconstructed by combining features of approximately 100-125 eigenfaces. Upon enrollment, the subject's eigenface is mapped to a series of numbers (coefficients). For 1-to-1 authentication, in which the image is being used to verify a claimed identity, one's "live" template is compared against the enrolled template to determine coefficient variation. The degree of variance from the template, of course, will determine acceptance or rejection. For 1-to-many identification, the same principle applies, but with a much larger comparison set. Like all facial recognition technology, eigenface technology is best utilized in well-lit, frontal image capture situations.

Feature analysis is perhaps the most widely utilized facial recognition technology. This technology is related to Eigenface, but is more capable of accommodating changes in appearance or facial aspect (smiling vs. frowning, for example). Visionics, a prominent facial recognition company, uses Local Feature Analysis (LFA), which can be summarized as an "irreducible set of building elements." LFA utilizes dozens of features from different regions of the face, and also incorporates the relative location of these features. The extracted (very small) features are building blocks, and both the type of blocks and their arrangement are used to identify/verify. It anticipates that the slight movement of a feature located near one's mouth will be accompanied by relatively similar movement of adjacent features. Since feature analysis is not a global representation of the face, it can accommodate angles up to approximately 25° in the horizontal plane, and approximately 15° in the vertical plane. Of course, a straight-ahead video image from a distance of three feet will be the most accurate. Feature analysis is robust enough to perform 1-1 or 1-many searches.

In Neural Network Mapping technology, features from both faces - the enrollment and verification face - vote on whether there is a match. Neural networks employ an algorithm to determine the similarity of the unique global features of live versus enrolled or reference faces, using as much of the facial image as possible. An incorrect vote, i.e. a false match, prompts the matching algorithm to modify the weight it gives to certain facial features. This method, theoretically, leads to an increased ability to identify faces in difficult conditions. As with all primary technologies, neural network facial recognition can do 1-1 or 1-many.

Automatic Face Processing (AFP) is a more rudimentary technology, using distances and distance ratios between easily acquired features such as eyes, end of nose, and corners of mouth. Though overall not as robust as eigenfaces, feature analysis, or neural network, AFP may be more effective in dimly lit, frontal image capture situations.

Facial recognition is deployed in large-scale citizen identification applications, surveillance applications, law enforcement applications such as booking stations, and kiosks. It is most often deployed in 1:N environments, searching databases of facial images for close matches. Facial recognition is not as adept at 1:1 verification; facial recognition vendors have attempted to penetrate the desktop login market, but the technology is not optimized for desktop authentication.

Project Description	Location	Vendor	Vertical Sector	Horizontal Application	Application Description	Additional Description
Manchester, NH Viisage	US-NH	Viisage	Travel and Transportation	Surveillance/ Screening	Screening	4th US airport to adopt solution
Cognitec 'SmartGate' Sydney Airport	Australia	Cognitec	Travel and Transportation	Phys Acc/T&A	Physical Access	6k Qantas aircrew, based on passport read
Virginia Beach Surveillance	US-VA	Identix	Law Enforcement	Criminal ID	Surveillance	600 image database, 10 subjects, alarm rate met with deployer approval
Berlin Airport	Germany	ZN	Travel and Transportation	Phys Acc/T&A	Physical Access	Face recognition terminal; template stored on SC
Diversity Visa Program	US-MA	Viisage	Government	Civil ID	Immig ID	Image first entered into system at time of green card registration to prevent duplicate apps, later used for security screening
CO DL	US-CO	Identix	Government	Civil ID	DL	duplicate enrollment detection
Zurich Airport Face	Switzerland	C-VIS	Travel and Transportation	Surveillance/ Screening	Screening	Zurich Airport Police running system; targeting illegal immigrants from W. Africa, M.East and Asia
City of Brentwood Police Dept.	US-CA	Imagis	Law Enforcement	Criminal ID	Forensic	ID-2000 and CABS system integrated into the Records Management System (RMS) of Data911

Facial recognition technology is expected to grow rapidly as customers deploy it for criminal and civil identification applications, including surveillance and screening, through 2007. Increased revenues will be primarily attributable to use in large-scale ID projects in which facial imaging already takes place and the technology can leverage existing processes, such as drivers' licensing, passport issuance applications, and voter registration. In addition, facial recognition technology's use in surveillance applications is expected to increase significantly in public and private sector applications. Because of its unique ability to perform surveillance, as well as the fact that facial images are acquired as part of nearly every document and ID issuance process, facial recognition stands to benefit strongly from post 9/11 deployment decisions. Facial recognition revenues are projected to grow from $34.4m in 2002 to $429.1m in 2007 and are expected to comprise approximately 10% of the entire biometric market.


External Links

http://www.research.ibm.com/ecvg/biom/facereco.html
http://www-white.media.mit.edu/vismod/demos/facerec
http://www.cs.colostate.edu/evalfacerec
http://www.merl.com/projects/face-rec
http://www-white.media.mit.edu/tech-reports/TR-516/node6.html
http://www.frvt.org
http://www.dodcounterdrug.com/facialrecognition/FERET/feret.htm