Face Tracking – FaceTrack
visage|SDK™ FaceTrack package is an extremely powerful, accurate and configurable face tracking engine. It automatically finds and tracks the face and facial features in video sequences in real time and returns full 2D and 3D head pose (translation and rotation), gaze information, 2D and 3D facial features coordinates, a full 3D face model, and a wealth of other information.
- For each processed video frame, returns 2D and 3D head pose and other information.
- Depending on the configuration, tracker can track the mouth contour, chin pose, eyebrow contours, eye closure and eye rotation (gaze direction).
- Fully automatic operation (manual fine-tuning in initial video frame available optionally for enhanced precision; results of such setup can be saved and reused).
- Robustly recovers from any losses due to occlusions, face turning away, tracked person coming and going etc.
- Automatically re-initialises if a new person appears in front of the camera.
- Tracks from webcam or video files.
- Raw image interface allows tracking from any video source.
- Tracks in color or grayscale video (internal processing performed on grayscale).
- No markers or makeup are needed on the face.
- Fully user-configurable to suit a number of different applications; default configurations include head tracking and facial features tracking. See details about configuring the tracker.
- Minimal size of the face in the video image is approx. 80 pixels wide.
- Minimum input video resolution is approx. 320×240. Higher resolutions (e.g. 640×480, 800×600) give better results.
- Head rotation is tracked up to approx. 50 degrees, though in good conditions it can be more.
- Extensive tracking volume (as a rough indication, with a 640×480 pixel camera the head can move up to approx. 120 cm away from the camera, approx. 50 cm left or right and approx. 30 cm up or down).
- Lightweight technology (example videos shown on these pages were recorded on an Intel Duo T7500 2.20 GHz processor with video capture running in parallel).
Face tracking outputs
The tracker offers easy-to-use API for accessing the tracking data on-the-fly during tracking operation. The available data includes:
- 2D and 3D head pose (translation and rotation).
- Facial feature coordinates in global 3D space, relative to the head or in 2D image space. The feature points are specified according to the MPEG-4 FBA standard.
- Gaze direction and information
- Eye closure.
- A set of Action Units (e.g. jaw drop, lips stretch, brow raise…) describing the current facial expression; these parameters can be used to animate a face model.
- Standard MPEG-4 FBA Face Animation Parameters available through API interface and as FBA file output.
- 3D model of the face in current pose and expression, returned as single textured 3D triangle mesh; this extremely powerful function enables applications such as drawing the model into the Z buffer to achieve correct occlusion of virtual objects by the head in AR applications; using the texture coordinates to cut out the face from the image; drawing the 3D model from a different perspective than the one in the actual video or inserting the face into another video or 3D scene.
Application development/platform availability
For developers wishing to integrate the tracker with the Unity 3D engine, the visage|SDK for Windows, iOS and Android packages include a working sample project integrating the tracker into Unity with full source code and documentation.
Fully configurable tracker
Face tracking is fully configurable through an extensive set of parameters in easily manageable configuration files. Each configuration file fully defines the tracker operation, in effect customising the tracker for a particular application. Default configuration files include:
- Head tracking configuration
- Facial features tracking configuration
- Off-line Facial features tracking configuration (the above video was recorded using this configuration)
Extensive documentation allows users to create own application-specific configurations. The documentation lists and documents all available configuration options. A partial list of main options is here:
- Input, work and display resolution settings.
- Camera mirror option (flip camera image horizontaly).
- Video file sync option (skip frames to sync or process all frames).
- Automatic or semi-autimatic initialisation.
- Smoothing filters to reduce noise in tracking results.
- Full control of the 3D head model internally used by the tracker, including the animation rig (this advanced option can potentially be used to completely replace the 3D model by a custom one).
- Control of regions of interest to be tracked.
- Control of number and positioning of actual track points on the face and the search precision (allowing tradeoffs between performance and precision).
- Control of facial actions to be tracked (e.g. jaw opening, eyebrow motion, eye rotation/closure etc.).
Furthermore, Visage Technologies consulting and custom development services are available to adapt the technology in terms of precision, performance and any other requirements in order to meet the needs of specific applications.