WRITTEN COMPONENT ENTRY Art of Tracking notes.
Tracking is the process of automatically locating a point (or series) frame-to-frame in a sequence, allowing the user to stabilise a shot, track to or solve object or camera movement. The process started as one point tracking which could stabilise a shot or add matching motion to a composite, but today it involves complex 3D camera solutions and extends to optical flow – the technology tracking every pixel.
By understanding the process it is possible to improve solutions and speed up work. This includes the inner algorithm’s that drive the software. There should however be precautions taken when filming so that tracking is not necessary, or a less complicated task. Doug Roble (Technical Academy Award winner) for his TRACK software at Digital Domain; “We go the other way, we measure everything on set, we are really exact. 20% of the shots we solve are easy, 80% are hard. We focus on changing those odds”.
Historical Overview
Prior to digital tracking, most effects shots required the camera to be locked off, it was impossible to track without motion control of the capturing cameras to align to shots in post. Hand tracking has been attempted but the eye isn’t precise enough to capture every movement.
The US Defence Department initially developed the concept of tracking for use in missile guidance systems, the earliest vfx application was in 1985 at NYIT Graphics Lab by Tom Bringham and J.P.Lewis, implementing a FFT-based tracker used for a number of television shows. ILM had an early 2D Tracking software called ‘MM2’, which was used on Hook. The tool was a manual tool which could alter the positions of the keyframes. ILM used this software as a base for their later technology used on Jurrasic Park which was the first 3D Tracking Software – developed by J.P.Lewis and Rob Bogart.
ILM allowed Lewis to publish the algorithm involved, which has also contributed the algorithm to other software packages Shake and Commotion.
Joe Alter developed one of the earliest markerless tracking system which was used for the morph plates on Star Trek The Next Generation.
Discreet first bought tracking software to the wider vfx community via ‘Flame’. Flame was initially ‘Flash’ – written in Melbourne, Aus by Gary Tregaskis. Flame was launched at Siggraph 1992. Flame was first used in feature film work on Super Mario Bros (1992).
“There’s a balance between a perfect track that takes forever and on interactive track that falls off from time to time. We decided to go with interactivity and the real thought went into making it easy to use, making it easy to stop and restart a track because it was never foing to be perfect”.
- Colin Wrey (first implementation of Quantel’s Auto Lock Follow [ALF])
Action was a general tracking tool which really allowed users to make the most of the technology. In the early days tracking a 30 second shot was a 30 minute batch process with no guaranteed success.
Shake was immediately popular because it offered the ability to work in a higher bit depth than anything else on the market. A very early version of the software was used on the opening shot of Titanic (a deep underwater shot) which had originally been filmed as an 8 bits/channel of colour info but slight tonal gradiations brought massive banding problems. Shake was able to glean a clean artifact-free shot.
In Flame/Fusion Version 5 (1997) you could now track as many points as you want, automatic triangulation stabilisation, corner tracking, perspective tracking.
In 1998 was when tracking began to be acknowledged by industry awards, the Technical and Scientific Academy Award to Gary Tregaskis (primary design) and Dominique Bolsvert, Phillipe Panzini and Andrew LeBlanc for the development and implementation of Flame and Inferno. Douglas R. Roble won the Technical Achievement Award for contributing to tracking technology and design and implementation of the TRACK system. TRACK is an integrated software at Digital Domain which uses computer-vision techniques to extract 2D & 3D information about the camera and the scene. TRACK feeds Digital Domain’s Nuke system with full 3D camera solutions. TRACK shift from 2D to 3D tracking was a reflection of the industry’s shift to 3D.
Luc Robert commented, “The main benefit from match-moving software is a marked increase in productivity. Automatic match-moving is a definitely a huge help in this respect, since on a significant fraction of shots it produces a solution which requires no human interaction at all. Most post-production companies using automatic match-moving have integrated it into their pipeline so that, for virtually no overhead cost, they can benefit from these automatic solutions. Despite this, there will always be shots which automatic match-moving software will not be able to solve. It is crucial for professionals to solve 100% of the shots they want to match-move, and for that, the software has to allow them to control and guide the process if necessary”.
3D Equalizer tech came about in 2000 which was an advanced camera and object match-moving software.
Boujou has allowed for seamless combination of live action and 3D.
2D Tracking
Most modern trackers are based to a luminance of black and white version of the track box. Digital Fusion alone uses colour information automatically, the interface shows the strength of each 3 colour channels and which it has decided to use. Shake allows the ability to use hue or saturation for tracking instead of luminance. Ron Brinkman pointed out that in some cases of multicoloured patterns this can be very useful and effective. Try a hue track if the monochrome version of the image has poor contrast. After Effects uses luminance as the preferred colour channel for the track (there is a switch to control RGB or saturation).
Ideal on-set Tracking Marker
Ensure that the tracking markers are not something which are not going to blow away. Common tracking markers are crosses, triangles or circles. Each software has a different algorithm, therefore prefers one shape to another. AE for example, doesn’t really prefer one shape to another. The question of which marker also relies on what the camera is doing, strictly speaking a circle provides a more robust track point (better for zooms, and focus changes). There is no right or wrong answer – but is dependant on many factors. LED markers were considered but were found were only more effective in low light conditions. LED’s have high contrast but there is the light and glare from the light. Most experts agree that there is little point in running special colour corrections, tracking algorithms generally already have some sort of colour correction built-in.
Dealing With Noise in tracks
A key aspect to tracking is reducing jitter. You can remove jitter by deleting keyframes and allowing an interpolator to fill in the gaps. A small blur could reduce noise. PFTrack has a very comprehensive noise reduction algorithm called ‘de-noise for noise reduction’. These are specifically designer algorithms that act similar to a median filter but maintain all edge detail. 3D Trackers produce a better track by tracking a video resolution image. 2D cannot.
3D Tracking
Steven D. Katz explained 3D Tracking; “Camer-matching software utilises a subset of projective geometry called epipolar geometry. This branch of mathematics is used to describe the geometric relationship between two optical systems viewing the same subject and can be used to locate points in space. Because a moving camera offers a new view every frame, epipolar geometry works for a single moving camera as well, and each new view is understood as a separate optical system:.
3D Tracking can provide solutions for measuring objects at a distance, projective geometry, camera angle projection. The 3D Tracking program is used to solve the camera position by using 2 camera views. Epipolar geometry is a type of triangulation. Epipolar geometry proposes that the point we are interested is actually constrained to a single line, which limits the search for that point.
Ideal Tracker for 3D Tracking
3D Trackers rarely use colour information. 3D Tracking experts have very strong opinions on the ideal markers, which strongly supports triangular markers. The advantage is that a triangle provides 3 well-spaced, high contrast corners for automatic trackers to pick up. Accurate recording of on-set measurements allow the user to track the room to the camera, so that the artist has a virtual set to move around.
3D Tracking falls into 2 categories;
- Automatic Trackers
- Manual Trackers
Automatic point generation is typical to 3D tracking. This allows the software to track highly complex footage, such as tracking water.
Dealing with Lens Distortion
Lens distortion has very little influence on 2D Tracking, but can have a huge influence on the calculation processes involved in 3D tracking. Most 3D camera packages have tools to adjust for lens distortion. Lens distortion only effects the camera solving.
In MatchMover, ld is represented by a mathematical model, whose parameters are computed automatically based on point tracks. If too complex a calibration pattern may be applied. The most powerful 3D packages additionally allow for extra information (or images) to be added.
The future to 3D Tracking will likely ne linked to optical flow technology, where every point is tracked and produces a set of floating point data once combined with 3D tracking. Some 2D features may be applied to calculate a 3D match-move and some may be used to calculate optical overflow. These provide very reliable solutions. 3D tracking is fundamentally about tracking a camera, not necessarily things in the scene.
Ron Brinkman commented that “There’s no doubt in my mind that tracking will continue to evolve in a direction that relies on characterising the entire image as much as possible, as opposed to specific feature-based analysis. Ultimately if your starting point includes information about the camera (derived from 3D analysis) than any additional tracking methods will have their accuracy improved dramatically”.
http://www.fxguide.com/article212.html
