Rich's Matchmoving Blog

The final matchmoved shot with overlay.

Conclusion

Despite initial problems due to the lack of clear areas for software tracking,  I’m pleased with the accuracy of the matchmove. There is some sliding of the ground plane but the camera moves too fast for it to be noticed, and since the only measurement used to solve it was the focal length I don’t think I’ve done too badly.

It’s a shame I didn’t get to include the first half of the video and get a chance to do some more object tracking and patchmoving as I would have liked. Being more aware of what I could do realistically by using a simpler environment would have allowed me to do this. In the near future I will post some examples demonstrating these techniques.

The original footage.

For the basis of my free move shot I was inspired by Cloverfield and the amatuer short film “What’s in the Box”, both take on a 1st person perspective with shaky handheld cameras and make great use of matchmoving to incorporate CG elements.

I wasn’t aiming to do anything as good looking but I like the idea of modifying the background and replacing objects with a new cg one. I decided to split the shot into 2 parts, an object track and standard free moving camera track so that I could complete either and still have a finished piece if time ran out.

At the beginning of the week I shot a few takes emerging from a few street corners looking onto a main road, looking at an object in my hand which was marked with crosses and then looking into the city in the background. Since I couldn’t measure the road I used Google Earth to find a measurement.

My prefered shot, which I worked on for a couple of days but eventually decided that I was getting nowhere since it the road, despite being flat occupies a tiny percentage of the frame, with most ground space filled by a side road which slopes downhill slightly- making alignment of the ground plane very difficult.

I started off with the camera matching segment since I found this to be more essential to the assignment.The shot was corrected in AE4 first by removing barrel distortion and a border, with reference taken from a distorion grid to speed things up.

Doing a track and solve without masking any areas was unsuprisingly, unusable as the software picked up the movement of the various elements that would be considered to be moving; the cars, traffic lights, man crossing the street and reflections in the shop window.

I masked these two areas off, attached them to user features to save time moving them by hand and tried again. The result was better but the amount of tracking points available in the shot was limited on the most important area- the ground- as the camera solve moved erratically at times at the start of the shot.

Lining the grid up for the first 20 or so frames to match the rest of the footage was a difficult process that could not solved by adding extra automatic features to improve the camera tracking solution process since the ground lacked distinguishing features to follow. I added several user features to locations that had obvious patterns that could be tracked manually by eye but were seemingly too random or subtle for the software to pick up, notably corners on the road markings, a manhole cover and coloured areas on the ground. This solved the worst of the cameras movement troubles, but the point cloud was still quite messy with features on the road and path which I constrained appearing to be on different height levels. However, I managed after some time to get a good match on the pavement.

Shooting Details:

Resolution: 720/ 50p

Focal Length: 4.3mm

Shutter Speed: 1/1000

Original Full Shot

Nodal Pan Shot

Here is the scene with basic geometry applied.

After several attempts at trying to get an accurate matchmore, I realised that the poor lighting conditions meant I had no clear reference with which to align the ground plane. As a result I decided to reshoot a similar movement in daylight and to correct all the mistakes I made the first time.

Shooting

The camera was placed further back than normal to reduce the parallax effect whist moving it. To assist with grid alignment later, a box with known dimensions was placed in the frame.

As measuring the size of the large buildings in the background of the location during the shoot would be impractical, I used Google Earths distance tool during the later phases to measure them instead. As I only needed them rounded to the nearest meter at most accuracy wasn’t such an issue as they could be corrected by eye if needed.

Footage was recorded at 1080p @25fps to capture the most detail possible. With this resolution, PFTrack has more data to use when tracking and solving the scene and therefore has a greater chance of accurately matching the camera movement.

Tracking and Solving

Before I could start solving the footage, I took the video and converted it into an image sequence in AE. I added a border to the composition and experimented with the optics compensation effect. Since this footage had little distortion I found that adding this did nothing to improve it. I then imported the unaltered footage into PFTrack.

The lighting conditions alone meant that PFTrack was able to create a much better solve for the shot, I only had to place 1 feature manually to mark a corner of the box so that the scene could be scaled at the end. No adjustments were made to the default settings in order to get a good solution.

Matching the ground plane to the scene was a lot easier this time round as edges and corners were more visible. A point on the reference box was used as the origin to align the grid correctly and small adjustments were made to stop it sliding in certain problem areas at the edge of the frame.

Matching Geometry

The scene was exported to Maya and the measurements taken on location were used to create basic models of the box, buildings and ground. The ground plane was set at 106cm  below the camera as it was in real life and from there the geometry was placed on top and rotated into position. Early versions of this scene had to be discarded as the plane in PFTrack was not set accurately enough, so work was done between the two until the ground and reference box were matched precisely.

Because the ground in the scene went uphill towards the right of the frame, the buildings had to be rotated slightly in order to become perpendicular to the horizon.

Once the chequered overlay footage had rendered, I place it above the original image sequence in AE and lowered its opacity. During the export, it was stretched from a 4:3 image into a full 16:9 1080p image

Conclusion

I’m very happy with the way the matchmove turned out; after a bad start when I spent hours trying to match footage I forgot to stretch to fit the resolution gate in Maya during first attempt at the matchmove, I managed to create an accurate track using better planning and higher quality footage. During work in Maya, it seemed that the track wasn’t as accurate as I would have liked, but the differences were only a few pixels apart and so are not very noticable in the final video. I feel that I have learnt a lot from my mistakes and have become more familiar with the matchmoving workflow.

Important Dimensions:

Focal Length: 8mm

Distance from Lens to ground: 106cm

Distance from Lens to acting pivot: 3.5cm- very little distortion as a result.

Box: 21.5cm x 26.5cm x 23.5cm

Car Park Barriers: 191cm Wide

Distant Tower Blocks: 30m wide, 18m deep

After watching Escape from City 17 a really like the idea of adding a large imposing building to an urban environment. I decided doing a shot in the evening would look good for this, both aesthetically and becasue the sky would contrast against the top of buildings and the larger shadows to give definate edges for the tracking software to work with.

Unfortunately Stoke doesn’t seem to have sunset as the sky yesterday went from grey to a darker grey and the same happened today, so the footage is quite noisy, but still usable.

When filming I increased the focal length to get rid of any noticable distortion and then applied an Optics Compensation effect in AE to remove the slight amount of curvature at the edge of the frame.

Once I had imported the footage and masked the tree, I let the software try  to solve the cameras movement, but because of the amount of noise there were very few usable tracking features. I placed some User Features and adjusted the tracking parameters to help the solve and tried again which gave better results. To check the orientation of the ground plane I placed some primitive shapes aligned to some buildings in the frame.

The week 6 tutorial, I found the idea of patches very interesting and its got me thinking about uses for it in my FYP to change the apearance of locations. Heres the result with a tree missing and a gobo applied to simulate light coming through the trees. I also decided to have another go at rendering in layers and using AO the correct way. After a couple of hours of wondering why the layers didn’t work, I looked back at a vid from last years modellng for film module and was reminded that mia_x materials can’t be rendered like this. A quick switch to a phong shader sorted things out.