Q&A: Pushing the Visual Limits of 3D

In their spare time, husband-wife visual effects team uses modern technology to create stunning renderings of historical Seattle landmarks

In 1905, Catholic Bishop Edward J. O'Dea laid the cornerstone on what would become St. James Cathedral in Seattle, Washington. More than 100 years later, as the structure still stands its ground, the husband-wife design duo of Matt and Danika Wright have pushed the limits of modern design technology and recreated the intricate beauty of the historical landmark in stunningly lifelike 3D renderings.

The Wrights are partners in Mattika Arts, a firm offering 3D modeling, rendering, illustration and photography services. As a visual effects team, they have created high-resolution 3D environment models for more than 11 movies, including Harry Potter, Master and Commander, XMEN2, Daredevil and Day After Tomorrow, as well as for projects in the video-gaming industry.

The pair didn't create the St. James Cathedral renderings for profit. They simply had a personal wish to challenge themselves as 3D artists. While they were at it, they also modeled Seattle's Mariner building and the Seattle skyline in a similar manner. Technology writer Brett Duesing spoke with Matt Wright about why he and his wife undertook these projects and how they achieved the amazing results.

Editor's note: Transcript has been edited for length, style and content.

br>

A lot of married couples might take up something like tennis in their free time. You and Danika chose to replicate St. James Cathedral.
Matt Wright: [laughs] Yes, all these projects we worked on in our spare time on evenings and weekends around our regular work schedules over the course of a year. The purpose was not related to any business end. Instead, it was a challenge to ourselves to push what we could do as 3D artists. We have worked in the film visual effects industry for a number of years, and also the video-games industry. We wanted to see how far we could take some current technology to produce the most accurate, lifelike work. Photogrammetry is a technique we were familiar with professionally, but we hadn't had the opportunity to explore all of its possibilities. We wanted to see if it could be applied on a very large scale and include a very high level of detail.

Photographs of Seattle's St. James Cathedral turn into accurate 3D measurements inside PhotoModeler. The key points in the 3D structure and the position of the cameras are then exported into Autodesk Maya for rendering.

The wireframe model over one of the original photographs of the cathedral. "Shots like this help us gauge the accuracy of the project, and show areas that need to be refined," says Matt Wright.

Final renderings of St. James Cathedral depict lifelike detail, created by Matt and Danika Wright, using Autodesk Maya.

Which software did you use?
We used photogrammetry software called PhotoModeler [from Eos Systems] to capture both the large-scale 3D measurements of the overall structure and the very fine details of the ornamentation. Essentially, whatever is in a photograph is measurable. PhotoModeler aligned our camera positions in 3D space and also helped generate 3D reference points. This camera and point data was then taken into Maya [from Autodesk], a modeler we use a lot in the film and game industry, which is good at handling very large, complex scenes. Inside Maya, we built all of the geometry, based on the data generated from PhotoModeler.

Why photogrammetry?
Whenever you're recreating a real setting, you have to decide on what technology to use to measure the sites in 3D. The most obvious is a tape measure; however, this is rather impractical on such a large scale. Another option might be laser scanning; however, this would have been too intrusive on the sites, especially in the case of the cathedral, because of the size and amount of equipment we would have to take to these places.

Photogrammetry seemed like the perfect solution. All you need is a camera, and you can survey the sites quickly. About 20 minutes in the cathedral yielded all the photographs required to model the entire interior. Photogrammetry, when used correctly, also yields great accuracy, especially for architecture. The cost is also a lot cheaper, since it only requires the software and a pretty good digital camera — which makes it far more practical for a couple of 3D artists like us doing a little experimental project on the side.

Was it difficult deriving the 3D geometries from photographs?
Not really. First you take your camera — we used a Canon EOS 10D, which is a regular digital SLR camera — and you run it through a few calibration processes within PhotoModeler to get accurate information about the lenses and distortion and all the camera's interior parameters. These camera specifics are how PhotoModeler can calculate the actual distances.

After that, it's nothing more complicated than taking a few pictures of what you want to model. You download those pictures to your computer and pop them inside PhotoModeler. Given the pictures and also the camera's technical parameters, PhotoModeler interprets the scene in three dimensions.

You start by matching up points between images — pick the corner of a chair in one image and the same corner in another image, for instance. You add maybe 20 matching points over the images. Doing that to a minimum of three images, you'll start seeing this 3D scene emerge in PhotoModeler. The software will also work out the camera positions (the point where you took the picture) relative to each other, which comes in handy later when you put the final model together in Maya. All of this information can then be exported to Maya, where you have the correctly aligned cameras/photos and all the reference points that you marked in 3D. From these, you can start modeling your scene using the tools inside Maya.

All these models have so much detail — millions of vertices in one scene. In reality, you said your 3D depiction of St. James Cathedral took the equivalent of about three months of solid work to create. What were some of the shortcuts you used?
Looks can be deceptive. We did reuse some of the components throughout the model, copying the pieces, like the crown molding on the columns, and repeating them around the nave. Architecture is all about repetition, so there's a lot you can copy. Some of the arches through the cathedral are the same, just different sizes, so all that is required is some scaling of the base curves. You can copy them over and make the modifications to it to make them fit. And, obviously, if you have 500 chairs that are all the same, there's no point in building each one.

One of the interesting problems we came across with all these projects: We are not dealing with brand-new construction, where everything is 90 degrees and just about vertical. This is old architecture that has settled over time. Copying details was good, but it took a lot of tricky alignment because walls aren't vertical and not perfectly square. The Mariner building represented about two months of work, and it turned out to be a lot harder than the cathedral because the structure had some real settling problems and hardly any 90 degree angles whatsoever. That's when it became extremely time-consuming. You weren't working on any flat plane to which you could align a modeling grid. Otherwise it would have been much quicker.

Seattle's Mariner building, with a wireframe model superimposed over an original photograph.

A final rendering of the Mariner building in Autodesk Maya. Matt Wright comments about the photogrammetry process: "One thing that we learned very quickly was that photogrammetry is very different to regular photography, or even photography for texture/reference work. You constantly have to think about angles between camera shots, what you have in frame, making sure there is enough detail and depth in the image."

Do you think that staying true to the actual building measurements — flaws and all -- made a difference in the final rendering? Was it worth the extra work?
In my opinion, yes, absolutely. We didn't want to make the building dead-vertical with proper 90-degree corners. That's not what that building was about — it isn't that way in reality. Visually, it's perhaps a little more of a subconscious thing. In the rendering, the walls may look perfectly vertical and the corners look perfect. But if everything was truly squared up, it probably would not look quite as realistic.

How did you deal with the enormous size of the 3D models? Did that present any limitations?
That's why we used Maya for the final modeling. Although you can model objects very well in PhotoModeler, Maya is designed for dealing with very complex scenes and huge amounts of geometry. If you work smart and keep your work organized, Maya can handle an almost unlimited amount of data. There was really no problem in that respect working on this whole thing.

One way to work smart in Maya is to choose your geometries beforehand — whether you'll model a section with NURBS surfaces, subdivision surfaces or polygons. This helps to manage the file sizes. We tried to keep everything in its original geometric form, right up until rendering. Traditionally at render time, you would convert the NURBS to polygons, and a lot of people would convert them to polygons long before that point. We actually kept everything in its place until the end, just to try to keep file sizes down.

Wherever possible, we tried to use instancing so at any one time only one full copy of a complex piece of geometry was stored in memory. So we didn't have the memory overhead for 200 column tops. Each column top might have 110,000 polygons. That alone, copied around the scene by itself, would be millions and millions of polygons. With instancing, we only need one version in memory.

The downtown Seattle skyline, captured by photogrammetry and rendered as a 3D model (top), then textured (bottom). Matt Wright says, "Modeling an entire city is a future project that we have. The downtown skyline was a bit of a test to see how far we could go. It would be interesting to see how far you can push this technology and how much time it would take to reproduce something as massive as a city."

PhotoModeler has been used for a wide variety of applications -- industrial and scientific measurement and reverse engineering -- but not so prominently in the animation industry. Why did you turn to this solution?
We actually started off using another product but discovered that it didn't take into account all principles of lens distortion and principal point of the camera — two factors that play a big part in the accuracy of photogrammetry. If your photogrametry solution isn't completely accurate, by the time you've added 10 or 15 cameras, the end solution won't solve, and you'll have no idea why.

That's when we found PhotoModeler, and right off the bat the calculations were a lot more accurate. It has amazing tools for analyzing error, and it has an incredible feature called Idealize, which corrects all camera distortion in your images and recalculates the scene directly. Maya and most other 3D software cannot deal with distortion, so you have to remove distortion from the images before taking them into your 3D software. PhotoModeler is one of the first tools that come to mind whenever we have to recreate architecture now, or recreate anything.