Behind the Scenes of Augmented Esquire

Augmented reality — which essentially means to layer data like audio, graphics, and animation over live video — existed long before we got the idea to use it in our 2009 Best and Brightest issue. The term was coined in 1992 by Tom Caudell while working for Boeing, where factory workers used AR to sort parts. Now, with video cameras in so many electronic devices — including the webcam on your computer — AR applications range from advertising to architecture and gaming to pizza boxes. But no one had taken advantage of the technology on any kind of editorial scale, until we got to thinking...

We've been experimenting with the physical nature of our cover lately: E-Ink last October (left), a window in February, some origami in May. But this spring, we sat down with digital-services firm The Barbarian Group looking for ways to make the entire magazine interactive. "We wanted to create something that wasn't just about showing off the technology but actually adds value to the story," says Barbarian CEO Benjamin Palmer. After reviewing several iterations, we began work on six experiences: a 3D cover, a weather-changing fashion portfolio, a time-sensitive Funny Joke from a Beautiful Woman, plus a song, a photo slideshow, and an ad from Lexus.

Conventional photo shoots didn't quite cut it for our December issue. So we brought in Pysop, the award-winning animation studio behind those jaw-dropping Coca-Cola ads. Over the summer, Psyop shot Robert Downey Jr.'s semi-improvised stand-up/lie-down/sit-on-the-cover routine against a green screen with a high-end HD camera. For Jeremy Renner's channeling of Charlie Chaplin, we used still photography in sequence. As for Gillian Jacobs (left), she told a highly choreographed joke to a handycam in an empty loft — and then another one for after midnight.

Our three celebrities would eventually live in landscapes of Psyop's fevered imagination: frame-by-frame animations on top of the edited footage, but with depth perception built in. That way, when you tilt our cover back toward you, the headlines would fly into a floating sculpture. For each turn of the first fashion page, you would see a seamless transition to a new animated climate. And Gillian Jacobs — well, she needed help telling her joke about a hunter in the woods.

This fall, Psyop combined its digital video and animations into a textured effects package in Maya 3D software that, once intercepted by The Barbarian Group to interact with your webcam, could be emulated at any angle. "When there's a joke on the printed page, you don't really understand how someone would have a certain timing to tell that joke," explains Psyop's Marie Hyon. "There's so much that you can explore in the visual language that you capitalize on — it's a richer experience."

For the last month, Barbarian's Manhattan office has been a frenzy of math equations, software development, and lots of coffee. "There's a little bit of magic, even if you know how it works," says Barbarian's Andrew Bell (left). "But it's friggin' complicated." As opposed to the limited capabilities of a Web-based Flash experience, Esquire's custom-coded C++ app (download time: five minutes) can talk to your graphics card for a level of AR sophistication that's never been built before.

Designing algorithms to display a transforming weather scene (left) at any angle or depth sounds complex, but "it's matrix math that you probably had to do at some unfortunate moment in your life," Bell says. Barbarian imported Psyop's animated keyframes, then replicated their visual designs as individual scenes. The software recognizes the black-and-white pattern visible to your webcam, which triggers the correct scene. When you turn the magazine and those squares become more like parallelograms, the activated algorithm "sort of plugs in the four sides of each square you're holding and solves the matrix," Bell says.

That googly-looking box on each AR-enabled page of the magazine is referred to as a marker. "The webcam has no idea it's being tricked into doing AR," Bell says, but it provides the algorithms inside Esquire's app environment with images. Those images trigger a number inside the digital-recognition software you downloaded, and suddenly Downey isn't sitting on that box anymore. He's jumping off it.

Tilt the magazine toward your webcam, and the already walking, talking Downey climbs on top of it to sing you a little song. Pull it toward yourself, and he stands under a cloud of letters from the cover. Turn it any which way, and mini-Downey's introduction to the issue (and a trailer for Sherlock Holmes) turn with you. That's no longer animated video you're seeing because of some box, it's augmented data you're interacting with because of... yourself. As Downey says, "it's about the artistry of technology."

In the coming months, Esquire plans to find even more inventive ways of using augmented reality. Advancements to further involve the user were happening even as we produced this issue, and while motion-sensor recognition already exists, so-called "natural-feature tracking" technology could soon put you inside AR without any googly-looking boxes at all. And that's to say nothing of what you can do on your cell phone. But we hope you enjoy this first leap in augmented magazines. It's a giant one.