Imagine being surrounded by the imagery of your dreams on a panoramic display. Now imagine being able to manipulate these images by pointing directly at them, like a conductor leading an orchestra of pixels. It sounds like something straight out of a science fiction film — which is exactly what it is.
In the Steven Spielberg-directed sci-fi thriller set in 2054, “Minority Report” follows a detective played by Tom Cruise who uses a sophisticated “spatial operating environment” to prevent future crimes. The system allows him to rapidly locate, organize and scrutinize multiple video sequences on a large, immersive display.
To help Spielberg create a cinematic vision rooted in realistic technology, the film hired MIT Ph.D. John Underkoffler as its chief science and technology advisor. A computer scientist with a deep love of cinema, Underkoffler invented a gestural command alphabet and a visual language that could, in theory, be built with existing sensing and visualization technologies. To instruct the actors, he produced detailed training materials, as if the product already existed.
But as the film’s simulated interface became increasingly influential, Underkoffler decided to devote himself to building a working system. He started developing g-speak™, a next-generation computing platform with support for motion-based input devices, flexible device networking and displaying data on multiple screens based on their locations in physical space. In 2007, he founded Oblong Industries to commercialize this technology.
Oblong’s first wave of g-speak systems used motion-capture cameras to track users’ hand positions as they navigated large projected displays. Each finger and hand had to be identified uniquely. To achieve this, Oblong’s VP of Hardware Engineering, Paul Yarin, turned to Protomold. “We needed custom-molded plates to attach motion-capture markers to fabric gloves. Protomold was ideal for its high quality and high speed.”
A team uses a Mezzanine system with multiple monitors filled with different forms of data as they video conference with an outside client.
Oblong deployed dozens of large-scale gesture tracking systems to organizations with advanced visualization needs — defense, oil and gas exploration, and research institutions. But it soon recognized a broader use for its immersive collaboration technology — the millions of conference rooms where people congregate daily to meet, present, brainstorm and implement new ideas. It used the g-speak platform as the basis for a new product called Mezzanine™.
“Mezzanine is a spatial computing environment that lets multiple people, local or remote, concurrently interact with the same data,” explains Yarin. “It allows a fluid sense of being able to manipulate data, and to be immersed in it, unmatched by other computer systems.”
A Mezzanine room is like a conference room on steroids — an environment where video data can be viewed across multiple displays. Mezzanine systems feature a tiled set of flat-panel displays at the front of the room, plus additional screens on one or more walls. Users can share images and video feeds from laptops, phones and video teleconferencing systems; in-room cameras allow instant capture of diagrams from whiteboards. Mezzanine rooms can establish real-time sharing connections such that all participants view and share a common workspace.
“The system allows you to arrange images on screen the way you see them in your mind,” says Yarin. This “infopresence” approach offers a marked improvement beyond video calls. But how could everyday meeting participants, often too busy to practice and to study a new interface, make use of these advanced capabilities?
Oblong drew on its experience with glove-based control to develop a control device for the conference room — a novel, motion-tracked “wand.” The wand provides an intuitive and natural interface, and like the glove-based tracking that preceded it, is responsive and precise. Like a laser pointer, the wand projects a cursor wherever it is pointed. But when held vertically, it becomes a zoom control; when rotated while pointing, special selection modes can be accessed. The wand’s unique design allows it to emulate some of the human hand’s flexibility.
The wand is based on a hybrid ultrasonic-inertial positioning technology. Ultrasonic emitters mounted on the ceiling act as position beacons that the wand detects using tiny microphones. The wand reports its position data via 900 MHz or 2.4 GHz radios, providing input data to Mezzanine’s “Perception” subsystem.
A pair of clear polycarbonate casing halves form the triangular ultrasonic wand. Two black microphones along the spine, a trio of round operational indicators (power, battery, radio link) and a green control button are also shown.
Given Oblong’s positive early experiences with Proto Labs, it was an easy decision to use both its CNC-machining and injection-molding services to realize the wand. Early prototype units were machined out of polycarbonate blocks by Firstcut, and production parts were produced by Protomold. The wand’s shell consists of two clear halves that are painted white on the inside by a separate vendor. The semi-translucent enclosure exposes some of the underlying molded features. The wand’s case was made to be easily opened and closed for access to its circuit boards or Lithium-ion battery. Protomold also produced Santoprene (TPE) control buttons for the wand in various colors, and enclosures for the ceiling-mounted emitters from both ABS and nylon.
“What’s great about Protomold is the speed and material flexibility on a very short notice, plus the automated quoting. We really like being able to quickly vet a design to catch all of the moldability issues,” says Yarin. To date, Protomold continues to produce the thousands of molded parts used in Mezzanine systems worldwide.
As a flagship product, Mezzanine has driven rapid growth at Oblong. To support an increasing roster of global customers, Oblong has built sales offices across the U.S. and is adding a London office to complement another in Barcelona. Mezzanine’s remote-collaboration capabilities make it a good fit for multinational corporations looking to improve their communications. Universities and health-care institutions are also well-suited to make use of Mezzanine.
Could all of our lives be enhanced by seamless collaboration? That’s a future vision that Oblong hopes to make very real.