Microsoft enters the marketplace with its futuristic mixed reality headset — HoloLens — which on a first glance looks like a Gadget straight out of a Bond movie. When most tech companies are focusing on developing a Virtual reality headset, Microsoft plans development of its augmented reality headset with holographic technology.
Microsoft foresees future in controlling computing devices with different user interfaces such as gesture, eyes gaze, and voice. Letting the user bridge the screen and physical space and extend the reality around them is Microsoft’s answer to several business insufficiencies. Recently, Ford’s design and engineering team was pictured using HoloLens to visualize a full-scale 3D model of car for building prototype without building it physically every time.
Although, the alternative reality technology that is virtual reality and augmented reality is not new and dates back to nearly 25 years. When Facebook acquired Oculus rift it raised many eyebrows in surprise. People started to talk about VR and AR devices and its implementation beyond the gaming world. Apart from Microsoft and Facebook, Google has also come up with Google glass which failed to lure the market.
Microsoft HoloLens stands at the better position against its rivals. Also, they have strategically defined their market and for now they aim only for that. In 2017, this technology has been much talked because of its limitless applications. We have tried to elaborate Hololens technological features from the perspective of the product as well the patents.
Here, in this article we will be focusing on dissecting the components of the HoloLens and then try to capture a perspective from the patent protection point of view.
Microsoft HoloLens enables users to engage with digital content and interact with holographic objects making it a mixed reality user experience. HoloLen is an all-in-one device that does not require to be tethered with wires, external camera or phone or laptop or PC connection and is claimed to be self-contained.
To simply the technology behind it, we have dissected it into multiple sub-technologies such as:
- Holographic Lens Technology
- Sensor Technology
- Holographic Processing Unit (HPU)
- Interactive Control System
1. Holographic Lens Technology
Microsoft insists on saying that their HoloLens is beyond screen and pixels. HoloLens has high-definition see-through holographic lenses which consist of three layers of glass of different primary colors i.e red, green and blue. And these layers have micro-thin corrugated grooves in them. A light-engine which has light particle bouncing around inside millions of times emits towards these lenses. The light particles hit those layer arrangements and enter the eye at a specific angle, intensity, and color to produce an image on the retina.
Recently, Microsoft was issued a patent disclosing a similar technology – US9759917, which discloses an interactive head-mounted eyepiece which displays content and images on the optical assembly of the eye-piece. Further, the eyepiece includes projecting optics to project an image onto a see-through or translucent lens through which user can also view the surrounding real-world objects. The projecting optics further includes RBG LED module that uses field sequential color technique to display a full-color image. The image is reflected into a two surface freeform waveguide for total internal reflection (TIR) which gives the image on the lens area for the viewer to view the image.
Interestingly, Microsoft pretty nicely sums up all the key components of HoloLens in this patent as you can see below:
2. Sensor Technology
For any VR or AR devices, sensors are the backbone. Sensors perform various activities like head tracking, eye tracking, depth sensing, and environment mapping. These sensors enhance the immersive experience of a user. HoloLens is equipped with advanced sensors and an inertial measurement unit (IMU). The sensor bar on the HoloLens comprises two environment understanding camera on each side, depth camera, ambient light camera, and 2MP photo/HD video camera.
The two environment understanding cameras on each side of sensor bar track head movement and also a time of flight (ToF). The centrally placed depth camera on sensor bar tracks hand gesture and also helps to place hologram on the real objects. Apart from these sensors, it has 4 microphones that better captures the user command and distinguishes it from ambient noise. The other kind of sensors is – gyroscope, magnetometer, and accelerometer together they form inertial measurement unit (IMU) which usually found in any AR head-mounted display.
This wide array of cutting-edge sensors constantly captures the data about the user’s external and internal environments.
Again, we were able to detect a patent from our analysis covering sensor technology issued to Microsoft – US9495801 titled “Pose tracking an augmented reality device” which discloses an augmented reality device with multiple sensors to capture the pose information of the augmented reality device. The augmented device filters out the tracking error from the pose information and further renders a virtual image on the display based upon the function of pose information.
In the context of the below image, the head-mounted display (HMD) device includes a display system and transparent display to project holographic objects to the wearer’s eye. The HMD device is configured with various sensors that can even track the eye or head movement of the viewer and acquire image data based on that.
3. Holographic Processing Unit (HPU)
Last year, Microsoft pulled the curtains over its secretive chip at hot chips conference in Cupertino, California. Microsoft’s holoLens optical sensor uses the power of advanced sensors to enable holographic processing unit (HPU) to work together. The sensor array constantly produces data which is processed by the holographic processing unit (HPU).
The holographic processing unit (HPU) has a custom designed TSMC fabricated 28nm coprocessor that has 24nm Tensilica DSP arranged in 12 clusters. It has about 65 million logic gates; 8MB of SRAM and 1GB of low power DDR3 RAM, all in a 12mm-by-12mm BGA package. Hololens run on window 10 operating system and based on x86architecture. Microsoft claims that HPU has a power to perform up to a trillion calculations per second and accelerates algorithm 200 times faster than pure software alone. In spite of that, it requires power less than 10W to handle gesture and sensing the environment.
Microsoft has aimed to go for the custom-made processing unit to optimize the performance, latency, scalability, flexibility and power supply of the device. Therefore, Microsoft has adopted Cadence’s Tensilica DSP platform for HoloLens. Cadence reserves all the rights for Tensilica and licenses to Microsoft for building and designing the chip.
As Microsoft holds a licensing deal with Cadence, we diverted our analysis for the HPU to Cadence and Tensilica assigned patents and were able to identify US8539399 assigned to Cadence, which discloses a novel computer-implemented method to design a microprocessor through user-defined interfaces. These interfaces allow the user to communicate status or control information between processor and external devices/processors.
The below figure illustrates a user-defined data type interface for configuring a processor design. The user can connect input data type interface and output data type interface through a processor pipeline execution path. The input and output interface connects via external agents to read/write the data, assert the handshake signal and receive the stall condition as defined for each interface.
For the second generation HoloLens, the artificial intelligence coprocessor will be built on its Holographic processing unit (HPU). The data from device’s onboard sensors will be analyzed by HPU using deep neural networks. The release date of the second generation HoloLens is still not on the card but expected to arrive in 2019.
4. Interactive Control System
The goal with which the HoloLens idea was conceived and then implemented by Microsoft, was to create a more natural and immersive interactions for the user to experience. Holograms are responsive with user’s eye gaze, hand gestures, voice and also user’s environment. The built-in sensors allow the user to control the cursor with the eye gaze. The sensors also track the head movement and the cursor accordingly moves.
A patent related to the tacking of eye is disclosed by Microsoft in US9759913. The patent describes a transparent waveguide for eye tracking which includes an input and output couplers. The input coupler has a plurality of curved varying lines that define the radially varying pitch. Further, when an eye is exposed to an infrared light, then the infrared light got reflected from the eye and incident on the input coupler. The incident light enters the waveguide of an input coupler where it propagates and exits through the waveguide by total internal reflection to the output coupler.
There was another publication which we came across, US20160080874 titled “Gaze-based audio direction”, which describes a technology that can assist the user in hearing by a gaze-tracking technique in a noisy environment. This patent resolves the problem of hearing in an environment of more than one source of the sound and allows the user to focus on the target speaker by gazing through the head-mounted display device. The device can track the gaze and assist in hearing the sound of the target speaker.
Sensors do not only keep track on your eyes but they also recognize your hand gestures in order to control any app, select or size the item and even allow to drag and drop the holograms in your environment.
Another Microsoft patent US9116666 titled “Gesture based region identification for holograms” discloses a technique to select a region within the virtual image by using hand gestures. The user can select a region by forming a closed loop with hand and the selected region rendered for zooming, filtering etc.
The figure below depicts an environment in which user wears a head-mounted display (HMD) device to select a region within a virtual image that could be a holographic image. In one embodiment, the HMD device has an image sensor that is forward looking to take the image of the user’s environment and allow a virtual image to coincide with the environment. Thus, it gives an augmented reality view to the user.
HoloLens is also powered by Cortana who can assist you to complete a task like navigate, select, open and control the app with your voice command. The four microphones make it easy to communicate even in a noisy environment.
It also has a built-in speaker so that user can hear the precise sound without the earphone. Microsoft HoloLens synthesize sound for holograms which you can hear anywhere from the room.
Other Salient Sub-technologies and the patents
Microsoft has not limited the scope of HoloLens only with the above described features. There are other interesting technologies that can enhance the performance of HoloLens and enrich the user’s experience. These technologies are described in another patent, for instance, a published application US20170169610 titled “Third party holographic portal”, authorize a third party to display holographic portal via a head-mounted display device in the user real-world environment.
Similarly, another intriguing patent US9390561, titled “Personal holographic billboard” discloses a unique way to share personalized information that showcases on a virtual billboard and displays the virtual billboard on the other’s user augment reality environment.
Microsoft’s patents disclose an impressive way to build a virtual object which has real-world permanence. The best characteristic of HoloLens is to anchor and position the virtual object in the spatial environment. Even Microsoft allows you to fill color to your augmented reality environment.
The future is shaping towards an exciting technology as is evident from one of this year’s patent publication US20170053447A1 titled “Augment reality” this enables augment reality system to create a plurality of local 3D-model of the real-world. How brilliant it will be to be able to experience a virtual world that too more realistically via the 3D-image rendering engine. I am sure as hell excited.
Few More patents in Microsoft’s kitty
Further, to avoid making this article from interesting to boring, we have succinctly put up some of the identified patents. Feel free to read them if it interests you.