HP Seeks Patent on Lip Reading Speech Recognition

By Steve Brachmann
January 17, 2014

Headquartered in Palo Alto, CA, the Hewlett-Packard Company is an American multinational corporation focused on developing products and software-based services for the information technology industry. Today marks the first time we have taken a look at HP patents and pending applications, having just added them to Companies We Follow.

The computer hardware manufacturer has been slightly beleaguered in recent years, and it was recently announced that it lost the title of world’s top personal computer manufacturer to Lenovo. Industry experts are optimistic that HP’s plans to renominate its entire board of directors for shareholder approval may help bring some much-needed corporate stability after a year in which HP saw one-quarter of its board change. The company may also make a grand re-entrance into the PC market with its unveiling of an Android-powered desktop to directly challenge Microsoft machines powered by Windows.

In today’s Companies We Follow column, we travel over to the West Coast to see what this Silicon Valley stalwart has been busy producing in its research and development programs. We’ve gone through the recent publications coming out of the U.S. Patent and Trademark Office to find the most intriguing HP innovations found in their patent applications and issued patents. In a world where the PC market has dwindled in recent years, Hewlett-Packard seems to be making strides to shore up its legal holdings for other technologies as well, especially imaging and printing tech, but the featured patent application today suggests HP has come up with software that can essentially read lips.

Our feature patent application today almost sounds impossible, or at least exceptionally futuristic. The application discloses a novel way of using image processing to aid in speech recognition services. The system described in this application could use a camera to analyze a user’s lip activity so that sounds picked up by a microphone can be differentiated as ambient background noise or genuine speech. We also discuss a few other patent applications, including one describing more efficient blade computer system architectures and another that discusses automated methods of delegating resources to individuals using an organization’s computer network.

Patent holdings are of great interest to us in this series, as is HP’s clear focus on printing technologies in their recently issued patents. A couple of issued patents that we explore today discuss improved methods of inkjet printing for pigment opacity or reducing abnormalities in print media. We also were intrigued by a novel system devised to ensure that networked printers handle print jobs with better respect to user printing preferences.

[Companies-1]


Lip Activity Detection

U.S. Patent Application No. 20140010418

Speech recognition technologies have been implemented by companies who want to offer some type of automated customer service to a large audience. These systems are capable of picking up human speech through a microphone and converting that response into digital data. In this way, businesses can provide a vocal user interface for delivering automated services to consumers.

However, there are times when the ambient sound in an environment can cause inaccuracies in speech recognition responses. If background conversations or audio playback continue at a certain volume level while a customer is accessing any speech recognition services, the system will likely not understand the response.

This patent application, filed by Hewlett-Packard with the USPTO, is designed to better differentiate between ambient noise and genuine human speech in computerized recognition systems. The application describes a system that uses image capture and processing to determine lip activity in response to sounds recorded by the speech recognition system. Discrepancies in lip activity and recorded sound would cause the system to identify the background ambient noise and prevent that from affecting a user’s vocal response.

This lip detection system can also incorporate directional audio detection to determine that a particular sound is coming from the customer or from some other part of the room. Even when multiple faces are positioned around a camera system, lip activity of all faces can be analyzed so as to attach speech to an individual user. This system can be implemented in computing systems from desktop computers to handheld mobile devices.

Claim 1 of this Hewlett-Packard patent application would provide protections over:

“A computer-implemented method of detecting lip activity, comprising: determining magnitude of optical flow in lip region of a detected face; determining magnitude of optical flow in at least one non-lip region of the detected face; comparing ratio of magnitude of optical flow in lip region of the detected face and magnitude of optical flow in at least one non-lip region of the detected face against a threshold; and if the ratio is greater than the threshold, recognizing lip activity of the detected face.”

 

Other Patent Applications

Hewlett-Packard is involved with a wide range of computing technologies, from desktops to laptops to printers and even business software services. Here at IPWatchdog, we’re noticing a couple of patent applications in recent weeks that would protect some novel improvements to business software and hardware. For instance, U.S. Patent Application No. 20140002988, entitled Blade Computer System, achieves better architectural flexibility and cooling ability over other blade computer systems by repositioning the cooling fan found on the blade servers. Content management systems for online work may become easier to manage with U.S. Patent Application No. 20140007197, which is titled Delegation Within a Computing Environment. This application describes a system that can automatically change access standards for network users by modifying network preferences, allowing them access to needed resources. The authentication process used for this delegation system is much less cumbersome than previous models and better supports individual user accounts in cloud application environments.

HP is also well known for its printing technologies, and a few patent applications published recently show that the company has been active in this area as well. U.S. Patent Application No. 20140010975, which is titled Inkjet Media, protects a method of printing with water soluble dyes on celluloid-based material in a way that doesn’t warp or otherwise damage the celluloid sheet.

U.S. Patent Application No. 20140000393, filed under the title Chain Positioning System, the  Background explains that inkjet printers typically contain a spittoon used to service the inkjet head. When the inkjet head needs to be cleaned the spittoon is moved to a cleaning location where the inkjet head can be serviced. The invention disclosed would protect a chain system that provides a better method of reaching an inkjet head for cleaning and maintenance. To that end the positioning system disclosed includes a chain that contains a plurality of links joined together, end-to-end, with a plurality of pins. The chain can be moved between a retracted position and an extended position, with the he links of the chain limit rotation about the pins in one direction.

U.S. Patent Application No. 20140000393, titled “Chain Positioning System.”

[Companies-4]
 

Issued Patents of Note

Nothing matters more in the world of intellectual property than the patent holdings amassed by a corporation, as we here at IPWatchdog well know. Considered one of the major founders of Silicon Valley back in the 1960s, HP is always busy bringing technological innovations to the USPTO for legal protections. This week, we’re noticing an overwhelming amount of patents issued recently to Hewlett-Packard that go much further with printing technologies than the few patent applications we’ve cited above.

A number of issued patents we’re picking up on today detail innovations that improve many of the physical materials involved with printing. U.S. Patent No. 8628833, entitled Stackable Ink-Jet Media, protects a system of stacking media sheets that are on their way to printing so that solvent vapors from the ink can be transported away from the sheets before they can cause distortions. U.S. Patent No. 8627638, which is titled Method of Packaging Media Used in a Document Producing Device, protects a wrapper guide that helps a user determine how to properly insert specialized print media into a printing device so that an image or document is printed on the proper side of the media sheet. We’re also intrigued by the new publishing capabilities brought forth by U.S. Patent No. 8617783, issued under the title Electrophotographic Ink, Liquid Toner Producing Methods, and Digital Printing Methods. This patent was issued to protect a method of printing with liquid electrophotographic (LEP) inks so that white pigments appear with more opacity on black or dark-colored media.

Figure 1 of U.S. Patent No. 8619297, which is titled “Printing Using a Profile Based on Printer Capabilities.”

HP has also scored some advantages in printing software technologies for image analysis and simpler user interfaces. U.S. Patent No. 8620092, titled Determining Similarity of Two Images, improves on current systems of determining printing defects in published documents or images. The patent describes a system that cuts down on waste over current systems that will identify a printing anomaly and order reprinting when no anomaly is noticeable to the human eye.

Finally, we decided to look at U.S. Patent No. 8619297, which is titled Printing Using a Profile Based on Printer Capabilities. The Detailed Description explains that “[t]he ability to utilize a variety of computing devices to print to a variety of web enabled printers presents challenges as to printer setup.” With that in mind, this patented innovation protects a system of automatically supplying different user printing preferences in a network environment to a web-based printer.

The Author

Steve Brachmann

Steve Brachmann is a freelance journalist located in Buffalo, New York. He has worked professionally as a freelancer for more than a decade. He writes about technology and innovation. His work has been published by The Buffalo News, The Hamburg Sun, USAToday.com, Chron.com, Motley Fool and OpenLettersMonthly.com. Steve also provides website copy and documents for various business clients and is available for research projects and freelance work.

Warning & Disclaimer: The pages, articles and comments on IPWatchdog.com do not constitute legal advice, nor do they create any attorney-client relationship. The articles published express the personal opinion and views of the author and should not be attributed to the author’s employer, clients or the sponsors of IPWatchdog.com. Read more.

Discuss this

There are currently 13 Comments comments.

  1. angry dude January 17, 2014 10:26 am

    “lip reading speech recognition” ???

    just give me a break…

    Claim 1 is clearly invalid – obvious and not novel: aren’t we humans do similar thing to determine if person is speaking ? computer-implemented :):):)

  2. Anon January 17, 2014 11:36 am

    As is often indicated by my namesake on another popular blog, humans are not machines.

  3. Gene Quinn January 17, 2014 11:52 am

    I agree with Anon.

    First, this is only claim 1 as it was published. This is not an issued patent.

    Second, humans reading lips clearly won’t be prior art to the first machine that reads lips. While we can argue whether the computer recited in the preamble is enough (I think it should be), saying that because a human can do something it is obvious for to get a machine to do it shows a lack of appreciation with respect to how difficult it is to create a useful system like this. Are we really going to say that artificial intelligence is obvious because some humans displace intelligence?

    -Gene

  4. angry dude January 17, 2014 12:27 pm

    Gene,

    they don’t have lip reading speech recognition and will not have it in the nearest future

    this is just a land grab by HP for future patent trolling – IBM is most famous for this

    making something like this to work in practical situations will require tremendous algorithmic effort – many PhDs mental labor years

    patent application does not disclose any of those complex image and signal processing algorithms required to make it work

    i can come up with applications like this every morning before breakfast

  5. Jonathan Stevens January 17, 2014 1:04 pm

    I remember when lip-reading technology and “type & talk” software was a huge trend, but no nobody seems to recall it. Maybe it’s making a comeback, albeit very slowly?

  6. Oneasasum January 17, 2014 9:32 pm

    There is a company in Israel called VocalZoom that has been testing a vision-based speech recognition system:

    http://vocalzoom.com/

    It does something quite a bit more sophisticated than read lips, however. It actually can screen out background noise, which leads to improved recognition rates.

    At the moment, the system looks rather bulky; but perhaps they can shrink it down quite a lot.

  7. Erik January 17, 2014 10:51 pm

    I would suggest to read the ENTIRE patent online before commenting on omly parts of it. Only if
    one has read it completely is it possible to comment about the scope in detail.

    http://ipquants.com/Home/View?page=1&number=US2014010418&Country=us&Search=pn%253Dus20140010418&Kind=A1

  8. Gene Quinn January 18, 2014 9:10 am

    Thanks for the comment Erik.

    You do realize your comment is completely non-informative, correct? The article you are commenting on mentions multiple issued patents and multiple patent applications and the comment does nothing to explain which one you are referring to and why you seem to think the characterization of the patent or application is incorrect. The only way to know what you are talking about is to follow the link you provided. If you want to comment on IPWatchdog.com please refrain from cryptic commentary that has no particular point.

  9. other January 18, 2014 5:36 pm

    As I recall there was a lip reading computer back in 2001. The computer was a HAL 9000.

  10. angry_dude January 18, 2014 11:09 pm

    VocalZoom ?

    sensing vibrations from persons mouth/face to supplement mics signal ?

    Reminds me good old laryngograph used for at least 50 years to measure vocal cord vbrations

    only now they can do it optically cause technology has advanced

    again, it’s far simpler to describe basic concepts in patent application then to come up with a working prototype

    usualy those things are protected with a combination of bs patent with some generic description and real know-how that is keeping the precious details of actual working technology as trade secrets

    you gotta be nuts to put actual working source code of a prototype as an appendix to your patent application

  11. Anon January 18, 2014 11:16 pm

    other,

    If we start letting science fiction in, all hell is going to break loose as prior art will get real complicated with time travel.

  12. Benny January 19, 2014 1:57 am

    Anon,
    I remember reading about a case in which a system described in a Disney animated film was cited as pior art in a patent examination, (referred to a method of raising sunken boats with inflatable balloons), so the gates of hell are already unlocked. (see http://www.iusmentis.com/patents/priorart/donaldduck/ )
    A patent application for time travel has already been submitted to the USPTO (US20090234788).

  13. Anon January 19, 2014 9:39 am

    Thanks Benny,

    My jest regarding time travel had nothing to do with an application filed at the patent office. All types of meaningless applications have been filed (one of my favorites is one for godly powers).

    I am not sure you got the point of my jest though.