USPTO experiences catastrophic failure of electronic patent and trademark systems

uspto-building-angle-335 copyEver since anthrax was mailed to several Capitol Hill Offices in the fall of 2001 the United States Patent and Trademark Office (USPTO) set out to become a paperless agency. On the trademark side of the building a paperless reality was far easier to achieve and has been the reality for a long time. After years of effort, today the patent side of the Office relies on the Electronic Filing System (EFS) for virtually all submissions to the USPTO, including new application filings and back and forth communications between patent examiners and patent practitioners. Extraordinarily few filings are made in paper form, few probably even remember how to do a paper filing. This means the unavailability of electronic information and filing systems at the USPTO is a big deal.

As many in the intellectual property community no doubt already know, on December 22, 2015, at approximately 7:00 pm, the USPTO experienced a catastrophic failure of electronic information systems due to what is being called a major power outage at the Office’s headquarters in Alexandria, Virginia. Initially it seemed that the USPTO was slow to provide information. In fact, according to some practitioners, the blog page dedicated to notify the industry about electronic filing issues was for a time displaying a 404 error message. When information was provided the Office posted only rather nebulous statements on the agency website.

Although I have no reason to question the veracity of the USPTO claim that a power outage caused the problems experienced by the electronic systems, my information is limited to that provided to the public at large. I reached out to the USPTO for comment on Wednesday, December 23, 2015, and was directed only to the updates that were periodically, and rather slowly, being posted to the website.

Could a power outage be responsible for electronic filing systems and information systems being down for nearly a week? Yes, it is certainly possible, but if you read between the lines it seems to me that the USPTO does not definitively know what happened or why.

At approximately 7:00pm on December 24, 2015, I was e-mailed a statement from Patrick Ross, Acting Chief Communications Officer, which was also posted to the USPTO website. In part it reads:

Power that comes into the USPTO’s main building feeds two power filtration systems that provide steady, “filtered” power so systems don’t suffer from damaging surges or drops in power supply. A malfunction in the power supply lines feeding these two systems caused significant damage to both systems. This is what we believe caused our systems to go down on Tuesday night.

Because of their size, these large and highly complex power filtration systems cannot be easily replaced. We are working with service providers to obtain a source of uninterrupted conditioned power to the data center as soon as possible.

As far as I can tell there is no updated explanation, or more detailed explanation, as to what caused this massive power outage. Thus, we are presently left with a belief that it was caused by a malfunction in power supply lines that caused two different filtration systems to suffer some kind of catastrophic failure. But what caused the malfunction?

Given the importance of the USPTO electronic systems why didn’t the USPTO have real, robust, and redundant contingency plans? Why did they choose not to answer questions? The one question I specifically posed was whether the Office could confirm that no information or files were lost. I have not yet received an answer to that question.

The USPTO electronic systems have been antiquated for years, difficult to use in any real world sense of what is acceptable in a 21st century e-commerce digital age. I have often said in private conversation that if Amazon.com operated with all the peculiarity of the electronic filing system (EFS), for example, the first thing the name “Amazon” would conjure up would be a river or jungle in South America, not the most dominant e-commerce company in the world. Sure, for those who use EFS everyday you become accustom to the quirks, the problems, and even the errors. But it shouldn’t be so difficult to send a file across the Internet to an agency that is supposed to be the hub of all things innovation.

The USPTO computer systems are widely known to be extremely fragile. For example, under the Kappos Administration Google accepted a no-fee contract to scrape data saved only in image files from the USPTO servers, converting the data to usable and readable text files. So fragile was the USPTO system that Google could only scrape data for several hours in the middle of the night due to concerns that additional traffic would cripple the system. Even further, each night Google would encounter numerous errors.

It is difficult to believe, but the USPTO computer systems for a variety of reasons have a long history of being woefully inadequate. EFS has no real back up system, which is a enormous problem when you have statutory deadlines and the EFS system goes down, and down it goes frequently. The USPTO will no doubt quibble with that previous statement, but based on the USPTO’s on answers to frequently asked questions, EFS Contingency is a contingency only in a Monty Python sort of way. According to the USPTO, filings cannot be made when EFS-Web is down for maintenance because “EFS-Web and EFS-Web Contingency share a common backend system. During scheduled maintenance activities, both EFS-Web and EFS-Web Contingency need to be brought offline to properly complete system maintenance.” That means the USPTO has a back up server that is guaranteed to go down when the main server goes down. To call that a contingency plan is insulting.

Over recent years the monumental task of updating and enhancing the computer systems has been undertaken. The Office has said that they have made exceptional strides in stabilizing their technology infrastructure, and the sad part is that they really have. Of course, when a power outage can cripple the Office for nearly an entire week a legitimate question needs to be raised about the integrity of the USPTO computer systems.

The good news is that the USPTO did declare that it will consider each day from Tuesday, December 22, 2015, through Thursday, December 24, 2015, to be a “Federal holiday within the District of Columbia” under 35 U.S.C. § 21 and 37 C.F.R. §§ 1.6, 1.7, 1.9, 2.2(d), 2.195, and 2.196. That means that any action or fee due on these days will be considered as timely if the action is taken, or the fee paid, on the next succeeding business day on which the USPTO is open (37 C.F.R. §§ 1.7(a) and 2.196).

EDITORIAL NOTE: Whether Director Lee has the authority to declare a federal holiday has come into question. The Director clearly can declare an emergency under § 21. For more on this please see The USPTO Director can legally extend filing deadlines for emergencies.

When the USPTO last updated its website on this catastrophic outage the Office reported that it would post another notice if the USPTO’s systems are not fully operational by Monday, December 28, 2015. As of the publication of this article no additional notice has been published, but the USPTO website continues to open with a warning notice that ends thanking the community for patience as the Office attempts to resume normal operations. Some operations are reportedly back on line, while others remain unavailable. Stay tuned.

Share

Warning & Disclaimer: The pages, articles and comments on IPWatchdog.com do not constitute legal advice, nor do they create any attorney-client relationship. The articles published express the personal opinion and views of the author as of the time of publication and should not be attributed to the author’s employer, clients or the sponsors of IPWatchdog.com.

Join the Discussion

34 comments so far.

  • [Avatar for Eleanor Meltzer]
    Eleanor Meltzer
    December 31, 2015 10:46 pm

    To Night Writer@31 & A Rational Person@32:

    I appreciate the points you each made and agree that common sense combined with existing information would have been adequate to identify both need and site location. Perhaps now the time will be right for USPTO to establish one or more backup sites.

  • [Avatar for Anonymous]
    Anonymous
    December 31, 2015 08:05 pm

    This presents a lot of pertinent information, I would think:
    https://govtribe.com/project/emergency-power-risk-assessment

  • [Avatar for A Rational Person]
    A Rational Person
    December 30, 2015 10:06 am

    Night Writer@31 Further to your point and Gene’s comments about Amazon, I would imagine it did not take a “study” for Amazon to decide it would be a good idea to have several backup locations for its data and operations scattered around the country. In fact, Amazon is in the backup and recovery data business:

    https://aws.amazon.com/backup-recovery/

    Wonder if the USPTO “studies” considered using existing commercial services that actually do this sort of thing for a living. But then again, all this backup and recovery technology is just abstract stuff that has no real world application. Just ask the USPTO.

  • [Avatar for Night Writer]
    Night Writer
    December 30, 2015 07:47 am

    Eleanor @28:Wow. Studies? Seriously! Any IT person worth their salt can figure all that out in a weekend without a study. This is the kind of stuff that was worked out pretty well 15-20 years ago.

  • [Avatar for step back]
    step back
    December 29, 2015 02:23 pm

    Rational Person @26
    Love your sarcastic humor style. Keep it up.
    No better way to criticize TPTB than with humor.

    ROFLMAO

    p.s. Still like your bit about how Justice Kennedy should have sent over his 2nd year engineering student to fix the e-commerce problem over the weekend! Genius.

  • [Avatar for Eleanor Meltzer]
    Eleanor Meltzer
    December 29, 2015 01:28 pm

    Further to my comment, the findings (both times) from the USPTO failover study were for the greater San Antonio, TX area – extending north to San Marcos, TX – but not actually up to Austin, TX.

  • [Avatar for Eleanor Meltzer]
    Eleanor Meltzer
    December 29, 2015 01:18 pm

    Gene –

    You are absolutely right to raise the question, “Why didn’t the USPTO have real, robust, and redundant contingency plans?”

    There are certainly a lot of answers, but I know – from personal involvement – that USPTO, aware of the significant risks posed by its single-point-of-failure reliance on a locally and regionally risky power source, conducted two significant studies (around 2006-2007, if memory serves) to find a “failover” location. Actually, the plan was for more than a failover system. The intention was full redundancy and load balancing, so the second site would also be an active location for USPTO data storage and processing. Criteria for the “failover” site included complete independence from the Eastern Interconnection, inexpensive real estate, geologic stability, general weather stability, and access to a local high-tech labor pool. In both studies, based on these general requirements, Austin, Texas was identified as the best location for an alternate USPTO data center. If USPTO still has the data from these two studies, such information might provide a cost-effective head-start on identifying a working failover location.

  • [Avatar for Gene Quinn]
    Gene Quinn
    December 29, 2015 01:00 pm

    @Winter is exactly correct, and why attorneys absolutely MUST still remember how to file in paper form to ensure filing dates when necessary. EFS always seems to be down when you need it most, and this latest episode doesn’t exactly instill confidence in the PTO electronic systems.

  • [Avatar for Anon]
    Anon
    December 29, 2015 12:22 pm

    ARP,

    I realize that I am preaching to the choir, but your views resonate with my recent post on the “Amici Ask Federal Circuit to Curb Misapplication of Alice” thread (post 12), which in part states:

    Because the requirement of a machine in a claim means that said machine may be “abstract.”

    Welcome to the ghost spirit world where realness is not real. Hard goods necessarily in a claim have been made to vanish into the nothingness of being “abstract.”

    Because the Court says so.

    And on top of that we have attorneys intent on NOT SEEING the problem with this….

  • [Avatar for A Rational Person]
    A Rational Person
    December 29, 2015 11:04 am

    Gene@21

    I respectfully disagree. A paper or fee that can be deposited in the the mail is something “tangible”.

    In contrast, as we have been repeatedly reminded by the Supreme Court, the Federal Circuit, various District Courts and the USPTO itself, the USPTO computer systems, including its electronic filing system, are merely an abstract idea, i.e., they involve generic computers running software and even implement business methods.

    So if the EFS system would be considered “abstract” and excluded under 35 USC 101, it should certainly be considered “abstract” and excluded under 35 USC 21(a) which is clearly directed only to tangible things being deposited with the US Postal service.

    Also, as a general principle, the language of 35 USC 21(a) is exactly the sort of thing that should have been updated to deal with today’s technology has the AIA really been about “patent reform”.

    Isn’t it amazing how “real world” and “tangible” how all of that “abstract” computer stuff, software stuff and business method stuff is when it goes down?

    Also, isn’t it amazing how when the computer systems go down, it seems to take more than a weekend’s work by a second year engineering student to fix them?

    I have “zero” sympathy for the current administration at the USPTO, who have repeatedly denigrated the work of software professionals and computer professionals in how they have trivialized the inventions in software patent applications and business method applications.

    However, I have much sympathy for the hard-working Examiners, inventors and patent professionals who are having to deal with this problem

  • [Avatar for Night Writer]
    Night Writer
    December 29, 2015 10:58 am

    Plus the PTO should waive the $400 paper filing fee for those of us that turned to paper filing to be sure we got the right date.

  • [Avatar for Night Writer]
    Night Writer
    December 29, 2015 10:57 am

    There is a difference here. Holidays are not “the data … or would have been deposited with the United States Postal Service but for postal service interruptions or emergencies designated by the Director.”

    This provision basically says the director can back date stuff because of emergencies. So, I agree the Director could say that anything filed electronically on the 23rd-28th will be treated as filed on the 22nd. I think that would be highly unlikely to fall in court. That it appears she has authority for.

    But, Holidays do not do that. Holidays say that it is OK to file later because of Holidays. Your provision of back dating would fix a host of problems with foreign dates and quarterly deadlines etc. The Holiday solution does not. If your filing date is the 26th at the PTO then it is the 26th in Japan. The holidays don’t slip it back to the 22nd.

    Not the same thing. Declaring a holiday is not the same things as declaring an emergency and back dating.

  • [Avatar for Winter]
    Winter
    December 29, 2015 10:50 am

    The US system is now first to file, and so defining days as holidays won’t help those trying to get the earliest filing date

  • [Avatar for Gene Quinn]
    Gene Quinn
    December 29, 2015 10:19 am

    EG, Victor-

    I don’t believe those who are questioning the authority of the Director are correct. Frankly, I didn’t give it any real consideration. Not because I didn’t think about it as an issue, but because I specifically did think about it and deemed it appropriate.

    35 U.S.C. 21(a) says: “The Director may by rule prescribe that any paper or fee required to be filed in the Patent and Trademark Office will be considered filed in the Office on the date on which it was deposited with the United States Postal Service or would have been deposited with the United States Postal Service but for postal service interruptions or emergencies designated by the Director.”

    Generally this rule has been used by the Office to extend time when there has been a natural disaster that has made filing by mail impossible. Think earthquakes or hurricanes where Post Offices in certain areas were destroyed. Generally the rule has been applied to certain zip codes only.

    It is true that MPEP 511 and the CFR seems to be written with an eye toward the emergency being mail related, but in my opinion the statute could easily be interpreted to give the Director the authority to identify a different type of emergency.

    -Gene

  • [Avatar for EG]
    EG
    December 29, 2015 10:03 am

    “The good news is that the USPTO did declare that it will consider each day from Tuesday, December 22, 2015, through Thursday, December 24, 2015, to be a “Federal holiday within the District of Columbia” under 35 U.S.C. § 21 and 37 C.F.R. §§ 1.6, 1.7, 1.9, 2.2(d), 2.195, and 2.196. That means that any action or fee due on these days will be considered as timely if the action is taken, or the fee paid, on the next succeeding business day on which the USPTO is open (37 C.F.R. §§ 1.7(a) and 2.196).”

    Hey Gene,

    As NW correctly observed, and others like David Boundy who I respect have also opined, the statutory authority for the USPTO to declare December 22, 23, and 24 subject to the Holiday rule is problematic at best and non-existent at worst. As Boundy astutely, that issue will be litigated for any patent that has to rely on an effective filing date based on the USPTO’s problematic/non-existent application of the Holiday rule.

    What should be done by Congress are two things: (1) a “nun pro tunc” law that approves the USPTO’s application of the Holiday rule to December 22-24 (and any other day where EFS is deemed bythe USPTO to be faulty); and (2) give the Director of the USPTO the clear STATUTORY authority to apply the Holiday rule in the case of such emergencies. Nothing else will give those who relied upon the USPTO’s application of the Holiday rule the “peace of mind” they should be entitled to.

  • [Avatar for Victor Suarez Rovere]
    Victor Suarez Rovere
    December 29, 2015 09:50 am

    Does the USPTO have authority to declare holydays? In this article that’s questioned and open to future litigation:

    https://blog.oppedahl.com/?p=1157

    Maybe the USPTO needs confirmation from the President, or declare also the office closed in such days.

  • [Avatar for NE Kramer]
    NE Kramer
    December 29, 2015 07:49 am

    Most big commercial data centers have backup generators and some even have a generator to back up the backup generator. Why does the US PTO not have this?
    Do they even have a disaster recovery plan and some kind of backup site like maybe the cloud? As someone who has been a data center operations manager this seems like gross incompetence to me at least from what I would expect in the commercial world.

  • [Avatar for Cal]
    Cal
    December 29, 2015 07:28 am

    Since you are bashing the PTO and its infrastructure, I just want to comment on the systems of the USPTO that I use on a daily basis. The stability of the EAST workstations in the Public Search Facility are vastly improved and very reliable. The PSF hosts a quarterly meeting with public users and the CIO John Owens attends the meeting to discuss search and IT issues. The guy is legitimately trying to bring the whole PTO IT systems up to modern standards. I will stay out of the budget debate around the PTO, but there are smart and talented people at PTO making things better.

  • [Avatar for Benny]
    Benny
    December 29, 2015 05:57 am

    According to Ross’s statement, the fault was in the power supply external to the USPTO building (high voltage transformer, perhaps). How many other government agencies in DC are fed power from identical infrastructure and thus equally vulnerable?

  • [Avatar for Jeff]
    Jeff
    December 29, 2015 01:04 am

    Night Writer, it’s completely unfair to blame Lee for this “because she is the boss.” You’re simply wrong: Google is the boss.

    Public Pair is still down today, Dec. 30. It redirects to http://www.uspto.gov/blog/ebiz/ that discusses the status of the website, but doesn’t say what the status is. I see nothing from the USPTO admitting that anything is actually still down right now.

  • [Avatar for Thinking Out Loud]
    Thinking Out Loud
    December 28, 2015 10:39 pm

    I don’t want to sound conspiracy theory alarms, but…

    In addition to damage to the electrical filters, some unrelated power sources were also damaged. There were no reported power outages or grid issues in Alexandria on Dec. 22. Damage to only high-voltage systems, all within a limited proximity, without any other symptoms elsewhere, starts to look an awful lot like an EM pulse attack.

  • [Avatar for Night Writer]
    Night Writer
    December 28, 2015 10:05 pm

    And, their half remedy of extending the holidays stinks. It doesn’t address all the problems and it isn’t clear they had the authority to do that. What does that mean? It means if you rely on that holiday extension, it may come back to bite your client in a law suit. So, it lessens the value of any issued patent that relies on the holiday extension.

    Sheesh. I wish Obama would have appointed a director who actually had experience prosecuting patents. Lee is responsible for this, because she is the boss.

  • [Avatar for Examiner]
    Examiner
    December 28, 2015 08:47 pm

    The true story has not come out. It would seem that every practitioner before the PTO should not only be calling their U.S. Representatives and Senators, but also the news outlets. This is at least a $20,000,000 mistake (given 20K personnel idled and repair costs) by an agency covering up its own problems by exerting power beyond its statutory limits (declaring a “federal holiday”) that put at risk any pending patents (allegedly the most important contribution of the US to the world these days).

  • [Avatar for Night Writer]
    Night Writer
    December 28, 2015 08:25 pm

    They should waive the $400 paper filing fee. We had to go to paper. 1) They didn’t extend the holiday until after the 23rd. 2) The holiday extension doesn’t help with foreign dates and with client end of quarters and end of year.

  • [Avatar for Lou]
    Lou
    December 28, 2015 08:21 pm

    The USPTO doesn’t spend nearly what it should on software systems. I could develop a robust architecture in about 3 weeks and implement a workable system in about 6 months. These people are fools.

  • [Avatar for step back]
    step back
    December 28, 2015 07:26 pm

    Cowboy @5

    Guess what?

    For those of “us” who have “Private” PAIR, the system is working.
    Just you inconsequential people in the “Public” are being kept out from government resources your tax dollars pay for.

    Long live the oligarchy. 😉

  • [Avatar for step back]
    step back
    December 28, 2015 07:23 pm

    MDT.

    Ha ha.
    Someone I know is a nurse at a hospital.
    The alarms go off all the time.
    Everyone ignores them.

  • [Avatar for Simon Elliott]
    Simon Elliott
    December 28, 2015 07:22 pm

    I was able to get on private PAIR today (Dec 28) and it showed recent updates, so at least some aspects of the PTO system must be working

  • [Avatar for MDT]
    MDT
    December 28, 2015 05:41 pm

    I’ve been in IT for over 20 years, and my wife is tech support (going on 5 years now). I’ve been in secure buildings all over the US working for government contractors, private corporations, and companies that are vital for our infrastructure (such as gas plants, etc).

    Anyone not in this industry would be utterly shocked at just how old and fragile a lot of the computer equipment is that’s running vital applications every day.

    And it’s not just that, a lot of times the people overseeing even the up to date tech are technological neophytes.

    Case in point, my wife took a call from a client about a month ago. Their entire system was down, they couldn’t process anything. Nothing would work. She tried to access their system remotely, and nothing was showing up. She happened to hear a high pitched buzzer in the background, and recognized it as the emergency backup battery system. The sound it makes when one of it’s batteries has died. She asked about it, and the person in charge of their server room said, and I quote, “Oh, yeah, that, it’s been doing that for a couple of months now, we just ignore it, nobody’s quite sure what it’s for.”.

    They had lost one of their batteries 6 months before, and the other failed 3 months after. They’d ignored the buzzer for 6 months, and then one morning, they had a power failure. The server farm crashed spectacularly, and the drives were garbaged. Took them a week to get back up and running from backups.

    The thought that someone has a power system that had some issue that was not looked into because nobody remembered what the little red light and buzzer meant doesn’t sound so hard to believe anymore does it? 🙂

  • [Avatar for American Cowboy]
    American Cowboy
    December 28, 2015 04:43 pm

    Ruh-Ro! Pair seems to be offline again, this Monday afternoon.

  • [Avatar for Gene Quinn]
    Gene Quinn
    December 28, 2015 04:37 pm

    AC-

    I’d cite to the FTC for certain things. I understand that in our little area they can be very political when it comes to patents and patent policy, but when it comes to going after companies and individuals engaged in deceptive or false advertising they seem to work like a well oiled machine. But even that example sort of proves the point.

    What is shocking is how long this PTO computer system problem has gone on and how bad it really is. That the PTO is supposed to be the agency in charge of innovation just adds a layer of irony.

    -Gene

  • [Avatar for American Cowboy]
    American Cowboy
    December 28, 2015 02:18 pm

    Gene, why should we be surprised? PTO is a government agency and Government agencies are virtually always slower, more expensive, more complicated, and more prone to breaking down than the private sector.

    I would be interested to see if anyone can post here the name of a government agency for which that is not true.

  • [Avatar for Gene Quinn]
    Gene Quinn
    December 28, 2015 01:03 pm

    Step-

    Yes. Justice Kennedy and is second year engineering intern to the rescue!

    -Gene

  • [Avatar for step back]
    step back
    December 28, 2015 12:58 pm

    Gene,

    A Rational Person already joked over on Patent Docs, the solution is easy.

    Call up Justice Kennedy and have him send over his 2nd year engineering student to the PTO building. You know, the one that can code up any e-commerce idea over the weekend. My idea: Fix it.