On August 31, 2016, the Office of the Inspector General (OIG) of the U.S. Department of Commerce (DOC) released a report titled Investigative Report on Analysis of Patent Examiners’ Time and Attendance, (the Report). While substantial improvements in examination quality, procedures and conduct are necessary, the Report provides no useful or relevant information to further this goal. The Report lacks a sound statistical basis for its conclusions and recommendations. It should be withdrawn and its errors corrected before further dissemination. Otherwise, the OIG’s credibility will be irreparably damaged and it will deserve not to be taken seriously by the public, the U.S. Patent & Trademark Office (PTO), or other DOC operating units.
The Report alleges that substantial examiner time and attendance abuse is occurring and that the PTO “is paying production bonuses to examiners who are possibly defrauding the agency.” Report at 22. The OIG alleges that examiners were paid for 288,479 hours that they did not work, equating “to over $18.3 million in potential waste.” Id. at 3. The OIG alleges widespread abuse—that 5,185 out of 8,067 examiners claimed to have worked some unsupported time. Report at 9.
These and other unjustified assertions of examiner time and attendance fraud are the subject of my detailed petition for correction to the OIG under the Information Quality Act (IQA). In addition to the unjustified harm done to the morale and perceived integrity of the examination corps, if left uncorrected the Report could be used by detractors of the patent system to attenuate the presumption of patent validity by claiming that widespread examiner fraud means that patents are issued after shorter examination time than expected, resulting in inferior patent quality.
OIG’s “findings” are not supported by the OIG’s own analysis
The OIG did not monitor nor audit actual examiner activities and compare them to the time examiners claimed on the PTO’s electronic time and attendance tracking system (WebTA). Instead, the OIG obtained digital “proxies” for work activities from various systems examiners use when entering or exiting their office building or while connected to the PTO’s computer and network systems. OIG’s analysis was limited to examiner-related computer activities that were covered by 66 activity codes. This work activity amounted to an average of 6 hours per examiner work day, leaving an average of 2 hours per day untracked by the study. But total OIG-derived unsupported hours were on average only 6 minutes per examiner work-day.
This minimal discrepancy is well within rounding error and mere misclassification errors across activity codes. Unlike similar previous OIG reports, this Report did not show that the OIG collected or analyzed corroborating evidence for such findings. Nor is there evidence to ascertain these measures’ reliability from OIG-conducted interviews with (i) examiners found to have claimed substantial number of “unsupported hours,” (ii) their supervisors, or (iii) PTO Employee Relations Office that records such discrepancies regularly and thus has experience in the reliability level of such “digital footprint” for hours worked.
The reliability of the “unsupported hours” estimates is further undermined because the OIG relied on low quality indicia of work activity that the OIG admits to having manipulated, ostensibly so as to interpret “the data in the light most favorable to the employees.” Report at 5. But the OIG’s methodology for counting “supportable” hours appears arbitrary and result-oriented. It considered (and could have used) a different badge-out methodology that would have trebled the total number of unsupported hours (Id.), or perhaps reduced this number using a third method. Yet each counting method would have been arbitrary and the Report gives no rational basis for using one method and not the other. The IQA requires that such critical statistics be objective, not settled by mere whim.
Even if the OIG’s method for deriving “unsupportable hours” were objective, it produces estimates having substantial but unacknowledged statistical variance. Several factors likely contributed to the inherent variability—for example, in examiners’ patterns of computer system use, the degree to which they over-report the share of their work day dedicated to any of the 66 activity codes and consequently under-report the rest of the work day in other activity codes. And there is inherent variability due to PTO system errors, which have been known to drop IT and workstation data. In short, “unsupportable hours” is a random variable resulting from the interplay of multiple stochastic processes, yet the Report does not disclose any statistical variance. If the variance is unknown, there is no way to infer that the OIG’s “findings” are statistically significant.
Faced with this key missing information, I reverse-engineered from the Report the mean and the standard deviation of the ratio between unsupported work hours and reported work hours over the entire examiner ensemble. I demonstrate that the difference between the mean “unsupported hours” percentage of ?1.6% and zero is statistically insignificant. This is because the estimated standard deviation for this term is 4.66% ¾ almost three times the reported percentage of unsupported hours.
My analysis shows that the probability p that the null hypothesis (that examiners do not engage in time and attendance abuse) is true given the observed statistics is about 0.36. Traditional practice in hypothesis testing calls for rejecting the null hypothesis only when there is strong evidence that it should be rejected—e.g., if p is less than 0.05. This means that OIG’s allegation that examiners engage in time and attendance fraud is not supported by OIG’s own data and analysis.
The OIG reports “quarterly peaks” and “fluctuation by day of the week” of “unsupported hours” (Report, at 18-20), but conspicuously does not report the variance of these estimates to enable a determination of their statistical significance. Due to the smaller number of observations, the relative variance of monthly or daily tallies of “unsupported hours” must be necessarily higher than that for the entire study period. For some of its results, the OIG even declares “statistical significance” (Report, at 14, 16), but conceals the variance of the results and the significance level probability it used in making such declarations. These “findings” have no credibility in view of the lack of disclosure and given the large unacknowledged variance in OIG’s measures.
The Report creates the unproven impression that the percentages of unsupported hours it derived are examiner-specific; in other words, that the spread in such percentages accurately reflects the actual spread in examiner meritless claims of hours worked. Commentators were thus tempted to conclude that it does not matter that the spread in percentages includes some examiners on the positive side (having worked more hours than claimed) because those on the negative side should be stopped regardless of offsetting contributions by examiners on the positive side.
The problem with this view is that it presumes noiseless estimation. Nowhere in the Report does the OIG show that its estimates are true individual attributes of the examiners in question, and not merely a particular realization of a measure subject to random perturbations inherent in the OIG’s “noisy” measure of the discrepancy in hours, which if measured over a different period will produce different results for the examiners in question. Stated differently, the OIG did not establish that the bulk of the 5,185 examiners that had at least some unsupported time during the 9-month period (Report at 9) are also those that would make up this category of examiners with unsupported time during a different non-overlapping period, say, the 6 months preceding period. Absent such a proof, examiner realizations on the negative side cannot be isolated and the unbiased statistics across the full distribution must be applied.
The alleged extreme time and attendance abusers apparently are also the most productive
In my petition for correction I show that the Report is structurally biased. The Report says that for 296 examiners, 10% or more of their claimed hours were unsupported over the 9-month sampled period. These examiners were paid more than $4.8 million in compensation and bonuses for those unsupported hours. Report, at 10. The activity of these examiners is depicted in Group A in the accompanying probability density plot for the full examiner ensemble. But the Report ignores an equal-sized group of examiners for which more hours were supported than claimed. These examiners’ activities are depicted in Group B and their unpaid activity enriched the PTO according to OIG’s criterion.
Examiners in Group A, who because they claimed 10% or more unsupported hours are deemed by the OIG to be the worst abusers, are also examiners who received above-average, “commendable,” or “exceptional” performance ratings. Report, at 22. Yet it seems highly implausible that top performing examiners working for the PTO are also committing the highest levels of apparent time and attendance fraud on the PTO.
Even if the findings are shown to be examiner-specific, the Report provides no analysis that could rule out more plausible explanations for what the OIG found for Group A. For example, the Report does not disclose whether or how OIG corrected its “supportable hours” estimates upwards to account for time spent when experienced senior examiners discuss and assist junior examiners with their cases. This is an unambiguously beneficial activity that would likely not be captured in either examiner’s “digital footprint” but for which examiners would rightfully claim time. Because experienced senior examiners assist multiple junior examiners, the resulting discrepancies in their “supportable hours” could be disproportionately large.
Instead of resolving this conundrum, the OIG hedges its bets. It stops short of transparently accusing these top performers of fraud. Instead, it characterizes them as guilty of “potential abuse” (Report, at 22). But which is it¾actual, or potential abuse? The OIG is supposed to investigate¾not speculate. Nevertheless, the OIG apparently was not about to let a serious alleged crisis go to waste, whether or not it can be substantiated: the OIG’s top recommendation was predicated on this speculative “finding” as described below.
The OIG says the entire examination corps should perform as if they were all in the top 4%
The Report alleges that the 296 super-performers met—or even exceeded their performance goals by completing their work assignments in less time than allotted by their production goals. Id. From this factoid the OIG concludes that all examiners have more time than they need, so “production goals need revision upwards.” Id.
This makes no sense. First, as noted earlier, the statistical complement to the examiner group with significant amount of unsupported hours (Group A in the plot), is a group of examiners with a significant amount of unclaimed (but worked) hours (Group B in the plot). The OIG ignored this latter group, and contrary to Group A, examiners in Group B appear to spend more time than allotted to meet their production goals.
Second, and most importantly, the OIG recommendation is based on the “above average” and “outstanding” performance of 296 examiners—less than 4% of the examining corps. Production goals, however, apply to the full examining corps and the Report ignores this fact. Even if the Report were correct, nothing in the Report suggests that the other 96% of examiners should (or could) be held to the same performance standard. All examiners cannot be above average, much less perform as well as the top 4%.
To sum up, the OIG Report casts terrible aspersions on thousands of diligent and hardworking patent examiners, and it does so based on flawed data, biased models, and inferences that cannot be supported even by these flawed data and biased models. The errors and omissions of key statistical facts are so blatant that it is hard not to conclude that the OIG crafted its conclusions fist and backfilled analysis to get there.
For more information, read my detailed IQA petition for correction of the OIG Report. Under the IQA, OIG should withdraw and correct the Report within 60 days or give a reasoned basis in writing for its refusal to do so.