Software Forensics: Qualifying Tools and Experts Who Use Them

In part 1 of this article I discussed software forensics, generally what it is and why it’s needed. One of the big reasons there is such a need for software forensics is to interject objectivity into what is otherwise a battle of experts who are supposed to be unbiased but who may be strongly influenced by, if not outright pressured to support, the positions of their clients. This is just as true of experts in other areas of litigation, but as more complex technologies are at issue in today’s IP cases, lay judges and juries are less capable of weeding through technical intricacies to weigh opposing views of experts. Compounding this reality is the ever increasing popularity of police dramas on television, which elevate the desire for juries to have some kind of objective information they can rely on; something of a smoking gun if you will. Software forensics can often provide that smoking gun and cut through the haze. But the question remains, how do we assure that software forensic tools are reliable and consistent and that the expert witnesses who use them are qualified and honest about their analyses?

Below are a few ideas about this, though each one carries with it potential problems. Perhaps not all of these ideas can definitely be implemented, but if we could insert some or all of them into the current legal system, we might have just results a higher percentage of the time. And applying these ideas to criminal cases might be a particularly good idea, where an expert’s opinion can be the difference between life and death for a person accused of a crime.

 

Certification

Certain states require that experts be certified in a field of engineering before being allowed to testify about that field in court. My understanding is that few states require certification, and it’s rare in those states that an expert is actually disqualified from testifying due to lack of certification. Perhaps if certification were required, there would be fewer “experts” who are simply looking for ways to make some extra money on the side. Similarly it might be more difficult for attorneys to find “experts” who support their case only because they’re not sophisticated enough to understand the technical or legal issues in depth.

One important question would be who runs the certification program. I imagine there might be some competition among organizations to implement the certification. There are definitely organizations available, such as the Association for Computing Machinery (ACM) and the Institute of Electrical and Electronics Engineering (IEEE) that could set certification standards for computer scientists and electrical engineers respectively. Other engineering groups could set standards for their own engineers. Perhaps the American Bar Association (ABA) or the American Intellectual Property Law Association (AIPLA) as well as state and federal government offices could also be involved.

A very important question would be under what circumstances would certification be revoked? There would need to be a hierarchy of actions and ramifications that ranged from fines to revocation. In reality, many penalties less than revocation would almost certainly result in the end of an expert’s career. Few attorneys would want to put an expert on the stand who had a record of having been found to be unqualified or dishonest. Also, would any behaviors lead to criminal charges against the expert? Perhaps unethical behavior in a criminal trial should carry stronger punishment, including criminal charges, than similar behavior in a civil trial. There is already the Daubert motion that can exclude an expert from a case, but such a motion relies upon the judge to make a decision rather than a knowledgeable certification organization.

There should be a no tolerance policy for dishonest, unethical, or illegal behavior by an expert. At a conference I once attended on digital forensics, a professor talked about a student who cheated on a test. The professor discovered the cheating and confronted the student. The student was sufficiently remorseful according to the teacher (in my experience most criminals are remorseful once they’re caught) and so the professor gave the student a second chance. This was simply a wrong decision. Remember that digital forensics is the study of sophisticated ways to hack into computer systems. This professor could very well be training a future criminal. Unfortunately only about half of the faculty members at the conference agreed with me, and some of the colleges had no official policies regarding cheating. All forensics education programs must have zero tolerance policies, in writing, and any certification program must too.

One issue that’s certain to arise is what to do if no certified expert in a particular field is available to work on the case. Perhaps the technology is very new or specialized. Or perhaps all of the certified experts are conflicted out or simply have no time. It seems that a judge could create an exception, allowing someone with experience in the field to testify in cases where certified experts are not available.

There may be resistance to a certification requirement by many experts themselves because they’re already earning a living that they wouldn’t want to interrupt in order to study for and take a test that they feel is unnecessary. I also used to think the certification was unnecessary, but seeing the shoddy or unethical work of some experts, I’m changing my mind. The government requires that a lawyer pass a bar exam before practicing law, yet the government requires no such similar test for experts despite their importance to the legal process.

 

Neutral Experts

Another way of dealing with this problem is to require neutral experts. Neutral experts are either contracted by the court or they are jointly contracted by the parties in the case in which case their costs are shared by both parties. Currently, there are typically two situations where neutral experts are used. One situation is where the judge decides that the issues involved are too complex for the judge or the jury to understand without an expert in the field to explain it. Another situation is where the parties agree on a single expert to do analysis and come to a conclusion. Hiring only one expert saves time and money in coming to a resolution, and it gives each party a limited ability to persuade the expert. Perhaps neutral experts should be required for every case. The parties could split the cost of the expert, or the loser could be required to pay the cost. This seems to me to be a good solution particularly if the neutral expert has been certified in his or her area of expertise. However, one drawback of having a neutral expert, that should be considered carefully, is that a biased neutral expert, or one whose skills are less than ideal, could draw an incorrect conclusion and there would be little ability for a party to challenge it on technical grounds. Of course having a neutral expert does not preclude the possibility that each party could additionally employ its own expert, though this might further obscure the issues rather than clarify them, given that there could potentially be three different opinions.

 

Testing of Tools and Techniques

It also seems that tools and techniques used by experts should be tested and certified by an official body. I’ve encountered instances of experts using the wrong tools, either unintentionally because they didn’t really understand what the tools did, or possibly on purpose to confuse the issues and get ambiguous or simply wrong results that actually obscured the facts. It seems that it would be good to require tools be tested, that their results be rigorously verified, and that experts be certified in the use of the tools before testimony can be introduced in court that relies on the results of the tools.

For example, my company Software Analysis and Forensic Engineering (S.A.F.E.) produces the CodeSuite® set of tools that have been tested extensively and used in over 80 court cases to date. We also offer certification training so that lawyers and litigants can be assured that the experts they hire are well-versed not only in the use of CodeSuite but also more generally in software analysis techniques and software intellectual property issues, and are skilled in writing up and presenting these results in court. By interjecting this type of objectivity into software analysis should improve results.

 

Conclusion

The field of software forensics is a very young field of science and provides great opportunities to be a pioneer. The field is growing in importance as software becomes a major component of all of the modern devices that we use and that affect the world. The value of software forensics is obvious from the value given to software development and software intellectual property as reflected in the market valuations of software companies and the stakes in software IP disputes.

Because of the importance of software, it is crucial that software litigation be as precise as the technology at issue. For this reason, I’ve proposed the use of certification programs for software forensics experts and for software forensics tools used in litigation. The ideas presented here are not fully formed, but I believe they deserve strong consideration and I hope others will write more extensively about them and consider ways of implementing them within our current legal system.

Along these lines, S.A.F.E. has recently begun a new project in partnership with the University of Warwick UK that we call the Depository of Universal Plagiarism Examples (DUPE). The project will bring together universities and corporations to contribute original code and plagiarized code to an online, publicly accessible database. We plan to use this database to create a definition of what it means for code to be “plagiarized,” perhaps requiring different definitions within the legal world and academic world. Organizations will be able to test various copy-detecting algorithms and tools against the database and be judged according to standardized benchmarks. These results will allow the comparison of various software comparison tools. It will allow algorithms to be honed and tools to be made more accurate so that their use, particularly within the legal system, will be more acceptable. If you’re interested in contributing effort or funding to this project, please contact me. This important project will have repercussions in computer science and law for years to come.

In conclusion, by using quantifiable algorithms, objective data, and industry standards, software forensics will yield much better results that are less likely to be overturned on appeal.

Share

Warning & Disclaimer: The pages, articles and comments on IPWatchdog.com do not constitute legal advice, nor do they create any attorney-client relationship. The articles published express the personal opinion and views of the author as of the time of publication and should not be attributed to the author’s employer, clients or the sponsors of IPWatchdog.com.

Join the Discussion

One comment so far.

  • [Avatar for tifoso]
    tifoso
    November 16, 2014 11:20 am

    It is unclear to what extent British or other systems of court rules are similar to those in the US but the Federal Rules of Evidence permit persons to testify as experts, but here there is no requirement of certification. All too often the term “expert” is applied to persons solely due to some academic achievement. A farmer who has not completed tenth grade can know much more about Charolais steers and their behavior than all the Ph.D’s at the state university. Would your plan of “certification” preclude that farmer from testifying. With respect, prior to 1969 no colleges offered programs in Computer Science. Many very good programmers majored in music, French, or other subjects. My Ph.D. adviser in Artificial Intelligence had a Ph.D. in Physics. One of the best software techies I ever knew majored in Chemistry. Bill Gates dropped out of college. Would your certification plan even allow these people to take the exam? And what would that exam cover? What if the code in question was written in Snobol or 1966 ANSI Fortran? How would a standardized exam certify someone in those languages?

    Some years ago I was the attorney/computer expert on a huge Cobol system. I never wrote Cobol programs but knew enough about it to find the “smoking gun” very quickly, resulting a multi-million dollar settlement for our clients. Would your proposals have permitted me to even be on the team?