Kia's general counsel, D. Casey Flaherty, was tired of paying lawyers to fumble around with Word and Excel and Acrobat. So he created a legal tech audit to show lawyers how much of his company's money they were squandering on renumbering paragraphs in contracts. The results were pretty spectacular. Everyone failed miserably, some taking hours longer to complete the audit as Flaherty thought it should take.
Now, if you have $250, you can take the audit yourself. We did. Lisa wrote about her experience with the audit yesterday. Shortly after her post went live, Flaherty emailed us to let us know that he was pretty unhappy with it. So last night, I took the audit myself. Because a lot of companies are probably going to read our post while researching whether or not to use the audit themselves, we are going to clarify and add context to Lisa's review later this week. The gist will not change. In the meantime, here are my own thoughts on what you should take away from our combined experience with the audit.
1. Our experience may be slightly difference than that you would experience if you were taking the audit in a training room at a big firm, which is the intended user experience. We don't actually think this would make a meaningful difference in the experience of taking the audit, but Flaherty thinks it could be an important difference.
2. Without yet addressing the merits of the audit, the audit-as-software itself is highly problematic. It requires Internet Explorer, first of all (although using IE would be an automatic fail in any audit I created), which means it is Windows-only. The interface is weird, too. But most important, the tasks you have to complete aren't always clear. Not in the way that instructions from clients aren't clear, but in a you-can't-complete-the-task-because-the-task-doesn't-make-sense kind of unclear.
My score on the Excel portion is completely worthless, for example, because the instructions were incomprehensible, even though I was perfectly capable of performing the sort of task required. And there is no back button so you can go back over previous instructions to try to get some clarity.
3. Let's assume Flaherty and Suffolk will fix these problems, though, so that the audit tests your technical ability instead of your ability to run and understand the audit itself. From emailing back and forth with Flaherty, it sounds like the audit is really still a work in progress. So assuming the problems are fixed, is it worthwhile?
In theory, absolutely. Lawyers are terrible at technology, and their terribleness wastes clients' money and can even harm outcomes. Getting lawyers to not suck at technology is a worthy goal, and I am fully on board with it. I have been citing Flaherty's audit with approval since I heard about it. But how useful is this audit to clients in assessing the terribleness/inefficiency/wastefulness of a particular lawyer?
I'm not sure.
I didn't really understand my scores, which came as an accuracy percentage (38% accuracy for the Word portion, for example) and a time (12:51 for the same section) that incorporates penalties for incorrect answers to accompanying questions. What I assume clients want to know is did I create an acceptable Word document or not, and did I do it in an acceptable amount of time? I'm not sure those results answer those questions.
I'm going to withhold a final judgment, though. Flaherty says that since the audit is basically still in beta, they are working on refining the scoring against the audit-takers' actual results. And once they get a sufficient sample size, they will include a score that shows performance relative to the average. That should be more useful.
As it is, unfortunately, the audit is painful to take with instructions and questions that are sometimes impossible to follow and answer. That makes the score — whatever it means — useless to lawyers and clients. I hope we will get a chance to take the audit again once Flaherty and Suffolk get a chance to tweak it, but right now I don't think anyone should make important decisions based on the results.