Malleus Maleficarum, or The Witches' Hammer.

All too often, management denies responsibility for causing or otherwise contributing to a problem of which their organization, policies, or procedures were a cause, if not the cause. Even when presented with facts supporting this conclusion, management may refuse to accept those facts, and, preferring illusion to reality, attempt to "spin" the facts so that they refute, rather than support, an auditor's conclusions.

Some organizations like to believe they are playing "hardball" with the auditor by doing this, or that they are in open negotiations with the auditor about the findings they are willing to accept. This is part of the audit reaction, more about which is documented elsewhere.

In such situations, I reluctantly employ a technique specifically designed to systematically refute any and all assertions that a problem does not exist. I call it the Malleus Maleficarum, or The Witches' Hammer. For example:

During an audit, I was asked to evaluate records which were required to be delivered to the customer at turnover, an event during which a completed portion of construction is delivered to the customer. Specifically, I was asked to determine if the database used to store and electronically reproduce records of construction was complete and accurate, in accordance with the requirements of the client's contract with the customer.

The use of a database to store and electronically reproduce records was a new process developed by the client and presented to the customer as a cost-saving measure, and which would ensure greater accuracy and a finer degree of information granularity than the traditional process, which was documenting information by hand using a printed form. As such, I viewed it as a process improvement.

As part of their agreement, the client was required to perform an independent and objective audit of the new process and report the auditor's conclusions to the customer. The customer requested I perform the audit. As a result, although I was effectively being paid by the client, I was really working for, and reporting to, the customer.

After developing the state map, I reviewed records stored by the database. After a brief review lasting one day, I documented approximately a dozen records which were not in compliance with the requirements of the client's contract with the customer, and were therefore discrepant, out of a sample of approximately thirty chosen at random, out of a total lot of over one thousand. Each discrepant record had one or more problems requiring correction. I grouped the problems into findings for which each discrepant record represented a single observation.

I reported over a dozen unique findings. For example: some required records were missing from the database, some records stored by the database were incomplete, and the information documented by the database for some records was incorrect.

Some of the findings were more serious than others. For example, because of the particular way valves are numbered, it was possible to determine the information documented by the database was a typographical error caused by errors in data entry. In other cases, a particular kind of valve with specific performance characteristics and maintenance requirements was misrepresented as another kind of valve. The potential consequences of such an error were significant, potentially representing tens of thousands of dollars in increased cost and loss of customer confidence.

I did not perform a comprehensive review of the database because sufficient objective evidence existed to support a conclusion that the entire lot of over one thousand records was rejectable, i.e., that it was statistically likely that the number and type of discrepancies in the sample were representative of the number and type of discrepancies in the lot as a whole. I intended to report a rejectable lot which would require the client to perform a detailed and thorough investigation. This is standard practice; it is not the job of an auditor to provide quality control on behalf of the client.

Because the use of a database to store and electronically reproduce records was an excellent idea, I wanted to encourage it, but the database should1 have contained all required records and those records it contained should be complete and accurate. I concluded that the new process did not ensure greater accuracy and a finer degree of information granularity than the traditional process and that any cost savings asserted by the client were therefore unverifiable. I reported my findings to the client and the customer.

When presented with my conclusion and given the opportunity to perform a detailed and thorough investigation, the client asserted that the contract did not require the records to be turned over as portions of the system were being turned over, but rather when the system as a whole was turned over.

Further, the client stated that the sample of approximately thirty records which I reviewed would not support a conclusion that the entire lot was rejectable when the database was complete, and contained three to four thousand records. The client asserted that I had reviewed an "intermediate work product" and that my findings were invalid, and that my conclusion was therefore invalid.

The client stated they would update their database to add the missing records documented by my report, now that the discrepancies had been brought to their attention, but that no additional action would be taken because I had not provided sufficient evidence that other records not documented by my report were missing. The client stated they would not investigate and determine how many records which should have been documented by the database were not documented by the database.

The client also stated they would update their database to complete and correct the incomplete and incorrect records documented by my report, now that the discrepancies had been brought to their attention, but that no additional action would be taken because I had not provided sufficient evidence that other records not documented by my report were incomplete or incorrect. The client stated they would not investigate and determine how many incomplete or incorrect records were documented by the database.

The client was unable to explain when the discrepant records would be completed or corrected if they were never detected, or how this would occur before the system as a whole was turned over. The client was unable to support their assertion that this was documented by the new process because it was not.

The client asserted they were in compliance with the requirements of the client's contract with the customer. The client was unable to support their assertion that the database used to store and electronically reproduce records of construction was complete and accurate, in accordance with the requirements of the client's contract with the customer.

In short, the client attempted to redefine what constituted a rejectable lot by asserting the number and type of discrepancies in a random sample were not representative of the number and type of discrepancies in the lot as a whole because the database was an "intermediate work product" and that my findings were invalid, and that my conclusion was therefore invalid. The client stated I was therefore unable to support an assertion that the discrepancies would not have been detected and corrected before the system as a whole was turned over to the customer.

This was then, as it is now, unacceptable.

So I very quickly and quietly cloned the database, and evaluated the lot of over one thousand records for compliance with the client's contract with the customer. This review required several days, and revealed that fully ten percent (over one hundred) of required records were missing, and approximately 25 percent were incomplete or incorrect. By doing this I refuted the client's assertion that the sample of approximately thirty records which I reviewed would not support a conclusion that the entire lot was rejectable when the database was complete.

I organized the few dozen findings into several significant findings specifically identifying weaknesses in the new process developed by the client, the client's corrective and preventive action program, and the client's compliance with the client's contract with the customer.

I sent my conclusion, which was unchanged, and the findings to the customer and suggested the customer contact the client directly and require the client to report what corrective action the client was taking to bring themselves into compliance with the contract.

This was The Witches' Hammer, the purpose of which was to systematically refute any and all assertions that a problem does not exist.

It should not have been necessary, and if the client had taken responsibility for the problem and not attempted to undermine the statistical basis for reported findings by asserting the number and type of discrepancies in a random sample were not representative of the number and type of discrepancies in the lot as a whole, it would not have been necessary.


Endnotes.

The word "should" is variously interpreted to establish a non-mandatory requirement, i.e., a strongly-worded suggestion. For example: to strongly suggest an organization use an approved process to clean a valve, but not require an approved process to be used. When questioned, an organization required to justify the decision not to follow such a suggestion inevitably disagrees that the word "should" establishes a requirement.
 
As a result, the word "should" is the enemy of clarity and, in general, I refuse to use it. However, in this case it is appropriate. When I attempted to explain the problem to the client, the discussion basically hinged on the definition of the words "complete" and "accurate". Because the client's agreement with the customer did not specify what constitutes a complete and accurate database, I was forced to describe what characteristics a complete and accurate database has, and use the word should in that context.

Last updated: Thursday, 28 July, 2011

Home