How to show that a PCOS voting machine is accurate enough

The COMELEC has already rejected 100% manual audit proposal. That should be the end of it.

Yet, it is hard ignore the voices of the country’s major information technology organizations, business oranizations, legal organizations, election watchdogs, church organizations, and other sectors – all counselling the COMELEC to do a 100% manual audit, since this is the first time the country is trying Smartmatic’s SAES 1800 PCOS machine.

Even in banks that have automated their operations, to the extent of using a bill counter that counts bills automatically, tellers still count the bills manually before handing out withdrawals to clients. And clients too would again count the money manually before leaving. When a depositor withdraws from an ATM, she would also manually count the money issued by the ATM machine, before she leaves the premises. We all do this, despite the fact that we have no reason to doubt the accuracy of the bill counters and ATMs.

Since we do have a strong reason to doubt the accuracy of PCOS machines, all the more should we double-check its count. Why do we doubt the accuracy of the PCOS?

The Comelec has already acknowledged the misalignment of ultraviolet security marks due to high-speed printing. Because of this misalignment, the Comelec decided on its own, without prodding, to ask election inspectors to conduct a 100% manual audit of the authenticity of the ballots, by visually determining with a UV lamp whether each ballot is authentic or not.

Since the Comelec already admitted that the security mark was misaligned, then it should, at the very least, double-check the possibility that the ovals in the ballot are likewise misaligned. If some ballots have misaligned ovals, then for exactly the same reason that the PCOS machine became unreliable in detecting the security marks, it will also become unreliable in detecting shaded ovals.

This is why we doubt the accuracy of the PCOS machines: they might be fed ballots with misaligned ovals caused by the high-speed printing.

The Comelec specified in its contract with Smartmatic that the PCOS should be able to read shaded marks correctly at least 99.995% of the time (in other words, that it makes at most one error for every 20,000 marks or .005%).

I propose a way to convincingly show, with 95% confidence, that a PCOS machine is 99.995% accurate. A 95% confidence level is the typical confidence level used in statistical tests, product quality control, and scientific experiments.

At this point, the math will get a little bit heavy. But since we are talking of P7.2 billion of taxpayers’ money, let us all hunker down and take the extra effort.

Let us suppose that Smartmatic supplied us with machines whose accuracy rates are all slightly lower than the Comelec specification of 99.995%, say, they make one error for every 19,999 instead of 20,000 marks. We want a set of test ballots that will detect these below-spec machines, because they will make one or more reading errors. Then we can reject these inaccurate machines and have them replaced by Smartmatic because they are all slightly below Comelec specifications (unless of course the Comelec was willing to adjust its specs downwards to accomodate Smartmatic).

If we feed one of these below-spec machines a test ballot with a single mark, we know that the chances of a correct reading is 19,998/19,999 or 99.9949997%, which is practically 100%. We’d expect the machine to read the ballot without error. If we used this ballot as our test set, this below-spec machine, which should fail the test, will pass it instead, where “fail” means at least one reading error and “pass” means zero reading error.

If we feed a test ballot with two marks into the same machine, the chances of two consecutive successes in reading marks are 99.99449997% x 99.99449997% or 99.990%. Our below-spec machine will most probably pass such a test ballot too.

If we feed a test ballot with three marks into the same machine, the chances of correctly reading all three are 99.99449997% x 99.99449997% x 99.99449997% or 99.985%. The machine will again probably pass the test.

Brace yourself: the above can be written as (99.99449997%)^3 = 99.985%, where “^3” means “multiplied by itself three times”.

A real ballot will contain an average of 34 marks (one each for president, vice president, party-list, congressman, governor, mayor, and so on, 12 for senators, etc.).

Hence the chances that the below-spec machine will accurately read a single test ballot with 34 marks correctly are (99.99449997%)^34 = 99.83%. This is still very high. The machine will most probably read this test ballot correctly too, and pass the test.

For ten ballots (340 marks): the chances of accurately reading all of them are (99.99449997%)^340 = 98.3%. Still very high.

Note that 3-7 days before election day, election inspectors, watchers and watchdogs in every precinct cluster will use a ten-ballot test set to determine if the PCOS machines meet the Comelec specification of 99.995%. This means they should be able to detect our below-spec machine, so that it may be replaced by Smartmatic. However, the chances that our below-spec machine will make no reading error and therefore pass the test is 98.3%. Thus, out of a hundred of our below-spec machines, the ten-ballot test will correctly detect two but will pass 98 machines that should have failed. This is obviously not acceptable.

For thirty ballots (1,020 marks): the chances of accurately reading all are (99.99449997%)^1020= 95.0%

For 408 ballots (13,872 marks): the chances are (99.99449997%)^13,872 = 50.0%. If we use a 408-ballot set to test the below-spec machines, half of them will be detected successfully, but half will still pass the test. Not good.

For 1,000-ballot test set (34,000 marks): the chances of our machines passing are now (99.99449997%)^34,000 = 18.3%. Out of a hundred below-spec machines, 18 will still pass undetected. Not acceptable.

Let me now jump to the magic number: 1,762 ballots (59,908 marks). The chances are now (99.99449997%)^59,908 = 5.0%. Out of a hundred, ninety-five of the below-spec machines will fail and only five will pass a 1,762-ballot test set. (In an earlier computation, I had used 99.9944% and got 1,573. A 1,762-ballot test set will detect marginally inaccurate machines better.)

This 95% confidence level is the most common standard used in statistical testing, product quality control, and scientific experiments. If we wanted even better quality standards, we can adopt a 99% instead of 95% confidence level. This will require a 2,708-ballot test set. Then, out of every hundred below-spec machines that Smartmatic delivers, 99 will make at least one error and will be correctly rejected. Only one will pass the test and be deployed for vote counting.(This would still be 822 below-spec PCOS machines out of the 82,200, but that’s probably a necessary risk, unless we want even stricter testing).

Let me now summarize how to test each PCOS machine for accuracy:

Prepare a test set of 1,762 official ballots with 34 marks each, making sure that every oval on the ballot is marked at least once. Tabulate the votes represented by the marks. The tally must be checked and rechecked for 100% accuracy, because it is the standard against which the accuracy of the machine will be measured.

Configure a PCOS machine to accept the ballots, and feed the test set into the machine. If the machine reads all the ballots without error, then we can be certain at the 95% confidence level that the machine has an accuracy rate of 99.995% or better. (If I were the test engineer, I would then repeat the test on every accepted machine, and I would only use as backup those machines that fail on the second test.)

If the COMELEC had done this acceptance testing when the machines were delivered, any machine which did not meet the 95% confidence level should have been returned to Smartmatic for replacement. Thus every PCOS machine deployed for the elections should have passed this test. Since such a test can take around seven hours and a half (assuming 15 seconds per ballot), and Smartmatic claimed that it was testing PCOS machines at the rate of 2,000 per day, it is highly unlikely that these machines went through this kind of acceptance testing. So, we do not know how many PCOS machines actually meet the Comelec specifications.

If Smartmatic is confident about the accuracy of its PCOS machines, and if the Comelec is confident about the alignment of the ovals in the 50.85 million ballots they just printed, they should welcome this 1,762-ballot test as a chance to prove that the PCOS machines truly meet the Comelec specifications for accuracy.