Tag Archives: failure of automation

Partial failure of automation has already occurred; how to determine if the failure is total

The printing of 50.85 million ballots for the May 10 elections is now complete. The COMELEC said they were two days ahead of schedule. This early finish, however, was achieved at terrible cost: the high-speed printing caused the misalignment of the ultraviolet security mark on the ballot by one to two millimeters. This led the failure of the automated PCOS ballot authentication system, which the COMELEC had to abandon in favor of a manual ballot authentication system. This manual system relies on election inspectors to shine a UV lamp on each ballot to determine ballot authenticity through visual inspection.

The unreliable scanning of UV marks is, in effect, already a partial failure of automation.

But if the UV marks were misaligned due to the high-speed printing, shouldn’t we all be concerned that the ovals themselves have been similarly misaligned? Misaligned ovals would have very serious consequences. In the 1998 automation pilot in ARMM, according to a report on the COMELEC website, similar ballot printing problems forced the COMELEC to manually recount ballots from Sulu and some municipalities of Lanao del Sur. However carefully 2010 voters will shade the ovals, if these ovals are misaligned, then the voters’ marks will also be misaligned. This will make the PCOS unreliable in registering the voters’ choices, in exactly the same way they had become unreliable in detecting the UV security marks.

If HALAL’s concern turns out to be true, that ovals themselves were also misaligned by the same high-speed printing that misaligned the UV security marks, then we cannot rely anymore on the PCOS to accurately register the voters’ choices. An inaccurate PCOS machine would render useless the rest of the automated election system. It will mean a total failure of automation.

Misalignment of ovals is far more serious than misalignment of UV marks for the following reason: misaligned UV marks will lead the PCOS machine to reject valid ballots, an obvious problem which voters will notice immediately and complain about. This failure of automation cannot be hidden from the voters. Thus, the COMELEC has no choice but to correct the problem. This is also true for PCOS machines that stop working. But inaccurate vote counts due to oval misalignment will still be registered by the machine. Voters will never know their votes were falsely registered. This failure of automation can be hidden from voters. Thus, no one will notice, and no one can complain if the COMELEC and Smartmatic opt to hide this problem. Firms or persons may want to do so, if they can be held legally or financially accountable for the failure of a P7.2 billion project.

Earlier field tests and mock elections were announced by COMELEC to be “almost perfect” and by Smartmatic to be “successful” despite numerous media reports of machines rejecting valid ballots and transmission problems even in Metro Manila. These false claims suggest that the COMELEC and Smartmatic are not averse to declaring “success” despite public knowledge of ballot rejections and transmission problems. Thus, it would not be far-fetched for them to likewise claim “successful” automation on May 10, despite inaccurate scans and false machine counts which no voter will notice and complain about.

Indeed, the COMELEC has managed to hide so far the PCOS machine’s true scanning accuracy, which has remained a complete mystery.

Four opportunities to measure the true PCOS accuracy had been lost: 1) the results of the acceptance tests remain inaccessible to the public; 2) the full reports of Systest Labs, which conducted a system audit and source code review, also remain inaccessible to the public (update: Commissioner Larrazabal has agreed to release the Systest Labs report); and 3) the voter verification feature of the PCOS was disabled, preventing voters from knowing if the machine accurately registered their choices; and 4) the COMELEC decided to reject the proposal for a 100% audit of PCOS accuracy, and to stick instead to a random audit that covers only 1.5% of machines.

Unfortunately, the fifth opportunity to measure PCOS accuracy – and the alignment of ovals – is woefully inadequate for detecting an inaccurate machine, which, according to COMELEC specifications, is one with an accuracy rate below 99.995%. Several days before election day, every board of election inspectors (BEI) is supposed to set up the PCOS machines and do a final test. The machines will be fed with ten pre-tabulated ballots and the expected results compared with the machine count. To pass, a machine must count all the test ballots without a single error. COMELEC rules are not clear what the BEI should do if the PCOS makes even a single error. The most logical thing to do – a more rigorous version of this test should have been done months ago – is to return the machine to Smartmatic for replacement. A ten-ballot set is not enough. It will miss 98% of marginally inaccurate machines, and detect only 2%. A test set of 1,762 ballots is needed to screen out 95% of marginally inaccurate machines. A 2,708-ballot test set can screen out 99% of marginally inaccurate machines. though it will still pass 1%, or 822 of the 82,200 PCOS machines. How I got these numbers will require a separate piece. (Google “how to show that a PCOS is accurate enough”)

Instead of testing machines randomly with a 1,762-ballot test set, the COMELEC’s best option might be to heed the following HALAL suggestion: increase the random manual audit (RMA) coverage from 1.5% to 30% of the machines.

A number of strong arguments justify this simple suggestion: 1) the no-legal-basis argument cannot be invoked, because the law provides for the RMA; 2) the COMELEC clearly has the power to change the coverage of the RMA, because it has already raised the coverage from 0.3% (one precinct per district) to 1.5% (five precincts per district); 3) the COMELEC has long announced that it was prepared to do a manual count in 30% of the precincts, so the 30% RMA can simply use the existing paraphernalia for the manual count; no new expenses or preparations are needed; since no precinct knows in advance if it will be drawn for the RMA, every precinct must prepare to conduct an RMA anyway, whether the COMELEC is going to cover 1.5% or 30% of precincts; 4) a 30% RMA may not satisfy all, but it will certainly placate some groups, particularly the business sector, that had wanted a 100% parallel count; the twenty-fold increase in coverage from 1.5% to 30% will surely go a long way in raising the credibility of the electoral process; and 5) auditing 30% of the PCOS machines should be enough to determine the extent of PCOS inaccuracy caused by any misalignment of ovals.

Unless a credible audit is done, we might never know whether the machines counted our votes accurately or not, and whether the failure of automation we suffered was partial or total.

The issue: failure of automation, not failure of election

The issue is not a failure of election, but a failure of automation.

Failure of election” is a narrow legal term describing a rare situation. The Omnibus Election Code defines it as a situation in which “the election in any polling place has not been held on the date fixed, or had been suspended before the hour fixed by law”. The suspension may also occur “after the voting and during the preparation and the transmission of the election returns”. The definition further requires that the failure “would affect the result of the election”, or “results in a failure to elect”. It has occurred in barangays or towns but never on a national scale.

Although some have raised its possibility in 2010, they were probably using the term loosely and were not aware of its legal definition.

Thus, Chairman Melo could say, with a straight face, that failure of election was “pure fantasy”. He is using its narrow legalistic definition. If voters were able to cast their votes and Comelec proclaimed a winner, there was no failure of election.

The issue in 2010 is the high risk of a failure of automation. This is what was raised by Halalang Marangal, an election watchdog which includes former Senator Wigberto Tañada, former Comelec Commissioner Mehol Sadain, and retired General Francisco Gudani among its convenors. We had in fact estimated the probability of failure as of March 8 at 75%, and we have seen no reason to substantially revise that estimate. We still consider the risk of failure “unacceptably high”.

Let me define what we mean by a failure of automation.

Election automation is a failure if the time it takes to determine the winners in the election is not significantly shorter than the manual method, or if the fraud that has chronically attended our elections is not significantly reduced.

Let me review the basis of our assessment that the election automation had a 25% chance of success. (You can find the details in Jarius Bondoc’s April 5, 7 and 9 columns in this paper.)

A March 8 full-page ad by Smartmatic in major national dailies had claimed “a vote of confidence” on the election automation project it was implementing in the Philippines. Smartmatic had claimed substantial achievements in the five sub-systems that comprised the whole Automated Election System (AES).

But when we scrutinized carefully the Smartmatic ad, we found gaps, delays, problems and at least one glaring false claim (“successful field tests and mock elections”).

In the Hardware sub-system, Smartmatic claimed they have completed the delivery of the machines, but glaringly omitted reference to testing. Clearly they have not tested the machines thoroughly. Neither did they have the time to do so. Former Comelec commissioner Mehol Sadain tells us that in 2004, it had taken them three months to fully test 1,990 automated counting machines. If deployed, some of the partially-tested machines are bound to cause problems on election day. We also found that Smartmatic had bought 21% more memory cards than necessary. In the wrong hands, these could be loaded with false precinct results and substituted for authentic cards. Because of these and other problems, we estimated the probability of success of this sub-system at 80%.

In the Software sub-system, we noted that no local stakeholders had managed to conduct a proper review of the source code, because of the Comelec’s obstinate refusal to implement the clear intent and letter of the law. We also noted that the Comelec released no certification documents or full report that would support its Feb. 9 claim that Systest Labs had completed its audit/review of the AES. Since time had run short for a thorough review, we estimated the probability of success of this sub-system at 70%.

For the Logistics sub-system, we cited media findings about the questionable capacity of the forwarders chosen by Smartmatic to deliver election paraphernalia throughout the Philippines. We estimated the Logistics probability of success at 80%.

For the Transmission sub-system, we cited among other things the 70% signal coverage in the Philippines, as Smartmatic itself found out. Smartmatic had transmission problems even within Metro Manila, suggesting poor quality of transmission equipment. We gave it 70%.

For the Ballot Printing sub-system, we cited the confidential Comelec memo which warned that it was impossible to finish ballot printing on time, given the rate they were printing them. We gave it 80%. The Comelec subsequently brought in a fifth ballot printer, raising its capacity by 20%, and making it possible – if no further glitches happened and the printing went on non-stop – to meet its April 25 deadline.

To get the overall probability of success of the entire AES project, the sub-system probabilities of success must be multiplied together. Yes, multiplied together, not averaged. And not just the lowest figure – the weakest link – either. Check it with your calculator: .8 x .7 x .8 x .7 x .8 = .25 or 25%. Note that we see the glass as one-fourth full, not three-fourths empty. We are optimists to a fault, not doomsayers.

So many things can go wrong with the AES that Murphy’s Law is bound to kick in. Like a toss of two coins, Chairman Melo is betting that two heads will come up. He bet P7.2 billion of the taxpayer’s money. If you count the whole election budget, P11 billion, all in.

Chairman Melo’s bet will lose 75% of the time. That makes failure of automation the issue in 2010.

The time is very short, but we still have a possible solution. Details in subsequent posts.