Tag Archives: PCOS

Too soon to call 2010 elections successful

It is too soon to declare the 2010 elections a success.

People want a successful election so badly, that it is easy to get carried away by flood of incoming election returns. Many want to believe that a clean and honest election has finally happened, at last.

But the vice-presidential election is yet to be settled. The contest between the 12th and 13th places in the senatorial race still has to be settled too. Hundreds, perhaps thousands, of local races also await to be settled.

Already news is coming in about delayed Election Returns (ERs), malfunctioning, missing or otherwise questionable memory cards, and other indicators of potential or emerging problems.

As in the manual system, the precinct level count is always the fastest. Even when election inspectors, watchers and the public counted votes by hand, most of the election results had always been available past midnight or early morning. Even under the manual method, the biggest challenge has always been at the municipal level and higher, where wholesale cheating operations occurred.

In fact, the automated election system failed spectacularly its first truly public test a week before election day, when many candidates got zero – a “bawas” — and some got more than the votes actually cast for them – a “dagdag”. The results were worse than most manual counts. Fortunately, the failures in the machine count were so obvious that the election inspectors and watchers noticed them immediately. An embarrassed Comelec quickly called off the public test, and traced the problem to misaligned ovals on the ballot. Because of a last-minute change from single-spacing to double-spacing in the ballot layout for local candidates, their oval locations did not anymore match the coordinates stored in a configuration file in a memory card within the PCOS machine.

Reconfiguring the memory cards was somewhat easier than reprinting ballots, so that is what the Comelec and Smartmatic tried to do.

Smartmatic only had 18,000 spare memory cards and there was little time to recall the rest, so in addition to the spares, Smartmatic recalled the cards that could still be recalled; imported the rest from Hongkong and Taiwan; edited each of the 1,631 ballot layout configuration files (unique for every town); programmed these configuration files into 76,340 memory cards (one for each machine); delivered the 76,340 newly reconfigured memory cards to the waiting machines all over the archipelago; found the right machines for the right memory cards; replaced the misconfigured memory card; and conducted a second round of public testing and sealing of the PCOS machines. All within a span of five days – 120 hours. Aside from some 400 machines that malfunctioned, the rest of the 76,340 machines worked fine and gave the country its first successful automated elections. So they say.

Can we now trust the machine results?

These machines had grievously failed to count a few days earlier. This was followed by a mad rush of recalls, importations, file reconfigurations, card reprogramming, deliveries, reinstallation, and a second round of testing and sealing. In the mad rush, were security procedures and chain of custody considerations still observed? Did anyone see an election inspector with an ultraviolet lamp to check for authentic ballots, for instance? (We have not found anyone who did.) Suppose there were also more subtle problems that a ten-ballot test set was insufficient to detect – ovals that were misaligned by only one or two millimeters, for example, just as the security marks were, or oval coordinates that were purposely changed slightly to shave votes from targetted candidates. Were tests done at all for these potential problems?

Suppose an ATM had earlier given you only half the money than it deducted from your account, and the bank tells you the machine is now ok. Wouldn’t you count your money at least once in subsequent withdrawals? Suppose most ATMs of a bank network shortchanged its clients, wouldn’t they demand that every ATM of that network be carefully tested and recertified for its counting accuracy?

For exactly the same reason, every candidate who lost – and won – in the machine-counted 2010 elections should demand thorough post-election testing and audit for accuracy of every counting machine and its results.

Losing candidates should demand it, because they might have actually won.

Winning candidates – especially those who lead by a huge margin – should demand it, because the gross machine errors a few days earlier and subsequent doubts about machine accuracy have devalued their victory.

Apparent president-elect Noynoy Aquino should demand it, if only for the sake of his running-mate Mar Roxas, who sacrificed his own presidential ambitions to give way to Noynoy.

There was no time for proper testing in the mad rush to the May 10 elections because few wanted the elections postponed. But we have fifty days before June 30, when the new set of elected officials are scheduled to take over. We still have enough time check, double-check, and be sure about the results of the 2010 elections.

In the meantime, the Comelec and local election authorities should not be in a hurry to proclaim winners and declare the elections a success.

Advertisements

Expect a flood of demands for recount from losing candidates

The PCOS fiasco a few days before the May 10 elections has shattered the credibility not only of election automation, but of the entire electoral process itself. The entire process hinges on an accurate count by the PCOS machine. In fact, COMELEC specifications require at least 99.995% accuracy, or at most one error for every 20,000 marks or around 600 ballots. Today, we have very little idea how accurately the PCOS will count our votes.

One minor mistake – changing from a single-spaced to a double-spaced layout – has created a crisis of credibility for the entire elections. Who will accept at face value the PCOS machine counts now? Even a candidate who wins with a narrow margin may suspect that the machine might have trimmed his margin. Certainly, losing candidates can now be expected to demand a manual recount, so they can see for themselves how they actually fared. No assurance from Smartmatic or the Comelec will now suffice, because everyone has seen how the PCOS machine made gross errors during the final testing and sealing.

Many may not realize it yet, but we are in a different ballgame now. May 10 election victories — whoever wins, in whatever position – have just suffered a major devaluation.

The PCOS fiasco has robbed the winners of the May 10 elections of a clear victory. It has created not only a cloud of doubt about the machine results, but also last-minute disruptions in the crucial final days before the elections that will surely create confusion and chaos in many areas.

Deliveries had to be suspended, because the PCOS machines had to stay put in the hubs or sub-hubs, so they can await the CF memory card replacements. The final testing and sealing had to be called off, to avoid further embarassment and damage to the credibility of the automation project. When the replacement memory cards eventually arrive for installation in the machines, only then can these machines leave the distribution hubs and sub-hubs for final deployment.

Designing machines and ballot sets that will only work with each other, but delivering them separately had earlier created a “logistical nightmare”. The nightmare just became worse, because machines, ballot sets, and memory cards are now being delivered separately, and under even greater time pressure.

If the PCOS machine, its associated set of ballots, and its associated memory card, do manage to find themselves reunited in their destination precinct cluster, on time for the May 10 elections, they still need to be tested again. It was obvious from the fiasco that the accuracy of the machines were not checked before they were deployed, so the final testing is our only chance to determine the accuracy of the PCOS. This final testing cannot be dispensed with.

If the final testing with a ten-ballot test set shows even a single error, then we cannot guarantee with 95% confidence that the machine’s error rate is lower than 1%. There’s a good chance it is higher than 1%. Then, the board of election inspectors must either ask for a replacement, or go ahead and use an inaccurate machine.

Where machines don’t arrive, stop working, reject too many valid ballots, or otherwise fail, the board of election inspectors must resort to a manual count. The Comelec says it has prepared the paraphernalia for up to 30% of the precincts. But the precinct election inspectors have ask for the paraphernalia to be delivered first. More delays. Given the confusion, more than 30% of precincts may need to resort to a manual count.

If the paraphernalia for a manual count don’t arrive, the ballot boxes will have to be sealed and watched over, to be counted later.

So, expect confusion and chaos in many areas on election day. Yet, even where everything works smoothly, the machine counts will remain under a cloud of doubt, and losing candidates will surely demand a recount, citing as reason the errors made by the PCOS machines earlier. Who can blame them?

All because of an innocuous-looking suggestion to change from a single-spaced to a double-spaced ballot layout.

The origin of this suggestion must be traced. Who first raised it? Why did Comelec approve it? Why didn’t Smartmatic or Comelec test the accuracy of the PCOS machine with the new layout?

If anyone had intentionally wanted to disrupt the elections and discredit its results, he – or she – couldn’t have done it any better.

Partial failure of automation has already occurred; how to determine if the failure is total

The printing of 50.85 million ballots for the May 10 elections is now complete. The COMELEC said they were two days ahead of schedule. This early finish, however, was achieved at terrible cost: the high-speed printing caused the misalignment of the ultraviolet security mark on the ballot by one to two millimeters. This led the failure of the automated PCOS ballot authentication system, which the COMELEC had to abandon in favor of a manual ballot authentication system. This manual system relies on election inspectors to shine a UV lamp on each ballot to determine ballot authenticity through visual inspection.

The unreliable scanning of UV marks is, in effect, already a partial failure of automation.

But if the UV marks were misaligned due to the high-speed printing, shouldn’t we all be concerned that the ovals themselves have been similarly misaligned? Misaligned ovals would have very serious consequences. In the 1998 automation pilot in ARMM, according to a report on the COMELEC website, similar ballot printing problems forced the COMELEC to manually recount ballots from Sulu and some municipalities of Lanao del Sur. However carefully 2010 voters will shade the ovals, if these ovals are misaligned, then the voters’ marks will also be misaligned. This will make the PCOS unreliable in registering the voters’ choices, in exactly the same way they had become unreliable in detecting the UV security marks.

If HALAL’s concern turns out to be true, that ovals themselves were also misaligned by the same high-speed printing that misaligned the UV security marks, then we cannot rely anymore on the PCOS to accurately register the voters’ choices. An inaccurate PCOS machine would render useless the rest of the automated election system. It will mean a total failure of automation.

Misalignment of ovals is far more serious than misalignment of UV marks for the following reason: misaligned UV marks will lead the PCOS machine to reject valid ballots, an obvious problem which voters will notice immediately and complain about. This failure of automation cannot be hidden from the voters. Thus, the COMELEC has no choice but to correct the problem. This is also true for PCOS machines that stop working. But inaccurate vote counts due to oval misalignment will still be registered by the machine. Voters will never know their votes were falsely registered. This failure of automation can be hidden from voters. Thus, no one will notice, and no one can complain if the COMELEC and Smartmatic opt to hide this problem. Firms or persons may want to do so, if they can be held legally or financially accountable for the failure of a P7.2 billion project.

Earlier field tests and mock elections were announced by COMELEC to be “almost perfect” and by Smartmatic to be “successful” despite numerous media reports of machines rejecting valid ballots and transmission problems even in Metro Manila. These false claims suggest that the COMELEC and Smartmatic are not averse to declaring “success” despite public knowledge of ballot rejections and transmission problems. Thus, it would not be far-fetched for them to likewise claim “successful” automation on May 10, despite inaccurate scans and false machine counts which no voter will notice and complain about.

Indeed, the COMELEC has managed to hide so far the PCOS machine’s true scanning accuracy, which has remained a complete mystery.

Four opportunities to measure the true PCOS accuracy had been lost: 1) the results of the acceptance tests remain inaccessible to the public; 2) the full reports of Systest Labs, which conducted a system audit and source code review, also remain inaccessible to the public (update: Commissioner Larrazabal has agreed to release the Systest Labs report); and 3) the voter verification feature of the PCOS was disabled, preventing voters from knowing if the machine accurately registered their choices; and 4) the COMELEC decided to reject the proposal for a 100% audit of PCOS accuracy, and to stick instead to a random audit that covers only 1.5% of machines.

Unfortunately, the fifth opportunity to measure PCOS accuracy – and the alignment of ovals – is woefully inadequate for detecting an inaccurate machine, which, according to COMELEC specifications, is one with an accuracy rate below 99.995%. Several days before election day, every board of election inspectors (BEI) is supposed to set up the PCOS machines and do a final test. The machines will be fed with ten pre-tabulated ballots and the expected results compared with the machine count. To pass, a machine must count all the test ballots without a single error. COMELEC rules are not clear what the BEI should do if the PCOS makes even a single error. The most logical thing to do – a more rigorous version of this test should have been done months ago – is to return the machine to Smartmatic for replacement. A ten-ballot set is not enough. It will miss 98% of marginally inaccurate machines, and detect only 2%. A test set of 1,762 ballots is needed to screen out 95% of marginally inaccurate machines. A 2,708-ballot test set can screen out 99% of marginally inaccurate machines. though it will still pass 1%, or 822 of the 82,200 PCOS machines. How I got these numbers will require a separate piece. (Google “how to show that a PCOS is accurate enough”)

Instead of testing machines randomly with a 1,762-ballot test set, the COMELEC’s best option might be to heed the following HALAL suggestion: increase the random manual audit (RMA) coverage from 1.5% to 30% of the machines.

A number of strong arguments justify this simple suggestion: 1) the no-legal-basis argument cannot be invoked, because the law provides for the RMA; 2) the COMELEC clearly has the power to change the coverage of the RMA, because it has already raised the coverage from 0.3% (one precinct per district) to 1.5% (five precincts per district); 3) the COMELEC has long announced that it was prepared to do a manual count in 30% of the precincts, so the 30% RMA can simply use the existing paraphernalia for the manual count; no new expenses or preparations are needed; since no precinct knows in advance if it will be drawn for the RMA, every precinct must prepare to conduct an RMA anyway, whether the COMELEC is going to cover 1.5% or 30% of precincts; 4) a 30% RMA may not satisfy all, but it will certainly placate some groups, particularly the business sector, that had wanted a 100% parallel count; the twenty-fold increase in coverage from 1.5% to 30% will surely go a long way in raising the credibility of the electoral process; and 5) auditing 30% of the PCOS machines should be enough to determine the extent of PCOS inaccuracy caused by any misalignment of ovals.

Unless a credible audit is done, we might never know whether the machines counted our votes accurately or not, and whether the failure of automation we suffered was partial or total.

HALAL April update: estimated chance of AES success is now 32%; PCOS accuracy remains a mystery

by Halalang Marangal (HALAL)

Last March 2010, Halalang Marangal issued an analysis of the estimated chance of success of the Automated Election System (AES) and put it at 25%. Based on developments in April, we are updating our estimates of the probabilities of success of the sub-projects and the AES itself as follows:

AES Sub-Project March April

  • Hardware 80% 80%
  • Software 70% 70%
  • Logistics 80% 90%
  • Transmission 70% 90%
  • Ballot Printing 80% 70%

Overall AES Project 25% 32%

Note that when estimating the overall chance of success of an entire project, comprising several sub-projects, each of which are essential to the success of the entire project, the individual probabilities of success of the sub-projects must be multiplied together, not averaged. Note too that we are estimating here the success or failure of automation, not the election itself.

In the hardware sub-project, there was no reason to modify our earlier assessment. Much of the hardware were still not fully tested, and neither were any test results made available to the public for scrutiny. Also, the purchase of 21% more memory cards than necessary remained unexplained, raising concerns that these extra memory cards, if they fall in the wrong hands, may be configured with false data and substituted for authentic cards.

In the software sub-project, no new developments occurred either, that might have led us to modify our assessment. The Systest Labs full report on its system audit and source code review remained inaccessible to the public, and no local group still has managed to conduct a source code review. PCOS software remained configured to disable the voter verification feature, an essential feature that enables voters to determine the accuracy of the PCOS with respect to the voters’ actual choices. The digital certification system remains in Smartmatic hands, instead of an independent third-party like the Department of Science and Technology.

In the logistics sub-project, HALAL has since learned that in addition to the three original small firms contracted to make nationwide deliveries of election paraphernalia for the Comelec, better capitalized forwarders like Air21, which have more experience in handling cargo, have also been contracted. This has led us to raise our estimate of this sub-project’s probability of success from 80% to 90%. Ensuring that paired ballots and PCOS machines, which are being delivered separately, will arrive on time in the right precincts remains a huge logistical problem.

In the transmission sub-project, HALAL has since learned that the Comelec will now be providing for 100% coverage of all precincts in terms of transmission capability, while the March 8 full-page ad of Smartmatic only reported enough transmission equipment to cover 70% of all precincts. Thus, we have raised our estimate of the probability of success of this sub-project from 70% to 90%. However, transmission problems even within Metro Manila as well as in remote provinces like Batanes still suggest that similar problems will occur on election day.

Under the ballot printing sub-project, the printing of 50.85 million ballots was reported by Comelec complete two days ahead of schedule.

It seems though that this early finish was attained at terrible cost. The Comelec says that the high-speed printing resulted in the “misalignment by one to two millimeters” of the ultraviolet security mark. The problem was serious enough that it led the Comelec to abandon the automatic PCOS authentication of ballots, in favor of a manual check for authenticity by shining a UV lamp on each ballot and letting the BEI determine ballot authenticity through visual inspection.

HALAL raises this important question: if the UV marks were misaligned due to the high-speed printing, could the ovals themselves have been similarly misaligned? Misaligned ovals would have very serious consequences. In the 1998 automation pilot in ARMM, according to a report on the Comelec website, similar ballot printing problems led the Comelec to manually recount ballots from Sulu and some municipalities of Lanao del Sur. However carefully voters will shade the ovals, if these ovals are misaligned, then the voters’ marks will also be misaligned, which will make the PCOS machine unreliable in scanning and counting the voters’ choices, in the same way it became so unreliable in scanning the UV marks that automatic scanning for ballot authenticity had to be abandoned.

Misalignment of ovals is far more serious than misalignment of UV marks for the following reason: misaligned UV marks will lead the PCOS machine to reject valid ballots, an obvious problem which voters will notice and complain about. Thus, the Comelec has no choice but to correct the problem. But misalignment of ballots leads to inaccurate vote counts, which will still be registered by the machine, although the voters will never know their votes were inaccurately registered. Thus if the Comelec chose to ignore this problem, no one will notice, and no one can complain. Earlier field tests and mock elections were announced by Comelec to be “almost perfect” and by Smartmatic to be “successful” despite numerous media reports of machines rejecting valid ballots and transmission problems even in Metro Manila. If they can make such false claims despite public knowledge of ballot rejections and transmission problems, it would be much easier for them to claim “successful” elections on May 10 despite inaccurate machine counts which no one will notice and complain about.

Because of this uncertain oval alignment, HALAL reduced its estimate of the probability of success of this sub-project from 80% to 70%.

Because of misaligned UV marks, the Comelec decided on its own, without prodding, to shift to a 100% manual audit for authenticity of the ballots, before any winner is proclaimed. The possible misalignment of ovals should have logically led to a similar 100% manual audit of ballots for accuracy of the machine counts, before any winner is proclaimed.

With the Comelec’s decision to reject the proposal for a 100% audit, and to stick instead to a random audit that covers only 1.5% of precincts, we have lost a fourth opportunity to ascertain the accuracy of the PCOS machines. Earlier, three other opportunities had also been lost: 1) the results of the acceptance tests remain inaccessible to the public; 2) the full reports of Systest Labs, which conducted a system audit and source code review, also remain inaccessible to the public; and 3) the voter verification feature of the machines was disabled. Sadly, the ten ballots that will be used by the BEI for testing three days before the elections are too few to reliably screen out inaccurate machines.

We will never know at all, it seems, how accurately these machines counted the voters’ choices.

April 29, 2010

How to show that a PCOS voting machine is accurate enough

The COMELEC has already rejected 100% manual audit proposal. That should be the end of it.

Yet, it is hard ignore the voices of the country’s major information technology organizations, business oranizations, legal organizations, election watchdogs, church organizations, and other sectors – all counselling the COMELEC to do a 100% manual audit, since this is the first time the country is trying Smartmatic’s SAES 1800 PCOS machine.

Even in banks that have automated their operations, to the extent of using a bill counter that counts bills automatically, tellers still count the bills manually before handing out withdrawals to clients. And clients too would again count the money manually before leaving. When a depositor withdraws from an ATM, she would also manually count the money issued by the ATM machine, before she leaves the premises. We all do this, despite the fact that we have no reason to doubt the accuracy of the bill counters and ATMs.

Since we do have a strong reason to doubt the accuracy of PCOS machines, all the more should we double-check its count. Why do we doubt the accuracy of the PCOS?

The Comelec has already acknowledged the misalignment of ultraviolet security marks due to high-speed printing. Because of this misalignment, the Comelec decided on its own, without prodding, to ask election inspectors to conduct a 100% manual audit of the authenticity of the ballots, by visually determining with a UV lamp whether each ballot is authentic or not.

Since the Comelec already admitted that the security mark was misaligned, then it should, at the very least, double-check the possibility that the ovals in the ballot are likewise misaligned. If some ballots have misaligned ovals, then for exactly the same reason that the PCOS machine became unreliable in detecting the security marks, it will also become unreliable in detecting shaded ovals.

This is why we doubt the accuracy of the PCOS machines: they might be fed ballots with misaligned ovals caused by the high-speed printing.

The Comelec specified in its contract with Smartmatic that the PCOS should be able to read shaded marks correctly at least 99.995% of the time (in other words, that it makes at most one error for every 20,000 marks or .005%).

I propose a way to convincingly show, with 95% confidence, that a PCOS machine is 99.995% accurate. A 95% confidence level is the typical confidence level used in statistical tests, product quality control, and scientific experiments.

At this point, the math will get a little bit heavy. But since we are talking of P7.2 billion of taxpayers’ money, let us all hunker down and take the extra effort.

Let us suppose that Smartmatic supplied us with machines whose accuracy rates are all slightly lower than the Comelec specification of 99.995%, say, they make one error for every 19,999 instead of 20,000 marks. We want a set of test ballots that will detect these below-spec machines, because they will make one or more reading errors. Then we can reject these inaccurate machines and have them replaced by Smartmatic because they are all slightly below Comelec specifications (unless of course the Comelec was willing to adjust its specs downwards to accomodate Smartmatic).

If we feed one of these below-spec machines a test ballot with a single mark, we know that the chances of a correct reading is 19,998/19,999 or 99.9949997%, which is practically 100%. We’d expect the machine to read the ballot without error. If we used this ballot as our test set, this below-spec machine, which should fail the test, will pass it instead, where “fail” means at least one reading error and “pass” means zero reading error.

If we feed a test ballot with two marks into the same machine, the chances of two consecutive successes in reading marks are 99.99449997% x 99.99449997% or 99.990%. Our below-spec machine will most probably pass such a test ballot too.

If we feed a test ballot with three marks into the same machine, the chances of correctly reading all three are 99.99449997% x 99.99449997% x 99.99449997% or 99.985%. The machine will again probably pass the test.

Brace yourself: the above can be written as (99.99449997%)^3 = 99.985%, where “^3” means “multiplied by itself three times”.

A real ballot will contain an average of 34 marks (one each for president, vice president, party-list, congressman, governor, mayor, and so on, 12 for senators, etc.).

Hence the chances that the below-spec machine will accurately read a single test ballot with 34 marks correctly are (99.99449997%)^34 = 99.83%. This is still very high. The machine will most probably read this test ballot correctly too, and pass the test.

For ten ballots (340 marks): the chances of accurately reading all of them are (99.99449997%)^340 = 98.3%. Still very high.

Note that 3-7 days before election day, election inspectors, watchers and watchdogs in every precinct cluster will use a ten-ballot test set to determine if the PCOS machines meet the Comelec specification of 99.995%. This means they should be able to detect our below-spec machine, so that it may be replaced by Smartmatic. However, the chances that our below-spec machine will make no reading error and therefore pass the test is 98.3%. Thus, out of a hundred of our below-spec machines, the ten-ballot test will correctly detect two but will pass 98 machines that should have failed. This is obviously not acceptable.

For thirty ballots (1,020 marks): the chances of accurately reading all are (99.99449997%)^1020= 95.0%

For 408 ballots (13,872 marks): the chances are (99.99449997%)^13,872 = 50.0%. If we use a 408-ballot set to test the below-spec machines, half of them will be detected successfully, but half will still pass the test. Not good.

For 1,000-ballot test set (34,000 marks): the chances of our machines passing are now (99.99449997%)^34,000 = 18.3%. Out of a hundred below-spec machines, 18 will still pass undetected. Not acceptable.

Let me now jump to the magic number: 1,762 ballots (59,908 marks). The chances are now (99.99449997%)^59,908 = 5.0%. Out of a hundred, ninety-five of the below-spec machines will fail and only five will pass a 1,762-ballot test set. (In an earlier computation, I had used 99.9944% and got 1,573. A 1,762-ballot test set will detect marginally inaccurate machines better.)

This 95% confidence level is the most common standard used in statistical testing, product quality control, and scientific experiments. If we wanted even better quality standards, we can adopt a 99% instead of 95% confidence level. This will require a 2,708-ballot test set. Then, out of every hundred below-spec machines that Smartmatic delivers, 99 will make at least one error and will be correctly rejected. Only one will pass the test and be deployed for vote counting.(This would still be 822 below-spec PCOS machines out of the 82,200, but that’s probably a necessary risk, unless we want even stricter testing).

Let me now summarize how to test each PCOS machine for accuracy:

Prepare a test set of 1,762 official ballots with 34 marks each, making sure that every oval on the ballot is marked at least once. Tabulate the votes represented by the marks. The tally must be checked and rechecked for 100% accuracy, because it is the standard against which the accuracy of the machine will be measured.

Configure a PCOS machine to accept the ballots, and feed the test set into the machine. If the machine reads all the ballots without error, then we can be certain at the 95% confidence level that the machine has an accuracy rate of 99.995% or better. (If I were the test engineer, I would then repeat the test on every accepted machine, and I would only use as backup those machines that fail on the second test.)

If the COMELEC had done this acceptance testing when the machines were delivered, any machine which did not meet the 95% confidence level should have been returned to Smartmatic for replacement. Thus every PCOS machine deployed for the elections should have passed this test. Since such a test can take around seven hours and a half (assuming 15 seconds per ballot), and Smartmatic claimed that it was testing PCOS machines at the rate of 2,000 per day, it is highly unlikely that these machines went through this kind of acceptance testing. So, we do not know how many PCOS machines actually meet the Comelec specifications.

If Smartmatic is confident about the accuracy of its PCOS machines, and if the Comelec is confident about the alignment of the ovals in the 50.85 million ballots they just printed, they should welcome this 1,762-ballot test as a chance to prove that the PCOS machines truly meet the Comelec specifications for accuracy.


The high-speed printing that misaligned UV marks can misalign the ballot ovals too

The ultraviolet (UV) scanner of the PCOS was disabled because the high-speed printing resulted in the UV mark on the ballot being “misaligned by one to two millimeters”, according to COMELEC Commissioner Larrazabal. As a result, the machine’s UV scanner often missed the mark and many valid ballots being being rejected.

Here’s the big question: If the misalignment of the UV mark was serious enough to make COMELEC turn off the UV scanner, then can’t the ovals be misaligned too? A misaligned oval means a misaligned a vote-mark. Which means the PCOS main scanner may have problems interpreting the voters’ choices.

The Comelec requires from the PCOS an error rate of less than .005%. That means less than five errors for every 100,000 marks. To determine if this Comelec specification is met, each PCOS should have been tested properly. But no test statistics have been released by the Comelec. And if these tests were done at all, they were probably done with perfectly aligned, not misaligned, ovals.

With misalignment, the two types of errors the scanner can make will both get worse: the false positives that register a vote/mark which is not there, and the false negatives that miss a vote/mark which is there. This means some candidates will gain votes (“dagdag”), while other candidates will lose votes (“bawas”). Does that sound familiar?

This problem is made worse by the Comelec decision to, in effect, blindfold voters while the machine is registering their choice. Originally, the PCOS was programmed to display the voter’s choices on its screen, so he can check if his choices were correctly registered by the PCOS and abort the process if the PCOS didn’t. This voter-verification is in fact required by the Automated Election Law (Section 7n): “Provide the voter a system of verification to find out whether or not the machine has registered his choice.” The Comelec ordered Smartmatic to disable this feature. Thus, the PCOS may falsely register voters’ choices, without voters knowing it. This is far worse than a PCOS that stops working or rejects valid ballots, problems which are apparent at once. An inaccurate PCOS will keep scanning and counting happily, with no indication or warning that it is miscounting votes.

According to a former Comelec official, similar problems had also occurred when the Comelec piloted automation in ARMM in 1998. He showed me a Comelec report entitled “Partial Automation of 1998 National and Local Elections”, which is also on the Comelec website. In 1998, said the report, Sulu ballots had to be manually recounted “due to an error in NPO’s printing procedure”. A similar recount was done with Lanao del Sur ballots, “also due to errors in printing of the ballots.”

The Comelec blamed “high-speed printing” for the misalignment. But misalignment is a common printing problem, and good operators know how to correct it. Why would it happen in such an important job as the printing of ballots?

It seems that the ballot printing is being done in such a hurry, that operators are not getting the time needed to stop the high-speed machines and correct any misalignments that may be happening. That the Comelec had insisted on printing ballots even on Maundy Thursday and Good Friday shows how disastrous even a slight delay might be. If they stopped the machines too often to correct for misalignments, they might miss their deadlines. Thus, the UV misalignment problem has gone on uncorrected.

As of April 16, according to Smartmatic, the NPO has printed 43.7 million ballots. If the high-speed printing has misaligned the UV marks, then how many of these ballots have misaligned ovals too? And in these misaligned ovals, how many false “dagdag-bawas” interpretations by the PCOS will occur?

The misalignment of UV marks and possibly the ovals too are the consequences of the Comelec violating the law and using a machine that has neither been piloted nor used widely. The SAES 1800 has never been piloted in the Philippines, as Section 6 of the Automated Election Law requires: “… the AES shall be used in at least two urbanized cities and two provinces each in Luzon, Visayas and Mindanao.” Nor has the PCOS been used widely in any other country, as Section 10 of the law also requires: “… the system procured must have demonstrated capability and been successfully used in a prior electoral exercise here or abroad.” The Supreme Court’s support for the Comelec’s stubborn insistence in using unpiloted Smartmatic machines must have reinforced the Comelec’s sense of impunity in violating provisions of the law.

Because of the lack of pilot, we lost the chance to detect early problems like these. As a result, they have put the entire national elections at risk.

Solution: both the PCOS and their associated ballots must be thoroughly checked for misalignment and accuracy in scanning. The tests must be done not by Smartmatic, but by independent third parties, say, the DOST, and witnessed by all stakeholders.

What if careful testing shows that the PCOS cannot reliably read properly shaded ovals? Then we may have no choice but to manually count the votes again, ballot by ballot.

[Note: This piece is based on the Halalang Marangal April 17 Statement on the problem of potentially-misaligned ovals, which I also drafted. It was published by the Philippine Star, p.18, on April 21, 2010.]

Roberto Verzola has a background in engineering and economics and a passion for social issues. He is recognized by the IT industry as an Internet pioneer in the Philippines and is often tapped by NGOs for technical advice. He currently lectures at the Institute of Mathematics of the University of the Philippines and is a convenor and secretary-general of the election watchdog Halalang Marangal (HALAL).

PCOS machines in Philippine automated elections: failure rates, error rates

According to the news, two of the twenty PCOS machines in Hongkong stopped working for a while. That is a 10% failure rate.

Cesar Flores of Smartmatic claims they expect a PCOS failure rate of 0.3- 0.5%. However, vendor claims must be taken with a grain of salt, more so if their goods were hurriedly made in China. The claim is also belied by Smartmatic’s own plans: they are deploying 8% of the total machines for backup. So, they must be expecting up to 8% of the machines to fail, which is more consistent with the failure rates in Hongkong.

The actual PCOS failure rate is, in fact, a big unknown.

First, it appears that Smartmatic had done most of the testing, not COMELEC. Due diligence requires COMELEC to do acceptance testing. Any buyer must double-check delivered goods before signing a receipt acknowledging that it was received in good working condition. Especially since the Smartmatic deliveries involved P7.2 billion of taxpayers’ money, the machines should have been independently tested if they meet COMELEC specifications as detailed in their contract. Those that didn’t meet specs should have been returned for replacement. If deployed, these can cause trouble during election day itself, as they did in Hongkong.

The 0.3-0.5% PCOS failure rate that Smartmatic claims is not backed by properly-witnessed test stats and is contradicted by the 10% failure rate reported in Hongkong and Smartmatic’s own preparations to replace up to 8% of machines that may fail on election day.

Second, the test stats have remained inaccessible to third parties like political parties, election watchdogs and media. Transparent test stats minimize potential insider collusion (as in the ballot secrecy folder contract), which can result in overpricing or payments for sub-standard equipment. Transparency also minimizes the possibility that insiders will selectively assign machines, depending on their quality, in order to cause trouble in targetted regions or provinces in favor one candidate or another. Just imagine if problemmatic machines or modems are selectively assigned to Aquino, Villar, Estrada, or Teodoro bailiwicks — whoever are disfavored.

At least five PCOS test results are so important that they should be publicly known:

  • Mean time between failures (MTBF). This is the average time a PCOS machine stays operational. Knowing the MTBF and the mean time to repair (or replace), we can determine the average failure rate. Instead of actual statistics, we have today media-reported field anecdotes and unsubstantiated vendor claims.
  • Average rejection rate of valid ballots. This is a specific but important case, when the PCOS stays operational but rejects a valid ballot. In Smartmatic demos, field tests and mock elections, the rejection rates were inordinately high, far above COMELEC specs.
  • Scan error rate. Just as PCOS machines fail, they make mistakes too. A PCOS scanning error can be a false positive (registering a vote that is not there) or a false negative (missing a vote that is there). When the PCOS is adjusted to read lighter shades, false positives increase because even a slight smudge may be falsely registered as a vote. When adjusted to read darker shades only, false negatives increase, because lightly or partially shaded ovals may be missed by the PCOS. Each machine has to be calibrated towards that ideal spot which minimizes the total errors from both false positives and false negatives. Based on COMELEC specs, this total should be lower than .005%, or five scanning errors for every 100,000 marks (at most one error per 1,000 ballots). Unfortunately, the calibration may change in transit or under environmental stresses like heat, humidity, or mechanical shocks. A PCOS machine that rejects valid ballots has, in effect, very high false negatives, because it misses all the shaded ovals, each representing one vote, in those rejected but valid ballots.
  • Transmission error rate. Because of ambient electrical and electronic noise, transmission is more susceptible to error than scanning, and therefore demands high quality equipment. That Smartmatic modems had transmission problems even within Metro Manila does not speak well of their quality. A poor quality modem is hopeless and should be replaced.
  • Battery backup life. COMELEC specified at least 16 hours of backup. A good quality control engineer would insist on batteries lasting up to 20 hours under test, a 25% margin for coping with unexpected operating and environmental extremes.

Smartmatic had earlier claimed it was testing 2,000 machines a day. Compare this to the three months it took COMELEC to thoroughly test some 1,900 automated counting machines in 2004. Even granting that the 2004 testing was done at a leisurely pace, the huge difference still makes one wonder how thorough the PCOS testing was.

In particular, the PCOS scan error rate is very important. If the error rate is, say, 5%, and the presidential winner’s margin is less than 5%, then we will again find ourselves in political limbo. In 2004, GMA’s supposed margin over FPJ was 3.48%. COMELEC specified .005%, which is quite low. But it doesn’t look like COMELEC actually measured each machine’s error rate. That is not possible when testing 2,000 machines a day (the necessary statistical test requires more than 1,700 test ballots per PCOS).

Each PCOS should pass various COMELEC tests before it is accepted, paid for, and deployed to a polling place. And stakeholders should have access to all the test statistics, including the number of machines that stopped operating, the number of valid ballots rejected, and the actual number of falsely registered voter choices, to prevent insiders from accepting bad machines and selectively assigning these to targetted areas.

Without the test statistics, we can only guess which is closer to the truth, the 10% failure rate shown by the machines in Hongkong, or the 0.3-0.5% failure rate claimed by Smartmatic.

It is not too late. COMELEC can still order the release to media of these test statistics, and improve its credibility before the voting public.