Tag Archives: voting machine

Smartmatic machines are not so smart after all

We are spending P7.2 billion to lease these “smart automatic” machines. It turns out that they are not so smart after all. In fact, they seem downright stupid.

They can’t recognize a check mark or a cross. They can’t recognize ballpen or pencil marks. They need full, dark shadings to be convinced that you want to mark an oval. Isn’t that stupid?

When the security marks were misaligned by a mere one to two millimeters, the machines had trouble finding them. They were making so many mistakes that Smartmatic decided to forget “smart automatic” and go back to manual instead. They will just give election inspectors ultraviolet lamps; the inspectors will shine the lamp on each ballot and decide after an ocular inspection if the ballot is authentic or not. Still better than a dumb machine that can’t find the security mark.

A few days before the May 10 elections, these “smart automatic” machines are supposed to be unsealed for a final test in the field by election inspectors. Reports are now flooding in that many can’t read some of the marks, and can’t count some of the votes. Read the reports:

For the sake of our elections, let us all hope and pray that these problems will be solved before May 10.

How to show that a PCOS voting machine is accurate enough

The COMELEC has already rejected 100% manual audit proposal. That should be the end of it.

Yet, it is hard ignore the voices of the country’s major information technology organizations, business oranizations, legal organizations, election watchdogs, church organizations, and other sectors – all counselling the COMELEC to do a 100% manual audit, since this is the first time the country is trying Smartmatic’s SAES 1800 PCOS machine.

Even in banks that have automated their operations, to the extent of using a bill counter that counts bills automatically, tellers still count the bills manually before handing out withdrawals to clients. And clients too would again count the money manually before leaving. When a depositor withdraws from an ATM, she would also manually count the money issued by the ATM machine, before she leaves the premises. We all do this, despite the fact that we have no reason to doubt the accuracy of the bill counters and ATMs.

Since we do have a strong reason to doubt the accuracy of PCOS machines, all the more should we double-check its count. Why do we doubt the accuracy of the PCOS?

The Comelec has already acknowledged the misalignment of ultraviolet security marks due to high-speed printing. Because of this misalignment, the Comelec decided on its own, without prodding, to ask election inspectors to conduct a 100% manual audit of the authenticity of the ballots, by visually determining with a UV lamp whether each ballot is authentic or not.

Since the Comelec already admitted that the security mark was misaligned, then it should, at the very least, double-check the possibility that the ovals in the ballot are likewise misaligned. If some ballots have misaligned ovals, then for exactly the same reason that the PCOS machine became unreliable in detecting the security marks, it will also become unreliable in detecting shaded ovals.

This is why we doubt the accuracy of the PCOS machines: they might be fed ballots with misaligned ovals caused by the high-speed printing.

The Comelec specified in its contract with Smartmatic that the PCOS should be able to read shaded marks correctly at least 99.995% of the time (in other words, that it makes at most one error for every 20,000 marks or .005%).

I propose a way to convincingly show, with 95% confidence, that a PCOS machine is 99.995% accurate. A 95% confidence level is the typical confidence level used in statistical tests, product quality control, and scientific experiments.

At this point, the math will get a little bit heavy. But since we are talking of P7.2 billion of taxpayers’ money, let us all hunker down and take the extra effort.

Let us suppose that Smartmatic supplied us with machines whose accuracy rates are all slightly lower than the Comelec specification of 99.995%, say, they make one error for every 19,999 instead of 20,000 marks. We want a set of test ballots that will detect these below-spec machines, because they will make one or more reading errors. Then we can reject these inaccurate machines and have them replaced by Smartmatic because they are all slightly below Comelec specifications (unless of course the Comelec was willing to adjust its specs downwards to accomodate Smartmatic).

If we feed one of these below-spec machines a test ballot with a single mark, we know that the chances of a correct reading is 19,998/19,999 or 99.9949997%, which is practically 100%. We’d expect the machine to read the ballot without error. If we used this ballot as our test set, this below-spec machine, which should fail the test, will pass it instead, where “fail” means at least one reading error and “pass” means zero reading error.

If we feed a test ballot with two marks into the same machine, the chances of two consecutive successes in reading marks are 99.99449997% x 99.99449997% or 99.990%. Our below-spec machine will most probably pass such a test ballot too.

If we feed a test ballot with three marks into the same machine, the chances of correctly reading all three are 99.99449997% x 99.99449997% x 99.99449997% or 99.985%. The machine will again probably pass the test.

Brace yourself: the above can be written as (99.99449997%)^3 = 99.985%, where “^3” means “multiplied by itself three times”.

A real ballot will contain an average of 34 marks (one each for president, vice president, party-list, congressman, governor, mayor, and so on, 12 for senators, etc.).

Hence the chances that the below-spec machine will accurately read a single test ballot with 34 marks correctly are (99.99449997%)^34 = 99.83%. This is still very high. The machine will most probably read this test ballot correctly too, and pass the test.

For ten ballots (340 marks): the chances of accurately reading all of them are (99.99449997%)^340 = 98.3%. Still very high.

Note that 3-7 days before election day, election inspectors, watchers and watchdogs in every precinct cluster will use a ten-ballot test set to determine if the PCOS machines meet the Comelec specification of 99.995%. This means they should be able to detect our below-spec machine, so that it may be replaced by Smartmatic. However, the chances that our below-spec machine will make no reading error and therefore pass the test is 98.3%. Thus, out of a hundred of our below-spec machines, the ten-ballot test will correctly detect two but will pass 98 machines that should have failed. This is obviously not acceptable.

For thirty ballots (1,020 marks): the chances of accurately reading all are (99.99449997%)^1020= 95.0%

For 408 ballots (13,872 marks): the chances are (99.99449997%)^13,872 = 50.0%. If we use a 408-ballot set to test the below-spec machines, half of them will be detected successfully, but half will still pass the test. Not good.

For 1,000-ballot test set (34,000 marks): the chances of our machines passing are now (99.99449997%)^34,000 = 18.3%. Out of a hundred below-spec machines, 18 will still pass undetected. Not acceptable.

Let me now jump to the magic number: 1,762 ballots (59,908 marks). The chances are now (99.99449997%)^59,908 = 5.0%. Out of a hundred, ninety-five of the below-spec machines will fail and only five will pass a 1,762-ballot test set. (In an earlier computation, I had used 99.9944% and got 1,573. A 1,762-ballot test set will detect marginally inaccurate machines better.)

This 95% confidence level is the most common standard used in statistical testing, product quality control, and scientific experiments. If we wanted even better quality standards, we can adopt a 99% instead of 95% confidence level. This will require a 2,708-ballot test set. Then, out of every hundred below-spec machines that Smartmatic delivers, 99 will make at least one error and will be correctly rejected. Only one will pass the test and be deployed for vote counting.(This would still be 822 below-spec PCOS machines out of the 82,200, but that’s probably a necessary risk, unless we want even stricter testing).

Let me now summarize how to test each PCOS machine for accuracy:

Prepare a test set of 1,762 official ballots with 34 marks each, making sure that every oval on the ballot is marked at least once. Tabulate the votes represented by the marks. The tally must be checked and rechecked for 100% accuracy, because it is the standard against which the accuracy of the machine will be measured.

Configure a PCOS machine to accept the ballots, and feed the test set into the machine. If the machine reads all the ballots without error, then we can be certain at the 95% confidence level that the machine has an accuracy rate of 99.995% or better. (If I were the test engineer, I would then repeat the test on every accepted machine, and I would only use as backup those machines that fail on the second test.)

If the COMELEC had done this acceptance testing when the machines were delivered, any machine which did not meet the 95% confidence level should have been returned to Smartmatic for replacement. Thus every PCOS machine deployed for the elections should have passed this test. Since such a test can take around seven hours and a half (assuming 15 seconds per ballot), and Smartmatic claimed that it was testing PCOS machines at the rate of 2,000 per day, it is highly unlikely that these machines went through this kind of acceptance testing. So, we do not know how many PCOS machines actually meet the Comelec specifications.

If Smartmatic is confident about the accuracy of its PCOS machines, and if the Comelec is confident about the alignment of the ovals in the 50.85 million ballots they just printed, they should welcome this 1,762-ballot test as a chance to prove that the PCOS machines truly meet the Comelec specifications for accuracy.


Four opportunities to check PCOS accuracy: all taken away by COMELEC

The Filipino voting public had four opportunities for checking the scanning accuracy of the PCOS machine, but the three were all taken away by COMELEC, while the fate of the fourth still hangs in the balance.

Acceptance testing by COMELEC

Everytime Smartmatic delivered a batch of machines, COMELEC should have tested the machines before accepting them, to screen out the lemons. After all, these machines were made in China in a hurry. But even if they were not. It is simple due diligence. The COMELEC specifications are clear in the contract, including the PCOS minimum accuracy rate of 99.995%, or a maximum error rate of .005%. Any machine that did not meet specs should have been returned for replacement. We lost this opportunity because the full results of the testing, if it was ever done by COMELEC at all, were not given to stakeholders like political parties, election watchdogs and the media.

Systest audit and source code review

One of the things Systest would have tested, because it is the heart of the ballot appreciation process, is the scanning accuracy of the PCOS. I had expected a certification from Systest that the PCOS, for instance, met all minimum functional requirements as specified in the Automated Election Law and the contract between COMELEC and Smartmatic. I had expected that the test procedures would be described in detail in the full report of Systest and the test results would be there too, for the scrutiny of all stakeholders. Finally, I had expected the Systest certifications and well as full report to be available to stakeholders like political parties, election watchdogs and the media. COMELEC claimed on Feb. 9 that Systest had finished its audit/review and given its certification, but released no proper Systest document or full report to back up its claim. We lost this opportunity because of the secrecy that surrounds the certifications and full report from Systest.

Voters verification that the PCOS correctly registered their choices

On election day itself, all voters would be in a position to check for themselves the accuracy of the PCOS, by observing whether their marks are properly interpreted by the machine. The PCOS was required by law to show voters their choices (that is, the names of those they voted for), so that they can verify if their choices were correctly registered on the machine. The PCOS, in fact, had this as a built-in feature, and it could show on its screen those choices. This was an excellect feature that empowered every voter to conduct a real-time audit of the accuracy of the machine. We lost this opportunity because COMELEC ordered Smartmatic to disable this feature.

Manual audit of the machine results

The ballots will be counted manually by an audit team, and this count compared with the machine count. Any discrepancy, after the manual count is double-checked, will mean an inaccurate scan by the PCOS, allowing us to actually measure its accuracy rate. We fear that this opportunity may be lost because COMELEC decided to conduct the manual audit of only 1.5% of precincts, and the machine winners may be proclaimed ation anyway, while the audit is still going on. If the machine winners are proclaimed while the audit is still going on, then the audit results won’t matter anymore. We all know how long election protests take to resolve, if they are ever resolved at all. Who would be interested in a futile exercise that is already moot and academic? The rush to proclaim within 48 hours (by May 13), is misplaced because the term of outgoing officials will end on June 30 yet. The manual audit can be given a June 22 deadline, the winners proclaimed on June 23, and still have one week to spare.

If 100% manual audit of UV marks is ok, why not 100% manual audit of ovals?

The question that bugs me is why would COMELEC plug every loophole to make sure that no test results from acceptance testing or the Systest audit/review can be used to question PCOS accuracy, that no inaccuracies are observable to voters on election day, and that when the inaccuracies are eventually discovered through a manual audit, it will be too late to correct them because the machine winners have been proclaimed.

When COMELEC found out that the ultraviolet security marks were misaligned, it ordered a 100% manual audit of every ballot before winners are proclaimed. COMELEC did so on its own, without any prodding.

If the ballot ovals are as misaligned as the ultraviolet security marks, by the same logic, the only solution is a 100% manual audit of every ballot before winners are proclaimed.

Despite Obama’s victory, problems with electronic voting machines should not be ignored

With Obama’s landslide victory over McCain in the 2008 U.S. presidential elections, I hope the problems of electronic voting will not be buried under the euphoria. U.S. media had been filled with all kinds of problems involving voting machines. These problems clearly indicated a trend of errors favoring McCain. There were so many reports in so many states that there seemed to be a machinery of cheating in place to make sure McCain would win.

Search the Web for “electronic voting machines in 2008 U.S. elections” and you will get these reports. Note that the search term given is completely neutral and does not include leading words like problem, error, failure and so forth. Yet, the bulk of the reports on the Internet are about problems associated with voting machines.

If we summarize the 2008 U.S. election experience from the perspective of clean and honest elections, this is how I’d put it: the threat of cheating came from those who controlled the electronic voting machines, and it was the massive turnout, the landslide for Obama, and the vigilance of U.S. election integrity activists which stopped the cheats from succeeding.

We were in a similar situation exactly ten years ago, in 1998, when the landslide victory of Joseph Estrada prevented any cheating effort by the administration party Lakas-NUCD although there were clear indications that the machinery to do so was in place.

We were not so lucky in 2004, when cheating was so rampant and brazen that President Gloria Macapagal-Arroyo herself was caught on tape micro-managing it. Yet, the whole system, including the business community, sections of the Church and even citizens’ watchdogs, colluded to cover up the cheating, probably because they thought “anyone but FPJ” would have been better.

I sure hope Philippine election authorities will get the correct lesson out of the U.S. 2008 experience.

Electronic voting, electronic cheating?

When I was awarded a six-week research fellowship by the University of Oxford’s Internet Institute, I chose to focus on electronic voting. (The term more commonly used in the Philippines is “automated elections”.) My research confirmed my initial suspicion that electronic voting and counting machines bring their own set of troubles. I realized that the COMELEC, as well as the media and the public, should therefore take extra steps to ensure the integrity of automated elections.

One of the things I did was review the experiences of countries that had earlier automated their elections. And I found well-documented cases of problems, errors and failures (download: Automated elections: voting machines have made mistakes too).

These cases included: uninitialized machines, which made ballot stuffing possible; votes not counted or lost; candidates’ votes reversed; contests not counted; ballots not counted; the wrong winner comes out; allowing voting more than once; vote totals that exceed the number of registered voters; negatives votes; unauthorized software replacement; and other problems.

I traced these troubles to deep-seated causes that were inherent with complex technologies, such as: software bugs, which are always present even in high-quality software; hardware problems such as miscalibration; environmental stresses that may worsen hardware problems; poor or flawed design; human errors; and malicious tampering. Since these factors were inherent with complex technologies, we can expect the electronic machine troubles to persist.

In my research, I also found out that insoluble problems associated with direct-recording electronic (DRE) voting machines have already led to their phase out in some states of the U.S.

I also compiled typical costs for DREs and optical scanners (download: The cost of automating elections), and found that DRE technology was much more expensive to implement that optical scanning. (However, because an increasing number of states are junking DREs, their prices are expected to go down, as they are dumped into the Third World.)

Halalang Marangal (HALAL), an election monitoring group that I work with, has already submitted two specific recommendations to the COMELEC as a result of my Oxford study:

1. Use double-entry accounting methods in election tabulation (download: Double-entry accounting in election tallies)to minimize the clerical errors that plague the COMELEC’s current single-entry tabulation system; and

2. Conduct a transparent post-election audit of machine results (download: Post-election audits using statistical sampling), by manually counting ballots from a random sample of precincts to confirm if the electronic voting machines are giving us correct results.

Given the reported problems in the August 2008 ARMM elections, which seem to confirm these troubles with automated elections and voting machines, I again strongly urge the COMELEC to heed our warnings and suggestions.