Tag Archives: smartmatic

Too soon to call 2010 elections successful

It is too soon to declare the 2010 elections a success.

People want a successful election so badly, that it is easy to get carried away by flood of incoming election returns. Many want to believe that a clean and honest election has finally happened, at last.

But the vice-presidential election is yet to be settled. The contest between the 12th and 13th places in the senatorial race still has to be settled too. Hundreds, perhaps thousands, of local races also await to be settled.

Already news is coming in about delayed Election Returns (ERs), malfunctioning, missing or otherwise questionable memory cards, and other indicators of potential or emerging problems.

As in the manual system, the precinct level count is always the fastest. Even when election inspectors, watchers and the public counted votes by hand, most of the election results had always been available past midnight or early morning. Even under the manual method, the biggest challenge has always been at the municipal level and higher, where wholesale cheating operations occurred.

In fact, the automated election system failed spectacularly its first truly public test a week before election day, when many candidates got zero – a “bawas” — and some got more than the votes actually cast for them – a “dagdag”. The results were worse than most manual counts. Fortunately, the failures in the machine count were so obvious that the election inspectors and watchers noticed them immediately. An embarrassed Comelec quickly called off the public test, and traced the problem to misaligned ovals on the ballot. Because of a last-minute change from single-spacing to double-spacing in the ballot layout for local candidates, their oval locations did not anymore match the coordinates stored in a configuration file in a memory card within the PCOS machine.

Reconfiguring the memory cards was somewhat easier than reprinting ballots, so that is what the Comelec and Smartmatic tried to do.

Smartmatic only had 18,000 spare memory cards and there was little time to recall the rest, so in addition to the spares, Smartmatic recalled the cards that could still be recalled; imported the rest from Hongkong and Taiwan; edited each of the 1,631 ballot layout configuration files (unique for every town); programmed these configuration files into 76,340 memory cards (one for each machine); delivered the 76,340 newly reconfigured memory cards to the waiting machines all over the archipelago; found the right machines for the right memory cards; replaced the misconfigured memory card; and conducted a second round of public testing and sealing of the PCOS machines. All within a span of five days – 120 hours. Aside from some 400 machines that malfunctioned, the rest of the 76,340 machines worked fine and gave the country its first successful automated elections. So they say.

Can we now trust the machine results?

These machines had grievously failed to count a few days earlier. This was followed by a mad rush of recalls, importations, file reconfigurations, card reprogramming, deliveries, reinstallation, and a second round of testing and sealing. In the mad rush, were security procedures and chain of custody considerations still observed? Did anyone see an election inspector with an ultraviolet lamp to check for authentic ballots, for instance? (We have not found anyone who did.) Suppose there were also more subtle problems that a ten-ballot test set was insufficient to detect – ovals that were misaligned by only one or two millimeters, for example, just as the security marks were, or oval coordinates that were purposely changed slightly to shave votes from targetted candidates. Were tests done at all for these potential problems?

Suppose an ATM had earlier given you only half the money than it deducted from your account, and the bank tells you the machine is now ok. Wouldn’t you count your money at least once in subsequent withdrawals? Suppose most ATMs of a bank network shortchanged its clients, wouldn’t they demand that every ATM of that network be carefully tested and recertified for its counting accuracy?

For exactly the same reason, every candidate who lost – and won – in the machine-counted 2010 elections should demand thorough post-election testing and audit for accuracy of every counting machine and its results.

Losing candidates should demand it, because they might have actually won.

Winning candidates – especially those who lead by a huge margin – should demand it, because the gross machine errors a few days earlier and subsequent doubts about machine accuracy have devalued their victory.

Apparent president-elect Noynoy Aquino should demand it, if only for the sake of his running-mate Mar Roxas, who sacrificed his own presidential ambitions to give way to Noynoy.

There was no time for proper testing in the mad rush to the May 10 elections because few wanted the elections postponed. But we have fifty days before June 30, when the new set of elected officials are scheduled to take over. We still have enough time check, double-check, and be sure about the results of the 2010 elections.

In the meantime, the Comelec and local election authorities should not be in a hurry to proclaim winners and declare the elections a success.

Expect a flood of demands for recount from losing candidates

The PCOS fiasco a few days before the May 10 elections has shattered the credibility not only of election automation, but of the entire electoral process itself. The entire process hinges on an accurate count by the PCOS machine. In fact, COMELEC specifications require at least 99.995% accuracy, or at most one error for every 20,000 marks or around 600 ballots. Today, we have very little idea how accurately the PCOS will count our votes.

One minor mistake – changing from a single-spaced to a double-spaced layout – has created a crisis of credibility for the entire elections. Who will accept at face value the PCOS machine counts now? Even a candidate who wins with a narrow margin may suspect that the machine might have trimmed his margin. Certainly, losing candidates can now be expected to demand a manual recount, so they can see for themselves how they actually fared. No assurance from Smartmatic or the Comelec will now suffice, because everyone has seen how the PCOS machine made gross errors during the final testing and sealing.

Many may not realize it yet, but we are in a different ballgame now. May 10 election victories — whoever wins, in whatever position – have just suffered a major devaluation.

The PCOS fiasco has robbed the winners of the May 10 elections of a clear victory. It has created not only a cloud of doubt about the machine results, but also last-minute disruptions in the crucial final days before the elections that will surely create confusion and chaos in many areas.

Deliveries had to be suspended, because the PCOS machines had to stay put in the hubs or sub-hubs, so they can await the CF memory card replacements. The final testing and sealing had to be called off, to avoid further embarassment and damage to the credibility of the automation project. When the replacement memory cards eventually arrive for installation in the machines, only then can these machines leave the distribution hubs and sub-hubs for final deployment.

Designing machines and ballot sets that will only work with each other, but delivering them separately had earlier created a “logistical nightmare”. The nightmare just became worse, because machines, ballot sets, and memory cards are now being delivered separately, and under even greater time pressure.

If the PCOS machine, its associated set of ballots, and its associated memory card, do manage to find themselves reunited in their destination precinct cluster, on time for the May 10 elections, they still need to be tested again. It was obvious from the fiasco that the accuracy of the machines were not checked before they were deployed, so the final testing is our only chance to determine the accuracy of the PCOS. This final testing cannot be dispensed with.

If the final testing with a ten-ballot test set shows even a single error, then we cannot guarantee with 95% confidence that the machine’s error rate is lower than 1%. There’s a good chance it is higher than 1%. Then, the board of election inspectors must either ask for a replacement, or go ahead and use an inaccurate machine.

Where machines don’t arrive, stop working, reject too many valid ballots, or otherwise fail, the board of election inspectors must resort to a manual count. The Comelec says it has prepared the paraphernalia for up to 30% of the precincts. But the precinct election inspectors have ask for the paraphernalia to be delivered first. More delays. Given the confusion, more than 30% of precincts may need to resort to a manual count.

If the paraphernalia for a manual count don’t arrive, the ballot boxes will have to be sealed and watched over, to be counted later.

So, expect confusion and chaos in many areas on election day. Yet, even where everything works smoothly, the machine counts will remain under a cloud of doubt, and losing candidates will surely demand a recount, citing as reason the errors made by the PCOS machines earlier. Who can blame them?

All because of an innocuous-looking suggestion to change from a single-spaced to a double-spaced ballot layout.

The origin of this suggestion must be traced. Who first raised it? Why did Comelec approve it? Why didn’t Smartmatic or Comelec test the accuracy of the PCOS machine with the new layout?

If anyone had intentionally wanted to disrupt the elections and discredit its results, he – or she – couldn’t have done it any better.

HALAL Statement: Smartmatic ballot production software was not certified

HALAL STATEMENT

Last April 30, in response to a request by candidate Joey de Venecia III, the Comelec made public a set of documents relating to the source code review of the Smartmatic software conducted by SysTest Labs Inc. One of the documents was “Certification Test Summary for AES May 2010 Rev. 1.00”, dated March 8, 2010. It summarized another SysTest report “Final AES Certification Test Report for the Smartmatic Automated Electon System (AES)”, which the COMELEC has not made public yet.

The Summary may help explain the recent spate of PCOS failures to read local votes which, Smartmatic admits, is due to their error in configuring the ballot design. Citing the problems SysTest had earlier found in its source code review, the Summary listed several “compensating controls” that were essential in mitigating the Smartmatic software problems that SysTest had identified.

In one compensating control, SysTest was very explicit: “The Ballot Production tool was not subjected to the full certification process; therefore it should not be utilized in the May 10, 2010 election process.” (Summary, p.6) Given the ballot printing problems of the COMELEC, from the misalignment of the ultraviolet security mark to the misconfiguration of the ballot design, HALAL asks the COMELEC and Smartmatic to clarify if they utilized Smartmatic’s Ballot Production tool despite the explicit warning of SysTest.

HALAL notes that the March 8 SysTest summary only gave a conditional endorsement of the Smartmatic software. HALAL further notes that the summary was submitted one month past the AES Law deadline for the legally-required certification “categorically stating that the AES … is operating properly, securely and accurately”. This was the SysTest recommendation in the summary (p.7): “Assuming the abovementioned [compensating] controls are put into practice and that the AES is properly configured, operated and supported, SysTest Labs finds the Smartmatic Automated Election System to be capable of operating properly, securely and accurately and therefore recommends the system for certification and use in the May 10, 2010 election.”

Instead of the categorical statement required by the AES Law R.A. 9369, SysTest’s conditional endorsement was premised on the crucial assumption that all “controls are put into practice”.

According to the AES Law R.A. 9369, the COMELEC Technical Evaluation Committee must “certify, through an established international certification entity, … categorically stating that the AES, including its hardware and software components, is operating properly, securely, and accurately, in accordance with the provisions of this Act based, among others, on the following documented results: 1) … ; 2) … ; 3) The successful completion of a source code review; 4) … “

Given all the problems cited in the Feb. 9 SysTest report (HALAL’s analysis of this report is attached), and the explicit warning in the Mar. 8 SysTest summary report against using Smartmatic’s ballot production tool, it is clear that no certification should have been issued to the Smartmatic software because it would put our national elections at an unacceptably high risk.

Smartmatic machines are not so smart after all

We are spending P7.2 billion to lease these “smart automatic” machines. It turns out that they are not so smart after all. In fact, they seem downright stupid.

They can’t recognize a check mark or a cross. They can’t recognize ballpen or pencil marks. They need full, dark shadings to be convinced that you want to mark an oval. Isn’t that stupid?

When the security marks were misaligned by a mere one to two millimeters, the machines had trouble finding them. They were making so many mistakes that Smartmatic decided to forget “smart automatic” and go back to manual instead. They will just give election inspectors ultraviolet lamps; the inspectors will shine the lamp on each ballot and decide after an ocular inspection if the ballot is authentic or not. Still better than a dumb machine that can’t find the security mark.

A few days before the May 10 elections, these “smart automatic” machines are supposed to be unsealed for a final test in the field by election inspectors. Reports are now flooding in that many can’t read some of the marks, and can’t count some of the votes. Read the reports:

For the sake of our elections, let us all hope and pray that these problems will be solved before May 10.

HALAL analysis of recently-released SysTest source code review

Halalang Marangal (HALAL) recently obtained a copy of the SysTest report on the source code review of the Smartmatic software that it conducted Oct. 26, 2009 to Feb. 9, 2010. This review was the basis for the Comelec concluding that the Smartmatic Automated Election System will count our May 10 votes properly, securely and accurately. The SysTest report and related documents may be downloaded here.

HALAL’s conclusion, after scrutinizing the SysTest report, is that the Smartmatic software should NOT have been certified. We should not have put our national elections at risk given the clear warnings of SysTest about problems in the Smartmatic software.

You may download the HALAL analysis here.

How to show that a PCOS voting machine is accurate enough

The COMELEC has already rejected 100% manual audit proposal. That should be the end of it.

Yet, it is hard ignore the voices of the country’s major information technology organizations, business oranizations, legal organizations, election watchdogs, church organizations, and other sectors – all counselling the COMELEC to do a 100% manual audit, since this is the first time the country is trying Smartmatic’s SAES 1800 PCOS machine.

Even in banks that have automated their operations, to the extent of using a bill counter that counts bills automatically, tellers still count the bills manually before handing out withdrawals to clients. And clients too would again count the money manually before leaving. When a depositor withdraws from an ATM, she would also manually count the money issued by the ATM machine, before she leaves the premises. We all do this, despite the fact that we have no reason to doubt the accuracy of the bill counters and ATMs.

Since we do have a strong reason to doubt the accuracy of PCOS machines, all the more should we double-check its count. Why do we doubt the accuracy of the PCOS?

The Comelec has already acknowledged the misalignment of ultraviolet security marks due to high-speed printing. Because of this misalignment, the Comelec decided on its own, without prodding, to ask election inspectors to conduct a 100% manual audit of the authenticity of the ballots, by visually determining with a UV lamp whether each ballot is authentic or not.

Since the Comelec already admitted that the security mark was misaligned, then it should, at the very least, double-check the possibility that the ovals in the ballot are likewise misaligned. If some ballots have misaligned ovals, then for exactly the same reason that the PCOS machine became unreliable in detecting the security marks, it will also become unreliable in detecting shaded ovals.

This is why we doubt the accuracy of the PCOS machines: they might be fed ballots with misaligned ovals caused by the high-speed printing.

The Comelec specified in its contract with Smartmatic that the PCOS should be able to read shaded marks correctly at least 99.995% of the time (in other words, that it makes at most one error for every 20,000 marks or .005%).

I propose a way to convincingly show, with 95% confidence, that a PCOS machine is 99.995% accurate. A 95% confidence level is the typical confidence level used in statistical tests, product quality control, and scientific experiments.

At this point, the math will get a little bit heavy. But since we are talking of P7.2 billion of taxpayers’ money, let us all hunker down and take the extra effort.

Let us suppose that Smartmatic supplied us with machines whose accuracy rates are all slightly lower than the Comelec specification of 99.995%, say, they make one error for every 19,999 instead of 20,000 marks. We want a set of test ballots that will detect these below-spec machines, because they will make one or more reading errors. Then we can reject these inaccurate machines and have them replaced by Smartmatic because they are all slightly below Comelec specifications (unless of course the Comelec was willing to adjust its specs downwards to accomodate Smartmatic).

If we feed one of these below-spec machines a test ballot with a single mark, we know that the chances of a correct reading is 19,998/19,999 or 99.9949997%, which is practically 100%. We’d expect the machine to read the ballot without error. If we used this ballot as our test set, this below-spec machine, which should fail the test, will pass it instead, where “fail” means at least one reading error and “pass” means zero reading error.

If we feed a test ballot with two marks into the same machine, the chances of two consecutive successes in reading marks are 99.99449997% x 99.99449997% or 99.990%. Our below-spec machine will most probably pass such a test ballot too.

If we feed a test ballot with three marks into the same machine, the chances of correctly reading all three are 99.99449997% x 99.99449997% x 99.99449997% or 99.985%. The machine will again probably pass the test.

Brace yourself: the above can be written as (99.99449997%)^3 = 99.985%, where “^3” means “multiplied by itself three times”.

A real ballot will contain an average of 34 marks (one each for president, vice president, party-list, congressman, governor, mayor, and so on, 12 for senators, etc.).

Hence the chances that the below-spec machine will accurately read a single test ballot with 34 marks correctly are (99.99449997%)^34 = 99.83%. This is still very high. The machine will most probably read this test ballot correctly too, and pass the test.

For ten ballots (340 marks): the chances of accurately reading all of them are (99.99449997%)^340 = 98.3%. Still very high.

Note that 3-7 days before election day, election inspectors, watchers and watchdogs in every precinct cluster will use a ten-ballot test set to determine if the PCOS machines meet the Comelec specification of 99.995%. This means they should be able to detect our below-spec machine, so that it may be replaced by Smartmatic. However, the chances that our below-spec machine will make no reading error and therefore pass the test is 98.3%. Thus, out of a hundred of our below-spec machines, the ten-ballot test will correctly detect two but will pass 98 machines that should have failed. This is obviously not acceptable.

For thirty ballots (1,020 marks): the chances of accurately reading all are (99.99449997%)^1020= 95.0%

For 408 ballots (13,872 marks): the chances are (99.99449997%)^13,872 = 50.0%. If we use a 408-ballot set to test the below-spec machines, half of them will be detected successfully, but half will still pass the test. Not good.

For 1,000-ballot test set (34,000 marks): the chances of our machines passing are now (99.99449997%)^34,000 = 18.3%. Out of a hundred below-spec machines, 18 will still pass undetected. Not acceptable.

Let me now jump to the magic number: 1,762 ballots (59,908 marks). The chances are now (99.99449997%)^59,908 = 5.0%. Out of a hundred, ninety-five of the below-spec machines will fail and only five will pass a 1,762-ballot test set. (In an earlier computation, I had used 99.9944% and got 1,573. A 1,762-ballot test set will detect marginally inaccurate machines better.)

This 95% confidence level is the most common standard used in statistical testing, product quality control, and scientific experiments. If we wanted even better quality standards, we can adopt a 99% instead of 95% confidence level. This will require a 2,708-ballot test set. Then, out of every hundred below-spec machines that Smartmatic delivers, 99 will make at least one error and will be correctly rejected. Only one will pass the test and be deployed for vote counting.(This would still be 822 below-spec PCOS machines out of the 82,200, but that’s probably a necessary risk, unless we want even stricter testing).

Let me now summarize how to test each PCOS machine for accuracy:

Prepare a test set of 1,762 official ballots with 34 marks each, making sure that every oval on the ballot is marked at least once. Tabulate the votes represented by the marks. The tally must be checked and rechecked for 100% accuracy, because it is the standard against which the accuracy of the machine will be measured.

Configure a PCOS machine to accept the ballots, and feed the test set into the machine. If the machine reads all the ballots without error, then we can be certain at the 95% confidence level that the machine has an accuracy rate of 99.995% or better. (If I were the test engineer, I would then repeat the test on every accepted machine, and I would only use as backup those machines that fail on the second test.)

If the COMELEC had done this acceptance testing when the machines were delivered, any machine which did not meet the 95% confidence level should have been returned to Smartmatic for replacement. Thus every PCOS machine deployed for the elections should have passed this test. Since such a test can take around seven hours and a half (assuming 15 seconds per ballot), and Smartmatic claimed that it was testing PCOS machines at the rate of 2,000 per day, it is highly unlikely that these machines went through this kind of acceptance testing. So, we do not know how many PCOS machines actually meet the Comelec specifications.

If Smartmatic is confident about the accuracy of its PCOS machines, and if the Comelec is confident about the alignment of the ovals in the 50.85 million ballots they just printed, they should welcome this 1,762-ballot test as a chance to prove that the PCOS machines truly meet the Comelec specifications for accuracy.


The high-speed printing that misaligned UV marks can misalign the ballot ovals too

The ultraviolet (UV) scanner of the PCOS was disabled because the high-speed printing resulted in the UV mark on the ballot being “misaligned by one to two millimeters”, according to COMELEC Commissioner Larrazabal. As a result, the machine’s UV scanner often missed the mark and many valid ballots being being rejected.

Here’s the big question: If the misalignment of the UV mark was serious enough to make COMELEC turn off the UV scanner, then can’t the ovals be misaligned too? A misaligned oval means a misaligned a vote-mark. Which means the PCOS main scanner may have problems interpreting the voters’ choices.

The Comelec requires from the PCOS an error rate of less than .005%. That means less than five errors for every 100,000 marks. To determine if this Comelec specification is met, each PCOS should have been tested properly. But no test statistics have been released by the Comelec. And if these tests were done at all, they were probably done with perfectly aligned, not misaligned, ovals.

With misalignment, the two types of errors the scanner can make will both get worse: the false positives that register a vote/mark which is not there, and the false negatives that miss a vote/mark which is there. This means some candidates will gain votes (“dagdag”), while other candidates will lose votes (“bawas”). Does that sound familiar?

This problem is made worse by the Comelec decision to, in effect, blindfold voters while the machine is registering their choice. Originally, the PCOS was programmed to display the voter’s choices on its screen, so he can check if his choices were correctly registered by the PCOS and abort the process if the PCOS didn’t. This voter-verification is in fact required by the Automated Election Law (Section 7n): “Provide the voter a system of verification to find out whether or not the machine has registered his choice.” The Comelec ordered Smartmatic to disable this feature. Thus, the PCOS may falsely register voters’ choices, without voters knowing it. This is far worse than a PCOS that stops working or rejects valid ballots, problems which are apparent at once. An inaccurate PCOS will keep scanning and counting happily, with no indication or warning that it is miscounting votes.

According to a former Comelec official, similar problems had also occurred when the Comelec piloted automation in ARMM in 1998. He showed me a Comelec report entitled “Partial Automation of 1998 National and Local Elections”, which is also on the Comelec website. In 1998, said the report, Sulu ballots had to be manually recounted “due to an error in NPO’s printing procedure”. A similar recount was done with Lanao del Sur ballots, “also due to errors in printing of the ballots.”

The Comelec blamed “high-speed printing” for the misalignment. But misalignment is a common printing problem, and good operators know how to correct it. Why would it happen in such an important job as the printing of ballots?

It seems that the ballot printing is being done in such a hurry, that operators are not getting the time needed to stop the high-speed machines and correct any misalignments that may be happening. That the Comelec had insisted on printing ballots even on Maundy Thursday and Good Friday shows how disastrous even a slight delay might be. If they stopped the machines too often to correct for misalignments, they might miss their deadlines. Thus, the UV misalignment problem has gone on uncorrected.

As of April 16, according to Smartmatic, the NPO has printed 43.7 million ballots. If the high-speed printing has misaligned the UV marks, then how many of these ballots have misaligned ovals too? And in these misaligned ovals, how many false “dagdag-bawas” interpretations by the PCOS will occur?

The misalignment of UV marks and possibly the ovals too are the consequences of the Comelec violating the law and using a machine that has neither been piloted nor used widely. The SAES 1800 has never been piloted in the Philippines, as Section 6 of the Automated Election Law requires: “… the AES shall be used in at least two urbanized cities and two provinces each in Luzon, Visayas and Mindanao.” Nor has the PCOS been used widely in any other country, as Section 10 of the law also requires: “… the system procured must have demonstrated capability and been successfully used in a prior electoral exercise here or abroad.” The Supreme Court’s support for the Comelec’s stubborn insistence in using unpiloted Smartmatic machines must have reinforced the Comelec’s sense of impunity in violating provisions of the law.

Because of the lack of pilot, we lost the chance to detect early problems like these. As a result, they have put the entire national elections at risk.

Solution: both the PCOS and their associated ballots must be thoroughly checked for misalignment and accuracy in scanning. The tests must be done not by Smartmatic, but by independent third parties, say, the DOST, and witnessed by all stakeholders.

What if careful testing shows that the PCOS cannot reliably read properly shaded ovals? Then we may have no choice but to manually count the votes again, ballot by ballot.

[Note: This piece is based on the Halalang Marangal April 17 Statement on the problem of potentially-misaligned ovals, which I also drafted. It was published by the Philippine Star, p.18, on April 21, 2010.]

Roberto Verzola has a background in engineering and economics and a passion for social issues. He is recognized by the IT industry as an Internet pioneer in the Philippines and is often tapped by NGOs for technical advice. He currently lectures at the Institute of Mathematics of the University of the Philippines and is a convenor and secretary-general of the election watchdog Halalang Marangal (HALAL).