HALAL analysis: automated election has 25% chance of success
[As of March 31, we have updated our assessment and now put the chance of success at 28%. See this post on the ballot-printing sub-project for details.]
by Halalang Marangal
Last March 8, Smartmatic-TIM full-page ads came out in some national dailies, claiming “a vote of confidence for the 2010 automated elections”, and listing the accomplishments of the five subsystems under the Automated Election System (AES). The five AES subsystems are: 1. Hardware, supplies, consumables; 2. Software, certification, voter education; 3. Logistics, support, preparations; 4. Telecommunications and Transmission; and 5. Ballot printing infrastructure.
Halalang Marangal (HALAL) carefully evaluated these Smartmatic-TIM claims. We have concluded that, in fact, serious problems beset each of the five subsystems, reducing the AES chances of success and creating opportunities for cheats to manipulate the election results, as they had routinely done in the past.
Remember that most election fraud are inside jobs. HALAL is less worried about hackers and other external threats. We are more worried about cheats who have inside access to the various AES subsystems to do what they have always done with impunity under the manual system.
Note also that by “success”, we mean the absence of significant cheating and similar problems that have chronically attended our elections and a canvassing period that is significantly shorter than the manual method. Otherwise, we would still consider the AES a failure. This “failure of authomation” is different from the legalistic term “failure of election”, by which election officials mean that no voting actually occurred. By their definition, if voters were able to cast ballots, then there was no failure of election.
Due to space limitations, we will cover only the most serious of the problems we identified.
Subsystem 1: Hardware, supplies and consumables
Claim: “82,200 PCOS machines [and batteries] manufactured and delivered”. Note the glaring omission – no mention of the number of machines tested and accepted by the Comelec. Due diligence requires that Comelec personnel – not Smartmatic-TIM – thoroughly test each of these machines for compliance with contract specifications. The Comelec should not accept, deploy or pay for machines which do not meet contract specs. Instead, it should ask Smartmatic-TIM to replace these machines.
Can the Comelec finish the testing on time? HALAL convenor and former Comelec Commissioner Mehol Sadain recalls that in 2004, they needed three months to thoroughly test 1,990 counting machines. Given this experience and the Smartmatic delivery delays, thorough testing of 82,200 machines is an imposing challenge indeed. If the tests are rushed – Smartmatic says they are testing 2,000 machines a day — then we risk deploying for May 10 hurriedly tested machines which can fail, reject valid ballots, or scan inaccurately.
Among the tests results, HALAL considers most important the following: failure rates (the machine mean time between failure or MTBF); the rate of rejection of valid ballots; and the scan error rate (less than 5 in 100,000 marks, according to contract specs). The failure and errors rates in the transmission equipment are also extremely important. We have tried asking the Comelec, political parties, as well as election watchdogs if they have obtained any test statistics. Aside from the field tests and mock elections, when media reported inordinately high ballot rejection rates and transmission problems, there seems to be a complete blackout regarding the test results. This is a bad sign.
Consider the implications of secret testing by Smartmatic-TIM: “good” machines can be selectively assigned to some regions and “bad” machines to other regions. This can easily bias voter turnout in favor of some candidates. Not to mention the Comelec (actually the Filipino taxpayer) paying for substandard machines. “Good” and “bad” modems can likewise be deployed selectively, causing more transmission problems in targetted areas.
Claim: “180.640 compact flash memory cards purchased”. Let us do the arithmetic. Some 82,200 PCOS machines will use two memory cards each. So only 164,400 are needed. Since these cards are solid-state devices, their failures rates are extremely low, compared to the PCOS, which include mechanical parts. Smartmatic-TIM bought 20% more memory cards than necessary. These extra cards, loaded with false results, may be surreptiously used to replace the authetic cards.
Given these and other concerns, HALAL assesses the probability of success of Subsystem 1 at 80%. That is actually a generous figure.
Subsystem 2: Software, certification, voter education
Claim: “Source code customization to meet the requirements of the Philippine elections finished”. The source code was actually customized in a way that violates the requirement of election law for voter verification. The PCOS has a built-in feature that displays on screen the names of candidates the voter has marked. Voters can then verify if their voting intentions were accurately interpreted by the machine. If not, they can abort, and feed their ballot again. If it did, voters can then confirm and press the CAST button. This feature is absolutely necessary to assure voters that the machine scanned their ballots accurately.
Smartmatic disabled this feature, taking away the only opportunity for voters to check the scanning accuracy of the machine on election day. Given the blackout in the results of pre-election testing, and the Comelec plan to conduct the post-election audit of machine results after the proclamation of candidates, we have lost all the three opportunities to determine the scanning accuracy of the machines. This is not reassuring.
Claim: “System audit … finished”; “source code public review process opened”. The law required both a system audit – which covers all the five subsystems of the AES – and a source code review – which is specific to the software programs that control the PCOS and the canvassing servers. The Comelec contracted for this purpose the U.S. firm Systest Labs. Last Feb. 9, the Comelec claimed that the system audit and source code review were done, meeting the Feb. 10 deadline set by the law.
Here’s the rub: neither Systest, Smartmatic-TIM nor the Comelec have released to the public any proper certification document. Such a document should state unequivocally that the AES and its five subsystems, as well as the source code, indeed meet the Comelec requirements of quality, reliability and security as specified in detail in the contract with Smartmatic-TIM. Where are these certification documents? Neither has the full report of Systest Labs been released to the public. Without them, we are justified in asking: are the Systest system audit and source code review actually done, or not yet? Comelec insiders have informed us of a “series of written exchanges” between Systest and the Comelec Technical Evaluation Committee on certain concerns regarding the Systest audit and review. What were these concerns? The only way we can be convinced that Systest has actually certified the AES and its source code is for the Comelec to release to the public the certification documents and full reports of Systest.
The Comelec did open the source code review process to the public. But the conditions it imposed are so unrealistically restrictive, that they make it extremely difficult to conduct a proper local review. Surely, the Comelec did not impose the same restrictive terms and conditions on Systest, when the latter conducted their review.
It is important to appreciate why the source code must be open to public review. The source code is Smartmatic’s general instructions to its machines, in the same way that the Comelec issues general instructions to election inspectors and canvassers. Just as it is totally unacceptable for the Comelec to keep its general instructions secret, it is also totally unacceptable for Smartmatic to keep its general instructions to its machines secret. This is a fundamental issue in a democracy. Our election law fortunately recognized this, and required the prompt release of the source code for public review as soon as the technology is selected. As of today, however, due to the restrictions imposed by the Comelec, no local group has yet conducted any review of the source code. Only two foreign companies – Smartmatic and Systest – have so far seen the general instructions to the machines that will determine our political future. Systest took more than four months to conduct its review. Less than two months before the elections, no local stakeholder including political parties or election watchdogs, have reviewed the source code yet. Even if the Comelec should relax its restrictions tomorrow, a proper review is hardly possible anymore. Smartmatic knew about the open source requirement of the law when they submitted their bid and signed the contract with the Comelec, they cannot invoke commercial confidentiality after winning the contract.
Claim: “Successful field tests and mock elections”. We have read the media reports on high ballot rejection rates and well as transmission problems right in Metro Manila. If Smartmatic can misrepresent results this way, what else are they misrepresenting?
Given these concerns, HALAL assesses the probability of success of Subsystem 2 at 70%.
Subsystem 3: Logistics, support and preparations
Claim: “Over 36,000 voting centers surveyed … [for] network signals, power.” etc. Since we have some 48,000 voting centers, that’s 75% of voting centers covered as of March 8.
Claim: “904 testing … employees working two shifts”. Even three shifts is not enough, knowing that 1,990 machines took the Comelec three months to complete their tests. Furthermore, vendor testing is the vendor’s problem. They should have tested these machines in China, before shipping them here. What we want is testing by the Comelec – due diligence. After all, it is our elections, and it is our money that will be paying for the machines.
Claim: “Contracts with logistics providers and forwarders signed.” According to reports by the newspapers Malaya and Daily Tribune, the three forwarders hired by Smartmatic are: Argo Intl Forwarders (P3.7M 2008 retained earnings, 0.42% of 2008 domestic cargo traffic, 11th place); Germalin Enterprises (P2.3M 2006 net income, 0.35% of 2008 domestic cargo traffic, 12th place); and ACF Logistics Worldwide (P1.1M 2008 cash balance; not in the top 30). Given the herculean task they are entrusted with, the financial capabilities of these companies do not inspire much confidence. Smartmatic should release the list of their field offices, so that stakeholders can double-check their capacity for delivering goods on time.
Claim: “Recruitment and training of over 48,000 field support technicians started.” Since the ad came out 60 days before election day, we can only gape in disbelief: “Started”?
Claim: “438 Comelec training personnel certified”. And at least 230,000 elections officials more to train in the next 60 days.
The Comelec needs to make public the results of the Smartmatic survey for signal and power, as well as the distribution of the forwarders field offices. We must be wary of “problems” in delivery, power availability, and signal transmission, lest these be used to selectively affect voter turnout in some regions or provinces, in a way that can bias the outcome of the election.
HALAL – quite generously – assesses the probability of success of Subsystem 3 at 80%.
Subsystem 4: Telecommunications and transmission
Claim: “48,000 modems for transmission manufactured and delivered”. Again, the missing word here is “tested”. If these made-in-China modems can cause transmission problems right in Metro Manila, something could be wrong with their quality. If Smartmatic delivers a mix of good and bad modems, these can be selectively assigned by region or province to cause transmission and other problems in areas where election cheats want to operate.
Claim: “46,000 SIM cards secured”. Only 46,000 SIM cards for 48,000 modems? “5,500 BGAN transmitters purchased” and “680 VSAT transmitters leased”. With the 48,000 modems, these add up to 54,180 tranmitting equipment, enough for 71.8% of the machines. An extremely serious problem actually hangs over the security of the transmission process: instead of an independept body, Smartmatic, controls the entire system of passwords and digital signatures, from generation to certification. In a business setting, this is equivalent to merging in a single person the duties of vendor, operator, accountant, cashier and auditor – an open invitation to fraud.
Claim: “Contract with major telcos … secured.” But Smartmatic’s own survey says the telcos can cover at most 70% of the precincts. If transmission problems can occur even in Metro Manila, as we all realized during the mock elections, what about smaller cities and municipalities?
HALAL estimates the probability of success of Subsystem 4 at 70%.
Subsystem 5: Ballot printing infrastructure
Claim: “Over 10 million ballots with invisible ultraviolet mark and unique barcode printed.” The printing of ballots started on Feb. 8. A confidential internal Comelec memo was recently leaked to the media which said: as of March 1 (20 days after Feb. 8), 7.9 million ballots had been printed. Let us do the arithmetic: 7.9 million ballots for 20 days is 394,000 ballots per day. At this rate, in 60 more days – March 2 to April 30 – some 21.3 million more ballots can still be printed, for a total of 39.2 million, not quite the 50 million needed for a 1:1 ballot-to-voter ratio.
Remember that this is not a single print job, but some 1,600 jobs, because each city/ municipality has its own list of candidates. The Comelec expects to do 20 different print jobs a day over 80 days; practically one print job every hour. Even the scheduling of the print jobs – if done in a biased way – can matter, because the ballots from print jobs scheduled later are in greater risk of late deliveries.
Aside from printing delays, another problem lurks. An important quality issue in any print run is “registration” — the ovals must be printed exactly where the PCOS expects them to be. Any misalignment in the ballot can cause the machine to scan some positions inaccurately, creating a slight bias in favor of one oval versus another. Such misalignments can occur to any candidate (tough luck!), but election cheats can also exploit this to favor some candidates over others.
HALAL has reason to believe that the poor quality of ultraviolet printing is a cause of the inordinate number of rejections of valid ballots by the PCOS. This is probably why the machine’s ultraviolet scanning feature, which helps distinguish authentic from fake ballots, has been disabled, as former Chief Justice Arturo Panganiban revealed in a column. That’s one less security measure election cheats have to worry about.
HALAL estimates the probability of success of Subsystem 5 at 80%.
Let us now summarize our assessment of Smartmatic-TIM’s five AES subsystems:
Subsystem Probability of Success
- Hardware, supplies and consumables — 80%
- Software, certification, voter education — 70%
- Logistics, support and preparations — 80%
- Telecommunications and transmission — 70%
- Ballot printing infrastructure — 80%
We now turn to a fundamental principle in project management: to get the overall probability of success of a project, which relies on a series of sub-projects, each of which is essential to the project, the sub-projects’ probabilities of success must be multiplied together. Essentially the same principle is followed in product design and reliability engineering.
Thus a system with five subsystems, each with a 99% probability of success, will have an overall probability of success of .99 x .99 x .99 x .99 x .99 or .95 (i.e., 95%). If each of the five subsystems had a 95% probability of success, the probability of success of the overall system is 77%. (Try it on your calculator!) Five subsystems with a success probability of 90% each will give the overall system a success probability of only 59%. Given the problems we pointed out above, we are not ready to assign such optimistic probabilities to the AES project. If some are willing to give Smartmatic the benefit of the doubt, and assign 80% probabilities of success to each of their subprojects, that is still a 33% probability of success overall, a two-to-one odds in favor of AES failure.
HALAL’s assessment of the subprojects’ chances of success leads to .8 x .7 x .8 x .7 x .8 or a 25% overall probability of success for the AES. Feel free to make your own estimates. The conclusion seems inescapable: the risk of a May 10 AES failure is unacceptably high, especially for an election at this important juncture of our political history when failure should not be an option.
Unfortunately the Comelec appears to remain in denial about these problems. It continues to put up a confident face, pretending that nothing is wrong and everything is going on as scheduled. Yet, the automated elections have so many points of vulnerability that Murphy’s Law will almost surely kick in. Thus, the Comelec needs to prepare every precinct for a back up manual system in case some machines fail or are delivered late, or their replacements don’t come on time, or valid ballots are rejected by the machines, and finally for the legally mandated post-election manual audit. But if the preparations for the automated elections are delayed, those for the manual back up are even more so.
Thus, the Comelec has painted the whole country into a corner. If we are lucky enough, the Comelec may still meet its deadlines. After all, a random throw of two coins does come up with two heads 25% of the time. But the same toss will not have two heads 75% of the time, the same risk of failure facing the automated elections. To prepare for this more probable eventuality, we must now wrack our heads and find a way out of this black hole that automation is threatening us with.
March 28, 2010
The Halalang Marangal (HALAL) convenors are: former Senator Wigberto Tañada, retired General Francisco Gudani, former Comelec Commissioner Mehol Sadain, PRRM president Isagani Serrano, former St. Scholastica’s College president Sr. Mary John Mananzan, TOYM awardee Atty. Ma. Paz Luna, and IT expert Roberto Verzola.