PCOS machines in Philippine automated elections: failure rates, error rates

According to the news, two of the twenty PCOS machines in Hongkong stopped working for a while. That is a 10% failure rate.

Cesar Flores of Smartmatic claims they expect a PCOS failure rate of 0.3- 0.5%. However, vendor claims must be taken with a grain of salt, more so if their goods were hurriedly made in China. The claim is also belied by Smartmatic’s own plans: they are deploying 8% of the total machines for backup. So, they must be expecting up to 8% of the machines to fail, which is more consistent with the failure rates in Hongkong.

The actual PCOS failure rate is, in fact, a big unknown.

First, it appears that Smartmatic had done most of the testing, not COMELEC. Due diligence requires COMELEC to do acceptance testing. Any buyer must double-check delivered goods before signing a receipt acknowledging that it was received in good working condition. Especially since the Smartmatic deliveries involved P7.2 billion of taxpayers’ money, the machines should have been independently tested if they meet COMELEC specifications as detailed in their contract. Those that didn’t meet specs should have been returned for replacement. If deployed, these can cause trouble during election day itself, as they did in Hongkong.

The 0.3-0.5% PCOS failure rate that Smartmatic claims is not backed by properly-witnessed test stats and is contradicted by the 10% failure rate reported in Hongkong and Smartmatic’s own preparations to replace up to 8% of machines that may fail on election day.

Second, the test stats have remained inaccessible to third parties like political parties, election watchdogs and media. Transparent test stats minimize potential insider collusion (as in the ballot secrecy folder contract), which can result in overpricing or payments for sub-standard equipment. Transparency also minimizes the possibility that insiders will selectively assign machines, depending on their quality, in order to cause trouble in targetted regions or provinces in favor one candidate or another. Just imagine if problemmatic machines or modems are selectively assigned to Aquino, Villar, Estrada, or Teodoro bailiwicks — whoever are disfavored.

At least five PCOS test results are so important that they should be publicly known:

• Mean time between failures (MTBF). This is the average time a PCOS machine stays operational. Knowing the MTBF and the mean time to repair (or replace), we can determine the average failure rate. Instead of actual statistics, we have today media-reported field anecdotes and unsubstantiated vendor claims.

• Average rejection rate of valid ballots. This is a specific but important case, when the PCOS stays operational but rejects a valid ballot. In Smartmatic demos, field tests and mock elections, the rejection rates were inordinately high, far above COMELEC specs.

• Scan error rate. Just as PCOS machines fail, they make mistakes too. A PCOS scanning error can be a false positive (registering a vote that is not there) or a false negative (missing a vote that is there). When the PCOS is adjusted to read lighter shades, false positives increase because even a slight smudge may be falsely registered as a vote. When adjusted to read darker shades only, false negatives increase, because lightly or partially shaded ovals may be missed by the PCOS. Each machine has to be calibrated towards that ideal spot which minimizes the total errors from both false positives and false negatives. Based on COMELEC specs, this total should be lower than .005%, or five scanning errors for every 100,000 marks (at most one error per 1,000 ballots). Unfortunately, the calibration may change in transit or under environmental stresses like heat, humidity, or mechanical shocks. A PCOS machine that rejects valid ballots has, in effect, very high false negatives, because it misses all the shaded ovals, each representing one vote, in those rejected but valid ballots.

• Transmission error rate. Because of ambient electrical and electronic noise, transmission is more susceptible to error than scanning, and therefore demands high quality equipment. That Smartmatic modems had transmission problems even within Metro Manila does not speak well of their quality. A poor quality modem is hopeless and should be replaced.

• Battery backup life. COMELEC specified at least 16 hours of backup. A good quality control engineer would insist on batteries lasting up to 20 hours under test, a 25% margin for coping with unexpected operating and environmental extremes.

Smartmatic had earlier claimed it was testing 2,000 machines a day. Compare this to the three months it took COMELEC to thoroughly test some 1,900 automated counting machines in 2004. Even granting that the 2004 testing was done at a leisurely pace, the huge difference still makes one wonder how thorough the PCOS testing was.

In particular, the PCOS scan error rate is very important. If the error rate is, say, 5%, and the presidential winner’s margin is less than 5%, then we will again find ourselves in political limbo. In 2004, GMA’s supposed margin over FPJ was 3.48%. COMELEC specified .005%, which is quite low. But it doesn’t look like COMELEC actually measured each machine’s error rate. That is not possible when testing 2,000 machines a day (the necessary statistical test requires more than 1,700 test ballots per PCOS).

Each PCOS should pass various COMELEC tests before it is accepted, paid for, and deployed to a polling place. And stakeholders should have access to all the test statistics, including the number of machines that stopped operating, the number of valid ballots rejected, and the actual number of falsely registered voter choices, to prevent insiders from accepting bad machines and selectively assigning these to targetted areas.

Without the test statistics, we can only guess which is closer to the truth, the 10% failure rate shown by the machines in Hongkong, or the 0.3-0.5% failure rate claimed by Smartmatic.

It is not too late. COMELEC can still order the release to media of these test statistics, and improve its credibility before the voting public.