Author: Phillip Stark
The second risk-limiting audit under California AB 2023 was conducted on May 6 in Monterey County. The contest was a Special all-mail election for Monterey Peninsula Water Management District Director, Division 1. Monterey uses Sequoia equipment. There were two candidates: Brenda Lewis and Thomas M. Mancini, and write-ins. 2111 ballots were cast in all. The reported totals were 1353 reported for Lewis, 742 for Mancini, and 13 write-ins. The remaining 3 ballots were recorded as undervotes and overvotes. Lewis was reported to have 64.18% of the valid votes.
Two members of the public observed the entire audit process, which took roughly 90 minutes including some preliminary explanation of the procedure. They confirmed that their interpretation of the ballots agreed with mine and the elections officials’, and they helped roll the dice used to select ballots at random. In conversations afterward, they seemed quite satisfied with the transparency of the procedure (although perhaps not utterly convinced by the mathematics that justified the details).
The audit was performed as follows. After the ballots had been tabulated officially, elections officials Bates-stamped each with a unique serial number (1962 ballots that were scanned had been stamped prior to audit day; the remaining 149 were stamped as part of the audit). It is my understanding that stamping the ballots took about 5 person-hours in all. The particular risk-limiting auditing method used was extremely simple, although statistically inefficient. It does not require exporting any data at all from the voting system. All it relies on is the audit trail. The calculations required are also very simple (only multiplication). Thus, it might be useful in a wide variety of settings, although the hand-counting burden can get high if the margin (as a percentage of valid votes) is small.
Conventional vote-tabulation audits generally have two parts: (1) confirming that batch subtotals as reported by the voting system add up to the contest totals as reported by the voting system, then (2) confirming that the subtotals are sufficiently accurate to give the right answer (by checking a random sample of subtotals against hand counts for those ballots). A risk-limiting audit at 10% risk based on checking the accuracy of precinct-level reports from the voting system would have required hand counting the majority of the ballots.
If the voting system reported cast vote records for individual ballots, roughly 30 ballots would have sufficed (if the CVRs were all accurate). I had hoped to use the Trachtenberg Election Verification System (TEVS) to obtain cast vote records from ballot scans and perform a “transitive audit” along the lines suggested by Calendrino et al., but moditied to be risk-limiting. Mitch Trachtenberg was extremely helpful in getting TEVS software working for Sequoia ballots. It performed correctly on a set of 25 ballots we had for testing. Moreover, Monterey County Elections scanned all their ballots on an office scanner so that I could process them with TEVS. As it turned out, however, when I tried to process the ballot images this morning to extract CVRs, roughly 10% of the ballot images could not be processed automatically (most likely because of the quality of the sheet-feeding in the scanner). Time did not permit further tweaking of the software settings, so I used a backup plan: “blind ballot polling.” This is the first time ballot polling has been used to perform a risk-limiting audit.
To perform ballot polling, physical ballots were selected at random (using the Bates stamp number to identify them) and interpreted by hand. This selection continued until the fraction of ballots in the sample was sufficiently high to give strong evidence that a full hand count would show that Lewis actually won. If strong evidence that Lewis won had not been forthcoming, or if the sample gave strong evidence that Lewis had not won, there would have been a full hand count. Blind ballot polling ignores the voting system completely and makes its own statistical assessment of who won directly from a random sample of the audit trail.
The method works as follows: Lewis really won if her share of votes among the ballots that showed valid votes was larger than the share of Mancini or write-ins. Her reported margin was so large that I could treat Mancini and write-ins as a single candidate: Lewis won if her share of the valid votes was greater than 50%. I used Wald’s sequential probability ratio test (published in 1945) to audit incrementally by sequentially testing the hypothesis that Lewis received less than 50% of the valid votes. To reject that hypothesis is to conclude that Lewis really won.
The algorithm is simple:
1. Set T = 1.
2. Select a ballot at random from the 2111 cast in the contest.
3. If the ballot is an undervote, overvote, or invalid ballot, go back to step 2.
4. If the ballot shows a vote for Lewis, multiply T by 64.08%/50%.
5. If the ballot shows a vote for Mancini or a write-in, multiply T by (100%-64.08%)/50%.
6. If T > 9.9, stop the audit and declare Lewis to be the winner.
7. If T < 0.011, stop the audit and perform a full hand count.
8. Go back to step 2.
It is a theorem that the chance is at most 10% that this stops short of a full hand count if Lewis did not really win. This method could terminate by looking at as few as 10 ballots if the first 10 all reported votes for Lewis. The expected number of ballots to examine if Lewis really got at least 64.08% of the vote is 58. In this case, it took drawing 92 ballots (89 of which were distinct) before T exceeded 9.9 (it ended up at 10.16), because there were a surprising number of votes for Mancini in the early draws. Among the 92, one was a deliberate undervote (a write-in with no name written in), 56 were for Lewis, and 35 were for Mancini. Among the 56 for Lewis, two ballots had been selected twice; among the 35 for Mancini, one had been selected twice. We thus looked at 89 ballots in all before the audit was done.
This method was practical for this contest because the margin was so large. The work load grows rapidly as the margin shrinks, but it does have the advantage of requiring nothing from the voting system other than the audit trail–no “data plumbing.” On the other hand, while it confirms the outcome with low risk, it does not directly check the accuracy of the voting system, only of the outcome: The voting system might have found the right outcome through a fortuitous cancellation of errors, rather than because it is intrinsically accurate.
The outcome was confirmed with a risk limit of 10%. That is, if the outcome was wrong, there was at least a 90% chance that the audit would lead to a full hand count, which would correct the outcome. The audit required looking at 89 individual ballots. The auditing method also ensured that if Lewis had at least 64.08% of the vote (0.1% less than reported), there was at most a 1% chance that the audit would lead to a full hand count.