Standardize CVR identification across audit logs and RCTab output #913

nurse-the-code · 2025-01-09T00:29:37Z

Key Changes:

Fixes getId method to use logic that prints "RCTab CVR Id" to audit log.
- Uses computedId as primary identifier with fallback to suppliedId
Added getSuppliedId method to CVR to print computeId to rctab_cvr.csv as `"Vendor Id"
Updates both audit logs and rctab_cvr.csv to use the getId method
Changes Dominion computedId delimiter from pipe (|) to dash (-) for improved readability
Adds column "RCTab CVR Id" to RCTab CVR output

Benefits:

Same ballot maintains the same identifier across multiple tabulations by using centralized getId logic
Easier to track specific ballots across audit logs and RCTab CVR output
More intuitive ID format using dashes instead of pipes
Moving toward consistent identification approach throughout the application

Known Issues:

See TODOs in code asking if we can to use the updated getId method, the getComputedId, or something else.

Testing Needed:

Verify ID consistency:
- ✅ Compare IDs in audit log with rctab_cvr.csv output
- ✅ Ensure id returned from getId method is consistent across multiple tabulations
- ✅ Check ID format follows tabulator-batch-record pattern for records from Dominion CVR sources
Test fallback behavior:
- ✅ Cases where computedId is null
- ✅ Cases where computedId is empty or whitespace
- ✅ Cases where only suppliedId is available
Test with each CVR provider
- ✅ CDF/Unisyn
- ✅ Clear Ballot
- ✅ Dominion (tested with 2024 Alaska US House race)
- ✅ ES&S (in test suite)
- ✅ Hart
- ✅ generic CSV
Review and update failing tests:
- ✅ Done for now

Feature Request: Add Standardized Unique Identifiers for Cast Vote Records #911

Replace UUID-based identification with a more consistent ID system that uses the same identifier across audit logs and RCTab CVR output. The ID is primarily derived from the computedId (tabulator-batch-record) with fallback to suppliedId when computedId is unavailable. Unlike UUIDs which change between tabulations, this approach ensures the same ballot maintains the same identifier across multiple tabulations, making it easier to track specific ballots across different runs. Changes: - Remove UUID generation for CVR identification - Add getPrimaryId() method to centralize ID logic - Update audit log and RCTab CVR to use the same ID format - Change ID delimiter from pipe to dash for better readability - Add TODO to review legacy getId() usage Note: This change ensures that CVR IDs are consistent between audit logs, RCTab CVR output, and repeated tabulations, making it easier to track individual ballots across different output formats and multiple runs.

nurse-the-code · 2025-01-09T00:48:01Z

src/main/java/network/brightspots/rcv/CastVoteRecord.java

+  String getPrimaryId() {
+    return !isNullOrBlank(computedId) ? computedId : suppliedId;
+  }


This is a refactoring of the computedId and suppliedId logic used below in the logStringBuilder (which should print a unique CVR Id to the audit log.

src/main/java/network/brightspots/rcv/CastVoteRecord.java

…ture/issue-911-rctab-cvr-primary-id

src/main/java/network/brightspots/rcv/CastVoteRecord.java

src/main/java/network/brightspots/rcv/OutputWriter.java

Dominion"

nurse-the-code · 2025-01-22T10:49:06Z

src/main/java/network/brightspots/rcv/OutputWriter.java

@@ -981,6 +984,7 @@ private Map<String, Object> generateCvrSnapshotMap(
            entry("@type", "CVR.CVRContest"));

    return Map.ofEntries(
+        // TODO: Do we want this to use the getSuppliedId or the getId method?
        entry("@id", generateCvrSnapshotId(sanitizeStringForOutput(cvr.getId()), round)),


Our choice here should be the same as in the TODO comment added before the sanitizedId definition, so long as we are offering a unique CVR id to include in @id.

nurse-the-code · 2025-01-22T11:03:19Z

src/main/java/network/brightspots/rcv/OutputWriter.java

@@ -875,6 +877,7 @@ private List<Map<String, Object>> generateCdfMapForCvrs(List<CastVoteRecord> cas
    for (CastVoteRecord cvr : castVoteRecords) {
      List<Map<String, Object>> cvrSnapshots = new LinkedList<>();
      cvrSnapshots.add(generateCvrSnapshotMap(cvr, null, null));
+      // TODO: Do we want this to use the getSuppliedId or the getId method?


In researching this question, I looked at the NIST Cast Vote Records Common Data Format Specification v 1.0 document. Here are some relevant parts I found.

on page 18:

on page 31:

This prints out the BallotPrePrintedId from the CDF's CVR class. It would seem to me that this should be a vendor-supplied id (e.g. suppliedId). I am not sure that we were using this correctly prior to this PR, because we would sometimes print the computedId (which was never pre-printed on any ballot). I am also not sure what this should look like if there is no vender-supplied id.

nurse-the-code · 2025-01-22T12:09:30Z

...ces/network/brightspots/rcv/test_data/conversions_from_ess/conversions_from_ess_expected.csv

-Source Filepath,CVR Provider,Contest Id,Tabulator Id,Batch Id,Record Id,Precinct,Precinct Portion,Rank 1,Rank 2,Rank 3
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-1,,,Mookie Blaylock,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-2,,,Mookie Blaylock,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-3,,,Mookie Blaylock,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-4,,,Mookie Blaylock,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-5,,,Yinka Dare,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-6,,,Yinka Dare,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-7,,,Yinka Dare,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-8,,,George Gervin,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-9,,,George Gervin,Yinka Dare,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-10,,,Sedale Threatt,George Gervin,undervote
+Source Filepath,CVR Provider,Contest Id,RCTab CVR Id,Tabulator Id,Batch Id,Vendor Id,Precinct,Precinct Portion,Rank 1,Rank 2,Rank 3
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-1,,,,,,Mookie Blaylock,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-2,,,,,,Mookie Blaylock,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-3,,,,,,Mookie Blaylock,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-4,,,,,,Mookie Blaylock,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-5,,,,,,Yinka Dare,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-6,,,,,,Yinka Dare,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-7,,,,,,Yinka Dare,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-8,,,,,,George Gervin,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-9,,,,,,George Gervin,Yinka Dare,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-10,,,,,,Sedale Threatt,George Gervin,undervote


For ES&S CVRs, it looks like our new getSuppliedId method (slightly changed from the original getId method results in no "Vendor Id" (formerly "Record Id") getting printed to the rctab_cvr.csv. Instead, that same value is printed as the "RCTab CVR Id" (using the new getId method).

This is because the old getId method (and the new getId method as well) returns a computedId for ES&S CVRs. The new getSuppliedId returns an empty string, because ES&S CVRs don't have suppliedId associated with them in RCTab.

methods.

src/test/java/network/brightspots/rcv/CastVoteRecordTests.java

…ture/issue-911-rctab-cvr-primary-id

nurse-the-code · 2025-01-30T15:28:52Z

src/main/java/network/brightspots/rcv/OutputWriter.java

@@ -696,14 +696,15 @@ String writeRcTabCvrCsv(
      CSVFormat format = CSVFormat.DEFAULT.builder().setNullString("").build();
      csvPrinter = new CSVPrinter(writer, format);
      // print header:
-      // ContestId, TabulatorId, BatchId, RecordId, Precinct, Precinct Portion, rank 1 selection,
-      // rank 2 selection, ... rank maxRanks selection
+      // RCTab CVR Id, ContestId, TabulatorId, BatchId, RecordId, Precinct, Precinct Portion,


reorder the variables in this comment

null. Null handling is now handled where getSuppliedId is called.

src/main/java/network/brightspots/rcv/OutputWriter.java

nurse-the-code added 2 commits January 8, 2025 02:47

Experimental implementation of GitHub issue 991:

7e7e29b

Feature Request: Add Standardized Unique Identifiers for Cast Vote Records #911

nurse-the-code mentioned this pull request Jan 9, 2025

Add Standardized Unique Identifiers for Cast Vote Records #912

Closed

nurse-the-code linked an issue Jan 9, 2025 that may be closed by this pull request

Feature Request: Add common id to both audit log and rctab_cvr.csv #911

Open

nurse-the-code commented Jan 9, 2025

View reviewed changes

src/main/java/network/brightspots/rcv/CastVoteRecord.java Outdated Show resolved Hide resolved

Removed depricated reference to UUID.

3dd2f90

nurse-the-code mentioned this pull request Jan 13, 2025

Filter out dominion partial ballots #910

Merged

nurse-the-code added 2 commits January 15, 2025 01:51

Merge branch 'develop' of https://github.com/BrightSpots/rcv into fea…

dde3587

…ture/issue-911-rctab-cvr-primary-id

Fixed all tests except "Test Convert to rctab_cvr works for Dominion"

d967d53

nurse-the-code commented Jan 15, 2025

View reviewed changes

src/main/java/network/brightspots/rcv/CastVoteRecord.java Outdated Show resolved Hide resolved

nurse-the-code commented Jan 15, 2025

View reviewed changes

src/main/java/network/brightspots/rcv/OutputWriter.java Outdated Show resolved Hide resolved

nurse-the-code commented Jan 15, 2025

View reviewed changes

src/main/java/network/brightspots/rcv/OutputWriter.java Outdated Show resolved Hide resolved

nurse-the-code added 3 commits January 15, 2025 11:41

Fixed broken TabulatorTest "Test Convert to rctab_cvr works for

276a79c

Dominion"

Updates per GitHub comments

dccb737

fixed broken tests

16941f7

nurse-the-code commented Jan 22, 2025

View reviewed changes

nurse-the-code added 2 commits January 24, 2025 08:06

Added unit tests for CastVoteRecord's getId() and getSuppliedId()

944d6ee

methods.

Removing TODOs associated with this PR

9e40a87

yezr reviewed Jan 28, 2025

View reviewed changes

src/test/java/network/brightspots/rcv/CastVoteRecordTests.java Outdated Show resolved Hide resolved

Merge branch 'develop' of https://github.com/BrightSpots/rcv into fea…

f69fb95

…ture/issue-911-rctab-cvr-primary-id

nurse-the-code commented Jan 30, 2025

View reviewed changes

CastVoteRecord's getSuppliedId now returns null when the suppliedId is

cf52345

null. Null handling is now handled where getSuppliedId is called.

yezr reviewed Jan 31, 2025

View reviewed changes

src/main/java/network/brightspots/rcv/OutputWriter.java Show resolved Hide resolved

Updated rctab_cvr.csv headers

f4272c9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standardize CVR identification across audit logs and RCTab output #913

Standardize CVR identification across audit logs and RCTab output #913

nurse-the-code commented Jan 9, 2025 •

edited

Loading

nurse-the-code Jan 9, 2025

nurse-the-code Jan 22, 2025 •

edited

Loading

nurse-the-code Jan 22, 2025 •

edited

Loading

nurse-the-code Jan 22, 2025 •

edited by yezr

Loading

nurse-the-code Jan 30, 2025

Standardize CVR identification across audit logs and RCTab output #913

Are you sure you want to change the base?

Standardize CVR identification across audit logs and RCTab output #913

Conversation

nurse-the-code commented Jan 9, 2025 • edited Loading

nurse-the-code Jan 9, 2025

Choose a reason for hiding this comment

nurse-the-code Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

nurse-the-code Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

nurse-the-code Jan 22, 2025 • edited by yezr Loading

Choose a reason for hiding this comment

nurse-the-code Jan 30, 2025

Choose a reason for hiding this comment

nurse-the-code commented Jan 9, 2025 •

edited

Loading

nurse-the-code Jan 22, 2025 •

edited

Loading

nurse-the-code Jan 22, 2025 •

edited

Loading

nurse-the-code Jan 22, 2025 •

edited by yezr

Loading