Fixing SF-SAC exports given historic data realities

[jadudm]

Story

We have completed migrating historic data into our database, but it is not (yet) documented for users to understand. And, some of that data (the migration records) play no role in the user's experience.

Particularly problematic is the GSA_MIGRATION tag. It is important as a placeholder, but without it being documented, it looks like we threw away data.

Now that the only source for data is GSA, we need to expose the migration data meaningfully.

Web views

We present records as summaries on the WWW. Some fields in these views might be GSA_MIGRATION; at this point, we need a way of flagging that the data is malformed, but the user also needs to see it. (For example, a malformed email address should be flagged, perhaps, but it should not be hidden from the view.)

Similarly, users need a way of seeing all of the values that changed on a given record. (Do they?) Should there be a collapsed view, or something, at the bottom of the page that highlights all of the data that is present-but-changed in this record?

Remember: not everything is public. Be cognizant of Tribal data in this work.

SF-SAC export

The SF-SAC export currently exports historic data with GSA_MIGRATION in many places.

These, too, need to update.

API

We could solve this by making the change/update table public. Simply providing the record of changes (with appropriate tribal controls) would make the API "compliant."

In short...

The only place for people to get SF-SAC data is now us.

If the data is part of the historic record, we are (in a way) obscuring that data. We have it, but it is "off to the side." If we're going to export data, and people need that data to do their jobs, we need to... give them the data, not the GSA_MIGRATION tag.

The exact solution isn't clear, but it isn't OK that an SF-SAC is full of GSA_MIGRATION tags.

Suggestion

We might split this out. It is possible we can "get away with" having the change records on the WWW page, even if the SF-SAC exports contain GSA_MIGRATION values. The API... might need to be fixed sooner. But, it is also an easier fix.

Point being: we want to ticket out the minimum increment that finishes in 2-3 weeks, and meets the minimum needs of "not obscuring the historical record." We will have to investigate, explore, iterate, and improve here over the next few months/quarters.

[jadudm]

If we are touching the SF-SAC export, it is worth noting that we are missing data from it:

#3303

That is a problem, because it is the primary way auditors review things before submission, and IGs/resolution officials do their work. We need to add the data that is not present.