Add Data Retention Flag

[MarkDWilliams]

Some records will need to be preserved for longer-term use than the current 30 day retention window allows. The vast majority of Messages we store are from automated testing, one-off queries, and other sources that don't need to be retained. However, some are referenced and need to be retained. I am proposing that the Message object be modified to contain a single Boolean "retain" flag at the top level. Queries submitted can contain a "retain: True" flag to mark those that users wish to retain (indefinitely?). I am unsure whether this would need to be formally included in the TRAPI spec, but will need to consult with ITRB about exempting such records from the regulat data cleanup.

[MarkDWilliams]

EDIT: Disregard. I believe I have thought of a better way that avoids this question. Explained in my comment below

@edeutsch Do you think it would be necessary to formally include the retain flag as part of the TRAPI spec, or could this call under an "additional fields" umbrella for the query graph? I don't imagine that anyone else in the consortium would make any use of such a flag, at least in terms of their tools retaining data. I suppose some could conceivably use it for retaining data in their own systems. However, I don't want to leave undocumented functionality floating around. Happy to propose it for a TRAPI change, leave it as an ARS specific functionality (and document it on our end), or to fast-track accepting it at the ARS level and then put it in as a low-priority item to be included in the next TRAPI version after 1.4, but I'll defer to you as the expert on the subject.

[MarkDWilliams]

Having given this a little more thought, it may be better for me to add a new endpoint that takes a PK and marks it internally for retention. This way, no TRAPI changes are needed, users can see their results before deciding whether to retain or not, and we avoid the possibility of a retain flag being accidentally left in a template or automated test query and retaining a lot of data that isn't actually needed

[edeutsch]

I agree that a "retain" endpoint would be the most flexible. Probably if a parent PK was flagged to retain, then that should cascade to all children?