Flaw in ProcessingHistory

[pavlis]

Started to attempt a serious test of the preprocessing workflow with a test set of about 20 USArray events each holding around 3800 channels per event. It was pretty inevitable I'd find some issue in working through that and indeed I did. One I fixed, and the other will take some more serious work.

1. The relatively easy fix was a mishandling in the bundle code with data previously marked dead. It was actually handling such data correctly (throwing out the bundle with incomplete data), but we were dropping the elog entries because of a logic error I had made in the handling of that case.

2. The second emerged from the above because it is a different elog issue. In debugging I was first mystified why some channels were being killed, but then I found it in a python code I had trying to match entries in channel. Some of those were failing. Why is not the point. What is the point is I discovered we are not always preserving error logs of parents as they give birth to children. I don't think this is true even for a map operation let alone an operation like bundle that is more of a reduce operator (multiple inputs (3)) for one output. This is somewhat a consequence of a mismatch in concept putting elog in ProcessingHistory. Recall I tried unsucessfully to fission elog from ProcessingHistory last fall. I don't think I want to revisit that approach, but I do think we need some modest redesign of ProcessingHistory to handle this problem. Right now I'm thinking ProcessingHistory just needs a private std::list<mspass::LogData container and whatever methods to manage putting things there when needed along with getters to retrieve the contents. That is a simple solution and one that shouldn't create to much memory bloat. Let me think about the design a bit before commit more to paper. Look for an update in this topic later.

[wangyinz]

Hmmm.... that is indeed a problem. I guess the question here is that whether the elogs from a parent should go into its children. I think it depends, right?

[pavlis]

Had another look and see there is a very clear (at least to me) way to implement what we need here. That is, I am 100% sure that all I really need to is add operator+= to ErrorLogger. Then all the methods in ProcessingHistory need small additions with this basic logic:

if(other.elog.size()>0)
{
  this->operator+=(other.elog);
 }

[pavlis]

The only question I think is if things marked dead need to be always be retained. I think the answer in all cases is yes, because otherwise data can mysteriously vanish with no record of why it vanished. I think ErrorLogger objects have what we need. The internal list of LogData is enough, I believe.

[wangyinz]

The += operator looks good to me. Let's go with that.