gridfs read/write robustness issue

[pavlis]

Something I have not yet figure out is causing the following exception to be thrown while reading Seismogram objects stored in gridfs:

ValueError: Database._read_data_from_gridfs: Size mismatch in sample data. Number of points in gridfs file = 12001 but expected 36003

That comes from the method posted to the ValueError exception message. For reference, here is the current _read_data_from_gridfs code with the docstring stripped:

def _read_data_from_gridfs(self, mspass_object, gridfs_id):
        gfsh = gridfs.GridFS(self)
        fh = gfsh.get(file_id=gridfs_id)
        if isinstance(mspass_object, (TimeSeries, Seismogram)):
            if not mspass_object.is_defined("npts"):
                raise KeyError(
                    "Database._read_data_from_gridfs:  Required key npts is not defined"
                )
            else:
                if mspass_object.npts != mspass_object["npts"]:
                    message = "Database._read_data_from_gridfs: "
                    message += "Metadata value for npts is {} but datum attribute npts is {}\n".format(
                        mspass_object["npts"], mspass_object.npts
                    )
                    message += "Illegal inconsistency that should not happen and is a fatal error"
                    raise ValueError(message)
            npts = mspass_object.npts
        if isinstance(mspass_object, TimeSeries):
            # fh.seek(16)
            float_array = array("d")
            float_array.frombytes(fh.read(mspass_object.get("npts") * 8))
            mspass_object.data = DoubleVector(float_array)
        elif isinstance(mspass_object, Seismogram):
            np_arr = np.frombuffer(fh.read(npts * 8 * 3))
            file_size = fh.tell()
            if file_size != npts * 8 * 3:
                # Note we can only detect the cases where given npts is larger than
                # the number of points in the file
                emess = (
                    "Database._read_data_from_gridfs: Size mismatch in sample data. Number of points in gridfs file = %d but expected %d"
                    % (file_size / 8, (3 * mspass_object["npts"]))
                )
                raise ValueError(emess)
            # v1 did a transpose on write that this reversed - unnecessary
            # np_arr = np_arr.reshape(npts, 3).transpose()
            np_arr = np_arr.reshape(3, npts)
            mspass_object.data = dmatrix(np_arr)
        else:
            raise TypeError("only TimeSeries and Seismogram are supported")
        if mspass_object.npts > 0:
            mspass_object.set_live()
        else:
            mspass_object.kill()

That provides the context for this error message. I am not at all sure what in my dataset is causing this error to be thrown. The issue I have isn't really what causing that error. I suspect it is something anomalous cause by a unrelated problem I'm chasing in the bundle algorithm that I suspect is creating the flawed document this method is using to try to read these data. i.e. this is a rare problem that may not indicate a flaw in the writer.

THE ISSUE is I think it is a mistake to have this method throw ValueError exceptions in both of the cases where that can happen in this code. Because the function has a mspass_object passed to it means the proper solution is to kill and post not throw an exception in this case. The reader will be more robust if we make that change.

[wangyinz]

Yeah, this could probably just be a kill. I think originally it is treated as as error because _read_data_from_gridfs is one of the underlying method called by the read_data methods. I think the newly implemented read_data can correctly handle the kill here.

[pavlis]

Ok, I'll make those changes. I'll see if I can figure out in debugging the original problem why that error was created but that will be a challenge.