python iterators an ensemble pybind11 wrappers

[pavlis]

In the latest work on the getting started tutorial I ran across something fundamental. I think it is a pybind11 wrapper issue, but it may be a feature of python itself I don't understand. Consider this code I just revised for the tutorial:

from mspasspy.algorithms.window import WindowData
from mspasspy.ccore.seismic import TimeWindow
from obspy.taup import TauPyModel
model = TauPyModel(model="iasp91")
from obspy.geodetics import gps2dist_azimuth,kilometers2degrees

# This a time window in time relative to P arrival around 
# which we cut each waveform
cutwin=TimeWindow(-10.0,300.0)
i=0
for d in ensemble.member:
    stalat=d['channel_lat']
    stalon=d['channel_lon']
    srclat=d['source_lat']
    srclon=d['source_lon']
    depth=d['source_depth']
    otime=d['source_time']
    georesult=gps2dist_azimuth(srclat,srclon,stalat,stalon)
    # obspy's function we just called returns distance in m in element 0 of a tuple
    # their travel time calculator it is degrees so we need this conversion
    dist=kilometers2degrees(georesult[0]/1000.0)
    arrivals=model.get_travel_times(source_depth_in_km=depth,distance_in_degree=dist,phase_list=['P'])
    d.rtoa()
    # Arrivals are returned in time order 0 is always the first arrival
    d.ator(otime+arrivals[0].time)
    #d=WindowData(d,cutwin)
    ensemble.member[i]=WindowData(d,cutwin)
    i+=1  
plotter.plot(ensemble)

In the form above it works fine. The problem comes from this construct in that code:

    #d=WindowData(d,cutwin)
    ensemble.member[i]=WindowData(d,cutwin)

Putting "ensemble.member[i]" on the right does the right thing. If I replace it by the line above I have commented out WindowData is ignored. I presume the reason is that the symbol d is assigned a new value, it no longer points at "member[i]" so it is just discarded the next time through the loop. Clearly the python iterator is not the same thing as the C++ version of a vector container iterator: vector<TimeSeries>::iterator d;.

This will confuse our users. If we can't fix it we must document it clearly.

[wangyinz]

Well, the point here is that Python is always pass-by-reference, so d=WindowData(d,cutwin) will just dereference the original d and assign a new reference of the return of WindowData. This might be confusing to C++ programmers, but for those that use Python, it shouldn't be a surprise.

[pavlis]

So, my guess was right. I more or less said that in a much more verbose and confusing way.

The question is how we should deal with this? This is an important oddity of using an ensemble container. I can start by adding some words for the getting started tutorial I'm working on.

I think this also brings up a need for another documentation section somewhere with the title "memory management". Serious users will need to understand some of the subtle issues of python and C memory management. The other topic that section could and should cover is avoiding memory bloat. With large ensemble you can eat a lot of memory fast if you don't watch what you are doing. A case in point is I may have told you I used spyder for debugging python but it has something that is inhibiting garbage collection that has caused my ubuntu machine to come to a standstill or from a spyder controlled test running out of memory.