mongodb behavior or a bug?

[pavlis]

I encountered a surprise when I tried running the following block of code working on the 2012 miniseed data on quakes.

from obspy import UTCDateTime
from obspy.taup import TauPyModel
from obspy.geodetics import locations2degrees
# could use a different earth model here but probably a small detail
model = TauPyModel(model="iasp91")
query={'source_id' : {'$exists' : True},
       'channel_id' : {'$exists' : True} }
#ntotal=db.wf_miniseed.count_documents(query)
#print('Total documents to be processed =',ntotal)
# This need a more elegant solution but for now we just grab all sources in the time period
# We know this has duplicates, but small overhead for now. Proper solution would find all 
# unique source_id values in wf_miniseed and use that list to drive the loop below
stimeutc=UTCDateTime('2012-01-01T00:00:00.0')
etimeutc=UTCDateTime('2013-01-01T00:00:00.0')
srcquery={'time' : {'$gt' : stimeutc.timestamp,'$lt' : etimeutc.timestamp}}
nsrctotal=db.source.count_documents(srcquery)
print('Total number of source documents in 2012=',nsrctotal)
# used to limit return by travel time calculator to only P phase
phase_list=['P']
# temporary -verify that worked - not clear
cursor=db.source.find(srcquery).limit(5)
for doc in cursor:
    srcid=doc['_id']
    srclat=doc['lat']
    srclon=doc['lon']
    srcdep=doc['depth']
    srctime=doc['time']
    query={'source_id' : srcid}
    # Note this query seems very slow - for fullscale test add an index on source_id
    n=db.wf_miniseed.count_documents(query)
    if n == 0:
        continue
    print('Number of docs for this event=',n)
    wfcurs=db.wf_miniseed.find(query)
    for wfdoc in wfcurs:
        net=wfdoc['net']
        sta=wfdoc['sta']
        chan=wfdoc['chan']
        # Ian, this section does bad things.  I'm going to post a discussion on this on github
        if 'loc' in wfcurs:
            stadoc=db.get_seed_channel(net,sta,chan,loc,srctime)
        else:
            stadoc=db.get_seed_channel(net,sta,chan,time=srctime)
        stalat=stadoc['lat']
        stalon=stadoc['lon']
        staelev=stadoc['elev']
        degdist=locations2degrees(srclat,srclon,stalat,stalon)
        print(srclat,srclon,stalat,stalon,degdist)
        arr=model.get_travel_times(distance_in_degree=degdist,
                                  source_depth_in_km=srcdep,
                                  phase_list=phase_list)
        atime=srctime+arr[0].time
        print(net,sta,chan,UTCDateTime(atime))
        print(net,sta,chan)

This produced this output:

Total number of source documents in 2012= 1867
Number of docs for this event= 3682
2G IUGFS BHE

The second print and the code means I should expect to see 3682 lines from the final print statement in this box from the notebook. Instead I got just one. I discovered by commenting out pieces of this that the problem is this:

        if 'loc' in wfcurs:
            stadoc=db.get_seed_channel(net,sta,chan,loc,srctime)
        else:
            stadoc=db.get_seed_channel(net,sta,chan,time=srctime)

The operation applied to the Database class seems to invalidate the cursor driving this loop so the loop terminates after the first entry in the cursor.

Is this a feature of MongoDB or is this a bug in get_seed_channel (or maybe my script above)? I can imagine it is a property of MongoDB but if so the behavior is surprising. Any ideas on this?

[wangyinz]

Shouldn't that line be if 'loc' in wfdoc: instead?

[pavlis]

Ok, slap me on that one. That definitely was all that was wrong. Jumped to the wrong conclusion there. I stared at that forever and didn't see it. Classic blindness