DLCPoseEstimation fails on integer epoch

[MichaelCoulter]

I'm trying to run the DLC pipeline with this command and i get the following error. I don't have trodes position for this session, so there is no entry in RawPosition, but I thought we found a way to run the code in this situation. Thanks.
Command:

camera_name = "MEC_run_camera"
matching_rows = sgc.VideoFile() & {"camera_name": camera_name} & {"nwb_file_name": "RS1220240809_.nwb"}

for row in matching_rows:
    col1val = row["nwb_file_name"]
    if "RS1220240809" in col1val:  # *** change depending on rat/day!!!
        col2val = row["epoch"]
        col3val = row["video_file_num"]
        # NOTE: need to add () after DLCPoseEstimationSelection
        ##insert pose estimation task
        pose_estimation_key = (
            sgp.DLCPoseEstimationSelection().insert_estimation_task(
                {
                    "nwb_file_name": col1val,
                    "epoch": col2val,
                    "video_file_num": col3val,
                    **model_key,
                },
                task_mode="trigger",  # load or trigger
                params={"gputouse": gputouse, "videotype": "mp4"},
            )
        )

        ##populate DLC Pose Estimation
        sgp.DLCPoseEstimation().populate(pose_estimation_key)

Error:

/home/mcoulter/mambaforge/envs/spyglass-position/lib/python3.9/site-packages/hdmf/container.py:420: UserWarning: The table for this DynamicTableRegion has not been added to the parent.
  warn(msg)
[17:35:51][INFO] Spyglass: inserted entry into Pose Estimation Selection
[17:35:51][INFO] Spyglass: ----------------------
[17:35:51][INFO] Spyglass: Pose Estimation
Config:
{'all_joints': [[0], [1], [2]],
 'all_joints_names': ['redLED_C', 'greenLED', 'tailBase'],
 'batch_size': 1,
 'crop_pad': 0,
 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_RS1220240809_runSep3/RS1220240809_run_mcoulter_section95shuffle1.mat',
 'dataset_type': 'imgaug',
 'deterministic': False,
 'fg_fraction': 0.25,
 'global_scale': 0.8,
 'init_weights': '/home/mcoulter/mambaforge/envs/spyglass-position/lib/python3.9/site-packages/deeplabcut/pose_estimation_tensorflow/models/pretrained/resnet_v1_50.ckpt',
 'intermediate_supervision': False,
 'intermediate_supervision_layer': 12,
 'location_refinement': True,
 'locref_huber_loss': True,
 'locref_loss_weight': 1.0,
 'locref_stdev': 7.2801,
 'log_dir': 'log',
 'mean_pixel': [123.68, 116.779, 103.939],
 'mirror': False,
 'net_type': 'resnet_50',
 'num_joints': 3,
 'optimizer': 'sgd',
 'pairwise_huber_loss': True,
 'pairwise_predict': False,
 'partaffinityfield_predict': False,
 'regularize': False,
 'scoremap_dir': 'test',
 'shuffle': True,
 'snapshot_prefix': '/nimbus/deeplabcut/projects/RS1220240809_run-mcoulter_section-2024-09-03/dlc-models/iteration-0/RS1220240809_runSep3-trainset95shuffle1/test/snapshot',
 'stride': 8.0,
 'weigh_negatives': False,
 'weigh_only_present_joints': False,
 'weigh_part_predictions': False,
 'weight_decay': 0.0001}
/home/mcoulter/mambaforge/envs/spyglass-position/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1694: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.
  warnings.warn('`layer.apply` is deprecated and '

Using snapshot-1030000 for model /nimbus/deeplabcut/projects/RS1220240809_run-mcoulter_section-2024-09-03/dlc-models/iteration-0/RS1220240809_runSep3-trainset95shuffle1

2024-09-05 17:35:54.506661: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 78948 MB memory:  -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:52:00.0, compute capability: 8.0

Starting to analyze %  /stelmo/nwb/deeplabcut/video/20240809_RS12_02_r1.mp4
Loading  /stelmo/nwb/deeplabcut/video/20240809_RS12_02_r1.mp4
Duration of video [s]:  1269.55 , recorded with  30.0 fps!
Overall # of frames:  38087  found with (before cropping) frame dimensions:  852 852
Starting to extract posture

100%|████████████████████████████████████▉| 38000/38087 [08:07<00:01, 77.87it/s]
[17:44:04][WARNING] Spyglass: DEPRECATION scheduled for version 0.6: dlc_reader: PoseEstimation
[17:44:04][INFO] Spyglass: getting raw position

Saving results in /stelmo/nwb/deeplabcut/output/RS1220240809/RS1220240809_02_model_RS1220240809_run_tutorial_01...
The videos are analyzed. Now your research can truly start! 
 You can create labeled videos with 'create_labeled_video'
If the tracking is not satisfactory for some videos, consider expanding the training set. You can use the function 'extract_outlier_frames' to extract a few representative outlier frames.

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[49], line 22
      8 pose_estimation_key = (
      9     sgp.DLCPoseEstimationSelection().insert_estimation_task(
     10         {
   (...)
     18     )
     19 )
     21 ##populate DLC Pose Estimation
---> 22 sgp.DLCPoseEstimation().populate(pose_estimation_key)
     24 ##start smooth interpolation
     25 si_params_name = "just_nan"

File ~/spyglass/src/spyglass/utils/dj_mixin.py:794, in SpyglassMixin.populate(self, *restrictions, **kwargs)
    792 if use_transact:  # Pass single-process populate to super
    793     kwargs["processes"] = processes
--> 794     return super().populate(*restrictions, **kwargs)
    795 else:  # No transaction protection, use bare make
    796     for key in keys:

File ~/mambaforge/envs/spyglass-position/lib/python3.9/site-packages/datajoint/autopopulate.py:241, in AutoPopulate.populate(self, suppress_errors, return_exception_objects, reserve_jobs, order, limit, max_calls, display_progress, processes, make_kwargs, *restrictions)
    237 if processes == 1:
    238     for key in (
    239         tqdm(keys, desc=self.__class__.__name__) if display_progress else keys
    240     ):
--> 241         error = self._populate1(key, jobs, **populate_kwargs)
    242         if error is not None:
    243             error_list.append(error)

File ~/mambaforge/envs/spyglass-position/lib/python3.9/site-packages/datajoint/autopopulate.py:292, in AutoPopulate._populate1(self, key, jobs, suppress_errors, return_exception_objects, make_kwargs)
    290 self.__class__._allow_insert = True
    291 try:
--> 292     make(dict(key), **(make_kwargs or {}))
    293 except (KeyboardInterrupt, SystemExit, Exception) as error:
    294     try:

File ~/spyglass/src/spyglass/position/v1/position_dlc_pose_estimation.py:213, in DLCPoseEstimation.make(self, key)
    209 """.populate() method will launch training for each PoseEstimationTask"""
    210 self.log_path = (
    211     Path(infer_output_dir(key=key, makedir=False)) / "log.log"
    212 )
--> 213 self._logged_make(key)

File ~/spyglass/src/spyglass/position/v1/dlc_utils.py:198, in file_log.<locals>.decorator.<locals>.wrapper(self, *args, **kwargs)
    196     logger.removeHandler(logger.handlers[0])
    197 try:
--> 198     return func(self, *args, **kwargs)
    199 finally:
    200     if not console:

File ~/spyglass/src/spyglass/position/v1/position_dlc_pose_estimation.py:252, in DLCPoseEstimation._logged_make(self, key)
    246 creation_time = datetime.fromtimestamp(
    247     dlc_result.creation_time
    248 ).strftime("%Y-%m-%d %H:%M:%S")
    250 logger.info("getting raw position")
    251 interval_list_name = (
--> 252     convert_epoch_interval_name_to_position_interval_name(
    253         {
    254             "nwb_file_name": key["nwb_file_name"],
    255             "epoch": key["epoch"],
    256         },
    257         populate_missing=False,
    258     )
    259 )
    260 spatial_series = (
    261     RawPosition() & {**key, "interval_list_name": interval_list_name}
    262 ).fetch_nwb()[0]["raw_position"]
    263 _, _, _, video_time = get_video_info(key)

File ~/spyglass/src/spyglass/common/common_behav.py:641, in convert_epoch_interval_name_to_position_interval_name(key, populate_missing)
    638     PositionIntervalMap()._no_transaction_make(key)
    639     pos_query = PositionIntervalMap & key
--> 641 if pos_query.fetch(pos_str)[0] == "":
    642     logger.info(f"No position intervals found for {key}")
    643     return []

IndexError: index 0 is out of bounds for axis 0 with size 0

[CBroz1]

Hi @MichaelCoulter - Please share your pose_estimation_key value.

The position pipeline expects epochs of the format pos # valid times. My guess is that convert_epoch_interval_name_to_position_interval is failing on a digit it can't substring.

[samuelbray32]

@CBroz1, there won't be a matchingpos # valid time interval for epochs without existing position tracking

It looks like the master branch has lost the changes from #973 that enabled the pipeline to run without existing position tracking. Looks like it happened in #870

ex)

• Current line requires a spatial series to exist for the epoch
• The version from #973 would work without it

[samuelbray32]

Looks like this happens in at least a couple places:

DLCCentroid.make()

• Current requires RawPosition entry
• Pre #870 works without it

My guess is that since #870 began before #973 but was merged after, the merge conflicts were resolved in a way that excluded #973's changes

[MichaelCoulter]

thanks for figuring this out! here is the post estimation key if you need it:
{'nwb_file_name': 'RS1220240809_.nwb',
'epoch': 2,
'video_file_num': 1,
'project_name': 'RS1220240809_run',
'dlc_model_name': 'RS1220240809_run_tutorial_01',
'dlc_model_params_name': 'default',
'task_mode': 'trigger'}

[rpswenson]

I'm getting this same error even though my spyglass is up to date

[samuelbray32]

You may just need to run this prior to populating:

convert_epoch_interval_name_to_position_interval_name(
  {
  "nwb_file_name": key["nwb_file_name"],
  "epoch": key["epoch"],
  },
  populate_missing=True, )

[samuelbray32]

Checking the code, @CBroz1 is there a reason that this needs populate_missing=False here?

Theres a no transaction make in convert_epoch_interval_name_to_position_interval_name available which should let it be populated even in the DLCPoseEstimation.make()

[rpswenson]

I still get the same error when running the function on its own:

from spyglass.common.common_behav import convert_epoch_interval_name_to_position_interval_name
for epoch in [2,4,6,8]:
    convert_epoch_interval_name_to_position_interval_name(
      {
      "nwb_file_name": "RS1220240810",
      "epoch": epoch,
      },
      populate_missing=True, )

output:

[16:33:23][INFO] Spyglass: Found 0 interval list names found for RS1220240810 epoch 2
INFO:spyglass:Found 0 interval list names found for RS1220240810 epoch 2

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[21], line 3
      1 from spyglass.common.common_behav import convert_epoch_interval_name_to_position_interval_name
      2 for epoch in [2,4,6,8]:
----> 3     convert_epoch_interval_name_to_position_interval_name(
      4       {
      5       "nwb_file_name": "RS1220240810",
      6       "epoch": epoch,
      7       },
      8       populate_missing=True, )

File ~/spyglass/src/spyglass/common/common_behav.py:647, in convert_epoch_interval_name_to_position_interval_name(key, populate_missing)
    644     PositionIntervalMap()._no_transaction_make(key)
    645     pos_query = PositionIntervalMap & key
--> 647 if pos_query.fetch(pos_str)[0] == "":
    648     logger.info(f"No position intervals found for {key}")
    649     return []

IndexError: index 0 is out of bounds for axis 0 with size 0

[rpswenson]

wait disregard I forgot to put the .nwb it works fine