data_preparation

Data Preparation

NOTE: It is being assumed that all commands are run from the top-level-directory of the project.

Overview

This module is primarily responsible for conducting Extract-Transform-Load (ETL) operations that processes the PhytoOracle data present in iRODs to a JSON format that is compatible with OpenSearch, to perform quicker and more efficient search operations.

Each file corresponds to an ETL operation for data specific to a sensor, and therefore works in its own unique way (see Usage). However, each operation, at its end adds JSON file(s) to the output/ directory which can then be used by search_configuration to populate the OpenSearch index.

Usage

• Drone

  python3 data_preparation/drone.py  /iplant/home/shared/phytooracle/season_<season>_<crop_name>_yr_<year>/level_<level>/drone/<crop_name>/
  

  The directory structure and level information may vary from one season to another. It only works for a specific level of processing. For Season 14 (2022, sorghum), the supported level was 2.

• flirIRCamera

   python3 data_preparation/flirIRCamera.py  /iplant/home/shared/phytooracle/season_<season>_<crop_name>_yr_<year>/level_<level>/flirIrCamera/<clustering_file_name>.csv
  

  The directory structure, level information and the clustering file name may vary from one season to another. It only works for a specific level of processing. For Season 14 (2022, sorghum), the supported level was 2 and the clustering file was named season_14_clustering_flir.csv.

• stereoTop

   python3 data_preparation/stereoTop.py  /iplant/home/shared/phytooracle/season_<season>_<crop_name>_yr_<year>/level_<level>/stereoTop/<clustering_file_name>.csv
  

  The directory structure, level information and the clustering file name may vary from one season to another. It only works for a specific level of processing. For Season 14 (2022, sorghum), the supported level was 2 and the clustering file was named season_14_clustering.csv.

• scanner3D.py

   python3 data_preparation/stereoTop.py  <path_to_fieldbook_csv> <path_to_scanner_parent_directory>
  

  NOTE: The current iteration is super complex and generates quite a lot of debug output. TO BE FIXED.