Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Look for the closest declination line and adequate NSB random forest model for DL2 production #280

Draft
wants to merge 103 commits into
base: main
Choose a base branch
from

Conversation

marialainez
Copy link
Collaborator

@marialainez marialainez commented Feb 28, 2024

Choose the adequate RF model in the [declination lines x NSB] MC grid.

Closes #275 (initially for just choosing the declination MC track closest to the target declination)

@marialainez marialainez changed the title Look for the closest declination line to choose the RF model to produ… Look for the closest declination line RF model to produce the DL2 files Feb 28, 2024
src/osa/paths.py Outdated Show resolved Hide resolved
src/osa/paths.py Outdated Show resolved Hide resolved
src/osa/paths.py Outdated Show resolved Hide resolved
src/osa/paths.py Outdated Show resolved Hide resolved
src/osa/paths.py Outdated Show resolved Hide resolved
src/osa/paths.py Outdated Show resolved Hide resolved
Copy link

codecov bot commented Mar 1, 2024

Codecov Report

Attention: Patch coverage is 85.18519% with 12 lines in your changes missing coverage. Please review.

Project coverage is 81.58%. Comparing base (e589deb) to head (cbdee3e).
Report is 14 commits behind head on main.

Current head cbdee3e differs from pull request most recent head 9b4b441

Please upload reports for the commit 9b4b441 to get more accurate results.

Files Patch % Lines
src/osa/paths.py 81.81% 8 Missing ⚠️
src/osa/utils/utils.py 66.66% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #280      +/-   ##
==========================================
+ Coverage   79.76%   81.58%   +1.81%     
==========================================
  Files          59       57       -2     
  Lines        5144     4984     -160     
==========================================
- Hits         4103     4066      -37     
+ Misses       1041      918     -123     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@morcuended
Copy link
Member

@moralejo, I was going to comment in today's meeting. The way we look for the adequate RF model declination track is by using the pointing information from the TCU database (specifically the target coordinates), not checking the average pointing information inside the data files as you mentioned. Is this fine?

@moralejo
Copy link

Since the relevant pointing is the one in celestial coordinates, the target coordinates are perfectly fine for standard observations (if we ever do something like drift scans we will anyway have to do some custom work).

src/osa/paths.py Outdated Show resolved Hide resolved
src/osa/paths.py Outdated Show resolved Hide resolved
@morcuended morcuended changed the title Look for the closest declination line RF model to produce the DL2 files Look for the closest declination line and adequate NSB random forest model for DL2 production Feb 6, 2025
@marialainez marialainez marked this pull request as draft February 7, 2025 10:47
@morcuended morcuended dismissed their stale review February 10, 2025 13:16

new changes regarding NSB tunning were included

Copy link
Member

@morcuended morcuended left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The NSB + declination directory scheme looks good to me. Just left some comments to improve the readability of the code.

I trigger again the CI, since now lstchain is available in conda forge.

EDIT on the CI run: it will not work because of an incompatibility with lstcam_calib package still requiring ctapipe_io_lst 0.24.x

src/osa/utils/utils.py Outdated Show resolved Hide resolved
src/osa/utils/utils.py Outdated Show resolved Hide resolved
src/osa/utils/utils.py Outdated Show resolved Hide resolved
src/osa/utils/utils.py Outdated Show resolved Hide resolved
src/osa/utils/utils.py Outdated Show resolved Hide resolved
@morcuended
Copy link
Member

You committed the changes from the catB branch to this one. I think it should be kept separated. If so, you can come back to the previous commit e6ea503, skip the merge branch CatB and directly go to the last one 8b005ba

@marialainez
Copy link
Collaborator Author

I did that because the dl2 files should be saved in directories like "...tailcutXY/nsb_tuning_X.XX/", and the cleaning levels are fixed in the dl1ab step with the find_tailcuts function.

@marialainez
Copy link
Collaborator Author

Using as pointing declination the median of the declination of the datacheck files was not working, because by the time the sequence*.py files are written, the datacheck files have not been produced yet. Therefore, the RF model could not be determined. So I made this change to use the declination given in the run catalogs, which is actually the pointing declination of the telescope.

@morcuended
Copy link
Member

Using as pointing declination the median of the declination of the datacheck files was not working, because by the time the sequence*.py files are written, the datacheck files have not been produced yet. Therefore, the RF model could not be determined. So I made this change to use the declination given in the run catalogs, which is actually the pointing declination of the telescope.

Have you checked if this is available for all runs?

@marialainez
Copy link
Collaborator Author

No, in any case, I am testing it and this scheme doesn't work, because the function get_RF_model() is called before having even launched the r0_to_dl1 step. Consequently, no dl1a files have been created yet by the time this function is called and therefore the "find_tailcuts" function inside get_RF_model cannot be called.
I am thinking on going back to the initial scheme and calling to the get_RF_model function inside datasequence in the dl1_to_dl2 step. In this way, we could still use the median declination from the datacheck files.
The reason why we moved to this new scheme was to avoid accessing to the TCU database for every subrun, but we do not get the declination from there anymore.

@morcuended
Copy link
Member

I agree, it seems like another analysis step that could fit in datasequence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DL2 production with standard dark NSB models
4 participants