Skip to content
This repository has been archived by the owner on Dec 20, 2024. It is now read-only.

Provide positional embedding on the hidden grid for transformer #96

Draft
wants to merge 8 commits into
base: develop
Choose a base branch
from

Conversation

sahahner
Copy link
Member

@sahahner sahahner commented Dec 5, 2024

At the moment the latitude and longitude coordinates of the grid points are provided to the encoder only.

This PR implements the positional embedding for that transformer in the processor.

Related to this issue: https://github.com/ecmwf/aifs-dev/issues/10

Open points

  • shape of positional embedding flexible
  • make use of positional embedding more flexible. use hydra instantiation
    • Lat, Cos(lon), Sin(lon)
    • Cos(lat), Sin(lat), Cos(lon), Sin(lon)

@codecov-commenter
Copy link

codecov-commenter commented Dec 5, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.85%. Comparing base (0deb66b) to head (59a5445).
Report is 2 commits behind head on develop.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop      #96   +/-   ##
========================================
  Coverage    99.85%   99.85%           
========================================
  Files           23       23           
  Lines         1374     1374           
========================================
  Hits          1372     1372           
  Misses           2        2           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sahahner
Copy link
Member Author

The latest commits make the positional encoding configurable and allow us to point to different ways to calculate the positional encodings given the latlons of the hidden grid.
I implemented the following two approaches:

  • Lat, Cos(lon), Sin(lon)
    
  • Cos(lat), Sin(lat), Cos(lon), Sin(lon)
    

@mchantry mchantry assigned mchantry and sahahner and unassigned mchantry Dec 19, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants