Code

Multihead attention#199

Code

Merged

milancurcic merged 73 commits intomodern-fortran:mainmodern-fortran/neural-fortran:mainfrom OneAdder:multihead_attentionOneAdder/neural-fortran:multihead_attentionCopy head branch name to clipboardFeb 21, 2025

+1,188-5

Commits

Commits on Feb 17, 2025

• linear2d_layer forward implementation

  OneAdder

  committed
  49e8507

  Copy full SHA for 49e8507

• linear2d_layer: temporarily remove api

  OneAdder

  committed
  feb7112

  Copy full SHA for feb7112

• Don't expose the concrete layer type via nf

  

  milancurcic

  authored and

  OneAdder

  committed
  8f320f0

  Copy full SHA for 8f320f0

• Plumbing of linear2d with input2d and linear2d

  

  milancurcic

  authored and

  OneAdder

  committed
  af4a5d7

  Copy full SHA for af4a5d7

• linear2d_layer: add flatten2d layer

  OneAdder

  committed
  549d4e6

  Copy full SHA for 549d4e6

• linear2d_layer: make linear2d layer work with input2d and flatten2d

  OneAdder

  committed
  3218be0

  Copy full SHA for 3218be0

• update cmake

  OneAdder

  committed
  39636f4

  Copy full SHA for 39636f4

• linear2d_layer: remove flatten2d layer

  OneAdder

  committed
  4cc7d1d

  Copy full SHA for 4cc7d1d

• linear2d_layer: remove public api

  OneAdder

  committed
  d863ce7

  Copy full SHA for d863ce7

• linear2d_layer: update cmakelists

  OneAdder

  committed
  78eb17a

  Copy full SHA for 78eb17a

• Add linear2d example

  

  milancurcic

  authored and

  OneAdder

  committed
  567abc4

  Copy full SHA for 567abc4

• linear2d_layer: remove redundant constructor args

  OneAdder

  committed
  32ac10d

  Copy full SHA for 32ac10d

• linear2d_layer: make example converge

  OneAdder

  committed
  edd169d

  Copy full SHA for edd169d

• linear2d_layer: add loss stopping and more iterations

  OneAdder

  committed
  aa5b83f

  Copy full SHA for aa5b83f

• start impementing MultiHeadAttention

  OneAdder

  committed
  dd3ce33

  Copy full SHA for dd3ce33

• scaled dot product attention

  OneAdder

  committed
  0ed77e8

  Copy full SHA for 0ed77e8

• combine attention heads

  OneAdder

  committed
  d6e6f3e

  Copy full SHA for d6e6f3e

• forward (not working)

  OneAdder

  committed
  eb58006

  Copy full SHA for eb58006

• rearrange attention dimensions in more efficient way

  OneAdder

  committed
  452032e

  Copy full SHA for 452032e

• initial forward implementation for multi-head attention

  OneAdder

  committed
  e06d39b

  Copy full SHA for e06d39b

• tests for multihead_attention%forward

  OneAdder

  committed
  519a6c8

  Copy full SHA for 519a6c8

• multihead_attention: move most logic to subroutines (performance)

  OneAdder

  committed
  9fdc7ae

  Copy full SHA for 9fdc7ae

• multihead_attention: update tests

  OneAdder

  committed
  bc67331

  Copy full SHA for bc67331

• multihead_attention: concurrency

  OneAdder

  committed
  a0a6fc4

  Copy full SHA for a0a6fc4

• multihead_attention: proof of concept backward (works, but not mathematically correct)

  OneAdder

  committed
  f8101af

  Copy full SHA for f8101af

• multihead_attention: fix minor scaling issue

  OneAdder

  committed
  63cce11

  Copy full SHA for 63cce11

• multihead_attention: complete backward implementation

  OneAdder

  committed
  dfb8842

  Copy full SHA for dfb8842

• multihead_attention: add comments for forward prop

  OneAdder

  committed
  adcf5e6

  Copy full SHA for adcf5e6

• multihead_attention: add tests for backward

  OneAdder

  committed
  650e47c

  Copy full SHA for 650e47c

• multihead_attention: adjust expected test values for updated scaling

  OneAdder

  committed
  3d16161

  Copy full SHA for 3d16161

• multihead_attention: calculate scaling factor only once

  OneAdder

  committed
  dcae5d6

  Copy full SHA for dcae5d6

• multihead_attention: use heap-allocated arrays during back prop

  OneAdder

  committed
  9fceae7

  Copy full SHA for 9fceae7

• multihead_attention: use heap-allocated arrays in forward

  OneAdder

  committed
  248e124

  Copy full SHA for 248e124

• multihead_attention: set values from correct shape to tests

  OneAdder

  committed
  4693028

  Copy full SHA for 4693028

• multihead_attention: fix issues with shapes (softmax prime became even more monstruos)

  OneAdder

  committed
  32dd628

  Copy full SHA for 32dd628

• multihead_attention: minor refactoring and optimization

  OneAdder

  committed
  33c33b9

  Copy full SHA for 33c33b9

• multihead_attention: fix comments

  OneAdder

  committed
  40c3f2b

  Copy full SHA for 40c3f2b

• multihead_attention: tests, add checks for attention weights

  OneAdder

  committed
  6a607b0

  Copy full SHA for 6a607b0

• multihead_attention: remove some of the copypaste comments

  OneAdder

  committed
  5fc5a5b

  Copy full SHA for 5fc5a5b

• multihead_attention: optimize shapes

  OneAdder

  committed
  65fd88d

  Copy full SHA for 65fd88d

• multihead_attention: params api

  OneAdder

  committed
  fbc132d

  Copy full SHA for fbc132d

• multihead_attention: fix incorrect dw bug

  OneAdder

  committed
  5422e4c

  Copy full SHA for 5422e4c

• multihead_attention: tests for updated parameters

  OneAdder

  committed
  39637e7

  Copy full SHA for 39637e7

• multihead_attention: remove reshape crutches

  OneAdder

  committed
  60a49db

  Copy full SHA for 60a49db

• multihead_attention: rename common forward and backward calls

  OneAdder

  committed
  7ab7769

  Copy full SHA for 7ab7769

• multihead_attention: tidy mha up

  OneAdder

  committed
  20c5eb0

  Copy full SHA for 20c5eb0

• multihead_attention: self attention

  OneAdder

  committed
  6098533

  Copy full SHA for 6098533

• multihead_attention: add cross attention

  OneAdder

  committed
  66b5023

  Copy full SHA for 66b5023

• multihead_attention: add more comments

  OneAdder

  committed
  ac813aa

  Copy full SHA for ac813aa

• multihead_attention: arrange attention into submodule

  OneAdder

  committed
  6b70f6b

  Copy full SHA for 6b70f6b

• multihead_attention: update cmakelists

  OneAdder

  committed
  b622d55

  Copy full SHA for b622d55

• multihead_attention: update attention in accordance with linear2d

  OneAdder

  committed
  ce03b39

  Copy full SHA for ce03b39

• multihead_attention: remove redundand constructor args for attention layers

  OneAdder

  committed
  41a80cd

  Copy full SHA for 41a80cd

• multihead_attention: use pure and elemental where necessary

  OneAdder

  committed
  a84efd3

  Copy full SHA for a84efd3

• multihead_attention: plumbing

  OneAdder

  committed
  52c94c4

  Copy full SHA for 52c94c4

• multihead_attention: add reference

  OneAdder

  committed
  66b539b

  Copy full SHA for 66b539b

• multihead_attention: remove rebase artifact

  OneAdder

  committed
  992da67

  Copy full SHA for 992da67

Commits on Feb 19, 2025

• multihead_attention: remove redundant args

  OneAdder

  committed
  d93be41

  Copy full SHA for d93be41

• multihead_attention: update tests

  OneAdder

  committed
  70272cb

  Copy full SHA for 70272cb

• multihead_attention: add the most important lines to tests

  OneAdder

  committed
  cb717f5

  Copy full SHA for cb717f5

• multihead_attention: simple MHA example

  OneAdder

  committed
  b7a6d06

  Copy full SHA for b7a6d06

• multihead_attention: update cmake

  OneAdder

  committed
  cb26afb

  Copy full SHA for cb26afb

• multihead_attention: remove debug line from tests

  OneAdder

  committed
  4c92e9c

  Copy full SHA for 4c92e9c

• multihead_attention: set slightly higher margin for fp imprecision (due to IEEE_DENORMAL)

  OneAdder

  committed
  df5f4cf

  Copy full SHA for df5f4cf

Commits on Feb 21, 2025

• Merge upstream/main

  milancurcic

  committed
  46786d6

  Copy full SHA for 46786d6

• Rename mha_simple example

  milancurcic

  committed
  6162783

  Copy full SHA for 6162783

• Update src/nf/nf_multihead_attention.f90

  Show description for 89abf22
  

  milancurcic

  and

  jvdp1

  authored
  89abf22

  Copy full SHA for 89abf22

• Update src/nf/nf_multihead_attention.f90

  Show description for 29b7d2e
  

  milancurcic

  and

  jvdp1

  authored
  29b7d2e

  Copy full SHA for 29b7d2e

• Update src/nf/nf_multihead_attention.f90

  Show description for e901479
  

  milancurcic

  and

  jvdp1

  authored
  e901479

  Copy full SHA for e901479

• Update src/nf/nf_multihead_attention.f90

  Show description for 1eaee95
  

  milancurcic

  and

  jvdp1

  authored
  1eaee95

  Copy full SHA for 1eaee95

• Tidy up

  milancurcic

  committed
  588ecb1

  Copy full SHA for 588ecb1

• Add self_attention to the layers table

  milancurcic

  committed
  20ffe05

  Copy full SHA for 20ffe05

• Merge branch 'multihead_attention' of github.com:OneAdder/neural-fortran into multihead_attention

  milancurcic

  committed
  e4c6548

  Copy full SHA for e4c6548