Skip to content

Commit

Permalink
adding hidden clariification
Browse files Browse the repository at this point in the history
  • Loading branch information
rhysdg committed Jun 21, 2024
1 parent 2d4e162 commit 2f59c5f
Showing 1 changed file with 1 addition and 2 deletions.
3 changes: 1 addition & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,12 +79,11 @@ Last of all the aim here is to keep up with the latest optimised foundation mode

- Notice also cosine similrity at `get_similarity_scores` is adusted to handle multiple context - in other words a handful of text embedding can be sent as 'contexts', and send to the function to be evaluated against a single image or a batch of images.

- hidden states are also available at `onnx_model.hidden_image` and `onnx_model.hidden_text` allowing for analysis, attention plotting and multi-point processing as input to SAM. Watch this space for more on this. The stock implementation below uses the second `pooler` output for each in a cosine similarity scenario.
- hidden states are also available at `onnx_model.hidden_image` and `onnx_model.hidden_text` when using `type=siglip` for extraction only - allowing for analysis, attention plotting and multi-point processing as input to SAM. Watch this space for more on this.

- Not also that an `OnnxSAM` class is also available with the same instantiation and automatic model download - further examples are on their way along with SigLIP integration



## Example usage (CLIP/SigLIP - SAM incoming) :

- For the full 384 SigLIP model go ahead and use the `.inference` method as follows. Noting that CLIP is avaiable via the same method. Either model will switch between softmax and sigmoid accordingly:
Expand Down

0 comments on commit 2f59c5f

Please sign in to comment.