feat: countgd sam2 video support #318
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hrnn
commented
Dec 6, 2024
dillonalaird
requested changes
Dec 7, 2024
| OWLV2 = "owlv2" | ||
|
|
||
|
|
||
| def od_sam2_video_tracking( |
vision_agent/tools/tools.py
Outdated
| """'countgd_sam2_video_tracking' is a tool that can segment multiple objects given a text | ||
| prompt such as category names or referring expressions. The categories in the text | ||
| prompt are separated by commas. It returns a list of bounding boxes, label names, | ||
| mask file names and associated probability scores of 1.0. |
There was a problem hiding this comment.
minro comment, only florence2 returns probability scores of 1.0, countgd and owlv2 will can return regular probability scores. So you can just say "and associated probability scores."
vision_agent/tools/tools.py
Outdated
| """'owlv2_sam2_video_tracking' is a tool that can segment multiple objects given a text | ||
| prompt such as category names or referring expressions. The categories in the text | ||
| prompt are separated by commas. It returns a list of bounding boxes, label names, | ||
| mask file names and associated probability scores of 1.0. |
There was a problem hiding this comment.
see comment above on prob scores
vision_agent/tools/tools.py
Outdated
Comment on lines
2732
to
2753
| List[Dict[str, Any]]: A list of dictionaries containing the score, label, | ||
| bounding box, and mask of the detected objects with normalized coordinates | ||
| (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left | ||
| and xmax and ymax are the coordinates of the bottom-right of the bounding box. | ||
| The mask is binary 2D numpy array where 1 indicates the object and 0 indicates | ||
| the background. | ||
|
|
||
| Example | ||
| ------- | ||
| >>> countgd_sam2_video_tracking("car, dinosaur", image) | ||
| [ | ||
| { | ||
| 'score': 1.0, | ||
| 'label': 'dinosaur', | ||
| 'bbox': [0.1, 0.11, 0.35, 0.4], | ||
| 'mask': array([[0, 0, 0, ..., 0, 0, 0], | ||
| [0, 0, 0, ..., 0, 0, 0], | ||
| ..., | ||
| [0, 0, 0, ..., 0, 0, 0], | ||
| [0, 0, 0, ..., 0, 0, 0]], dtype=uint8), | ||
| }, | ||
| ] |
There was a problem hiding this comment.
Use the return values and examples from florence2_sam2_video_tracking. It's actually a list of list of dictionaries where the inner list is a frame
vision_agent/tools/tools.py
Outdated
Comment on lines
2776
to
2798
| Returns: | ||
| List[Dict[str, Any]]: A list of dictionaries containing the score, label, | ||
| bounding box, and mask of the detected objects with normalized coordinates | ||
| (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left | ||
| and xmax and ymax are the coordinates of the bottom-right of the bounding box. | ||
| The mask is binary 2D numpy array where 1 indicates the object and 0 indicates | ||
| the background. | ||
|
|
||
| Example | ||
| ------- | ||
| >>> countgd_sam2_video_tracking("car, dinosaur", image) | ||
| [ | ||
| { | ||
| 'score': 1.0, | ||
| 'label': 'dinosaur', | ||
| 'bbox': [0.1, 0.11, 0.35, 0.4], | ||
| 'mask': array([[0, 0, 0, ..., 0, 0, 0], | ||
| [0, 0, 0, ..., 0, 0, 0], | ||
| ..., | ||
| [0, 0, 0, ..., 0, 0, 0], | ||
| [0, 0, 0, ..., 0, 0, 0]], dtype=uint8), | ||
| }, | ||
| ] |
There was a problem hiding this comment.
see comment above on return comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added
countgd_sam2_video_trackingtool, to usecountgdfor object detection and pass the results tosam2to track the objects in the entire video.Link to Colab
Note to the reviewer:
mypycomplains when using too generic parameters.