vision_agent.tools
vision_agent.tools
register_tool
Source code in vision_agent/tools/__init__.py
vision_agent.tools.tools
COLORS
module-attribute
COLORS = [
(158, 218, 229),
(219, 219, 141),
(23, 190, 207),
(188, 189, 34),
(199, 199, 199),
(247, 182, 210),
(127, 127, 127),
(227, 119, 194),
(196, 156, 148),
(197, 176, 213),
(140, 86, 75),
(148, 103, 189),
(255, 152, 150),
(152, 223, 138),
(214, 39, 40),
(44, 160, 44),
(255, 187, 120),
(174, 199, 232),
(255, 127, 14),
(31, 119, 180),
]
FUNCTION_TOOLS
module-attribute
FUNCTION_TOOLS = [
owlv2_object_detection,
owlv2_sam2_instance_segmentation,
owlv2_sam2_video_tracking,
countgd_object_detection,
countgd_sam2_instance_segmentation,
countgd_sam2_video_tracking,
florence2_ocr,
florence2_object_detection,
florence2_sam2_instance_segmentation,
florence2_sam2_video_tracking,
claude35_text_extraction,
document_extraction,
document_qa,
ocr,
qwen2_vl_images_vqa,
qwen2_vl_video_vqa,
activity_recognition,
depth_anything_v2,
generate_pose_image,
vit_nsfw_classification,
flux_image_inpainting,
siglip_classification,
minimum_distance,
]
UTIL_TOOLS
module-attribute
UTIL_TOOLS = [
extract_frames_and_timestamps,
save_json,
load_image,
save_image,
save_video,
overlay_bounding_boxes,
overlay_segmentation_masks,
]
sam2
'sam2' is a tool that can segment multiple objects given an input bounding box, label and score. It returns a set of masks along with the corresponding bounding boxes and labels.
PARAMETER | DESCRIPTION |
---|---|
image |
The image that contains multiple instances of the object.
TYPE:
|
detections |
A list of dictionaries containing the score, label, and bounding box of the detected objects with normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, bounding box, and mask of the detected objects with normalized coordinates (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. |
Example
>>> sam2(image, [
{'score': 0.49, 'label': 'flower', 'bbox': [0.1, 0.11, 0.35, 0.4]},
])
[
{
'score': 0.49,
'label': 'flower',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
]
Source code in vision_agent/tools/tools.py
od_sam2_video_tracking
od_sam2_video_tracking(
od_model,
prompt,
frames,
box_threshold=0.3,
chunk_length=50,
deployment_id=None,
)
Source code in vision_agent/tools/tools.py
232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 |
|
owlv2_object_detection
'owlv2_object_detection' is a tool that can detect and count multiple objects given a text prompt such as category names or referring expressions on images. The categories in text prompt are separated by commas. It returns a list of bounding boxes with normalized coordinates, label names and associated probability scores.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The prompt to ground to the image.
TYPE:
|
image |
The image to ground the prompt to.
TYPE:
|
box_threshold |
The threshold for the box detection. Defaults to 0.10.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, and bounding box of the detected objects with normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. |
Example
>>> owlv2_object_detection("car, dinosaur", image)
[
{'score': 0.99, 'label': 'dinosaur', 'bbox': [0.1, 0.11, 0.35, 0.4]},
{'score': 0.98, 'label': 'car', 'bbox': [0.2, 0.21, 0.45, 0.5},
]
Source code in vision_agent/tools/tools.py
owlv2_sam2_instance_segmentation
'owlv2_sam2_instance_segmentation' is a tool that can detect and count multiple instances of objects given a text prompt such as category names or referring expressions on images. The categories in text prompt are separated by commas. It returns a list of bounding boxes with normalized coordinates, label names, masks and associated probability scores.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The object that needs to be counted.
TYPE:
|
image |
The image that contains multiple instances of the object.
TYPE:
|
box_threshold |
The threshold for detection. Defaults to 0.10.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, bounding box, and mask of the detected objects with normalized coordinates (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. |
Example
>>> owlv2_sam2_instance_segmentation("flower", image)
[
{
'score': 0.49,
'label': 'flower',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
]
Source code in vision_agent/tools/tools.py
owlv2_sam2_video_tracking
'owlv2_sam2_video_tracking' is a tool that can track and segment multiple objects in a video given a text prompt such as category names or referring expressions. The categories in the text prompt are separated by commas. It returns a list of bounding boxes, label names, masks and associated probability scores and is useful for tracking and counting without duplicating counts.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The prompt to ground to the image.
TYPE:
|
frames |
The list of frames to ground the prompt to.
TYPE:
|
box_threshold |
The threshold for the box detection. Defaults to 0.10.
TYPE:
|
chunk_length |
The number of frames to re-run owlv2 to find new objects.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[List[Dict[str, Any]]]
|
List[List[Dict[str, Any]]]: A list of list of dictionaries containing the label, segmentation mask and bounding boxes. The outer list represents each frame and the inner list is the entities per frame. The detected objects have normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. The label names are prefixed with their ID represent the total count. |
Example
>>> owlv2_sam2_video_tracking("car, dinosaur", frames)
[
[
{
'label': '0: dinosaur',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
],
...
]
Source code in vision_agent/tools/tools.py
florence2_object_detection
'florence2_object_detection' is a tool that can detect multiple objects given a text prompt which can be object names or caption. You can optionally separate the object names in the text with commas. It returns a list of bounding boxes with normalized coordinates, label names and associated confidence scores of 1.0.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The prompt to ground to the image. Use exclusive categories that do not overlap such as 'person, car' and NOT 'person, athlete'.
TYPE:
|
image |
The image to used to detect objects
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, and bounding box of the detected objects with normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The scores are always 1.0 and cannot be thresholded |
Example
>>> florence2_object_detection('person looking at a coyote', image)
[
{'score': 1.0, 'label': 'person', 'bbox': [0.1, 0.11, 0.35, 0.4]},
{'score': 1.0, 'label': 'coyote', 'bbox': [0.34, 0.21, 0.85, 0.5},
]
Source code in vision_agent/tools/tools.py
florence2_sam2_instance_segmentation
'florence2_sam2_instance_segmentation' is a tool that can segment multiple objects given a text prompt such as category names or referring expressions. The categories in the text prompt are separated by commas. It returns a list of bounding boxes, label names, mask file names and associated probability scores of 1.0.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The prompt to ground to the image. Use exclusive categories that do not overlap such as 'person, car' and NOT 'person, athlete'.
TYPE:
|
image |
The image to ground the prompt to.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, bounding box, and mask of the detected objects with normalized coordinates (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. |
Example
>>> florence2_sam2_instance_segmentation("car, dinosaur", image)
[
{
'score': 1.0,
'label': 'dinosaur',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
]
Source code in vision_agent/tools/tools.py
florence2_sam2_video_tracking
'florence2_sam2_video_tracking' is a tool that can track and segment multiple objects in a video given a text prompt such as category names or referring expressions. The categories in the text prompt are separated by commas. It returns a list of bounding boxes, label names, masks and associated probability scores and is useful for tracking and counting without duplicating counts.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The prompt to ground to the image. Use exclusive categories that do not overlap such as 'person, car' and NOT 'person, athlete'.
TYPE:
|
frames |
The list of frames to ground the prompt to.
TYPE:
|
chunk_length |
The number of frames to re-run florence2 to find new objects.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[List[Dict[str, Any]]]
|
List[List[Dict[str, Any]]]: A list of list of dictionaries containing the label, segmentation mask and bounding boxes. The outer list represents each frame and the inner list is the entities per frame. The detected objects have normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. The label names are prefixed with their ID represent the total count. |
Example
>>> florence2_sam2_video_tracking("car, dinosaur", frames)
[
[
{
'label': '0: dinosaur',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
],
...
]
Source code in vision_agent/tools/tools.py
746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 |
|
florence2_ocr
'florence2_ocr' is a tool that can detect text and text regions in an image. Each text region contains one line of text. It returns a list of detected text, the text region as a bounding box with normalized coordinates, and confidence scores. The results are sorted from top-left to bottom right.
PARAMETER | DESCRIPTION |
---|---|
image |
The image to extract text from.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the detected text, bbox with normalized coordinates, and confidence score. |
Example
>>> florence2_ocr(image)
[
{'label': 'hello world', 'bbox': [0.1, 0.11, 0.35, 0.4], 'score': 0.99},
]
Source code in vision_agent/tools/tools.py
countgd_object_detection
'countgd_object_detection' is a tool that can detect multiple instances of an object given a text prompt. It is particularly useful when trying to detect and count a large number of objects. You can optionally separate object names in the prompt with commas. It returns a list of bounding boxes with normalized coordinates, label names and associated confidence scores.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The object that needs to be counted.
TYPE:
|
image |
The image that contains multiple instances of the object.
TYPE:
|
box_threshold |
The threshold for detection. Defaults to 0.23.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, and bounding box of the detected objects with normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. |
Example
>>> countgd_object_detection("flower", image)
[
{'score': 0.49, 'label': 'flower', 'bbox': [0.1, 0.11, 0.35, 0.4]},
{'score': 0.68, 'label': 'flower', 'bbox': [0.2, 0.21, 0.45, 0.5},
{'score': 0.78, 'label': 'flower', 'bbox': [0.3, 0.35, 0.48, 0.52},
{'score': 0.98, 'label': 'flower', 'bbox': [0.44, 0.24, 0.49, 0.58},
]
Source code in vision_agent/tools/tools.py
countgd_sam2_instance_segmentation
'countgd_sam2_instance_segmentation' is a tool that can detect multiple instances of an object given a text prompt. It is particularly useful when trying to detect and count a large number of objects. You can optionally separate object names in the prompt with commas. It returns a list of bounding boxes with normalized coordinates, label names, masks associated confidence scores.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The object that needs to be counted.
TYPE:
|
image |
The image that contains multiple instances of the object.
TYPE:
|
box_threshold |
The threshold for detection. Defaults to 0.23.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, bounding box, and mask of the detected objects with normalized coordinates (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. |
Example
>>> countgd_sam2_instance_segmentation("flower", image)
[
{
'score': 0.49,
'label': 'flower',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
]
Source code in vision_agent/tools/tools.py
countgd_sam2_video_tracking
'countgd_sam2_video_tracking' is a tool that can track and segment multiple objects in a video given a text prompt such as category names or referring expressions. The categories in the text prompt are separated by commas. It returns a list of bounding boxes, label names, masks and associated probability scores and is useful for tracking and counting without duplicating counts.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The prompt to ground to the image.
TYPE:
|
frames |
The list of frames to ground the prompt to.
TYPE:
|
box_threshold |
The threshold for detection. Defaults to 0.23.
TYPE:
|
chunk_length |
The number of frames to re-run countgd to find new objects.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[List[Dict[str, Any]]]
|
List[List[Dict[str, Any]]]: A list of list of dictionaries containing the label, segmentation mask and bounding boxes. The outer list represents each frame and the inner list is the entities per frame. The detected objects have normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. The label names are prefixed with their ID represent the total count. |
Example
>>> countgd_sam2_video_tracking("car, dinosaur", frames)
[
[
{
'label': '0: dinosaur',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
],
...
]
Source code in vision_agent/tools/tools.py
1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 |
|
countgd_visual_object_detection
'countgd_visual_object_detection' is a tool that can detect multiple instances of an object given a visual prompt. It is particularly useful when trying to detect and count a large number of objects. You can optionally separate object names in the prompt with commas. It returns a list of bounding boxes with normalized coordinates, label names and associated confidence scores.
PARAMETER | DESCRIPTION |
---|---|
visual_prompts |
Bounding boxes of the object in format [xmin, ymin, xmax, ymax] with normalized coordinates. Up to 3 bounding boxes can be provided.
TYPE:
|
image |
The image that contains multiple instances of the object.
TYPE:
|
box_threshold |
The threshold for detection. Defaults to 0.23.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, and bounding box of the detected objects with normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. |
Example
>>> countgd_visual_object_detection(
visual_prompts=[[0.1, 0.1, 0.4, 0.42], [0.2, 0.3, 0.25, 0.35]],
image=image
)
[
{'score': 0.49, 'label': 'object', 'bounding_box': [0.1, 0.11, 0.35, 0.4]},
{'score': 0.68, 'label': 'object', 'bounding_box': [0.2, 0.21, 0.45, 0.5},
{'score': 0.78, 'label': 'object', 'bounding_box': [0.3, 0.35, 0.48, 0.52},
{'score': 0.98, 'label': 'object', 'bounding_box': [0.44, 0.24, 0.49, 0.58},
]
Source code in vision_agent/tools/tools.py
countgd_sam2_visual_instance_segmentation
'countgd_sam2_visual_instance_segmentation' is a tool that can precisely count multiple instances of an object given few visual example prompts. It returns a list of bounding boxes, label names, masks and associated probability scores.
PARAMETER | DESCRIPTION |
---|---|
visual_prompts |
Bounding boxes of the object in format [xmin, ymin, xmax, ymax] with normalized coordinates. Up to 3 bounding boxes can be provided.
TYPE:
|
image |
The image that contains multiple instances of the object.
TYPE:
|
box_threshold |
The threshold for detection. Defaults to 0.23.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, bounding box, and mask of the detected objects with normalized coordinates (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. |
Example
>>> countgd_sam2_visual_instance_segmentation(
visual_prompts=[[0.1, 0.1, 0.4, 0.42], [0.2, 0.3, 0.25, 0.35]],
image=image
)
[
{
'score': 0.49,
'label': 'object',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
]
Source code in vision_agent/tools/tools.py
custom_object_detection
'custom_object_detection' is a tool that can detect instances of an object given a deployment_id of a previously finetuned object detection model. It is particularly useful when trying to detect objects that are not well detected by generalist models. It returns a list of bounding boxes with normalized coordinates, label names and associated confidence scores.
PARAMETER | DESCRIPTION |
---|---|
deployment_id |
The id of the finetuned model.
TYPE:
|
image |
The image that contains instances of the object.
TYPE:
|
box_threshold |
The threshold for detection. Defaults to 0.1.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, and bounding box of the detected objects with normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. |
Example
>>> custom_object_detection("abcd1234-5678efg", image)
[
{'score': 0.49, 'label': 'flower', 'bbox': [0.1, 0.11, 0.35, 0.4]},
{'score': 0.68, 'label': 'flower', 'bbox': [0.2, 0.21, 0.45, 0.5]},
{'score': 0.78, 'label': 'flower', 'bbox': [0.3, 0.35, 0.48, 0.52]},
{'score': 0.98, 'label': 'flower', 'bbox': [0.44, 0.24, 0.49, 0.58]},
]
Source code in vision_agent/tools/tools.py
1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 |
|
custom_od_sam2_video_tracking
'custom_od_sam2_video_tracking' is a tool that can segment multiple objects given a custom model with predefined category names. It returns a list of bounding boxes, label names, mask file names and associated probability scores.
PARAMETER | DESCRIPTION |
---|---|
deployment_id |
The id of the deployed custom model.
TYPE:
|
image |
The image to ground the prompt to.
TYPE:
|
chunk_length |
The number of frames to re-run florence2 to find new objects.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[List[Dict[str, Any]]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, bounding box, and mask of the detected objects with normalized coordinates (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. |
Example
>>> custom_od_sam2_video_tracking("abcd1234-5678efg", frames)
[
[
{
'label': '0: dinosaur',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
],
...
]
Source code in vision_agent/tools/tools.py
agentic_object_detection
'agentic_object_detection' is a tool that can detect multiple objects given a text prompt such as object names or referring expressions on images. It's particularly good at detecting specific objects given detailed descriptive prompts but runs slower. It returns a list of bounding boxes with normalized coordinates, label names and associated probability scores.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The prompt to ground to the image, only supports a single prompt with no commas or periods.
TYPE:
|
image |
The image to ground the prompt to.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, and bounding box of the detected objects with normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. |
Example
>>> agentic_object_detection("a red car", image)
[
{'score': 0.99, 'label': 'a red car', 'bbox': [0.1, 0.11, 0.35, 0.4]},
{'score': 0.98, 'label': 'a red car', 'bbox': [0.2, 0.21, 0.45, 0.5},
]
Source code in vision_agent/tools/tools.py
agentic_sam2_instance_segmentation
'agentic_sam2_instance_segmentation' is a tool that can detect multiple instances given a text prompt such as object names or referring expressions on images. It's particularly good at detecting specific objects given detailed descriptive prompts but runs slower. It returns a list of bounding boxes with normalized coordinates, label names, masks and associated probability scores.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The object that needs to be counted, only supports a single prompt with no commas or periods.
TYPE:
|
image |
The image that contains multiple instances of the object.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, bounding box, and mask of the detected objects with normalized coordinates (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. |
Example
>>> agentic_sam2_instance_segmentation("a large blue flower", image)
[
{
'score': 0.49,
'label': 'a large blue flower',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
]
Source code in vision_agent/tools/tools.py
agentic_sam2_video_tracking
'agentic_sam2_video_tracking' is a tool that can track and segment multiple objects in a video given a text prompt such as object names or referring expressions. It's particularly good at detecting specific objects given detailed descriptive prompts but runs slower, and returns a list of bounding boxes, label names, masks and associated probability scores and is useful for tracking and counting without duplicating counts.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The prompt to ground to the image, only supports a single prompt with no commas or periods.
TYPE:
|
frames |
The list of frames to ground the prompt to.
TYPE:
|
chunk_length |
The number of frames to re-run agentic object detection to to find new objects.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[List[Dict[str, Any]]]
|
List[List[Dict[str, Any]]]: A list of list of dictionaries containing the label, segmentation mask and bounding boxes. The outer list represents each frame and the inner list is the entities per frame. The detected objects have normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. The label names are prefixed with their ID represent the total count. |
Example
>>> agentic_sam2_video_tracking("a runner with yellow shoes", frames)
[
[
{
'label': '0: a runner with yellow shoes',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
],
...
]
Source code in vision_agent/tools/tools.py
1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 |
|
glee_object_detection
'glee_object_detection' is a tool that can detect multiple objects given a text prompt such as object names or referring expressions on images. It's particularly good at detecting specific objects given detailed descriptive prompts. It returns a list of bounding boxes with normalized coordinates, label names and associated probability scores.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The prompt to ground to the image, only supports a single prompt with no commas or periods.
TYPE:
|
image |
The image to ground the prompt to.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, and bounding box of the detected objects with normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. |
Example
>>> glee_object_detection("person holding a box", image)
[
{'score': 0.99, 'label': 'person holding a box', 'bbox': [0.1, 0.11, 0.35, 0.4]},
{'score': 0.98, 'label': 'person holding a box', 'bbox': [0.2, 0.21, 0.45, 0.5},
]
Source code in vision_agent/tools/tools.py
glee_sam2_instance_segmentation
'glee_sam2_instance_segmentation' is a tool that can detect multiple instances given a text prompt such as object names or referring expressions on images. It's particularly good at detecting specific objects given detailed descriptive prompts. It returns a list of bounding boxes with normalized coordinates, label names, masks and associated probability scores.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The object that needs to be counted, only supports a single prompt with no commas or periods.
TYPE:
|
image |
The image that contains multiple instances of the object.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label, bounding box, and mask of the detected objects with normalized coordinates (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. |
Example
>>> glee_sam2_instance_segmentation("a large blue flower", image)
[
{
'score': 0.49,
'label': 'a large blue flower',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
]
Source code in vision_agent/tools/tools.py
glee_sam2_video_tracking
'glee_sam2_video_tracking' is a tool that can track and segment multiple objects in a video given a text prompt such as object names or referring expressions. It's particularly good at detecting specific objects given detailed descriptive prompts and returns a list of bounding boxes, label names, masks and associated probability scores and is useful for tracking and counting without duplicating counts.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The prompt to ground to the image, only supports a single prompt with no commas or periods.
TYPE:
|
frames |
The list of frames to ground the prompt to.
TYPE:
|
chunk_length |
The number of frames to re-run agentic object detection to to find new objects.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[List[Dict[str, Any]]]
|
List[List[Dict[str, Any]]]: A list of list of dictionaries containing the label, segmentation mask and bounding boxes. The outer list represents each frame and the inner list is the entities per frame. The detected objects have normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. The label names are prefixed with their ID represent the total count. |
Example
>>> glee_sam2_video_tracking("a runner with yellow shoes", frames)
[
[
{
'label': '0: a runner with yellow shoes',
'bbox': [0.1, 0.11, 0.35, 0.4],
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
],
...
]
Source code in vision_agent/tools/tools.py
1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 |
|
qwen25_vl_images_vqa
'qwen25_vl_images_vqa' is a tool that can answer any questions about arbitrary images including regular images or images of documents or presentations. It can be very useful for document QA or OCR text extraction. It returns text as an answer to the question.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The question about the document image
TYPE:
|
images |
The reference images used for the question
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
str
|
A string which is the answer to the given prompt.
TYPE:
|
Example
>>> qwen25_vl_images_vqa('Give a summary of the document', images)
'The document talks about the history of the United States of America and its...'
Source code in vision_agent/tools/tools.py
qwen25_vl_video_vqa
'qwen25_vl_video_vqa' is a tool that can answer any questions about arbitrary videos including regular videos or videos of documents or presentations. It returns text as an answer to the question.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The question about the video
TYPE:
|
frames |
The reference frames used for the question
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
str
|
A string which is the answer to the given prompt.
TYPE:
|
Example
>>> qwen25_vl_video_vqa('Which football player made the goal?', frames)
'Lionel Messi'
Source code in vision_agent/tools/tools.py
qwen2_vl_images_vqa
'qwen2_vl_images_vqa' is a tool that can answer any questions about arbitrary images including regular images or images of documents or presentations. It can be very useful for document QA or OCR text extraction. It returns text as an answer to the question.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The question about the document image
TYPE:
|
images |
The reference images used for the question
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
str
|
A string which is the answer to the given prompt.
TYPE:
|
Example
>>> qwen2_vl_images_vqa('Give a summary of the document', images)
'The document talks about the history of the United States of America and its...'
Source code in vision_agent/tools/tools.py
qwen2_vl_video_vqa
'qwen2_vl_video_vqa' is a tool that can answer any questions about arbitrary videos including regular videos or videos of documents or presentations. It returns text as an answer to the question.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The question about the video
TYPE:
|
frames |
The reference frames used for the question
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
str
|
A string which is the answer to the given prompt.
TYPE:
|
Example
>>> qwen2_vl_video_vqa('Which football player made the goal?', frames)
'Lionel Messi'
Source code in vision_agent/tools/tools.py
ocr
'ocr' extracts text from an image. It returns a list of detected text, bounding boxes with normalized coordinates, and confidence scores. The results are sorted from top-left to bottom right.
PARAMETER | DESCRIPTION |
---|---|
image |
The image to extract text from.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the detected text, bbox with normalized coordinates, and confidence score. |
Example
>>> ocr(image)
[
{'label': 'hello world', 'bbox': [0.1, 0.11, 0.35, 0.4], 'score': 0.99},
]
Source code in vision_agent/tools/tools.py
claude35_text_extraction
'claude35_text_extraction' is a tool that can extract text from an image. It returns the extracted text as a string and can be used as an alternative to OCR if you do not need to know the exact bounding box of the text.
PARAMETER | DESCRIPTION |
---|---|
image |
The image to extract text from.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
str
|
The extracted text from the image.
TYPE:
|
Source code in vision_agent/tools/tools.py
document_extraction
'document_extraction' is a tool that can extract structured information out of documents with different layouts. It returns the extracted data in a structured hierarchical format containing text, tables, pictures, charts, and other information.
PARAMETER | DESCRIPTION |
---|---|
image |
The document image to analyze
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the extracted information. |
Example
>>> document_analysis(image)
{'pages':
[{'bbox': [0, 0, 1.0, 1.0],
'chunks': [{'bbox': [0.8, 0.1, 1.0, 0.2],
'label': 'page_header',
'order': 75
'caption': 'Annual Report 2024',
'summary': 'This annual report summarizes ...' },
{'bbox': [0.2, 0.9, 0.9, 1.0],
'label': 'table',
'order': 1119,
'caption': [{'Column 1': 'Value 1', 'Column 2': 'Value 2'},
'summary': 'This table illustrates a trend of ...'},
],
Source code in vision_agent/tools/tools.py
2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 2132 2133 2134 2135 2136 2137 2138 2139 2140 2141 2142 2143 2144 2145 2146 2147 2148 2149 2150 2151 2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 |
|
document_qa
'document_qa' is a tool that can answer any questions about arbitrary documents, presentations, or tables. It's very useful for document QA tasks, you can ask it a specific question or ask it to return a JSON object answering multiple questions about the document.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The question to be answered about the document image.
TYPE:
|
image |
The document image to analyze.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
str
|
The answer to the question based on the document's context.
TYPE:
|
Example
>>> document_qa(image, question)
'The answer to the question ...'
Source code in vision_agent/tools/tools.py
2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 |
|
activity_recognition
'activity_recognition' is a tool that can recognize activities in a video given a text prompt. It can be used to identify where specific activities or actions happen in a video and returns a list of 0s and 1s to indicate the activity.
PARAMETER | DESCRIPTION |
---|---|
prompt |
The event you want to identify, should be phrased as a question, for example, "Did a goal happen?".
TYPE:
|
frames |
The reference frames used for the question
TYPE:
|
model |
The model to use for the inference. Valid values are 'claude-35', 'gpt-4o', 'qwen2vl'.
TYPE:
|
chunk_length_frames |
length of each chunk in frames
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[float]
|
List[float]: A list of floats with a value of 1.0 if the activity is detected in the chunk_length_frames of the video. |
Example
>>> activity_recognition('Did a goal happened?', frames)
[0.0, 0.0, 0.0, 1.0, 1.0, 0.0]
Source code in vision_agent/tools/tools.py
2322 2323 2324 2325 2326 2327 2328 2329 2330 2331 2332 2333 2334 2335 2336 2337 2338 2339 2340 2341 2342 2343 2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 2357 2358 2359 2360 2361 2362 2363 2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 2377 2378 2379 2380 2381 2382 2383 2384 2385 2386 2387 2388 2389 2390 2391 2392 2393 2394 2395 2396 2397 2398 2399 2400 2401 2402 2403 |
|
vit_image_classification
'vit_image_classification' is a tool that can classify an image. It returns a list of classes and their probability scores based on image content.
PARAMETER | DESCRIPTION |
---|---|
image |
The image to classify or tag
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the labels and scores. One dictionary contains a list of labels and other a list of scores. |
Example
>>> vit_image_classification(image)
{"labels": ["leopard", "lemur, otter", "bird"], "scores": [0.68, 0.30, 0.02]},
Source code in vision_agent/tools/tools.py
vit_nsfw_classification
'vit_nsfw_classification' is a tool that can classify an image as 'nsfw' or 'normal'. It returns the predicted label and their probability scores based on image content.
PARAMETER | DESCRIPTION |
---|---|
image |
The image to classify or tag
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the labels and scores. One dictionary contains a list of labels and other a list of scores. |
Example
>>> vit_nsfw_classification(image)
{"label": "normal", "scores": 0.68},
Source code in vision_agent/tools/tools.py
detr_segmentation
'detr_segmentation' is a tool that can segment common objects in an image without any text prompt. It returns a list of detected objects as labels, their regions as masks and their scores.
PARAMETER | DESCRIPTION |
---|---|
image |
The image used to segment things and objects
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score, label and mask of the detected objects. The mask is binary 2D numpy array where 1 indicates the object and 0 indicates the background. |
Example
>>> detr_segmentation(image)
[
{
'score': 0.45,
'label': 'window',
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
{
'score': 0.70,
'label': 'bird',
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
},
]
Source code in vision_agent/tools/tools.py
2479 2480 2481 2482 2483 2484 2485 2486 2487 2488 2489 2490 2491 2492 2493 2494 2495 2496 2497 2498 2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 2515 2516 2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 2535 2536 2537 2538 2539 2540 2541 2542 2543 2544 |
|
depth_anything_v2
'depth_anything_v2' is a tool that runs depth anything v2 model to generate a depth image from a given RGB image. The returned depth image is monochrome and represents depth values as pixel intensities with pixel values ranging from 0 to 255.
PARAMETER | DESCRIPTION |
---|---|
image |
The image to used to generate depth image
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
ndarray
|
np.ndarray: A grayscale depth image with pixel values ranging from 0 to 255 where high values represent closer objects and low values further. |
Example
>>> depth_anything_v2(image)
array([[0, 0, 0, ..., 0, 0, 0],
[0, 20, 24, ..., 0, 100, 103],
...,
[10, 11, 15, ..., 202, 202, 205],
[10, 10, 10, ..., 200, 200, 200]], dtype=uint8),
Source code in vision_agent/tools/tools.py
generate_pose_image
'generate_pose_image' is a tool that generates a open pose bone/stick image from a given RGB image. The returned bone image is RGB with the pose amd keypoints colored and background as black.
PARAMETER | DESCRIPTION |
---|---|
image |
The image to used to generate pose image
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
ndarray
|
np.ndarray: A bone or pose image indicating the pose and keypoints |
Example
>>> generate_pose_image(image)
array([[0, 0, 0, ..., 0, 0, 0],
[0, 20, 24, ..., 0, 100, 103],
...,
[10, 11, 15, ..., 202, 202, 205],
[10, 10, 10, ..., 200, 200, 200]], dtype=uint8),
Source code in vision_agent/tools/tools.py
template_match
'template_match' is a tool that can detect all instances of a template in a given image. It returns the locations of the detected template, a corresponding similarity score of the same
PARAMETER | DESCRIPTION |
---|---|
image |
The image used for searching the template
TYPE:
|
template_image |
The template image or crop to search in the image
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Any]]
|
List[Dict[str, Any]]: A list of dictionaries containing the score and bounding box of the detected template with normalized coordinates between 0 and 1 (xmin, ymin, xmax, ymax). xmin and ymin are the coordinates of the top-left and xmax and ymax are the coordinates of the bottom-right of the bounding box. |
Example
>>> template_match(image, template)
[
{'score': 0.79, 'bbox': [0.1, 0.11, 0.35, 0.4]},
{'score': 0.38, 'bbox': [0.2, 0.21, 0.45, 0.5},
]
Source code in vision_agent/tools/tools.py
flux_image_inpainting
'flux_image_inpainting' performs image inpainting to fill the masked regions, given by mask, in the image, given image based on the text prompt and surrounding image context. It can be used to edit regions of an image according to the prompt given.
PARAMETER | DESCRIPTION |
---|---|
prompt |
A detailed text description guiding what should be generated in the masked area. More detailed and specific prompts typically yield better results.
TYPE:
|
image |
The source image to be inpainted. The image will serve as the base context for the inpainting process.
TYPE:
|
mask |
A binary mask image with 0's and 1's, where 1 indicates areas to be inpainted and 0 indicates areas to be preserved.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
ndarray
|
np.ndarray: The generated image(s) as a numpy array in RGB format with values ranging from 0 to 255. |
Example: >>> # Generate inpainting >>> result = flux_image_inpainting( ... prompt="a modern black leather sofa with white pillows", ... image=image, ... mask=mask, ... ) >>> save_image(result, "inpainted_room.png")
Source code in vision_agent/tools/tools.py
2684 2685 2686 2687 2688 2689 2690 2691 2692 2693 2694 2695 2696 2697 2698 2699 2700 2701 2702 2703 2704 2705 2706 2707 2708 2709 2710 2711 2712 2713 2714 2715 2716 2717 2718 2719 2720 2721 2722 2723 2724 2725 2726 2727 2728 2729 2730 2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 |
|
siglip_classification
'siglip_classification' is a tool that can classify an image or a cropped detection given a list of input labels or tags. It returns the same list of the input labels along with their probability scores based on image content.
PARAMETER | DESCRIPTION |
---|---|
image |
The image to classify or tag
TYPE:
|
labels |
The list of labels or tags that is associated with the image
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Dict[str, Any]
|
Dict[str, Any]: A dictionary containing the labels and scores. One dictionary contains a list of given labels and other a list of scores. |
Example
>>> siglip_classification(image, ['dog', 'cat', 'bird'])
{"labels": ["dog", "cat", "bird"], "scores": [0.68, 0.30, 0.02]},
Source code in vision_agent/tools/tools.py
minimum_distance
'minimum_distance' calculates the minimum distance between two detections which can include bounding boxes and or masks. This will return the closest distance between the objects, not the distance between the centers of the objects.
PARAMETER | DESCRIPTION |
---|---|
det1 |
The first detection of boxes or masks.
TYPE:
|
det2 |
The second detection of boxes or masks.
TYPE:
|
image_size |
The size of the image given as (height, width).
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
float
|
The closest distance between the two detections.
TYPE:
|
Example
>>> closest_distance(det1, det2, image_size)
141.42
Source code in vision_agent/tools/tools.py
closest_mask_distance
'closest_mask_distance' calculates the closest distance between two masks.
PARAMETER | DESCRIPTION |
---|---|
mask1 |
The first mask.
TYPE:
|
mask2 |
The second mask.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
float
|
The closest distance between the two masks.
TYPE:
|
Example
>>> closest_mask_distance(mask1, mask2)
0.5
Source code in vision_agent/tools/tools.py
closest_box_distance
'closest_box_distance' calculates the closest distance between two bounding boxes.
PARAMETER | DESCRIPTION |
---|---|
box1 |
The first bounding box.
TYPE:
|
box2 |
The second bounding box.
TYPE:
|
image_size |
The size of the image given as (height, width).
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
float
|
The closest distance between the two bounding boxes.
TYPE:
|
Example
>>> closest_box_distance([100, 100, 200, 200], [300, 300, 400, 400])
141.42
Source code in vision_agent/tools/tools.py
extract_frames_and_timestamps
'extract_frames_and_timestamps' extracts frames and timestamps from a video which can be a file path, url or youtube link, returns a list of dictionaries with keys "frame" and "timestamp" where "frame" is a numpy array and "timestamp" is the relative time in seconds where the frame was captured. The frame is a numpy array.
PARAMETER | DESCRIPTION |
---|---|
video_uri |
The path to the video file, url or youtube link
TYPE:
|
fps |
The frame rate per second to extract the frames. Defaults to 5.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
List[Dict[str, Union[ndarray, float]]]
|
List[Dict[str, Union[np.ndarray, float]]]: A list of dictionaries containing the extracted frame as a numpy array and the timestamp in seconds. |
Example
>>> extract_frames("path/to/video.mp4")
[{"frame": np.ndarray, "timestamp": 0.0}, ...]
Source code in vision_agent/tools/tools.py
2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 |
|
save_json
'save_json' is a utility function that saves data as a JSON file. It is helpful for saving data that contains NumPy arrays which are not JSON serializable.
PARAMETER | DESCRIPTION |
---|---|
data |
The data to save.
TYPE:
|
file_path |
The path to save the JSON file.
TYPE:
|
Example
>>> save_json(data, "path/to/file.json")
Source code in vision_agent/tools/tools.py
load_image
'load_image' is a utility function that loads an image from the given file path string or an URL.
PARAMETER | DESCRIPTION |
---|---|
image_path |
The path or URL to the image.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
ndarray
|
np.ndarray: The image as a NumPy array. |
Example
>>> load_image("path/to/image.jpg")
Source code in vision_agent/tools/tools.py
save_image
'save_image' is a utility function that saves an image to a file path.
PARAMETER | DESCRIPTION |
---|---|
image |
The image to save.
TYPE:
|
file_path |
The path to save the image file.
TYPE:
|
Example
>>> save_image(image)
Source code in vision_agent/tools/tools.py
save_video
'save_video' is a utility function that saves a list of frames as a mp4 video file on disk.
PARAMETER | DESCRIPTION |
---|---|
frames |
A list of frames to save.
TYPE:
|
output_video_path |
The path to save the video file. If not provided, a temporary file will be created.
TYPE:
|
fps |
The number of frames composes a second in the video.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
str
|
The path to the saved video file.
TYPE:
|
Example
>>> save_video(frames)
"/tmp/tmpvideo123.mp4"
Source code in vision_agent/tools/tools.py
overlay_bounding_boxes
'overlay_bounding_boxes' is a utility function that displays bounding boxes on an image. It will draw a box around the detected object with the label and score.
PARAMETER | DESCRIPTION |
---|---|
medias |
The image or frames to display the bounding boxes on.
TYPE:
|
bboxes |
A list of dictionaries or a list of list of dictionaries containing the bounding boxes.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Union[ndarray, List[ndarray]]
|
np.ndarray: The image with the bounding boxes, labels and scores displayed. |
Example
>>> image_with_bboxes = overlay_bounding_boxes(
image, [{'score': 0.99, 'label': 'dinosaur', 'bbox': [0.1, 0.11, 0.35, 0.4]}],
)
Source code in vision_agent/tools/tools.py
3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 |
|
overlay_segmentation_masks
'overlay_segmentation_masks' is a utility function that displays segmentation masks. It will overlay a colored mask on the detected object with the label.
PARAMETER | DESCRIPTION |
---|---|
medias |
The image or frames to display the masks on.
TYPE:
|
masks |
A list of dictionaries or a list of list of dictionaries containing the masks, labels and scores.
TYPE:
|
draw_label |
If True, the labels will be displayed on the image.
TYPE:
|
secondary_label_key |
The key to use for the secondary tracking label which is needed in videos to display tracking information.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
Union[ndarray, List[ndarray]]
|
np.ndarray: The image with the masks displayed. |
Example
>>> image_with_masks = overlay_segmentation_masks(
image,
[{
'score': 0.99,
'label': 'dinosaur',
'mask': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
}],
)
Source code in vision_agent/tools/tools.py
3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 3355 3356 3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 |
|
overlay_heat_map
'overlay_heat_map' is a utility function that displays a heat map on an image.
PARAMETER | DESCRIPTION |
---|---|
image |
The image to display the heat map on.
TYPE:
|
heat_map |
A dictionary containing the heat map under the key 'heat_map'.
TYPE:
|
alpha |
The transparency of the overlay. Defaults to 0.8.
TYPE:
|
RETURNS | DESCRIPTION |
---|---|
ndarray
|
np.ndarray: The image with the heat map displayed. |
Example
>>> image_with_heat_map = overlay_heat_map(
image,
{
'heat_map': array([[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 125, 125, 125]], dtype=uint8),
},
)