vision_agent.models

vision_agent.models.AgentMessage

Bases: BaseModel

AgentMessage encompases messages sent to the entire Agentic system, which includes both LMMs and sub-agents.

user: The user's message. assistant: The assistant's message. observation: An observation made after conducting an action, either by the user or assistant. interaction: An interaction between the user and the assistant. For example if the assistant wants to ask the user for help on a task, it could send an interaction message. interaction_response: The user's response to an interaction message. conversation: Messages coming from the conversation agent, this is a type of assistant messages. planner: Messages coming from the planner agent, this is a type of assistant messages. coder: Messages coming from the coder agent, this is a type of assistant messages.

role instance-attribute

role

content instance-attribute

content

media class-attribute instance-attribute

media = None

vision_agent.models.CodeContext

Bases: BaseModel

CodeContext is a data model that represents final code and test cases.

code: The final code that was written. test: The test cases that were written. success: A boolean value indicating whether the code passed the test cases. test_result: The result of running the test cases.

code instance-attribute

code

test instance-attribute

test

success instance-attribute

success

test_result instance-attribute

test_result

vision_agent.models.ErrorContext

Bases: BaseModel

ErrorContext is a data model that represents an error message. These errors can happen in the planning phase when a model does not output correctly formatted messages (often because it considers some response to be a safety issue).

error: The error message.

error instance-attribute

error

vision_agent.models.InteractionContext

Bases: BaseModel

InteractionContext is a data model that represents the context of an interaction.

chat: A list of messages exchanged between the user and the assistant.

chat instance-attribute

chat

vision_agent.models.PlanContext

Bases: BaseModel

PlanContext is a data model that represents the context of a plan.

plan: A description of the overall plan. instructions: A list of step-by-step instructions. code: Code snippets that were used during planning.

plan instance-attribute

plan

instructions instance-attribute

instructions

code instance-attribute

code

vision_agent.models.Message module-attribute

Message = Dict[str, Union[TextOrImage, Execution]]

vision_agent.models.TextOrImage module-attribute

TextOrImage = Union[
    str, Sequence[Union[str, Path, Image, ndarray]]
]

vision_agent.models.BboxInput

Bases: BaseModel

image_path instance-attribute

image_path

labels instance-attribute

labels

bboxes instance-attribute

bboxes

vision_agent.models.BboxInputBase64

Bases: BaseModel

image instance-attribute

image

filename instance-attribute

filename

labels instance-attribute

labels

bboxes instance-attribute

bboxes

vision_agent.models.BoundingBoxes module-attribute

BoundingBoxes = list[ODResponseData]

vision_agent.models.Florence2FtRequest

Bases: BaseModel

model_config class-attribute instance-attribute

model_config = ConfigDict(populate_by_name=True)

image class-attribute instance-attribute

image = None

video class-attribute instance-attribute

video = None

task instance-attribute

task

prompt class-attribute instance-attribute

prompt = ''

chunk_length_frames class-attribute instance-attribute

chunk_length_frames = None

postprocessing class-attribute instance-attribute

postprocessing = None

job_id class-attribute instance-attribute

job_id = Field(None, alias='jobId')

serialize_job_id

serialize_job_id(job_id, _info)

Source code in vision_agent/models/tools_types.py

@field_serializer("job_id")
def serialize_job_id(self, job_id: UUID, _info: SerializationInfo) -> str:
    return str(job_id)

vision_agent.models.JobStatus

Bases: str, Enum

The status of a fine-tuning job.

CREATED

The job has been created and is waiting to be scheduled to run.

STARTING: The job has started running, but not entering the training phase. TRAINING: The job is training a model. EVALUATING: The job is evaluating the model and computing metrics. PUBLISHING: The job is exporting the artifact(s) to an external directory (s3 or local). SUCCEEDED: The job has finished, including training, evaluation and publishing the artifact(s). FAILED: The job has failed for some reason internally, it can be due to resources issues or the code itself. STOPPED: The job has been stopped by the use locally or in the cloud.

CREATED class-attribute instance-attribute

CREATED = 'CREATED'

STARTING class-attribute instance-attribute

STARTING = 'STARTING'

TRAINING class-attribute instance-attribute

TRAINING = 'TRAINING'

EVALUATING class-attribute instance-attribute

EVALUATING = 'EVALUATING'

PUBLISHING class-attribute instance-attribute

PUBLISHING = 'PUBLISHING'

SUCCEEDED class-attribute instance-attribute

SUCCEEDED = 'SUCCEEDED'

FAILED class-attribute instance-attribute

FAILED = 'FAILED'

STOPPED class-attribute instance-attribute

STOPPED = 'STOPPED'

vision_agent.models.ODResponseData

Bases: BaseModel

label instance-attribute

label

score instance-attribute

score

bbox class-attribute instance-attribute

bbox = Field(alias='bounding_box')

model_config class-attribute instance-attribute

model_config = ConfigDict(populate_by_name=True)

vision_agent.models.PromptTask

Bases: str, Enum

Valid task prompts options for the Florence2 model.

PHRASE_GROUNDING class-attribute instance-attribute

PHRASE_GROUNDING = '<CAPTION_TO_PHRASE_GROUNDING>'