[wip] feat(agents-api,sdks): multimodal support #343
Conversation
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
👍 Looks good to me! Reviewed everything up to 953f9d9 in 1 minute and 24 seconds
More details
- Looked at
1021lines of code in25files - Skipped
3files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_fb9m71EE7XAZQd2M
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on a8440fc in 1 minute and 35 seconds
More details
- Looked at
516lines of code in22files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. sdks/ts/src/api/schemas/$doc_id.ts:6
- Draft comment:
type: "string",
format: "uuid",
- Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 50%.
Workflow ID: wflow_I6RfMLrQ71IevVKm
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
❌ Changes requested. Incremental review on 0f67e4a in 1 minute and 13 seconds
More details
- Looked at
72lines of code in4files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_5pHlbzIhjLzYVg4q
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
creatorrr
changed the title
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on 5ed236e in 2 minutes and 8 seconds
More details
- Looked at
22lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. sdks/python/julep/api/client.py:829
- Draft comment:
The PR is supposed to introduce multimodal support for ChatML messages, but the changes in this Python SDK file do not reflect any updates related to handling new content types such as images or other multimedia elements. Please ensure that the SDK updates align with the PR's intent to support multimodal features. - Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 50%.
Workflow ID: wflow_ojKvKpVHiIxyuPYR
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
❌ Changes requested. Incremental review on dfad045 in 2 minutes and 36 seconds
More details
- Looked at
180lines of code in2files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_c5M5AHPajdjBHoks
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
❌ Changes requested. Incremental review on b11cc4a in 1 minute and 22 seconds
More details
- Looked at
56lines of code in2files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_2sZKR0Qt37xsaRUF
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on fb5cd60 in 2 minutes and 40 seconds
More details
- Looked at
370lines of code in12files - Skipped
1files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. sdks/python/julep/api/types/chat_ml_message_content.py:13
- Draft comment:
The union type forChatMlMessageContentcorrectly includesstrandtyping.List[ChatMlMessageContentItem], aligning with the multimodal support described in the PR. This allows for both text and image content parts within ChatML messages. - Reason this comment was not posted:
Confidence changes required:0%
The PR description mentions the introduction of multimodal support for ChatML messages, which includes handling both text and image content. The PR modifies the SDKs to handle these new content types. The changes in the SDKs should reflect the ability to handle these new types of content effectively. The PR also mentions updates to the Python and TypeScript SDKs, which should include the necessary changes to support these new features. The PR also mentions a new debug utility, which should be reviewed for proper implementation and usefulness in debugging.
Workflow ID: wflow_V9dqLo71FBbFGMZn
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
❌ Changes requested. Incremental review on 7d6288b in 1 minute and 32 seconds
More details
- Looked at
22lines of code in1files - Skipped
2files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_uixLAGX4yqXNSGq4
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
❌ Changes requested. Incremental review on 9975838 in 1 minute and 59 seconds
More details
- Looked at
122lines of code in3files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_sAqlt4ejFzXMfhau
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
❌ Changes requested. Incremental review on ce03bb4 in 3 minutes and 25 seconds
More details
- Looked at
15lines of code in1files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_FxQmhHyeE3bruubZ
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on 627f494 in 2 minutes and 50 seconds
More details
- Looked at
85lines of code in5files - Skipped
0files when reviewing. - Skipped posting
2drafted comments based on config settings.
1. sdks/python/julep/api/types/chat_ml_message_content.py:7
- Draft comment:
The removal of imports related to image content handling in this file and 'input_chat_ml_message_content.py' seems inconsistent with the PR's goal of introducing multimodal support. Please ensure that all necessary components for handling different types of content, such as images, are correctly integrated. - Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 50%.
2. sdks/python/julep/api/types/input_chat_ml_message_content.py:7
- Draft comment:
The removal of imports related to image content handling in this file and 'chat_ml_message_content.py' seems inconsistent with the PR's goal of introducing multimodal support. Please ensure that all necessary components for handling different types of content, such as images, are correctly integrated. - Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 50%.
Workflow ID: wflow_cbzMGY0oiWhm9rE6
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on fdb5515 in 1 minute and 25 seconds
More details
- Looked at
153lines of code in4files - Skipped
1files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. agents-api/agents_api/routers/sessions/session.py:449
- Draft comment:
The session memory feature is disabled with a FIXME comment. This could lead to unexpected behavior where session data is not being stored even if theremembersetting is true. Please address this to ensure correct session management.
# if not final_settings.remember:
# return
- Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 50%.
Workflow ID: wflow_LITdxXRTxqr6csJZ
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
❌ Changes requested. Incremental review on 9b63985 in 2 minutes and 12 seconds
More details
- Looked at
34lines of code in2files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_q9ULOk5f3IvYWrxm
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
Signed-off-by: Diwank Tomer <[email protected]>
Signed-off-by: Diwank Tomer <[email protected]>
Signed-off-by: Diwank Tomer <[email protected]>
Signed-off-by: Diwank Tomer <[email protected]>
Signed-off-by: Diwank Tomer <[email protected]>
Signed-off-by: Diwank Tomer <[email protected]>
Signed-off-by: Diwank Tomer <[email protected]>
Signed-off-by: Diwank Singh <[email protected]>
Signed-off-by: Diwank Tomer <[email protected]>
Signed-off-by: Diwank Tomer <[email protected]>
Signed-off-by: Diwank Tomer <[email protected]>
whiterabbit1983
force-pushed
the
f/multimodal-support
branch
from
9b63985
to
f2640e6
Compare
There was a problem hiding this comment.
❌ Changes requested. Incremental review on f2640e6 in 1 minute and 43 seconds
More details
- Looked at
34lines of code in2files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_TQOtgFPLrmTsXfOA
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on 2e897c1 in 1 minute and 12 seconds
More details
- Looked at
13lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. agents-api/agents_api/common/protocol/entries.py:45
- Draft comment:
The change to calculate the length of the JSON string representation of the dictionary content is correct and necessary for proper token count estimation. - Reason this comment was not posted:
Confidence changes required:0%
The code change in thetoken_countmethod of theEntryclass is intended to fix the calculation ofcontent_lengthwhen the content is a dictionary. Previously, the method was incorrectly assigning the JSON string representation of the dictionary tocontent_lengthinstead of its length. The change correctly calculates the length of the JSON string, which is necessary for estimating the token count when the content is stored in a dictionary format.
Workflow ID: wflow_2T8zF2HaCHBmQF80
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
Summary:
Enhances ChatML messages with multimodal support, updates SDKs, revises the database schema, adds a new debug utility, and fixes content length calculation in the 'Entry' class.
Key points:
pdb_on_exceptionfor better error handling.get_developer_idto accept UUIDs directly.Entryclass for dictionary types.Generated with ❤️ by ellipsis.dev