Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for streaming with Anthropic Tools #1322

Open
Ashton-Sidhu opened this issue Oct 11, 2024 · 4 comments
Open

Add support for streaming with Anthropic Tools #1322

Ashton-Sidhu opened this issue Oct 11, 2024 · 4 comments
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@Ashton-Sidhu
Copy link

When using the Anthropic Agent with streaming: true, you get the error: "Anthropic does not support streaming . If you remove the code block

LlamaIndexTS/packages/llamaindex/src/agent/anthropic.ts

Lines 37 to 40 in 45bcf51

if (stream) {
// Anthropic does support this, but looks like it's not supported in the LITS LLM
throw new Error("Anthropic does not support streaming");
}
streaming does work fine but if you use tools, it says tools are not supported via Anthropic.

Anthropic does support streaming with tools and it would be great to get it integrated https://docs.anthropic.com/en/api/messages-streaming#streaming-request-with-tool-use
@himself65 himself65 added bug Something isn't working enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed and removed bug Something isn't working labels Oct 11, 2024
@SandipGyawali
Copy link

@Ashton-Sidhu is this open to contribute ?

@Ashton-Sidhu
Copy link
Author

@Ashton-Sidhu is this open to contribute ?

Absolutely! I don't have time at the moment to do it

@SandipGyawali
Copy link

@Ashton-Sidhu i can do it assign me ??

@gongzhaoman
Copy link

I think this feat is not simple, as the streaming response from Anthropic does not indicate whether a tool should be called at the first moment. This means that it is impossible to distinguish whether it is the final message output at the beginning of the streaming response or even during the streaming response process. Currently, I believe it seems that only after the streaming response is completely finished can we make a good judgment on whether to call a tool or if it is the final output, but this has already lost the meaning of streaming response.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Ashton-Sidhu @himself65 @SandipGyawali @gongzhaoman