Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement AsyncTextIteratorStreamer for asynchronous streaming #34931

Open
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

CISC
Copy link
Contributor

@CISC CISC commented Nov 25, 2024

Asynchronous text generation streaming

Added a new AsyncTextIteratorStreamer class that allows you to have fully asynchronous text generation streaming in your application. Works identically to TextIteratorStreamer except that you iterate the stream as follows:

generated_text = ""
async for new_text in streamer:
    generated_text += new_text

Requires Python 3.11+ due to use of asyncio.timeout to handle timeout. Falls back to asyncio.wait_for if asyncio.timeout is not available.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Any hints on how best to implement an async test would be appreciated. :) pytest-asyncio seems to be the way to go...

Who can review?

@gante @ArthurZucker

@CISC
Copy link
Contributor Author

CISC commented Nov 25, 2024

Hmmm, looks like I'm missing something important regarding documentation, not sure what though?

Exception: The following objects are in the public init so should be documented:

  • AsyncTextIteratorStreamer

Was missing autodoc entry.

@CISC
Copy link
Contributor Author

CISC commented Nov 27, 2024

@gante @ArthurZucker Ready for review.

The check_code_quality workflow failure makes no sense to me, not sure how to fix?

@CISC
Copy link
Contributor Author

CISC commented Dec 3, 2024

@ArthurZucker @gante ping

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant