Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Add support for GitHub Models #1719

Closed
jdubois opened this issue Sep 6, 2024 · 9 comments · Fixed by #1807
Closed

[FEATURE] Add support for GitHub Models #1719

jdubois opened this issue Sep 6, 2024 · 9 comments · Fixed by #1807
Labels
enhancement New feature or request P2 High priority

Comments

@jdubois
Copy link
Contributor

jdubois commented Sep 6, 2024

GitHub Models is a new feature provided by GitHub, which can be found at https://github.com/marketplace/models

It allows to access a variety of models using a GitHub API key, and they are running behind the scenes on Azure OpenAI.

It makes it easy for all GitHub users to use those models, and to support them here is my proposal:

  • Create a new "langchain4j-github-models" Maven module
  • Have this module depend on "langchain4j-azure-open-ai": as it's the same system underneath, that should simplify a lot testing and dependency management
  • Implement a specific mechanism for authentication (it's just a bearer token), and use the endpoint "https://models.inference.ai.azure.com" -> this should be most of the work
  • Have a list of supported models to help people configure their model names

If all goes well, the resulting API would be something like:

GitHubModelsChatModel.builder()
                .aipKey(gitHubApiKey)
                .deploymentName(GitHubModelsCahtModelName.PHI_3_5_MINI_INSTRUCT)
                .logRequestsAndResponses(true)
                .build();

Once this is all done, I'll try to have this in the official documentation, when you click on Get Started at https://github.com/marketplace/models/azureml/Phi-3-5-mini-instruct as they have currently 0 Java documentation.
@jdubois jdubois added the enhancement New feature or request label Sep 6, 2024
@langchain4j
Copy link
Owner

@jdubois sounds good!

@langchain4j langchain4j added the P2 High priority label Sep 9, 2024
@hrhrng
Copy link
Contributor

hrhrng commented Sep 16, 2024

@langchain4j Could you assign this to me? I just been granted access to GitHub Models.

@langchain4j
Copy link
Owner

@jdubois did you plan to work on this?

@jdubois
Copy link
Contributor Author

jdubois commented Sep 16, 2024

I haven't started coding, but I have started doing some research, and this uses a new Azure API called the "Azure AI Inference API", that is not accessible publicly yet for Java (it's already there in Python, JavaScript and .NET). So this wouldn't depend on the Azure OpenAI implementation, as I initially thought.
I'm in contact (internally at Microsoft) with the team working on this, and I'll have access to non-public releases -> not sure how @hrhrng can work on this right now.

@hrhrng
Copy link
Contributor

hrhrng commented Sep 16, 2024 •
edited
Loading

I thought I can work on this according to your proposal mentioned before. If that can't works, then just ignore my assign request😂. @jdubois

@jdubois
Copy link
Contributor Author

jdubois commented Sep 16, 2024

I also thought the same :-(
Then this new API should be public soon.

@niksbansode
Copy link
Contributor

I'm also looking to contribute to this. First, I will let @hrhrng spend time analyzing the approach.

@niksbansode
Copy link
Contributor

@jdubois @hrhrng @langchain4j Do we really need this? GitHub models are compatible with Open AI API. The below client works fine for me

ChatLanguageModel chatModel = OpenAiChatModel.builder()
   .apiKey(gitHubApiKey)
   .baseUrl("https://models.inference.ai.azure.com/")
   .modelName("Phi-3.5-mini-instruct")
   .temperature(0.8)
   .build();

@jdubois
Copy link
Contributor Author

jdubois commented Sep 19, 2024

It’s a different Java SDK than the Azure OpenAI SDK, as it supports other models than OpenAI (I’m unsure of what changes at the moment), and as the security part is different (much simpler, and with less dependencies).
The Azure AI Inference API is what is being used in JavaScript, Python and .NET and we’ll soon have it officially for Java.

jdubois added a commit to jdubois/langchain4j that referenced this issue Sep 20, 2024
@jdubois
First working test using the unreleased Azure AI Inference API
71f33d8 
Fix langchain4j#1719
@jdubois jdubois mentioned this issue Sep 20, 2024
Merged
langchain4j pushed a commit that referenced this issue Sep 24, 2024
@jdubois
Support for GitHub Models using the Azure AI Inference API (#1807)
d546c64 
Fix #1719

This adds GitHub Models (see https://github.com/marketplace/models )
support with the new Azure AI Inference API Java SDK.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request P2 High priority
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support for GitHub Models using the Azure AI Inference API jdubois/langchain4j
4 participants
@jdubois @niksbansode @hrhrng @langchain4j