Skip to content

Lrq/promethues integration#332

Open
lrq619 wants to merge 3 commits into
v1-betafrom
lrq/promethues_integration
Open

Lrq/promethues integration#332
lrq619 wants to merge 3 commits into
v1-betafrom
lrq/promethues_integration

Conversation

@lrq619

@lrq619 lrq619 commented Jan 18, 2026

Copy link
Copy Markdown
Member

Description

  • Add Prometheus support with two initial Router metrics: number of instances and number of requests.
  • Enable dynamic discovery of backend endpoints (e.g., vLLM/SGLang) using Prometheus file-based service discovery, updated on scale up/down.

Motivation

Integrate prometheus

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update

Checklist

  • I have read the CONTRIBUTION guide.
  • I have updated the tests (if applicable).
  • I have updated the documentation (if applicable).

@gemini-code-assist

gemini-code-assist Bot commented Jan 18, 2026

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello @lrq619, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the observability of the ServerlessLLM system by integrating Prometheus. It provides crucial insights into router performance and backend instance health through exposed metrics and enables automatic discovery of dynamic backend endpoints, streamlining monitoring operations and improving system reliability.

Highlights

  • Prometheus Metrics Integration: Added Prometheus support to the head node, exposing two initial router metrics: sllm_router_requests_total (total requests) and sllm_router_instances (number of healthy backend instances).
  • Dynamic Service Discovery: Implemented Prometheus file-based service discovery, allowing Prometheus to dynamically discover and monitor backend inference endpoints (e.g., vLLM/SGLang) as they scale up or down.
  • CLI and Documentation Updates: Introduced a new --prometheus-sd-path CLI option for the sllm start command and added comprehensive documentation for observability, covering Prometheus setup, available metrics, and file-based service discovery configuration.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jan 18, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces Prometheus integration for metrics and service discovery, which is a valuable addition for observability. The implementation is well-structured, with a new sllm/prometheus.py module that includes robust logic like atomic file writes for service discovery. The necessary changes are correctly wired into the Reconciler, APIGateway, and CLI components. The documentation and tests have also been appropriately updated to reflect these new features. I have one suggestion to improve efficiency by removing a redundant metric update in the Router, which I've detailed in a specific comment. Overall, this is a solid contribution.

Comment thread sllm/router.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@future-xy future-xy Awaiting requested review from future-xy
@SecretSettler SecretSettler Awaiting requested review from SecretSettler
1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lrq619