Skip to content

Feat/new model manager#2251

Draft
grzegorz-roboflow wants to merge 279 commits into
mainfrom
feat/new-model-manager
Draft

Feat/new model manager#2251
grzegorz-roboflow wants to merge 279 commits into
mainfrom
feat/new-model-manager

Conversation

@grzegorz-roboflow

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adding inference_model_manager

Type of Change

  • New feature (non-breaking change that adds functionality)

Testing

Initial suit of tests added

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code where necessary, particularly in hard-to-understand areas
  • My changes generate no new warnings or errors
  • I have updated the documentation accordingly (if applicable)

Additional Context

N/A

Comment thread inference_server/inference_server/app.py Fixed
Comment thread inference_server/inference_server/server.py Fixed
Comment thread inference_server/inference_server/server.py Fixed
Comment thread inference_server/inference_server/server.py Fixed
Comment thread inference_server/inference_server/server.py Fixed
Comment thread inference_server/inference_server/server.py Fixed
Comment thread inference_server/inference_server/server.py Fixed
Comment thread inference_server/inference_server/server.py Fixed
Comment thread inference_server/inference_server/server.py Fixed
Comment thread inference_server/inference_server/server.py Fixed
Comment thread inference_server/inference_server/server.py Fixed
Comment thread inference_server/inference_server/app.py Fixed
Comment thread inference_server/inference_server/server.py Fixed
Comment thread inference_server/inference_server/server.py Fixed
Comment thread inference_server/inference_server/auth.py Dismissed
Comment thread inference_models/inference_models/models/base/classification.py Outdated
Comment thread inference_models/inference_models/models/auto_loaders/core.py Outdated
Comment thread inference_models/inference_models/models/auto_loaders/core.py Outdated
Comment thread inference_models/inference_models/models/base/documents_parsing.py Outdated
Comment thread inference_models/inference_models/models/moondream2/moondream2_hf.py Outdated
@socket-security

socket-security Bot commented Jun 22, 2026

Copy link
Copy Markdown

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action Severity Alert  (click "▶" to expand/collapse)
Warn High
Obfuscated code: pypi orjson is 90.0% likely obfuscated

Confidence: 0.90

Location: Package overview

From: inference_server/pyproject.tomlpypi/[email protected]

ℹ Read more on: This package | This alert | What is obfuscated code?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at [email protected].

Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore pypi/[email protected]. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

  - inject shared dependency instance via SuppliedDependency, strict identity match
  vs metadata
  - PreloadedDependencyMismatchError; re-raise in auto-cache path instead of
  swallowing
  - shallow-copy model_init_kwargs per package attempt
  - resolve_shared_base() detects head with supported base dep, negotiates concrete
  package, derives base_key
  - base_key excludes api_key for whitelisted public bases
  - cached metadata fetch; fail-open except auth denials
  - one worker hosts a base model + many heads; head_index slot routing, monotonic
  never-recycled indexes
  - SharedHeadControlPlane load_head/drop_head over req_id-keyed futures; fail on
  timeout and worker death
  - owner backend: eager base load, per-head slot tracker, ref-count teardown,
  in-flight load reservation
  - SharedHeadBackend view: submit_slot/result-callback/drain delegate to owner;
  worker_pid None
  - ModelManager.load_shared_head: create-or-reuse owner under base-key lock,
  empty-owner reaping
  - constructor cleanup on failure; complete in-flight slots on death; stats refresh
  path
  - _load_model_inner resolves shareable head before admission, loads via
  load_shared_head, reuses OOM/evict loop
  - skip admission when base already resident so base VRAM is counted once
  - track head before register_backend so worker death never sees an untracked loaded
  head
  - shared-base worker death marks all hosted heads not-loaded, clears inflight,
  drops backends; heads reload lazily
  - graceful unload/eviction untrack heads; ModelManager.has_shared_base accessor +
  death hook
  - attribute shared owner worker_pid to base_key in stats, so base+heads GPU shows
  once
  - shared head footprint is 0; reclaimable VRAM lives in the base worker
  - eviction works on units (normal models + base_keys); base_key evictable only when
  all heads cold
  - evicting a base_key drops all heads so the worker and its VRAM are actually freed
  - OOM retry evicts the cold unit, never a sibling head that frees nothing
  - ModelManager.shared_owners accessor; drop orphaned _is_still_evictable
This reverts commit 53b0d31.
  - ModelState.pinned; admin/preload T_LOAD pins, even an already-loaded model
  - eviction blocked at _flavor_loaded_idle chokepoint, so all planners honor it
  - a pinned head keeps its shared base resident; explicit T_UNLOAD still unloads
  - no T_LOAD device field: preload and default request both resolve DEFAULT_DEVICE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants