Feat/new model manager#2251
Conversation
b0de01d to
c27b669
Compare
…_pool flag; unload() only detaches pool
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
- inject shared dependency instance via SuppliedDependency, strict identity match vs metadata - PreloadedDependencyMismatchError; re-raise in auto-cache path instead of swallowing - shallow-copy model_init_kwargs per package attempt
- resolve_shared_base() detects head with supported base dep, negotiates concrete package, derives base_key - base_key excludes api_key for whitelisted public bases - cached metadata fetch; fail-open except auth denials
- one worker hosts a base model + many heads; head_index slot routing, monotonic never-recycled indexes - SharedHeadControlPlane load_head/drop_head over req_id-keyed futures; fail on timeout and worker death - owner backend: eager base load, per-head slot tracker, ref-count teardown, in-flight load reservation - SharedHeadBackend view: submit_slot/result-callback/drain delegate to owner; worker_pid None - ModelManager.load_shared_head: create-or-reuse owner under base-key lock, empty-owner reaping - constructor cleanup on failure; complete in-flight slots on death; stats refresh path
- _load_model_inner resolves shareable head before admission, loads via load_shared_head, reuses OOM/evict loop - skip admission when base already resident so base VRAM is counted once - track head before register_backend so worker death never sees an untracked loaded head - shared-base worker death marks all hosted heads not-loaded, clears inflight, drops backends; heads reload lazily - graceful unload/eviction untrack heads; ModelManager.has_shared_base accessor + death hook
- attribute shared owner worker_pid to base_key in stats, so base+heads GPU shows once - shared head footprint is 0; reclaimable VRAM lives in the base worker - eviction works on units (normal models + base_keys); base_key evictable only when all heads cold - evicting a base_key drops all heads so the worker and its VRAM are actually freed - OOM retry evicts the cold unit, never a sibling head that frees nothing - ModelManager.shared_owners accessor; drop orphaned _is_still_evictable
This reverts commit 53b0d31.
- ModelState.pinned; admin/preload T_LOAD pins, even an already-loaded model - eviction blocked at _flavor_loaded_idle chokepoint, so all planners honor it - a pinned head keeps its shared base resident; explicit T_UNLOAD still unloads - no T_LOAD device field: preload and default request both resolve DEFAULT_DEVICE
This reverts commit f017056.
95a3a0c to
c7c0af0
Compare
What does this PR do?
Adding
inference_model_managerType of Change
Testing
Initial suit of tests added
Checklist
Additional Context
N/A