Mood: 😊
Category: Requests and ideas

MULTI-MODEL SUBAGENT ORCHESTRATION SYSTEM

1. Problem Statement

Currently, when a user interacts within a Session configured with a text-only Large Language Model (LLM), the system triggers an unsupported error if the input contains multimodal data such as images. This limits user experience and restricts the system's flexibility in handling complex Tasks that require distinct strengths from different models.

2. Proposed Solution

We propose a mechanism for Dynamic Input-Based Model Configuration and Task-Based Subagent Orchestration. Instead of forcing the main Session to process all types of data and tasks, the system will automatically decompose and delegate workloads to specialized Subagents running dedicated models, then aggregate the results back to the main Session.

3. Detailed Workflow

Phase 1: Input Type-Based Routing

• Trigger Condition: The user uploads an image into a Session currently configured with a text-only model.
• Workflow:
  1. The system detects that the input contains an image.
  2. The system looks up the Configuration to find the pre-defined Vision-supported model for this specific data type.
  3. A Subagent is initialized in the background, sending the image to the Vision model for analysis.
  4. The Subagent converts the image analysis results into a detailed textual description.
  5. This textual description is returned and fed into the Context of the original text-only model in the main Session to generate the final response for the user.

Phase 2: Extension - Task-Based Routing

• Trigger Condition: The system identifies a complex workflow consisting of multiple stages (e.g., Analysis followed by Coding).
• Workflow:
  1. The user configures the workflow pipeline, or the system automatically recognizes the task type via the Prompt.
  2. During the "Analysis" stage, the system activates a Subagent running a model optimized for reasoning (e.g., GPT-5.3).
  3. Once the analysis is complete, based on the task definition configuration, the system proactively triggers another Subagent running a model optimized for source code (e.g., Claude Opus) to execute the coding task.
  4. All intermediate results are seamlessly consolidated and delivered back to the main Session.

4. Key Benefits

• Cost and Performance Optimization: Eliminates the need to use expensive Multimodal models for the entire conversation when only a few simple images need to be processed.
• Enhanced Accuracy: Leverages the specialized strengths of individual models (e.g., combining a strong logic/analysis model with a powerful coding model).
• Seamless User Experience: Users no longer encounter "format not supported" errors, as the system automatically handles the processing in the background through seamless integration.