Improve Planner to Support invalid ASK - Issue 159#406
Conversation
rogerbarreto
added
enhancement
PR: ready for review
rogerbarreto
changed the title
| // TODO - consider adding the relevancy score for functions added to manual | ||
|
|
||
| var plan = await this._functionFlowFunction.InvokeAsync(context); | ||
| if (await this.VerifyCanCreatePlanForGoalAsync(goal, context)) |
There was a problem hiding this comment.
I don't think we should hit the model before creating a plan, just to ask if it can. Extra time, expense. This should be merged into the Planner prompt itself: "Create a plan. If you can't generate a plan, respond with..."
There was a problem hiding this comment.
@shawncal I wasn't able to achieve this with the current plan prompt (adding variations without compromising the result) in GPT-3.
It was always assuming the few shots as potential solutions and using that or the available functions as a valid plan even if it wouldn't resolve the plan.
There was a problem hiding this comment.
I think there are some other ways we can do this, let's connect.
lemillermicrosoft
added
PR: feedback to address
|
See also #523 which wants to remove the planner altogether |
PlannerSkill - not Planner though :) But yeah, we will still want to do something to address this issue. Did not do in that PR to reduce scope of changes. |
|
@rogerbarreto I think we can close this draft. Can you confirm? |
4 participants
Motivation and Context
Resolve Related Issue: Improve Planner prompt to support invalid ASK
This change mitigate the potential of generated plans with non existing Skills / Functions not provided.
If you ask for a Plan with insufficient Skills to fulfill, it will fail with a reason why.
context.ErrorDescriptionDescription
This change makes an additional query to the LLM providing only the available skills and the Goal without mixing it with the plan rules.
This provides clarity when asking for plans that cannot be created and avoid the potential of LLM wrong assumptions, hallucinations using the few shots on the planning creation prompt.
This changes adds also a post verification when the plan is created to ensure the functions returned by the plan are indeed present in the SkillCollection.
Contribution Checklist
dotnet format