Skip to content

Commit cfd3dfe

Browse files
giles17Copilot
giles17
and
Copilot
authored
.NET: CI hardening — split Functions tests, re-enable skipped integration tests (microsoft#5717)
* Split DurableTask/AzureFunctions integration tests into dedicated CI job - Add -TestProjectNameExclude parameter to New-FilteredSolution.ps1 - Add 'functions' and 'core' path filters to paths-filter job - Exclude DurableTask/AzureFunctions from main dotnet-test job - Remove emulator setup from dotnet-test (no longer needed) - Add new dotnet-test-functions job (ubuntu/net10.0 only, path-conditional) - Update merge gate and report job to include dotnet-test-functions Co-authored-by: Copilot <[email protected]> * Address PR feedback: add Workflows.Generators to core filter, drop dotnetChanges gate from functions job Co-authored-by: Copilot <[email protected]> * Re-enable Anthropic integration tests Co-authored-by: Copilot <[email protected]> * Upgrade Anthropic SDK 12.13.0 -> 12.20.0 to fix M.E.AI incompatibility Fixes MissingMethodException on WebSearchToolResultContent.get_Results() caused by Anthropic 12.13.0 being compiled against an older Microsoft.Extensions.AI.Abstractions version. Suppress RT0003 in AI.Abstractions.csproj as the transitive reference from the upgraded Anthropic SDK conflicts with the explicit one. Co-authored-by: Copilot <[email protected]> * Fix Anthropic unit test mocks for SDK 12.20.0 interface changes Add missing interface members: IAnthropicClient.WebhookKey, IBetaService.MemoryStores, IBetaService.Webhooks, IBetaService.UserProfiles Co-authored-by: Copilot <[email protected]> * Re-enable CheckSystem declarative integration tests The CheckSystem.yaml tests were temporarily skipped in PR microsoft#4270 during the Azure.AI.Projects 2.0.0-beta.1 SDK update. Since then, the system variable plumbing (SystemScope, SetLastMessageAsync, conversation initialization) has been significantly updated and stabilized. The other tests in these same files pass reliably using the same infrastructure. Co-authored-by: Copilot <[email protected]> * Fix CheckSystem test case to expect 1 response The CheckSystem workflow sends a 'PASSED!' SendActivity when all system variables are populated, producing 1 AgentResponseEvent. The test case had min_response_count: 0 with no max, so the assertion defaulted max to 0 and failed with 'Response count greater than expected: 0 (Actual: 1)'. Updated to expect exactly 1 response, matching the SendActivity pattern. Co-authored-by: Copilot <[email protected]> * Re-enable Foundry OpenAPI server-side tool integration test Remove Skip="For manual testing only" from AsAIAgent_WithOpenAPITool_NativeSDKCreation_InvokesServerSideToolAsync. The test already uses RetryFact(3 retries, 5s delay) to handle transient failures from the external restcountries.com API. Co-authored-by: Copilot <[email protected]> * Include workflow file in functions/core path filters A PR editing only dotnet-build-and-test.yml would skip dotnet-test-functions because the workflow path was missing from both the functions and core path filter lists. Co-authored-by: Copilot <[email protected]> * Rename filter parameters for consistency TestProjectNameFilter -> TestProjectNameIncludeFilter TestProjectNameExclude -> TestProjectNameExcludeFilter Co-authored-by: Copilot <[email protected]> * Remove unnecessary RT0003 warning suppression The RT0003 suppression was added during the Anthropic SDK 12.20.0 upgrade but the warning no longer fires. Removing it to keep the NoWarn list minimal. Co-authored-by: Copilot <[email protected]> * Remove duplicate WebhookKey properties from merge Both our branch and main added WebhookKey to the Anthropic test mock classes, resulting in CS0102 duplicate definition errors. Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]>
1 parent 3b6a457 commit cfd3dfe

6 files changed

Lines changed: 163 additions & 20 deletions

File tree

‎.github/workflows/dotnet-build-and-test.yml‎

Lines changed: 124 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,8 @@ jobs:
3838
dotnetChanges: ${{ steps.filter.outputs.dotnet }}
3939
cosmosDbChanges: ${{ steps.filter.outputs.cosmosdb }}
4040
foundryHostingChanges: ${{ steps.filter.outputs.foundryHosting }}
41+
functionsChanged: ${{ steps.filter.outputs.functions }}
42+
coreChanged: ${{ steps.filter.outputs.core }}
4143
steps:
4244
- uses: actions/checkout@v6
4345
- uses: dorny/paths-filter@v3
@@ -64,6 +66,24 @@ jobs:
6466
- 'dotnet/Directory.Packages.props'
6567
- 'dotnet/tests/Foundry.Hosting.IntegrationTests/scripts/it-build-image.ps1'
6668
- '.github/workflows/dotnet-build-and-test.yml'
69+
functions:
70+
- 'dotnet/src/Microsoft.Agents.AI.DurableTask/**'
71+
- 'dotnet/src/Microsoft.Agents.AI.Hosting.AzureFunctions/**'
72+
- 'dotnet/tests/Microsoft.Agents.AI.DurableTask.IntegrationTests/**'
73+
- 'dotnet/tests/Microsoft.Agents.AI.Hosting.AzureFunctions.IntegrationTests/**'
74+
- '.github/actions/azure-functions-integration-setup/**'
75+
- '.github/workflows/dotnet-build-and-test.yml'
76+
core:
77+
- 'dotnet/src/Microsoft.Agents.AI/**'
78+
- 'dotnet/src/Microsoft.Agents.AI.Abstractions/**'
79+
- 'dotnet/src/Microsoft.Agents.AI.OpenAI/**'
80+
- 'dotnet/src/Microsoft.Agents.AI.Workflows/**'
81+
- 'dotnet/src/Microsoft.Agents.AI.Workflows.Generators/**'
82+
- 'dotnet/eng/scripts/New-FilteredSolution.ps1'
83+
- 'dotnet/tests/Directory.Build.props'
84+
- 'dotnet/Directory.Packages.props'
85+
- 'dotnet/global.json'
86+
- '.github/workflows/dotnet-build-and-test.yml'
6787
# run only if 'dotnet' files were changed
6888
- name: dotnet tests
6989
if: steps.filter.outputs.dotnet == 'true'
@@ -211,10 +231,11 @@ jobs:
211231
Verbose = $true
212232
}
213233
./dotnet/eng/scripts/New-FilteredSolution.ps1 @commonArgs `
214-
-TestProjectNameFilter "*UnitTests*" `
234+
-TestProjectNameIncludeFilter "*UnitTests*" `
215235
-OutputPath dotnet/filtered-unit.slnx
216236
./dotnet/eng/scripts/New-FilteredSolution.ps1 @commonArgs `
217-
-TestProjectNameFilter "*IntegrationTests*" `
237+
-TestProjectNameIncludeFilter "*IntegrationTests*" `
238+
-TestProjectNameExcludeFilter "*DurableTask.IntegrationTests*","*AzureFunctions.IntegrationTests*" `
218239
-OutputPath dotnet/filtered-integration.slnx
219240
220241
- name: Run Unit Tests
@@ -256,14 +277,6 @@ jobs:
256277
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
257278
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
258279

259-
# This setup action is required for both Durable Task and Azure Functions integration tests.
260-
# We only run it on Ubuntu since the Durable Task and Azure Functions features are not available
261-
# on .NET Framework (net472) which is what we use the Windows runner for.
262-
- name: Set up Durable Task and Azure Functions Integration Test Emulators
263-
if: github.event_name != 'pull_request' && matrix.integration-tests && matrix.os == 'ubuntu-latest'
264-
uses: ./.github/actions/azure-functions-integration-setup
265-
id: azure-functions-setup
266-
267280
- name: Run Integration Tests
268281
shell: pwsh
269282
working-directory: dotnet
@@ -416,11 +429,110 @@ jobs:
416429
AZURE_SEARCH_INDEX_NAME: ${{ secrets.AZURE_SEARCH_INDEX_NAME }}
417430
# IT_HOSTED_AGENT_IMAGE was exported into $GITHUB_ENV by the previous step.
418431

432+
# DurableTask and AzureFunctions integration tests (ubuntu/net10.0 only).
433+
# Split from main dotnet-test job for path-based filtering and parallelism.
434+
dotnet-test-functions:
435+
needs: [paths-filter]
436+
if: >
437+
github.event_name != 'pull_request' &&
438+
(needs.paths-filter.outputs.functionsChanged == 'true' ||
439+
needs.paths-filter.outputs.coreChanged == 'true' ||
440+
github.event_name == 'schedule' ||
441+
github.event_name == 'workflow_dispatch')
442+
runs-on: ubuntu-latest
443+
environment: integration
444+
steps:
445+
- uses: actions/checkout@v6
446+
with:
447+
persist-credentials: false
448+
sparse-checkout: |
449+
.
450+
.github
451+
dotnet
452+
python
453+
declarative-agents
454+
455+
- name: Setup dotnet
456+
uses: actions/[email protected]
457+
with:
458+
global-json-file: ${{ github.workspace }}/dotnet/global.json
459+
460+
- name: Build functions integration test projects
461+
shell: bash
462+
working-directory: dotnet
463+
run: |
464+
dotnet build ./tests/Microsoft.Agents.AI.DurableTask.IntegrationTests -c Release -f net10.0 --warnaserror
465+
dotnet build ./tests/Microsoft.Agents.AI.Hosting.AzureFunctions.IntegrationTests -c Release -f net10.0 --warnaserror
466+
467+
- name: Azure CLI Login
468+
uses: azure/login@v2
469+
with:
470+
client-id: ${{ secrets.AZURE_CLIENT_ID }}
471+
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
472+
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
473+
474+
- name: Set up Durable Task and Azure Functions Integration Test Emulators
475+
uses: ./.github/actions/azure-functions-integration-setup
476+
id: azure-functions-setup
477+
478+
- name: Run Functions Integration Tests
479+
shell: pwsh
480+
working-directory: dotnet
481+
run: |
482+
# Run DurableTask integration tests
483+
dotnet test `
484+
--project ./tests/Microsoft.Agents.AI.DurableTask.IntegrationTests `
485+
-f net10.0 `
486+
-c Release `
487+
--no-build -v Normal `
488+
--report-xunit-trx `
489+
--report-junit `
490+
--results-directory ../IntegrationTestResults/ `
491+
--ignore-exit-code 8 `
492+
--filter-not-trait "Category=IntegrationDisabled" `
493+
--parallel-algorithm aggressive `
494+
--max-threads 2.0x
495+
496+
# Run AzureFunctions integration tests
497+
dotnet test `
498+
--project ./tests/Microsoft.Agents.AI.Hosting.AzureFunctions.IntegrationTests `
499+
-f net10.0 `
500+
-c Release `
501+
--no-build -v Normal `
502+
--report-xunit-trx `
503+
--report-junit `
504+
--results-directory ../IntegrationTestResults/ `
505+
--ignore-exit-code 8 `
506+
--filter-not-trait "Category=IntegrationDisabled" `
507+
--parallel-algorithm aggressive `
508+
--max-threads 2.0x
509+
env:
510+
# OpenAI Models
511+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
512+
OPENAI_CHAT_MODEL_NAME: ${{ vars.OPENAI_CHAT_MODEL_NAME }}
513+
OPENAI_REASONING_MODEL_NAME: ${{ vars.OPENAI_REASONING_MODEL_NAME }}
514+
# Azure OpenAI Models
515+
AZURE_OPENAI_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_DEPLOYMENT_NAME }}
516+
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_DEPLOYMENT_NAME }}
517+
AZURE_OPENAI_ENDPOINT: ${{ vars.AZURE_OPENAI_ENDPOINT }}
518+
# Azure AI Foundry
519+
AZURE_AI_PROJECT_ENDPOINT: ${{ vars.AZURE_AI_PROJECT_ENDPOINT }}
520+
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ vars.AZURE_AI_MODEL_DEPLOYMENT_NAME }}
521+
AZURE_AI_BING_CONNECTION_ID: ${{ vars.AZURE_AI_BING_CONNECTION_ID }}
522+
523+
- name: Upload functions test results
524+
if: always()
525+
uses: actions/upload-artifact@v7
526+
with:
527+
name: dotnet-test-results-functions-net10.0-ubuntu-latest
528+
path: IntegrationTestResults/**/*.junit
529+
if-no-files-found: ignore
530+
419531
# This final job is required to satisfy the merge queue. It must only run (or succeed) if no tests failed
420532
dotnet-build-and-test-check:
421533
if: always()
422534
runs-on: ubuntu-latest
423-
needs: [dotnet-build, dotnet-test, dotnet-foundry-hosted-it]
535+
needs: [dotnet-build, dotnet-test, dotnet-foundry-hosted-it, dotnet-test-functions]
424536
steps:
425537
- name: Get Date
426538
shell: bash
@@ -467,7 +579,7 @@ jobs:
467579
github.event_name != 'pull_request' &&
468580
(contains(join(needs.*.result, ','), 'success') ||
469581
contains(join(needs.*.result, ','), 'failure'))
470-
needs: [dotnet-test]
582+
needs: [dotnet-test, dotnet-test-functions]
471583
runs-on: ubuntu-latest
472584
defaults:
473585
run:

‎dotnet/eng/scripts/New-FilteredSolution.ps1‎

Lines changed: 32 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,15 @@
2121
.PARAMETER Configuration
2222
Optional MSBuild configuration used when querying TargetFrameworks. Defaults to Debug.
2323
24-
.PARAMETER TestProjectNameFilter
24+
.PARAMETER TestProjectNameIncludeFilter
2525
Optional wildcard pattern to filter test project names (e.g., *UnitTests*, *IntegrationTests*).
2626
When specified, only test projects whose filename matches this pattern are kept.
2727
28+
.PARAMETER TestProjectNameExcludeFilter
29+
Optional wildcard pattern(s) to exclude test projects by name (e.g., *DurableTask.IntegrationTests*).
30+
When specified, test projects whose filename matches any of these patterns are removed.
31+
Applied after TestProjectNameIncludeFilter. Can be a single string or an array of strings.
32+
2833
.PARAMETER ExcludeSamples
2934
When specified, removes all projects under the samples/ directory from the solution.
3035
@@ -38,11 +43,15 @@
3843
3944
.EXAMPLE
4045
# Generate a solution with only unit test projects
41-
./dotnet/eng/scripts/New-FilteredSolution.ps1 -Solution dotnet/agent-framework-dotnet.slnx -TargetFramework net10.0 -TestProjectNameFilter "*UnitTests*" -OutputPath filtered-unit.slnx
46+
./dotnet/eng/scripts/New-FilteredSolution.ps1 -Solution dotnet/agent-framework-dotnet.slnx -TargetFramework net10.0 -TestProjectNameIncludeFilter "*UnitTests*" -OutputPath filtered-unit.slnx
4247
4348
.EXAMPLE
4449
# Inline usage with dotnet test (PowerShell)
4550
dotnet test --solution (./dotnet/eng/scripts/New-FilteredSolution.ps1 -Solution dotnet/agent-framework-dotnet.slnx -TargetFramework net472) --no-build -f net472
51+
52+
.EXAMPLE
53+
# Generate integration tests excluding DurableTask and AzureFunctions
54+
./dotnet/eng/scripts/New-FilteredSolution.ps1 -Solution dotnet/agent-framework-dotnet.slnx -TargetFramework net10.0 -TestProjectNameIncludeFilter "*IntegrationTests*" -TestProjectNameExcludeFilter "*DurableTask.IntegrationTests*","*AzureFunctions.IntegrationTests*" -OutputPath filtered-other-integration.slnx
4655
#>
4756

4857
[CmdletBinding()]
@@ -55,7 +64,9 @@ param(
5564

5665
[string]$Configuration = "Debug",
5766

58-
[string]$TestProjectNameFilter,
67+
[string]$TestProjectNameIncludeFilter,
68+
69+
[string[]]$TestProjectNameExcludeFilter,
5970

6071
[switch]$ExcludeSamples,
6172

@@ -100,13 +111,30 @@ foreach ($proj in $allProjects) {
100111
$isTestProject = $projRelPath -like "*tests/*"
101112

102113
# Filter test projects by name pattern if specified
103-
if ($isTestProject -and $TestProjectNameFilter -and ($projFileName -notlike $TestProjectNameFilter)) {
114+
if ($isTestProject -and $TestProjectNameIncludeFilter -and ($projFileName -notlike $TestProjectNameIncludeFilter)) {
104115
Write-Verbose "Removing (name filter): $projRelPath"
105116
$removed += $projRelPath
106117
$proj.ParentNode.RemoveChild($proj) | Out-Null
107118
continue
108119
}
109120

121+
# Exclude test projects matching any exclusion pattern
122+
if ($isTestProject -and $TestProjectNameExcludeFilter) {
123+
$excluded = $false
124+
foreach ($pattern in $TestProjectNameExcludeFilter) {
125+
if ($projFileName -like $pattern) {
126+
$excluded = $true
127+
break
128+
}
129+
}
130+
if ($excluded) {
131+
Write-Verbose "Removing (exclude filter): $projRelPath"
132+
$removed += $projRelPath
133+
$proj.ParentNode.RemoveChild($proj) | Out-Null
134+
continue
135+
}
136+
}
137+
110138
if (-not (Test-Path $projFullPath)) {
111139
Write-Verbose "Project not found, keeping in solution: $projRelPath"
112140
$kept += $projRelPath

‎dotnet/tests/Foundry.IntegrationTests/FoundryVersionedAgentCreateTests.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -183,7 +183,7 @@ and report the SECRET_NUMBER value it prints. Respond only with the number.
183183
/// invokes the server-side OpenAPI function through <c>RunAsync</c>.
184184
/// Regression test for https://github.com/microsoft/agent-framework/issues/4883.
185185
/// </summary>
186-
[RetryFact(Constants.RetryCount, Constants.RetryDelay, Skip = "For manual testing only")]
186+
[RetryFact(Constants.RetryCount, Constants.RetryDelay)]
187187
public async Task AsAIAgent_WithOpenAPITool_NativeSDKCreation_InvokesServerSideToolAsync()
188188
{
189189
// Arrange — create agent version with OpenAPI tool using native Azure.AI.Projects SDK types.

‎dotnet/tests/Microsoft.Agents.AI.Workflows.Declarative.IntegrationTests/DeclarativeCodeGenTest.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ namespace Microsoft.Agents.AI.Workflows.Declarative.IntegrationTests;
1414
public sealed class DeclarativeCodeGenTest(ITestOutputHelper output) : WorkflowTest(output)
1515
{
1616
[Theory]
17-
[InlineData("CheckSystem.yaml", "CheckSystem.json", Skip = "Temporarily skipped")]
17+
[InlineData("CheckSystem.yaml", "CheckSystem.json")]
1818
[InlineData("SendActivity.yaml", "SendActivity.json")]
1919
[InlineData("InvokeAgent.yaml", "InvokeAgent.json")]
2020
[InlineData("InvokeAgent.yaml", "InvokeAgent.json", true)]

‎dotnet/tests/Microsoft.Agents.AI.Workflows.Declarative.IntegrationTests/DeclarativeWorkflowTest.cs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ namespace Microsoft.Agents.AI.Workflows.Declarative.IntegrationTests;
1515
public sealed class DeclarativeWorkflowTest(ITestOutputHelper output) : WorkflowTest(output)
1616
{
1717
[Theory]
18-
[InlineData("CheckSystem.yaml", "CheckSystem.json", Skip = "Temporarily skipped")]
18+
[InlineData("CheckSystem.yaml", "CheckSystem.json")]
1919
[InlineData("ConversationMessages.yaml", "ConversationMessages.json")]
2020
[InlineData("ConversationMessages.yaml", "ConversationMessages.json", true)]
2121
[InlineData("InputArguments.yaml", "InputArguments.json")]

‎dotnet/tests/Microsoft.Agents.AI.Workflows.Declarative.IntegrationTests/Testcases/CheckSystem.json‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,10 @@
1010
"conversation_count": 1,
1111
"min_action_count": 2,
1212
"max_action_count": -1,
13-
"min_response_count": 0,
13+
"min_response_count": 1,
14+
"max_response_count": 1,
15+
"min_message_count": 0,
16+
"max_message_count": 0,
1417
"actions": {
1518
"start": [
1619
"check_system"

0 commit comments

Comments
 (0)