Skip to content

Commit acc4919

Browse files
eavanvalkenburgCopilot
eavanvalkenburg
and
Copilot
authored
Python: updated integration tests and guidance (#4181)
* updated integration tests and guidance * fixed merge test * updated integration tests * fix: remove duplicate --dist loadfile flag from pytest-xdist config Only one --dist mode can be active at a time; the second value silently overrides the first. Keep --dist worksteal (dynamic load balancing) and remove the redundant --dist loadfile from all workflow files and pyproject.toml configs. Co-authored-by: Copilot <[email protected]> * docs: add keep-in-sync notes for merge and integration test workflows Both python-merge-tests.yml and python-integration-tests.yml share the same parallel job structure. Added sync reminders in workflow file comments, the python-testing SKILL.md, and CODING_STANDARD.md. Co-authored-by: Copilot <[email protected]> * refactor: remove RUN_INTEGRATION_TESTS flag Integration test gating now uses two mechanisms: - `@pytest.mark.integration` for test selection via `-m` filtering - `skip_if_*_disabled` for credential/service availability checks The RUN_INTEGRATION_TESTS env var was redundant since the marker handles selection and the skip decorators already check for actual credentials. Co-authored-by: Copilot <[email protected]> * fix: sync missing env vars from merge-tests to integration-tests Add OPENAI_EMBEDDINGS_MODEL_ID and AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME to python-integration-tests.yml to match python-merge-tests.yml. Co-authored-by: Copilot <[email protected]> * fix: remove remaining RUN_INTEGRATION_TESTS from embedding tests and docs Missed test_openai_embedding_client.py and vector-stores README in the earlier cleanup. Co-authored-by: Copilot <[email protected]> * set functions tests to 3.10 --------- Co-authored-by: Copilot <[email protected]>
1 parent 6305e3e commit acc4919

52 files changed

Lines changed: 718 additions & 199 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/python-integration-tests.yml‎

Lines changed: 163 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
11
#
22
# Dedicated Python integration tests workflow, called from the manual integration test orchestrator.
3-
# Runs all tests (unit + integration).
3+
# Runs all tests (unit + integration) split into parallel jobs by provider.
4+
#
5+
# NOTE: This workflow and python-merge-tests.yml share the same set of parallel
6+
# test jobs. Keep them in sync — when adding, removing, or modifying a job here,
7+
# apply the same change to python-merge-tests.yml.
48
#
59

610
name: python-integration-tests
@@ -19,25 +23,160 @@ permissions:
1923

2024
env:
2125
UV_CACHE_DIR: /tmp/.uv-cache
22-
RUN_INTEGRATION_TESTS: "true"
26+
UV_PYTHON: "3.13"
2327

2428
jobs:
25-
python-tests-core:
26-
name: Python Integration Tests - Core
29+
# Unit tests: all non-integration tests across all packages
30+
python-tests-unit:
31+
name: Python Integration Tests - Unit
32+
runs-on: ubuntu-latest
33+
environment: integration
34+
timeout-minutes: 60
35+
defaults:
36+
run:
37+
working-directory: python
38+
steps:
39+
- uses: actions/checkout@v6
40+
with:
41+
ref: ${{ inputs.checkout-ref }}
42+
persist-credentials: false
43+
- name: Set up python and install the project
44+
id: python-setup
45+
uses: ./.github/actions/python-setup
46+
with:
47+
python-version: ${{ env.UV_PYTHON }}
48+
os: ${{ runner.os }}
49+
- name: Test with pytest (unit tests only)
50+
run: >
51+
uv run poe all-tests
52+
-m "not integration"
53+
-n logical --dist worksteal
54+
--timeout=120 --session-timeout=900 --timeout_method thread
55+
--retries 2 --retry-delay 5
56+
57+
# OpenAI integration tests
58+
python-tests-openai:
59+
name: Python Integration Tests - OpenAI
2760
runs-on: ubuntu-latest
2861
environment: integration
2962
timeout-minutes: 60
3063
env:
31-
UV_PYTHON: "3.10"
3264
OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI__CHATMODELID }}
3365
OPENAI_RESPONSES_MODEL_ID: ${{ vars.OPENAI__RESPONSESMODELID }}
66+
OPENAI_EMBEDDINGS_MODEL_ID: ${{ vars.OPENAI_EMBEDDING_MODEL_ID }}
3467
OPENAI_API_KEY: ${{ secrets.OPENAI__APIKEY }}
68+
defaults:
69+
run:
70+
working-directory: python
71+
steps:
72+
- uses: actions/checkout@v6
73+
with:
74+
ref: ${{ inputs.checkout-ref }}
75+
persist-credentials: false
76+
- name: Set up python and install the project
77+
id: python-setup
78+
uses: ./.github/actions/python-setup
79+
with:
80+
python-version: ${{ env.UV_PYTHON }}
81+
os: ${{ runner.os }}
82+
- name: Test with pytest (OpenAI integration)
83+
run: >
84+
uv run pytest --import-mode=importlib
85+
packages/core/tests/openai
86+
-m integration
87+
-n logical --dist worksteal
88+
--timeout=120 --session-timeout=900 --timeout_method thread
89+
--retries 2 --retry-delay 5
90+
91+
# Azure OpenAI integration tests
92+
python-tests-azure-openai:
93+
name: Python Integration Tests - Azure OpenAI
94+
runs-on: ubuntu-latest
95+
environment: integration
96+
timeout-minutes: 60
97+
env:
98+
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHATDEPLOYMENTNAME }}
99+
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
100+
AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__EMBEDDINGDEPLOYMENTNAME }}
101+
AZURE_OPENAI_ENDPOINT: ${{ vars.AZUREOPENAI__ENDPOINT }}
102+
defaults:
103+
run:
104+
working-directory: python
105+
steps:
106+
- uses: actions/checkout@v6
107+
with:
108+
ref: ${{ inputs.checkout-ref }}
109+
persist-credentials: false
110+
- name: Set up python and install the project
111+
id: python-setup
112+
uses: ./.github/actions/python-setup
113+
with:
114+
python-version: ${{ env.UV_PYTHON }}
115+
os: ${{ runner.os }}
116+
- name: Azure CLI Login
117+
uses: azure/login@v2
118+
with:
119+
client-id: ${{ secrets.AZURE_CLIENT_ID }}
120+
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
121+
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
122+
- name: Test with pytest (Azure OpenAI integration)
123+
run: >
124+
uv run pytest --import-mode=importlib
125+
packages/core/tests/azure
126+
-m integration
127+
-n logical --dist worksteal
128+
--timeout=120 --session-timeout=900 --timeout_method thread
129+
--retries 2 --retry-delay 5
130+
131+
# Misc integration tests (Anthropic, Ollama, MCP)
132+
python-tests-misc-integration:
133+
name: Python Integration Tests - Misc
134+
runs-on: ubuntu-latest
135+
environment: integration
136+
timeout-minutes: 60
137+
env:
35138
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
36139
ANTHROPIC_CHAT_MODEL_ID: ${{ vars.ANTHROPIC_CHAT_MODEL_ID }}
140+
LOCAL_MCP_URL: ${{ vars.LOCAL_MCP__URL }}
141+
defaults:
142+
run:
143+
working-directory: python
144+
steps:
145+
- uses: actions/checkout@v6
146+
with:
147+
ref: ${{ inputs.checkout-ref }}
148+
persist-credentials: false
149+
- name: Set up python and install the project
150+
id: python-setup
151+
uses: ./.github/actions/python-setup
152+
with:
153+
python-version: ${{ env.UV_PYTHON }}
154+
os: ${{ runner.os }}
155+
- name: Test with pytest (Anthropic, Ollama, MCP integration)
156+
run: >
157+
uv run pytest --import-mode=importlib
158+
packages/anthropic/tests
159+
packages/ollama/tests
160+
packages/core/tests/core/test_mcp.py
161+
-m integration
162+
-n logical --dist worksteal
163+
--timeout=120 --session-timeout=900 --timeout_method thread
164+
--retries 2 --retry-delay 5
165+
166+
# Azure Functions + Durable Task integration tests
167+
python-tests-functions:
168+
name: Python Integration Tests - Functions
169+
runs-on: ubuntu-latest
170+
environment: integration
171+
timeout-minutes: 60
172+
env:
173+
UV_PYTHON: "3.10"
174+
OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI__CHATMODELID }}
175+
OPENAI_RESPONSES_MODEL_ID: ${{ vars.OPENAI__RESPONSESMODELID }}
176+
OPENAI_API_KEY: ${{ secrets.OPENAI__APIKEY }}
37177
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHATDEPLOYMENTNAME }}
38178
AZURE_OPENAI_RESPONSES_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__RESPONSESDEPLOYMENTNAME }}
39179
AZURE_OPENAI_ENDPOINT: ${{ vars.AZUREOPENAI__ENDPOINT }}
40-
LOCAL_MCP_URL: ${{ vars.LOCAL_MCP__URL }}
41180
FUNCTIONS_WORKER_RUNTIME: "python"
42181
DURABLE_TASK_SCHEDULER_CONNECTION_STRING: "Endpoint=http://localhost:8080;TaskHub=default;Authentication=None"
43182
AzureWebJobsStorage: "UseDevelopmentStorage=true"
@@ -49,37 +188,38 @@ jobs:
49188
with:
50189
ref: ${{ inputs.checkout-ref }}
51190
persist-credentials: false
52-
53191
- name: Set up python and install the project
54192
id: python-setup
55193
uses: ./.github/actions/python-setup
56194
with:
57-
python-version: "3.10"
195+
python-version: ${{ env.UV_PYTHON }}
58196
os: ${{ runner.os }}
59-
env:
60-
UV_CACHE_DIR: /tmp/.uv-cache
61-
62197
- name: Azure CLI Login
63198
uses: azure/login@v2
64199
with:
65200
client-id: ${{ secrets.AZURE_CLIENT_ID }}
66201
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
67202
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
68-
69203
- name: Set up Azure Functions Integration Test Emulators
70204
uses: ./.github/actions/azure-functions-integration-setup
71205
id: azure-functions-setup
206+
- name: Test with pytest (Functions + Durable Task integration)
207+
run: >
208+
uv run pytest --import-mode=importlib
209+
packages/azurefunctions/tests/integration_tests
210+
packages/durabletask/tests/integration_tests
211+
-m integration
212+
-n logical --dist worksteal
213+
--timeout=120 --session-timeout=900 --timeout_method thread
214+
--retries 2 --retry-delay 5
72215
73-
- name: Test with pytest
74-
run: uv run poe all-tests -n logical --dist loadfile --dist worksteal --timeout=120 --session-timeout=900 --timeout_method thread --retries 2 --retry-delay 5
75-
216+
# Azure AI integration tests
76217
python-tests-azure-ai:
77218
name: Python Integration Tests - Azure AI
78219
runs-on: ubuntu-latest
79220
environment: integration
80221
timeout-minutes: 60
81222
env:
82-
UV_PYTHON: "3.10"
83223
AZURE_AI_PROJECT_ENDPOINT: ${{ secrets.AZUREAI__ENDPOINT }}
84224
AZURE_AI_MODEL_DEPLOYMENT_NAME: ${{ vars.AZUREAI__DEPLOYMENTNAME }}
85225
LOCAL_MCP_URL: ${{ vars.LOCAL_MCP__URL }}
@@ -91,33 +231,32 @@ jobs:
91231
with:
92232
ref: ${{ inputs.checkout-ref }}
93233
persist-credentials: false
94-
95234
- name: Set up python and install the project
96235
id: python-setup
97236
uses: ./.github/actions/python-setup
98237
with:
99-
python-version: "3.10"
238+
python-version: ${{ env.UV_PYTHON }}
100239
os: ${{ runner.os }}
101-
env:
102-
UV_CACHE_DIR: /tmp/.uv-cache
103-
104240
- name: Azure CLI Login
105241
uses: azure/login@v2
106242
with:
107243
client-id: ${{ secrets.AZURE_CLIENT_ID }}
108244
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
109245
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
110-
111246
- name: Test with pytest
112247
timeout-minutes: 15
113-
run: uv run --directory packages/azure-ai poe integration-tests -n logical --dist loadfile --dist worksteal --timeout=120 --session-timeout=900 --timeout_method thread --retries 2 --retry-delay 5
248+
run: uv run --directory packages/azure-ai poe integration-tests -n logical --dist worksteal --timeout=120 --session-timeout=900 --timeout_method thread --retries 2 --retry-delay 5
114249

115250
python-integration-tests-check:
116251
if: always()
117252
runs-on: ubuntu-latest
118253
needs:
119254
[
120-
python-tests-core,
255+
python-tests-unit,
256+
python-tests-openai,
257+
python-tests-azure-openai,
258+
python-tests-misc-integration,
259+
python-tests-functions,
121260
python-tests-azure-ai
122261
]
123262
steps:

0 commit comments

Comments
 (0)