Skip to content

Implement batch API with changeset, upsert, and DataFrame integration#129

Open
sagebree wants to merge 24 commits intomainfrom
users/sagebree/batch
Open

Implement batch API with changeset, upsert, and DataFrame integration#129
sagebree wants to merge 24 commits intomainfrom
users/sagebree/batch

Conversation

@sagebree
Copy link
Copy Markdown
Contributor

@sagebree sagebree commented Feb 27, 2026

Summary

  • Adds client.batch namespace -- a deferred-execution batch API that packs multiple
    Dataverse Web API operations into a single POST $batch HTTP request
  • Adds client.batch.dataframe namespace -- pandas DataFrame wrappers for batch operations
  • Adds client.records.upsert() and client.batch.records.upsert() backed by the
    UpsertMultiple bound action with alternate-key support
  • Fixes a bug where alternate key fields were merged into the UpsertMultiple request
    body, causing 400 Bad Request on the create path

Batch API Design

Implements the Batch API Design spec from @sagebree:

Capability How to use Status
Record CRUD (create / update / delete / get) batch.records.* Done
Upsert by alternate key batch.records.upsert(...) Done
Table metadata (create / delete / columns / relationships) batch.tables.* Done
SQL queries batch.query.sql(...) Done
Atomic write groups batch.changeset() Done
Continue past failures batch.execute(continue_on_error=True) Done
DataFrame integration batch.dataframe.create/update/delete Done (new)

Design constraints enforced:

  • Maximum 1000 operations per batch (validated before sending)
  • records.get paginated overload not supported -- single-record only
  • GET operations cannot be placed inside a changeset (enforced by API design)
  • Content-ID references are only valid within the same changeset
  • File upload operations not batchable
  • tables.create returns no table metadata on success (HTTP 204)
  • tables.add_columns / tables.remove_columns do not flush the picklist cache
  • client.flush_cache() not supported in batch (client-side operation)

What's included

New: client.batch API

  • batch.records.create / get / update / delete / upsert
  • batch.tables.create / get / list / add_columns / remove_columns / delete
  • batch.tables.list(filter=..., select=...) -- parity with client.tables.list() from Add filter and select parameters to client.tables.list() #112
  • batch.tables.create_one_to_many_relationship / create_many_to_many_relationship / delete_relationship / get_relationship / create_lookup_field
  • batch.query.sql
  • batch.changeset() context manager for transactional (all-or-nothing) operations
  • Content-ID reference chaining inside changesets (globally unique across all changesets via shared counter)
  • execute(continue_on_error=True) for mixed success/failure batches
  • BatchResult with .responses, .succeeded, .failed, .created_ids, .has_errors

New: client.batch.dataframe API

  • batch.dataframe.create(table, df) -- DataFrame rows to CreateMultiple batch item
  • batch.dataframe.update(table, df, id_column) -- DataFrame rows to update batch items
  • batch.dataframe.delete(table, ids_series) -- pandas Series to delete batch items

Existing: Refactored existing APIs

  • Payload generation shared between batch and direct API via _build_* / _RawRequest pattern
  • Execution of batch operations deferred to execute()

OData $batch spec compliance

  • Audited against Microsoft Learn docs
  • Content-Transfer-Encoding: binary per part
  • Content-Type: application/http per part
  • Content-Type: application/json; type=entry for POST/PATCH bodies
  • CRLF line endings throughout
  • Absolute URLs in batch parts
  • Empty changesets silently skipped (prevents invalid multipart)
  • Top-level batch error handling (non-multipart 4xx/5xx raises HttpError with parsed Dataverse error details)
  • Accepts 200, 202 Accepted, 207 Multi-Status, and 400 batch response codes

Review comment fixes

  • Fixed expected status codes to include 202/207 for all Dataverse environments
  • Fixed _split_multipart / _parse_mime_part return type annotations: List[Tuple[Dict[str, str], str]]
  • Fixed OptionSet string check regression: now uses dict key lookup instead of JSON string search
  • Fixed _build_get to lowercase select column names (consistency with _get_multiple)
  • Added RFC 3986 %20 encoding documentation in _build_sql docstring
  • Fixed content-id response parsing for non-changeset parts
  • Fixed test assertions after merge: data bytes instead of json kwarg
  • Exception type parity: batch.records.upsert() raises TypeError (matching client.records.upsert())

Testing

Unit tests -- 579 tests passing:

  • test_batch_operations.py -- BatchRequest, BatchRecordOperations, BatchTableOperations, BatchQueryOperations, ChangeSet, BatchItemResponse, BatchResult
  • test_batch_serialization.py -- multipart serialization, response parsing, intent resolution, upsert dispatch, batch size limit, content-ID uniqueness, top-level error handling
  • test_batch_edge_cases.py -- 40 edge case tests: empty changeset, changeset rollback, content-ID in standalone parts, mixed batch, multiple changesets, batch size limits, top-level errors, continue-on-error, serialization compliance, multipart parsing, content-ID references, intent validation
  • test_batch_dataframe.py -- 18 tests: DataFrame create/update/delete, validation, NaN handling, empty series, bulk delete
  • test_odata_internal.py -- _build_upsert_multiple body exclusion, conflict detection, URL/method correctness

E2E tests -- 14 tests passing against live Dataverse (crm10.dynamics.com):

  1. Basic batch CRUD (single create + CreateMultiple, update, get, delete)
  2. Changeset happy path (create + update via $ref content-ID)
  3. Changeset rollback (failing op rolls back entire changeset)
  4. Multiple changesets (globally unique content-IDs)
  5. Continue-on-error (mixed success/failure)
  6. Batch SQL query
  7. Batch tables.get + tables.list
  8. DataFrame batch create
  9. DataFrame batch update
  10. DataFrame batch delete
  11. Mixed batch (changeset + standalone GET)
  12. Empty changeset (silently skipped)
  13. Content-ID chaining (2 creates + 2 updates via $ref)
  14. Table setup/teardown

Examples & docs

  • examples/advanced/batch.py -- reference examples for all batch operation types
  • examples/advanced/walkthrough.py -- batch section added (section 11)
  • examples/basic/functional_testing.py -- test_batch_all_operations() covering all operation categories against a live environment

@sagebree sagebree requested a review from a team as a code owner February 27, 2026 18:53
Copilot AI review requested due to automatic review settings February 27, 2026 18:53
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements a deferred-execution $batch API (client.batch) for Dataverse Web API operations, including transactional changesets and upsert support via UpsertMultiple, while refactoring OData request construction into reusable _build_* helpers.

Changes:

  • Added client.batch namespace with BatchRequest builder, record/table/query batch operation namespaces, and changeset() transactional grouping.
  • Introduced internal batch intent types + multipart serializer/parser, plus public BatchResult/BatchItemResponse models.
  • Refactored _ODataClient to build requests via _build_* returning _RawRequest, enabling shared payload generation for direct and batch execution (including the UpsertMultiple alternate-key body fix).

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/PowerPlatform/Dataverse/client.py Exposes new client.batch namespace.
src/PowerPlatform/Dataverse/operations/batch.py Adds public batch builder + operation namespaces + changeset API.
src/PowerPlatform/Dataverse/models/batch.py Adds public result models for batch responses.
src/PowerPlatform/Dataverse/data/_raw_request.py Introduces internal _RawRequest dataclass for deferred request construction.
src/PowerPlatform/Dataverse/data/_batch.py Implements batch intent resolution, multipart serialization, and response parsing.
src/PowerPlatform/Dataverse/data/_odata.py Refactors CRUD/metadata/query to use _build_* + _execute_raw; adds _build_upsert_multiple fix and _build_sql encoding.
tests/unit/test_batch_operations.py Unit tests for client.batch API surface and result models.
tests/unit/data/test_batch_serialization.py Unit tests for multipart serialization and response parsing, plus intent resolution routing.
tests/unit/data/test_odata_internal.py Adds tests for _build_upsert_multiple payload rules and conflict detection.
tests/unit/data/test_sql_parse.py Adds tests for _build_sql URL encoding/round-tripping.
tests/unit/data/test_format_key.py Adds tests for _format_key behavior.
README.md Documents batch feature and examples.
examples/advanced/batch.py Adds a full batch usage example script.
examples/advanced/walkthrough.py Adds a batch section to the walkthrough.
examples/basic/functional_testing.py Adds a live-environment batch functional test routine.
src/PowerPlatform/Dataverse/claude_skill/dataverse-sdk-use/SKILL.md Updates usage skill docs with batch API.
src/PowerPlatform/Dataverse/claude_skill/dataverse-sdk-dev/SKILL.md Updates dev skill docs to include new namespace/module.
.claude/skills/dataverse-sdk-use/SKILL.md Mirrors usage skill docs update.
.claude/skills/dataverse-sdk-dev/SKILL.md Mirrors dev skill docs update.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@sagebree
Copy link
Copy Markdown
Contributor Author

Batch API Design:

Batch API Design

Dataverse Python SDK — client.batch namespace

Overview

The batch API lets callers bundle multiple Dataverse operations into a single HTTP POST to $batch. This reduces round-trips, improves throughput for bulk workflows, and enables atomic execution of write groups via changesets.

The public surface mirrors the existing client.records, client.tables, and client.query namespaces exactly — same method names, same signatures. Developers can copy-paste existing SDK calls, prepend batch., and call execute() with no other changes.

Zero-learning-curve design: batch.records.create("account", {...}) accepts the same arguments as client.records.create("account", {...}). The only difference: batch methods return None immediately; results are available via BatchResult after execute().
Capability | How to use -- | -- Record CRUD (create / update / delete / get) | batch.records.* Upsert by alternate key | batch.records.upsert(...) Table metadata (create / delete / columns / relationships) | batch.tables.* SQL queries | batch.query.sql(...) Atomic write groups | batch.changeset() Continue past failures | batch.execute(continue_on_error=True)

Pre-resolution metadata GETs and the final POST $batch all share the same x-ms-correlation-id. The operations bundled inside the multipart body are not separate HTTP requests and receive no SDK-level tracking IDs.

batch.execute() — one correlation scope

All HTTP calls within a single execute() share the same x-ms-correlation-id.

GETEntityDefinitions — MetadataId pre-resolutioncorrelation-id: sharedclient-request-id: unique-A
GETEntityDefinitions — MetadataId pre-resolutioncorrelation-id: sharedclient-request-id: unique-B
POST/$batch — the entire batch payloadcorrelation-id: sharedclient-request-id: unique-C
└─ multipart body
    ├─GETaccounts(guid)← bundled part, no SDK tracking ID
    ├─POSTaccounts← bundled part, no SDK tracking ID
    └─DELcontacts(guid)← bundled part, no SDK tracking ID

Limitations

  • Maximum 1000 operations per batch (validated before sending; counts HTTP requests, not SDK calls)
  • records.get paginated overload is not supported — only the single-record form
  • GET operations cannot be placed inside a changeset
  • Content-ID references are only valid within the same changeset
  • File upload operations (upload_file) are not batchable
  • tables.create returns no table metadata on success (HTTP 204)
  • tables.add_columns and tables.remove_columns do not flush the picklist cache after execution
  • client.flush_cache() is not supported in batch (client-side operation)

Examples

Record CRUD in a single batch

batch = client.batch.new()

# Single create
batch.records.create("account", {"name": "Contoso Ltd", "telephone1": "555-0100"})

# Bulk create — one batch item via CreateMultiple
batch.records.create("contact", [
{"firstname": "Alice", "lastname": "Smith"},
{"firstname": "Bob", "lastname": "Jones"},
])

# Update
batch.records.update("account", account_id, {"telephone1": "555-9999"})

# Broadcast update (same changes applied to all IDs)
batch.records.update("account", [id1, id2, id3], {"statecode": 0})

# Get — response in BatchItemResponse.data
batch.records.get("account", account_id, select=["name", "telephone1"])

# Delete
batch.records.delete("account", old_id)

result = batch.execute()
print(f"OK: {len(result.succeeded)}, ERR: {len(result.failed)}")
for guid in result.created_ids:
print(f"Created: {guid}")

Transactional changeset with content-ID chaining

batch = client.batch.new()

with batch.changeset() as cs:
lead_ref = cs.records.create("lead", {"firstname": "Ada", "lastname": "Lovelace"})
contact_ref = cs.records.create("contact", {"firstname": "Ada"})

cs.records.<span class="fn" style="box-sizing: border-box; margin: 0px; padding: 0px; color: rgb(137, 180, 250);">create</span>(<span class="st" style="box-sizing: border-box; margin: 0px; padding: 0px; color: rgb(166, 227, 161);">"account"</span>, {
    <span class="st" style="box-sizing: border-box; margin: 0px; padding: 0px; color: rgb(166, 227, 161);">"name"</span>: <span class="st" style="box-sizing: border-box; margin: 0px; padding: 0px; color: rgb(166, 227, 161);">"Babbage &amp; Co."</span>,
    <span class="st" style="box-sizing: border-box; margin: 0px; padding: 0px; color: rgb(166, 227, 161);">"[email protected]"</span>:   lead_ref,
    <span class="st" style="box-sizing: border-box; margin: 0px; padding: 0px; color: rgb(166, 227, 161);">"[email protected]"</span>: contact_ref,
})

cs.records.<span class="fn" style="box-sizing: border-box; margin: 0px; padding: 0px; color: rgb(137, 180, 250);">update</span>(<span class="st" style="box-sizing: border-box; margin: 0px; padding: 0px; color: rgb(166, 227, 161);">"contact"</span>, contact_ref, {<span class="st" style="box-sizing: border-box; margin: 0px; padding: 0px; color: rgb(166, 227, 161);">"lastname"</span>: <span class="st" style="box-sizing: border-box; margin: 0px; padding: 0px; color: rgb(166, 227, 161);">"Lovelace"</span>})

result = batch.execute()

if result.has_errors:
print("Changeset rolled back")
else:
print(f"{len(result.created_ids)} records created atomically")

Upsert by alternate key

from PowerPlatform.Dataverse.models.upsert import UpsertItem

batch = client.batch.new()

# Single item → PATCH entity_set(accountnumber='ACC-001')
batch.records.upsert("account", [
UpsertItem(
alternate_key={"accountnumber": "ACC-001"},
record={"name": "Contoso Ltd", "telephone1": "555-0100"},
)
])

# Multiple items → POST entity_set/UpsertMultiple (one batch item)
batch.records.upsert("account", [
UpsertItem(alternate_key={"accountnumber": "ACC-001"}, record={"name": "Contoso Ltd"}),
UpsertItem(alternate_key={"accountnumber": "ACC-002"}, record={"name": "Fabrikam Inc"}),
])

# Plain dicts also accepted
batch.records.upsert("account", [
{"alternate_key": {"accountnumber": "ACC-003"}, "record": {"name": "Woodgrove"}},
])

result = batch.execute()

Upsert uses PATCH without an If-Match header — the server creates the record if it does not exist, or updates it if it does. This differs from records.update, which includes If-Match: * and fails if the record is absent.

Table metadata operations

batch = client.batch.new()

# Create a table
batch.tables.create(
"new_Product",
{"new_Price": "decimal", "new_InStock": "bool"},
solution="MySolution",
)

# Add columns to an existing table (MetadataId resolved transparently)
batch.tables.add_columns("new_Order", {"new_ShipDate": "datetime", "new_Notes": "string"})

# Read table metadata
batch.tables.get("new_Product")

result = batch.execute()

# add_columns for 2 columns → 2 entries in result.responses
print([(r.status_code, r.is_success) for r in result.responses])

Error handling

from PowerPlatform.Dataverse.core.errors import HttpError, ValidationError, MetadataError

batch = client.batch.new()
batch.records.create("account", {"name": "Test"})
batch.records.delete("account", some_id)

try:
result = batch.execute(continue_on_error=True)
except ValidationError as e:
# Batch too large, unsupported column type, etc.
print(f"Validation: {e.message}")
except MetadataError as e:
# tables.delete / add_columns / remove_columns — table or column not found
print(f"Metadata: {e.message}")
except HttpError as e:
# Auth failure, server error — entire batch could not execute
print(f"HTTP {e.status_code}: {e.message}")
print(f" correlation_id: {e.details.get('correlation_id')}")
else:
for item in result.failed:
print(f"[ERR] {item.status_code}: {item.error_message} (code={item.error_code})")
for item in result.succeeded:
print(f"[OK] {item.status_code} entity_id={item.entity_id}")

SDK-level errors (ValidationError, MetadataError, HttpError) are raised from execute() before or after the HTTP call respectively. Per-operation failures inside the batch appear as non-2xx entries in result.responses — they are not raised as exceptions.

@sagebree sagebree changed the title implement batch API with upsert, changeset implement batch API including with changeset Feb 27, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

tests/unit/data/test_batch_serialization.py:1

  • This test is in the wrong file. _targets calls self.od._build_upsert_multiple, which is an _ODataClient method. This test class (TestBuildUpsertMultiple) belongs in test_odata_internal.py and is already defined there (lines 344–421). The copy in test_batch_serialization.py is a duplicate that validates the same logic. More importantly, the test's docstring says "it passes through to body" when the alternate key value matches — but looking at _build_upsert_multiple in _odata.py, keys present in both alt_key_lower and record_processed with equal values are not explicitly excluded; however, the code does not explicitly include them either unless they were already in record_processed. The test does not actually assert that accountnumber is present in target (the body), so the description "passes through to body" is not verified. The test should either assert assertIn("accountnumber", target) or be renamed to reflect what is actually checked.
# Copyright (c) Microsoft Corporation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

@saurabhrb saurabhrb self-assigned this Mar 18, 2026
saurabhrb pushed a commit that referenced this pull request Mar 18, 2026
- Add 202/207 to expected batch response status codes
- Fix return type annotations: List[tuple] -> List[Tuple[Dict[str, str], str]]
- Fix OptionSet check: use dict key lookup instead of string search in JSON body
- Lowercase select column names in _build_get for consistency with _get_multiple
- Add select/filter params to batch tables.list (parity with PR #112)
- Update _build_list_entities to accept filter/select parameters
- Add docstring note on RFC 3986 %20 encoding in _build_sql
- Fix merge-related test failures: parse data bytes instead of json kwarg
- Add _TableList dataclass fields for filter/select
@saurabhrb saurabhrb changed the title implement batch API including with changeset Implement batch API with changeset, upsert, and DataFrame integration Mar 18, 2026
@saurabhrb saurabhrb requested a review from Copilot March 18, 2026 23:06
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 6 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

@saurabhrb saurabhrb requested a review from Copilot March 19, 2026 01:24
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 7 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

@saurabhrb saurabhrb requested a review from Copilot March 19, 2026 02:15
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Add unit tests for batch serialization, OData key formatting, SQL parsing, and batch operations

- Implemented unit tests for internal batch multipart serialization and response parsing in `test_batch_serialization.py`.
- Added tests for `_ODataClient._format_key` functionality in `test_format_key.py`.
- Enhanced SQL parsing tests in `test_sql_parse.py` to cover URL encoding scenarios.
- Created comprehensive tests for batch operations, including record and table operations, in `test_batch_operations.py`.
Samson Gebre and others added 20 commits March 20, 2026 14:13
…into the body; add unit tests for _ODataClient._build_upsert_multiple validation
- Add 202/207 to expected batch response status codes
- Fix return type annotations: List[tuple] -> List[Tuple[Dict[str, str], str]]
- Fix OptionSet check: use dict key lookup instead of string search in JSON body
- Lowercase select column names in _build_get for consistency with _get_multiple
- Add select/filter params to batch tables.list (parity with PR #112)
- Update _build_list_entities to accept filter/select parameters
- Add docstring note on RFC 3986 %20 encoding in _build_sql
- Fix merge-related test failures: parse data bytes instead of json kwarg
- Add _TableList dataclass fields for filter/select
… fixes

Spec compliance and correctness:
- Skip empty changesets in _resolve_all instead of producing invalid multipart
- Extract content-id from non-changeset response parts (was passing None)

Edge case tests (40 new tests in test_batch_edge_cases.py):
- Empty changeset handling (skipped silently)
- Changeset error/rollback response parsing
- Content-ID in standalone and changeset response parts
- Mixed batch: changeset writes + standalone GETs
- Multiple changesets with globally unique content IDs
- Batch size limit counting across changesets
- Top-level batch error handling (JSON, non-JSON, empty body)
- Batch without continue-on-error (first failure stops)
- Batch with continue-on-error (mixed success/failure)
- OData multipart serialization compliance (CRLF, boundaries, headers)
- BatchResult computed properties edge cases
- Multipart response parsing edge cases (REQ_ID header, GUID formats)
- Content-ID reference format and usage ( in @odata.bind, update, delete)
- Intent validation for unknown types
- Batch boundary format validation

DataFrame + Batch integration:
- New BatchDataFrameOperations class (batch.dataframe namespace)
- batch.dataframe.create(table, df) -- DataFrame rows to CreateMultiple
- batch.dataframe.update(table, df, id_column) -- DataFrame to updates
- batch.dataframe.delete(table, ids_series) -- pandas Series to deletes
- 18 new tests in test_batch_dataframe.py covering all operations

Total: 579 tests passing (58 new tests added)
BatchResult.created_ids now extracts GUIDs from two sources:
- entity_id from OData-EntityId header (individual POST creates)
- data['Ids'] array from CreateMultiple/UpsertMultiple action responses

Previously, bulk creates via CreateMultiple returned 200 OK with
{'Ids': [...]} in the body but created_ids only looked at the
OData-EntityId header (which is absent for action responses).

Added 7 new unit tests covering:
- CreateMultiple response body parsing
- Mixed single + bulk creates in one batch
- Non-string ID filtering
- Failed CreateMultiple exclusion
- Full multipart response simulation
- Add DataFrame integration section to README batch docs
- Add Example 7 (DataFrame batch operations) to examples/advanced/batch.py
Design improvement: created_ids now ONLY returns entity_id values from
OData-EntityId response headers (the standard OData response mechanism).
It no longer auto-extracts IDs from CreateMultiple/UpsertMultiple response
body data['Ids'].

Rationale: A batch can contain heterogeneous operations (creates, updates,
deletes, queries, table ops). Mixing two different response formats
(OData-EntityId header vs action body) into one property was non-standard
and could mislead callers. The OData Web API pattern is for callers to
iterate result.responses and handle each response by type.

For CreateMultiple/UpsertMultiple bulk IDs, callers access them via:
    for resp in result.succeeded:
        if resp.data and 'Ids' in resp.data:
            bulk_ids = resp.data['Ids']

This aligns with the .NET SDK batch response model (BatchResponse returns
raw HttpResponseMessages for caller iteration) and follows OData spec
conventions.

Updated tests to verify the correct access pattern.
- Use VALIDATION_SQL_EMPTY constant instead of string literal in batch sql()
- Use VALIDATION_UNSUPPORTED_COLUMN_TYPE constant in all 3 _odata.py locations
- Import the constant from _error_codes.py
- Add input validation to _build_create_multiple (all items must be dicts)
- Narrow test assertions from Exception to ValidationError in test_odata_internal
The OData-EntityId header is returned by both POST (create) and PATCH
(update) operations, not just creates. The name created_ids was misleading.

entity_ids accurately reflects that it collects GUIDs from the standard
OData-EntityId response header across all successful operations that
return it (creates and updates). GET and DELETE operations do not return
this header.

For CreateMultiple/UpsertMultiple bulk action responses, callers access
IDs via response.data['Ids'] as documented in the property's docstring.

Updated all references across src, tests, examples, README, and SKILL
files.
New test_batch_scenarios.py (20 tests) - executable documentation covering:
- Response ordering matches operation order
- CreateMultiple IDs in data['Ids'], not entity_ids
- Update responses also return entity_id
- GET response: data, not entity_id
- DELETE response: no data, no entity_id
- SQL query result rows in data['value']
- Empty batch returns empty result (no HTTP call)
- Double execute is safe (sends two requests)
- Content-ID scope: only within same changeset
- add_columns: N columns = N responses
- tables.create returns 204, no metadata
- continue_on_error: without vs with
- Changeset rollback error shape
- DataFrame create: IDs in data['Ids']
- Mixed batch: changeset responses then standalone
- Individual response status checking pattern
- Batch max size validation (pre-flight)
- Error response field availability

Updated examples/advanced/batch.py:
- Example 8: Response data patterns -- shows how to handle each
  response type (single create, bulk create, query, delete)
- Remove unused imports in test_batch_scenarios.py (json, _RecordCreate,
  _RecordDelete, _RecordUpdate, _QuerySql, _CRLF, BatchRequest)
- Remove unused imports in test_batch_dataframe.py (patch, BatchRecordOperations)
- Remove unused import in test_batch_edge_cases.py (patch)
- Fix BatchRequest._items type annotation: list -> List[Any]
- Fix example GUID: nonexistent-guid -> 00000000-0000-0000-0000-000000000000
- Fix SKILL.md docs: entity_ids includes creates AND updates, not just creates
7 new tests in TestRobustnessEdgeCases covering:
- Malformed JSON body in batch response (silently handled)
- Truncated JSON body (silently handled)
- Exception in changeset context manager (changeset still in items)
- Empty string table name (accepted, validated downstream)
- Single-quote escaping in OData filter (_escape_odata_quotes)
- Non-dict JSON body (list) in response (data stays None)
- Boundary strings with special characters (+, /)

Verified by systematic audit:
- All user values in OData filters use _escape_odata_quotes
- No bare except in batch code (only except Exception with fallback)
- No mutable default arguments
- json.dumps handles serialization safely
- requests library auto-encodes URLs for non-batch path
- Fix pip install command in batch example (PowerPlatform-Dataverse-Client)
- Hoist entity_set call above if-check in _resolve_record_create
- Extract _require_entity_metadata helper for duplicate table lookups
- Add inline comment explaining 400 in expected status codes
- Add annotation explaining why example 5 is commented out
- Populate __all__ in operations/batch.py with public classes
- Use _GUID_RE from _odata.py for consistent GUID regex
- Add _resolve_record_update tests (single, multiple, invalid changes)
- Add delete tests (multiple IDs, no-bulk, empty list, empty strings)
@saurabhrb saurabhrb force-pushed the users/sagebree/batch branch from 33a45bc to be47e9e Compare March 20, 2026 21:18
…ne regex

- Remove unused imports in test_batch_edge_cases.py (_RecordCreate, _RecordUpdate, _QuerySql, _TableList)
- Remove unused BatchResult import in test_batch_serialization.py
- Remove dead _mock_batch_response function in test_batch_scenarios.py
- Compile boundary regex to module-level _BOUNDARY_RE constant in _batch.py
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Saurabh Badenkal added 2 commits March 20, 2026 14:57
…ext manager

- Add :type: directives to all batch docstrings (Microsoft Learn compatibility)
- Fix _resolve_record_update: raise TypeError (not ValidationError) to match
  RecordOperations.update() convention
- Add :raises: directive to BatchQueryOperations.sql()
- Fix :type sql: format from double backticks to :class:\str\
- Use context manager in batch example (with DataverseClient(...) as client:)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants