…a digests

Problem
The ID model had accreted overlapping digest concepts:
- call digest (recipe identity)
- custom digest override (isCustomDigest / WithDigest)
- contentDigest field
- additional digest strings

That mix made it hard to reason about what was identity vs metadata, and it complicated the e-graph direction where recipe identity should be canonical and other digests should be attached evidence.

It also encoded distinctions in call DAG nodes that we no longer want as foundational behavior:
calls with the same recipe but different non-recipe digests were represented as distinct call nodes.

Solution
Refactor call IDs so recipe digest is the definitive call identity, and represent non-recipe digests as structured attached metadata.

Specifically:
- Remove Call.isCustomDigest and Call.contentDigest from the protobuf model.
- Replace repeated string additionalDigests with repeated ExtraDigest { digest, label }.
- Keep call digest calculation recipe-only.
- Keep compatibility wrappers (WithContentDigest, WithDigest, ContentDigest) by mapping them onto labeled ExtraDigest entries.
- Canonicalize same-recipe call nodes during DAG gather by merging attached extras/effects rather than creating distinct nodes.

Implementation details
Proto / generated code
- Updated dagql/call/callpbv1/call.proto:
  - deleted fields isCustomDigest/contentDigest
  - added message ExtraDigest { digest, label }
  - replaced additionalDigests with extraDigests
- Regenerated dagql/call/callpbv1/call.pb.go.

ID semantics and APIs (dagql/call/id.go)
- Added call.ExtraDigest struct and ID.ExtraDigests().
- Removed ID.AdditionalDigests() and migrated all users to ExtraDigests().
- Kept compatibility behavior:
  - WithContentDigest(d) adds labeled extra digest label="content"
  - ContentDigest() returns the last "content" labeled digest
  - WithDigest(d)/WithCustomDigest(d) map to label="custom"
- Reworked digest derivation:
  - calcDigest() now uses recipe-only fields (receiver recipe digest, type/field/args/nth/module/view)
  - no custom digest override branch in apply()
  - literal ID hashing in recipe digest paths uses referenced ID.Digest() (recipe)
- Kept e-graph term input behavior using inputDigest() to include attached extra evidence where intended for equivalence modeling.

DAG canonicalization
- ID.gatherCalls now treats recipe digest as canonical node key.
- When a node already exists for that recipe, it merges:
  - effect IDs (dedup)
  - extra digests (dedup + deterministic normalization)
- Added helper functions for clone/append/remove/merge/normalize of ExtraDigest collections.
- Added explicit note that mergeExtraDigests is currently O(n^2) and should be revisited if lists grow.

Cache/egraph callsite migration
- Replaced legacy AdditionalDigests() usage with ExtraDigests() across dagql cache/egraph paths.
- Preserved existing digest-value matching behavior by deduping digest strings where those paths use digest-only keys.

Tests
- Updated dagql tests to match new canonical model:
  - same recipe + different attached extra digests now share recipe digest and merge into one call node in serialized DAG
- Updated cache tests affected by digest-model changes.
- Ran and passed:
  - go test ./dagql/call/... -count=1
  - go test ./dagql -count=1
  - go test ./core/schema -count=1
  - go test ./core -count=1
  - go test -race ./dagql -run 'TestCacheEgraphAliasesContentHitIDsForDownstreamTerms|TestCacheSecondaryIndexesCleanedOnRelease|TestIDAdditionalDigestsMergeOnSameRecipeCallInDAG|TestIDWithContentDigestAddsKnownDigest' -count=1

Result
This commit establishes recipe digest as foundational call identity and moves non-recipe digests into structured, labeled metadata, creating a cleaner base for the next cache/e-graph integration steps.

Signed-off-by: Erik Sipsma <[email protected]>

Problem

Cache scoping policies (per-client, per-session, per-call, per-schema, etc.)
were implemented by rewriting call digests via GetCacheConfig handlers. That
creates a split identity model where recipe identity is no longer the sole
call identity source, makes cache behavior harder to reason about, and adds
special-case coupling between cache policy and digest mutation.

As we move toward egraph-style equivalence, we want recipe digest semantics to
stay consistent: contextual scoping should be represented as inputs, not as
post-hoc digest rewriting.

Solution

Add first-class implicit call inputs and model generic cache-scoping policies
as implicit inputs. This makes scoped behavior part of recipe identity and
eliminates digest-rewrite logic for generic policies.

The implementation is intentionally staged:
- Generic policy callsites were migrated to implicit inputs.
- Custom cache-key callsites (e.g. container.from/withExec/cacheVolume) are
  intentionally left on existing GetCacheConfig logic for follow-up work.

Implementation details

1) Call/ID model
- Added `repeated Argument implicitInputs = 15` to `Call` proto.
- Regenerated protobuf bindings.
- Extended `call.ID` to track implicit inputs and expose:
  - `ImplicitInputs()`
  - `WithImplicitInputs(...*Argument)`
- Included implicit inputs in all identity/traversal surfaces where they must
  participate:
  - recipe digest calculation (`calcDigest`)
  - self digest decomposition (`SelfDigestAndInputs`)
  - input traversal (`Inputs`)
  - module/effect traversal (`Modules`, `AllEffectIDs`)
  - gather/decode/clone/proto paths
- Kept human-readable display output unchanged (implicit inputs are not shown).

2) dagql field plumbing
- Added field-level implicit input support:
  - `FieldSpec.ImplicitInputs []ImplicitInput`
  - `ImplicitInputResolver`
  - `Field.WithInput(...)`
- In `preselect`:
  - resolve implicit inputs from context + resolved explicit args
  - build deterministic argument list (sorted by name)
  - attach with `call.WithImplicitInputs(...)`
- After `GetCacheConfig` returns (and may rewrite ID args), decode final args
  from the returned ID and recompute implicit inputs so execution/cache identity
  stay aligned with the final call ID.

3) Cache policy helpers
- Replaced generic digest-rewrite helpers in `dagql/cachekey.go` with implicit
  input providers:
  - `CachePerClient`
  - `CachePerSession`
  - `CachePerCall`
  - `CachePerSchema`
  - `CacheAsRequested(argName)`
- Added bool argument decoding support for `Boolean`, `Optional[Boolean]`, and
  `DynamicOptional` in `CacheAsRequested`.

4) Callsite migration (generic policies)
- Migrated generic policy callsites from `*FuncWithCacheKey(..., dagql.Cache*)`
  to `.WithInput(dagql.Cache*)` across schema/introspection paths, including
  host/address no-cache toggles via `CacheAsRequested("noCache")` and combined
  client+schema policy via `.WithInput(dagql.CachePerClient,
  dagql.CachePerSchema(srv))`.
- Left custom cache-key callsites untouched for follow-up:
  - `core/schema/container.go` (`from`, `withExec`)
  - `core/schema/cache.go` (`cacheVolume`)

Tests

Added and updated tests to validate both ID-level and resolver-level behavior:
- `dagql/call/id_test.go`
  - implicit inputs affect digest
  - implicit inputs survive encode/decode round-trip
- `dagql/dagql_test.go`
  - per-client/session/call/schema behaviors via implicit inputs
  - combined client+schema behavior
  - `CacheAsRequested` behavior (explicit and default arg path)
  - implicit input recomputation after cache-config ID rewrite

Validation run

Used explicit package targets (no wildcard integration sweep):
- `go test ./dagql -run 'TestImplicitInput|TestCacheConfigReturnedIDRewritesExecutionArgs' -count=1`
- `go test ./dagql ./dagql/call ./dagql/introspection ./core/schema`

Signed-off-by: Erik Sipsma <[email protected]>

This continues the cache identity migration away from ad-hoc `WithDigest`
rewrites and toward explicit argument-based identity.

Problem
- `cacheVolume` and `container.from` still scoped cache identity by replacing
  call digests with custom hashes.
- That approach makes cache identity less transparent and harder to reason
  about as we move toward recipe-first IDs and egraph-based equivalence.

Solution
- Keep call identity in the ID structure and rewrite cache-relevant arguments
  instead of rewriting the digest itself.

Implementation details
- `core/schema/cache.go`
  - `cacheVolumeCacheKey` now rewrites the internal `namespace` argument on
    `CacheKey.ID` via `WithArgument(...)`.
  - removed `hashutil` usage for this cache key path.

- `core/schema/container.go`
  - added internal `ResolvedAddress` to `containerFromArgs` to carry canonical
    refs for execution while preserving normalized cache identity.
  - added `containerFromEffectiveAddress(args)` helper to choose
    `ResolvedAddress` when present.
  - `fromCacheKey` now:
    - parses and normalizes the effective address,
    - when canonical, rewrites ID args to:
      - `address = "digest:<sha256...>"`
      - `resolvedAddress = "<canonical ref>"`
    - does not use `WithDigest`.
  - `from(...)` now parses from `containerFromEffectiveAddress(args)` so
    execution uses canonical refs when provided.

- `skills/cache-expert/references/debugging.md`
  - expanded focused test-run guidance for `engine-dev test`.
  - documented `/tmp` log redirection and panic scanning.
  - added an explicit inner-engine `SIGQUIT` workflow to dump goroutines by
    locating the non-pid1 `/usr/local/bin/dagger-engine` process in `/proc`.

Validation
- `go test ./core/schema ./dagql`
- `dagger --progress=plain call engine-dev test --pkg ./core/integration --run='^TestContainer/TestFrom$'`

Signed-off-by: Erik Sipsma <[email protected]>

Problem
- `Container.withExec` used a custom cache-key digest rewrite (`withExecCacheKey`) that
  mixed parent digest with a manually computed args digest.
- This bypassed normal ID/recipe identity flow and made cache behavior harder to reason
  about while migrating toward recipe+implicit-input based identity.
- `execMD` carried runtime metadata that should not directly shape structural recipe identity,
  except for explicit cache partitioning via `CacheMixin`.

Solution
- Remove `withExec` custom digest rewrite and model cache identity directly through call ID
  + implicit inputs.
- Keep `execMD` runtime-only by marking it sensitive so it is excluded from encoded ID args.
- Introduce `execCacheMixin` implicit input for `withExec` that contributes only
  `execMD.CacheMixin` to the call identity.

Implementation
- Switched `withExec` from `NodeFuncWithCacheKey` to `NodeFunc(...).WithInput(...)`.
- Added `withExecCacheMixinInput` implicit input resolver:
  - reads `execMD` from decoded inputs,
  - returns empty string when absent,
  - returns `execMD.CacheMixin.String()` when present.
- Marked `containerExecArgs.ExecMD` as `sensitive:"true"` and documented intent.
- Removed `withExecCacheKey` entirely.

Validation
- `go test ./core/schema ./dagql`
- `dagger --progress=plain call engine-dev test --pkg ./core/integration --run='^TestContainer/TestFrom$'`
- `dagger --progress=plain call engine-dev test --pkg ./core/integration --run='^TestContainer/TestWithUnixSocket$'`

Signed-off-by: Erik Sipsma <[email protected]>

…from

Problem

The recent ID/equivalence work exposed two conflicting behaviors in cache identity: tag-based container.from calls should resolve once per session, while digest-addressed results should converge on shared content identity. At the same time, several dag-op paths still derived cache identity from ad-hoc digest rewriting, which leaked caller-specific scoping into places where we want equivalence-based reuse.

Solution

Rework ID digest helpers and dag-op keying so cache identity follows equivalent identity rules, and move container.from scoping to an implicit input instead of custom cache-key rewriting.

Implementation details

- dagql/call/id.go: add EquivalentDigest, SelfDigestAndEquivalentInputs, and equivalent-literal hashing helpers; add CustomDigest accessor; keep equivalent identity preference as content > custom > self+equivalent inputs.

- core/schema/wrapper.go: centralize dag-op ID construction in dagOpIDWithCustomDigest so dag-op internal args are included while digest derivation uses equivalent inputs rather than raw rewritten digests.

- core/dagop.go: switch dag-op CacheMap/Digest and stored ref metadata to helpers that prefer custom digest for cache-key scoping, and content/custom/equivalent digest for value identity.

- core/schema/container.go: remove fromCacheKey and resolvedAddress rewrite flow; add fromSessionScope implicit input that returns session ID for mutable tag refs and empty value for canonical digest refs; preserve recursive canonical re-invoke pattern; set from result content digest from container.from + resolved image digest + platform.

- dagql/cache.go: when returning equivalent/content hits, preserve the caller's recipe ID while copying non-custom extra digests and content digest from the cached constructor so downstream calls still benefit from known equivalences without leaking custom scope digests.

- skills/cache-expert/references/debugging.md: document that ./dagql/idtui and ./dagql/idtui/multiprefixw are integration-style tests and should be avoided in fast unit/debug loops.

Signed-off-by: Erik Sipsma <[email protected]>

… keys

This commit tightens cache-hit reuse rules and normalizes cache identity for
module-related calls so equivalent hits stay correct under session/client
boundaries and nested runtime/module loading.

Problem

The current WIP egraph/known-digest integration could return equivalent hits
that were digest-compatible but payload-incompatible:

- Container payload reuse across exact hits could share mutable buildkit refs,
  leading to finalize/snapshot race behavior.
- Equivalent hits across different call paths could leak contextual/provenance
  payloads (notably Module/ModuleSource/object-constructor payloads).
- Module/runtime calls still had mixed custom-digest vs recipe-digest behavior,
  making cache identity unstable across clients/sessions.
- Some module/runtime paths surfaced only opaque "exit code" errors and made
  root causes harder to diagnose.

Solution

1) Add explicit cache-reuse policy checks for exact and equivalent hits.
2) Normalize module function cache IDs around recipe structure and normalized
   receiver identity (instead of ad-hoc digest rewrites).
3) Keep legacy custom-digest override semantics where still required.
4) Make module content-digest registration and lookup behavior resilient across
   equivalent/cache-hit paths.
5) Plumb withExec runtime metadata through dagop execution without letting it
   affect cache identity.

Implementation details

- dagql/cache.go
  - Added guard helpers:
    - shouldReuseEquivalentObjectPayload
    - shouldReuseExactObjectPayload
    - shouldReuseEquivalentHitForCall
    - shouldPreserveRequestIDOnEquivalentHit
    - plus small helpers for constructor/type checks and dagop detection.
  - Prevent exact payload reuse for Container results.
  - Prevent equivalent-hit payload reuse for Module/ModuleSource and for
    identity/provenance selectors (`id`, `asString`).
  - Avoid equivalent reuse across path mismatches, except constrained
    Container cases; also reject dagop vs non-dagop container mixing.
  - Build known-digest lookup key list once and use it for both lookup and TTL
    persistence aliases.
  - Persist TTL rows under both the call key and known-digest-derived keys so
    known-digest lookups can resolve to the same storage entry.
  - Always derive the term proto for equivalent lookup (not only when
    storageKey==callKey).

- dagql/cache_egraph.go
  - termProtoForID now ignores module metadata in the self term shape via
    `id.With(call.WithModule(nil)).SelfDigestAndInputs()` to avoid
    client/session module metadata skew in term matching.

- dagql/call/id.go
  - CacheDigest now explicitly prefers CustomDigest when present, preserving
    existing custom-digest cache semantics while those rewrites still exist.

- core/modfunc.go
  - Reworked ModuleFunction cache-key derivation:
    - derive from `SelfDigestAndEquivalentInputs` on a normalized base ID,
      then append normalized receiver digest, module source digest, and module
      name.
    - keep per-session scoping for `FunctionCachePolicyPerSession`.
    - scope constructors (metadata.Name == "") per session to avoid reusing
      stale contextual-default payload IDs.
  - Added normalizedReceiverDigest recursion to normalize receiver identity
    deterministically.
  - loadContextualArg now receives precomputed module content cache key so
    all contextual loads for the call share one module-content identity.
  - Tightened safe-to-persist logic for named secrets: only allow persistence
    when returned top-level IDs are Secret/Socket; disable otherwise.

- core/module.go / core/query.go
  - Module ID generation now attaches content digest via WithContentDigest.
  - Query.IDDeps prefers ContentDigest when available, falling back to Digest.
  - Removed module call custom-digest rewrite in Module.CacheConfigForCall.

- core/schema/modulesource.go / core/schema/module.go
  - Added CachePerClient implicit input to module and moduleSource `sync`.
  - moduleSource.asModule now registers module-by-content-digest both
    immediately and as a post-call hook so equivalent/cache hits re-register
    mapping in the active session.

- core/schema/container.go / core/schema/wrapper.go / core/dagop.go
  - Plumbed withExec `execMD` to dagop execution as runtime-only metadata:
    - `containerExecArgs.ExecMD` marked internal+sensitive.
    - Added DagOpExecutionMetadata interface path in wrapper.
    - ContainerDagOp now optionally carries ExecMD and injects it into the
      internal load ID only when evaluating dagop withExec.
    - ExecMD remains excluded from cache identity derivation.

- core/sdk/module.go
  - Expanded sdk object init errors with module metadata context (object/
    interface/enum counts + result digest) for debugging.

Net effect

Cache hit behavior is stricter where payload identity matters and more stable
where recipe-equivalent identity should converge. Module/runtime cache keying
is more coherent with the implicit-input model and better scoped for
cross-session safety.

Signed-off-by: Erik Sipsma <[email protected]>

Problem
- Query.http still rewrote the call digest via WithDigest, which conflicts with the new ID model where recipe digest is authoritative and extra/content digests carry equivalence knowledge.
- A naive migration to stable returned IDs introduced behavior conflicts: HTTP needed to remain per-client for execution/revalidation, while service hostname stability expected equivalent outputs to converge.

Solution
- Keep Query.http recipe identity intact and attach HTTP output identity as a content digest instead of overriding the recipe digest.
- Keep refID as execution-only transport for dag-op evaluation, but return the caller-facing result ID without using digest rewrite semantics.
- Explicitly prevent equivalent/known-digest short-circuit reuse for Query.http so HTTP execution still follows per-client/session cache scope.
- Derive service hostnames from EquivalentDigest (fallback to recipe digest) so equivalent services converge on stable hostnames across sessions.

Implementation details
- core/schema/http.go
  - replaced WithDigest(hash(...)) with WithContentDigest(hash(...))
  - split execution ID (with refID) from returned result ID
  - preserved dag-op effect propagation on returned ID
- dagql/cache.go
  - added a targeted guard in shouldReuseEquivalentHitForCall for field == "http"
- core/service.go
  - Hostname now hashes id.EquivalentDigest() with fallback to id.Digest()

Validation
- go test ./core -run TestNonExistent -count=1
- go test ./dagql -run TestImplicitInputCachePerClient -count=1
- dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestHTTP/TestHTTPCachePerSessions'
- dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestHTTP/TestHTTPServiceStableDigest'
- dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestHTTP'
- dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestServices/(TestHostnamesAreStable|TestHostnameEndpoint)'

Signed-off-by: Erik Sipsma <[email protected]>

ModuleFunction caching was still centered on recipe override via WithDigest in CacheConfigForCall. That blocked the ID-consolidation direction where recipe digest stays definitive and equivalence is expressed via inputs/extra digests rather than replacing the recipe key.

This change moves module-function cache scoping to field-level implicit inputs and removes the custom-digest cache-key rewrite for module calls.

What changed:

- Added ModuleFunction.cacheImplicitInputs() to define module-function cache scope declaratively.

  - always adds a stable module scope input (module source digest + module name)

  - adds CachePerSession for per-session policy

  - adds CachePerSession for constructors (preserving existing constructor/session behavior)

- Wired these implicit inputs into module function FieldSpecs at install time for both constructors and regular object functions.

- In ModuleFunction.CacheConfigForCall, removed WithDigest(customDigest) rewrite flow.

- Kept canonical receiver normalization, and now attach a canonical module-function equivalent digest as ExtraDigest(label=moduleFunctionEquivalent) for non-constructors so cross-session equivalent lookups still work without recipe override.

Important gotcha fixed:

Constructor cache scoping is now represented by cachePerSession implicit input. That session input was unintentionally leaking into receiver normalization and made downstream function equivalence session-specific. normalizedReceiverDigest now filters cachePerSession implicit inputs before hashing canonical receiver identity.

Tests:

- go test ./dagql -run 'TestImplicitInputRecomputedAfterCacheConfigIDRewrite|TestImplicitInputCachePerSession|TestImplicitInputCachePerClient'

- go test ./core -run 'TestModuleFunctionCacheImplicitInputs|TestModuleFunctionCacheImplicitInputsNilSafe|TestModuleFunctionNormalizedReceiverDigestIgnoresSessionScopeInput'

- dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestModule/TestFunctionCacheControl/go'

- dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestModule/TestCrossSessionFunctionCaching/args'

- dagger --progress=plain call engine-dev test --pkg ./core/integration --run='Test/TestCrossSession'

- dagger --progress=plain call engine-dev test --pkg ./core/integration --run='TestModule'

Signed-off-by: Erik Sipsma <[email protected]>

…oped identity

Problem

The module-source SDK paths (codegen, module init/runtime) were still using
legacy digest override mechanisms (`WithDigest` / `WithObjectDigest`) to scope
cache identity to `ContentScopedDigest()`.

That path no longer matches the desired identity model:
- recipe digest should remain pure call recipe
- scope/equivalence metadata should be attached explicitly
- temporary module-init scoping should not depend on custom-digest rewrite
  behavior

In particular, `runModuleDefInSDK` derived `OverrideStorageKey` from
`tmpModInst.ID().Digest()` after applying `WithDigest(...)`, which is brittle
under the new ID semantics.

Solution

Move module-source scoping to explicit content-scoped IDs and explicit implicit
input scoping for temporary module-init IDs.

1) Use content-scoped source IDs consistently
- Replace `WithObjectDigest(...)` usage with `WithContentDigest(...)` in:
  - `moduleSourceSchema.runCodegen`
  - `moduleSourceSchema.initializeSDKModule`
  - `Module.LoadRuntime`
- Prime cache entries with the same content-scoped source ID that is passed into
  SDK calls.

2) Replace temp module-init custom digest rewrite
- In `runModuleDefInSDK`, stop using `CurrentID().WithDigest(...)`.
- Compute deterministic `moduleInitScope = hash(ContentScopedDigest, "modInit")`.
- Attach that scope as an explicit implicit input (`moduleInitScope`) to build
  the temporary module ID.

3) Decouple function storage-key override from ID digest mutation
- Use `moduleInitScope.String()` directly for `CallOpts.OverrideStorageKey`
  instead of `tmpModInst.ID().Digest().String()`.

Why this is better

- Keeps recipe identity and scoped/equivalence identity concerns separated.
- Makes module-init scoping explicit in ID inputs instead of legacy custom
  digest behavior.
- Preserves the original intent: scope SDK/module-init caches by
  content+provenance, not by incidental caller details.

Validation

Compile checks:
- `go test ./core -run TestNoSuchTest -count=1`
- `go test ./core/schema -run TestNoSuchTest -count=1`
- `go test ./dagql -run TestNoSuchTest -count=1`

Targeted integration:
- `TestModule/TestCustomSDK/module_initialization` (pass)
- `TestModule/TestFunctionCacheControl/go` (pass)
- `Test/TestCrossSession` (pass)

Full `TestModule` run had one `TestDaggerListen/with_mod` failure with
connection-refused/session-not-found symptoms; focused rerun of that exact
subtest passed (appears flaky).

Signed-off-by: Erik Sipsma <[email protected]>

ResourceTransferPostCall primes destination cache entries for transferred\nsecrets so downstream calls can still resolve those secrets if the source\nclient disconnects.\n\nThat path still rewrote the call ID with WithDigest(secretDigest), which\ncreates a custom recipe digest override and conflicts with the current\ndirection where recipe digest remains definitive and equivalence is handled\nvia known/extra digests.\n\nThis change removes the WithDigest(secretDigest) rewrite and keeps only the\ncontent digest annotation via WithContentDigest(secretDigest). That preserves\nthe intended lookup behavior (known-digest/content-based reuse) without\nminting an alternate recipe identity for the same call.\n\nValidated with engine-backed integration runs:\n- TestSecret\n- TestGit

Signed-off-by: Erik Sipsma <[email protected]>

Problem
- We were still conflating two different identities:
  1) operation identity ("is this the same call shape + equivalent inputs?")
  2) output equivalence ("can this output be reused across recipes?")
- DagOp paths depended on custom-digest rewriting in wrapper code to stabilize
  BuildKit-facing identity. That kept the old override model alive and made ID
  semantics harder to reason about as foundational behavior.

Solution
- Introduce explicit digest roles on call IDs:
  - StructuralEquivalentDigest(): call self shape + equivalent input digests.
  - OutputEquivalentDigest(): content digest, else custom digest, else
    structural-equivalent digest.
- Keep EquivalentDigest() as a compatibility alias to
  OutputEquivalentDigest().
- Move DagOp identity/effect keying onto structural-equivalent digests and
  remove wrapper-level custom-digest rewriting.

Implementation details
- dagql/call/id.go
  - Added StructuralEquivalentDigest() and OutputEquivalentDigest().
  - Kept EquivalentDigest() as alias to OutputEquivalentDigest().
  - Updated equivalent-input derivation to use OutputEquivalentDigest() for
    receiver and literal ID inputs.
- dagql/call/id_test.go
  - Added coverage that:
    - structural digest preserves self-shape differences,
    - structural digest matches self+equivalent-input hashing,
    - EquivalentDigest aliases OutputEquivalentDigest.
- core/schema/wrapper.go
  - Replaced dagOpIDWithCustomDigest with dagOpIDWithInternalArgs.
  - Removed wrapper-side WithDigest(...) rewrite.
  - Switched returned DagOp effect IDs from CacheDigest() to
    StructuralEquivalentDigest().
- core/dagop.go
  - Replaced custom/cache-digest helpers with structural-focused helpers.
  - Updated DagOp digest/cache map/content metadata paths to use:
    - structural digest for op identity,
    - content-or-structural digest where output content identity is needed.
- engine/buildkit/op.go
  - NewCustomLLB now uses StructuralEquivalentDigest() for effect ID.
- core/service.go
  - Service hostname derivation now uses OutputEquivalentDigest() explicitly.
- core/modfunc.go
  - Removed legacy WithCustomDigest("") normalization calls.

Validation
- Unit/package checks:
  - go test ./dagql/call -count=1
  - go test ./engine/buildkit -count=1
  - go test ./core/schema -count=1
  - go test ./core -run TestNope -count=1
- Targeted integration slices:
  - TestContainer/TestFileCaching
  - TestContainer/TestImageRef
  - Test/TestCrossSession
  - TestModule/TestFunctionCacheControl/go
- Full TestModule run still showed TestModule/TestDaggerUp/service_random
  timing out (read |0: i/o timeout), including isolated rerun.

Signed-off-by: Erik Sipsma <[email protected]>

Problem
- Several dagql APIs still used legacy wording from the custom-digest override
  era, which made the newer digest model harder to reason about.
- In particular:
  - cache-hook docs implied only custom digest rewriting
  - result helper methods centered on WithDigest/WithObjectDigest naming
  - call.ID exposed a stale custom-only helper surface (including HasCustomDigest)

Solution
- Keep compatibility for existing call sites, but add explicit APIs that match
  the current model (extra digests and legacy custom digests as one labeled
  variant), and update docs accordingly.

Implementation details
- dagql/objects.go
  - Updated FuncWithCacheKey / NodeFuncWithCacheKey comments to describe the
    full cache-config hook behavior (ID rewrites, TTL, do-not-cache,
    concurrency key) rather than just custom digest.

- dagql/cache.go
  - Added Result helpers:
    - WithExtraDigest(call.ExtraDigest)
    - WithAdditionalDigest(digest.Digest)
    - WithLegacyCustomDigest(digest.Digest)
  - Kept Result.WithDigest as a compatibility alias to
    WithLegacyCustomDigest.
  - Added ObjectResult counterparts:
    - WithExtraDigest
    - WithAdditionalDigest
    - WithLegacyCustomDigest
  - Kept ObjectResult.WithObjectDigest as compatibility alias to
    WithLegacyCustomDigest.

- dagql/call/id.go
  - Added explicit ID methods:
    - WithExtraDigest(call.ExtraDigest)
    - WithAdditionalDigest(digest.Digest)
    - WithLegacyCustomDigest(digest.Digest)
  - Kept ID.WithDigest as compatibility alias to WithLegacyCustomDigest.
  - Removed HasCustomDigest (unused and custom-specific surface area).
  - Clarified WithCustomDigest comment as legacy compatibility behavior.

Validation
- go test ./dagql/call -count=1
- go test ./dagql -run 'TestNope|TestIDAdditionalDigestsMergeOnSameRecipeCallInDAG|TestIDWithContentDigestAddsKnownDigest' -count=1
- go test ./core/schema -run TestNope -count=1
- dagger --progress=plain call engine-dev test --pkg ./core/integration --run='Test/TestCrossSession'

Signed-off-by: Erik Sipsma <[email protected]>

This checkpoints the current egraph/cache migration state. The tree is in a
working-but-not-fully-polished phase, and this commit is meant to preserve the
current known-good behavior before the next cleanup/refinement pass.

What is now working well

- Call ID digest model has moved toward the intended direction:
  - extra digests are first-class in call proto/ID handling
  - ID merge behavior now preserves/combines digest facts more consistently
- DagQL cache matching now better reflects equivalence semantics:
  - added execution-mode guardrails so dagop and non-dagop results do not
    cross-match incorrectly
  - improved handling of known digest/equivalence lookup paths
- Function cache-control regression fixed:
  - TTL storage-key mismatch is now allowed for non-persistable results
    (where strict storage-key matching was over-constraining and causing false
    misses)
- Contextual/module-source identity handling improved:
  - contextual loaders now attach content digests so output equivalence is
    stable across sessions where content is the same
- Cross-session socket transfer bug fixed:
  - source ID loads in client resource transfer now carry source query context,
    fixing `load unixSocket(...): no query in context`

Validation run in this checkpoint

- `TestModule/TestCrossSessionSockets` passes after the query-context fix.
- Combined integration slice passes:
  `TestModule/(TestFunctionCacheControl|TestCrossSessionFunctionCaching|TestCrossSessionServices|TestCrossSessionSockets|TestCrossSessionSecrets|TestCrossSessionContextualDirWithPrivate|TestCrossSessionContextualDirChange|TestCrossSessionContextualDirCacheHit|TestCrossSessionGitSockets)`
  (excluding `TestCrossSessionDedupeOfNestedExec` intentionally).

Honest state / cleanup still needed

- `dagql/cache.go` currently contains heavy debug tracing and transitional
  instrumentation that should be reduced once behavior is fully stabilized.
- The cache/egraph path still carries transitional complexity from migration;
  there is cleanup to do to make the matching logic easier to reason about.
- Test surface validated here is strong for module/cache/cross-session paths,
  but this is still a checkpoint commit, not final polish.

This commit is intentionally a large checkpoint so we can continue cleanup and
refinement from a known passing baseline.

Signed-off-by: Erik Sipsma <[email protected]>

Add explicit cache tests that freeze current safety boundaries when
output-equivalence digests overlap:

- implicit-input scope must prevent cross-hit reuse
- isDagOp execution mode must prevent cross-hit reuse

These tests are targeted at transitional matcher behavior in GetOrInitCall
and protect upcoming simplification steps from accidentally widening
cache hit eligibility.

Validation:
- go test ./dagql -run 'TestCacheOutputEquivalenceLookup(RespectsImplicitInputScope|SeparatesDagOpExecutionMode)$'
- go test ./dagql -run '^TestCache'
- go test ./dagql/call

Centralize cache-hit return shaping in a single helper and route all hit
branches through it:

- storage hits
- equivalent (egraph) hits
- known-digest hits
- wait-path success returns

This is a mechanical consolidation step with no intended matching semantic
changes. It removes duplicated handling for nil-result tracking, hit flags,
presentation ID application, and object reconstruction.

Validation:
- go test ./dagql -run '^TestCache'
- go test ./dagql/call

…nction cache hacks

Move cache identity to a single structural model and remove compatibility paths
that depended on ad hoc module-function normalization. The resulting behavior is
"structural recipe + equivalent inputs" throughout lookup, while output digest
facts remain post-exec equivalence evidence used for union/repair only.

Design invariants encoded by this change:
- Canonical lookup key is structural term identity: `(selfDigest, inputEqIDs)`.
- Module identity is an input-layer invariant, not a self-digest mutation.
- `pb.Module` remains metadata; module name/ref/pin do not affect identity.
- Post-exec discovered output digests are used to merge equivalence classes,
  not as separate query-time witness lanes.
- No compatibility special-casing for legacy module-function cache-key quirks.

ID and identity model changes:
- Treat module scope as a synthetic reserved input (`__dagger.module`) in recipe
  and input/equivalence derivation paths.
- Remove module fields from `SelfDigestAndInputs` / `SelfDigestAndEquivalentInputs`
  self-shape bytes; module contributes only via the input lane.
- Keep module metadata attached for transport/introspection semantics only.
- Add coverage proving:
  - module metadata changes do not change identity,
  - module ID changes do change identity,
  - module contributes as exactly one synthetic input digest.

Field identity construction cleanup:
- Introduce `FieldSpec.IdentityOpt(ctx, inputArgs)` as the single way to apply
  field-scoped identity (module + implicit inputs).
- Resolve implicit inputs deterministically (stable name ordering, replacement on
  duplicate names) and propagate resolver errors consistently.
- Use `IdentityOpt` in:
  - normal object preselect ID construction,
  - post-cache-config ID rebuild,
  - core object reconstruction from SDK values,
  - synthetic IDs for container rootfs/directory/file mount helpers.

Module-function cache simplification:
- Remove module-function-specific implicit `moduleFunctionScope` input.
- Remove synthetic module-function equivalent digest synthesis.
- Remove receiver normalization recursion and module-source-specific overrides.
- Keep per-session scoping only when requested by cache policy.
- Update module-function tests to assert absence of legacy scope behavior.

Cache lookup and e-graph simplification:
- Collapse call lookup to one structural term path (`termProtoForID` + e-graph),
  including zero-input calls.
- Remove known-digest side indexes and known-digest lookup branches.
- Remove request aliasing paths that rewrote lookup behavior via side channels.
- Keep term indexing for both request and constructor IDs, and merge output
  equivalence digests into output classes for convergence.
- Make term digest construction explicit and delimiter-safe; preserve input order
  as part of identity.
- Prefer storage-key-matching candidates first, then deterministic lowest-term-id
  fallback.

TTL/session and hit materialization behavior:
- Centralize acceptance in `evaluateCacheLookupCandidate`.
- For TTL lookups, enforce storage-key boundaries unless mismatch reuse is
  explicitly safe (non-persistable same-session reuse or still-valid TTL data).
- Track `sessionID` and `expiration` on shared results to make those decisions
  explicit.
- Keep constructor identity as authoritative on equivalent hits by disabling
  recipe remap in hit presentation.
- Preserve output-equivalence digest facts on presented IDs without rewriting
  incompatible recipe paths.

Dynamic-schema and contextual identity fixes:
- Prefer resolver-attached DAGQL server when loading IDs from SDK results so
  dynamic SDK-as-module schemas resolve in the right server context.
- Add query helper fallback ordering to respect context-local DAGQL server.
- Include contextual argument digest in module source contextual content digests
  (directory/file/git repo/ref) to avoid collisions.
- Use direct content digest for git tree identity instead of ad hoc output-eq
  hashing.

Refactor and test coverage:
- Move arbitrary in-memory cache entry flow into `cache_arbitrary.go`.
- Extend cache tests around structural-only lookup guarantees:
  - no cross-recipe lookup from shared content/additional/aux digests,
  - zero-input structural lookup behavior,
  - exact candidate preference over equivalent candidates,
  - output-eq digest cannot bypass structural lookup,
  - deterministic fallback behavior,
  - downstream reuse after post-exec content union,
  - TTL non-persistable entries do not cross sessions.
- Add targeted identity option tests for determinism and error propagation.

This consolidates cache semantics around the structural model and removes
module-specific cache identity hacks so future cache behavior is derived from the
same explicit invariants end to end.