daemon: instrument with more tracing; plumb more contexts by corhere · Pull Request #51967 · moby/moby

corhere · 2026-01-30T04:53:28Z

- What I did

Added tracing spans to the event handlers, callback functions and background goroutines so all the operations related to that operation are correlated into a single trace, like we already have with API endpoint handlers.
Instrumented libnetwork with even more tracing spans to help with debugging.
Plumbed contexts all over the place to propagate trace context all through the call stack.

- How I did it

grepping for context.TODO()
adding context arguments to the Backend interfaces in daemon/server/... and iterating on adding context arguments to functions until it passes go vet got me surprisingly far, quickly.

- How to verify it
Point a daemon at an OTEL ingestion pipeline and see what happens.

- Human readable description for the release notes

- A picture of a cute animal (not mandatory but encouraged)

Add spans to functions at the "root" of an event to logically group spans into a single trace per API call, Swarmkit callback, event handled, or background operation. And trace lots of interesting libnetwork functions to assist in debugging. Signed-off-by: Cory Snider <[email protected]>

Signed-off-by: Cory Snider <[email protected]>

Lots of interesting setup happens during the bridge and overlay driver registration. Rough-in context plumbing for more correlated tracing of driver initialization. Signed-off-by: Cory Snider <[email protected]>

vvoland

Initial pass, my main concerns are around not handling cancellations properly during cleanups

vvoland · 2026-01-30T10:32:41Z

daemon/builder/dockerfile/internals.go

 		return err
 	}
-	defer rwLayer.Release()
+	defer rwLayer.Release(ctx)


This could leak layer if ctx is cancelled.

Suggested change

defer rwLayer.Release(ctx)

defer rwLayer.Release(context.WithoutCancel(ctx))

vvoland · 2026-01-30T10:44:50Z

daemon/containerd/image_builder.go

Suggested change

if err := rl.c.LeasesService().Delete(context.WithoutCancel(ctx), lease); err != nil {

vvoland · 2026-01-30T10:46:49Z

daemon/images/image_builder.go

+	fs, err := newLayer.Mount(ctx, "")
 	if err != nil {
-		rwLayer.Release()
+		rwLayer.Release(ctx)


Suggested change

rwLayer.Release(ctx)

rwLayer.Release(context.WithoutCancel(ctx))

vvoland · 2026-01-30T10:47:18Z

daemon/images/image_commit.go

 	archive, err := rwlayer.TarStream()
 	if err != nil {
-		_ = rwlayer.Unmount()
+		_ = rwlayer.Unmount(ctx)


Suggested change

_ = rwlayer.Unmount(ctx)

_ = rwlayer.Unmount(context.WithoutCancel(ctx))

vvoland · 2026-01-30T10:47:37Z

daemon/images/image_commit.go

 	return ioutils.NewReadCloserWrapper(archive, func() error {
 		_ = archive.Close()
-		err := rwlayer.Unmount()
+		err := rwlayer.Unmount(ctx)


Suggested change

err := rwlayer.Unmount(ctx)

err := rwlayer.Unmount(context.WithoutCancel(ctx))

vvoland · 2026-01-30T10:59:59Z

daemon/archive_windows.go

 		err := data.Close()
 		container.DetachAndUnmount(daemon.LogVolumeEvent)
-		daemon.Unmount(container)
+		daemon.Unmount(ctx, container)


WithoutCancel?

vvoland · 2026-01-30T11:00:12Z

daemon/archive_windows.go

 		return err
 	}
-	defer daemon.Unmount(container)
+	defer daemon.Unmount(ctx, container)


WithoutCancel?

vvoland · 2026-01-30T11:00:16Z

daemon/archive_windows.go

 			container.DetachAndUnmount(daemon.LogVolumeEvent)
 			// unmount the container's rootfs
-			daemon.Unmount(container)
+			daemon.Unmount(ctx, container)


WithoutCancel?

vvoland · 2026-01-30T11:00:21Z

daemon/archive_windows.go

 		err := archv.Close()
 		container.DetachAndUnmount(daemon.LogVolumeEvent)
-		daemon.Unmount(container)
+		daemon.Unmount(ctx, container)


WithoutCancel?

vvoland · 2026-01-30T11:02:36Z

daemon/daemon.go


 // DaemonLeavesCluster informs the daemon has left the cluster
-func (daemon *Daemon) DaemonLeavesCluster() {
+func (daemon *Daemon) DaemonLeavesCluster(ctx context.Context) {


Do we need WithoutCancel for this whole method?

Surprisingly, it is desirable for DaemonLeavesCluster calls to be cancelable. This function is called both when the daemon leaves a Swarm cluster and when a clustered daemon shuts down. The latter runs under a deadline.

moby/daemon/command/daemon.go

Lines 550 to 572 in fabb016

// shutdownDaemon just wraps daemon.Shutdown() to handle a timeout in case

// d.Shutdown() is waiting too long to kill container or worst it's

// blocked there

func shutdownDaemon(ctx context.Context, d *daemon.Daemon) {

var cancel context.CancelFunc

if timeout := d.ShutdownTimeout(); timeout >= 0 {

ctx, cancel = context.WithTimeout(ctx, time.Duration(timeout)*time.Second)

} else {

ctx, cancel = context.WithCancel(ctx)

}

go func() {

defer cancel()

d.Shutdown(ctx)

}()

<-ctx.Done()

if errors.Is(ctx.Err(), context.DeadlineExceeded) {

log.G(ctx).Error("Force shutdown daemon")

} else {

log.G(ctx).Debug("Clean shutdown succeeded")

}

}

Draw the rest of the owl. Wire up contexts from the API handlers, Swarmkit callbacks, event handler callbacks, and background goroutines all the way into libnetwork and the containerd client. Signed-off-by: Cory Snider <[email protected]>

Signed-off-by: Cory Snider <[email protected]>

vvoland · 2026-02-02T11:46:25Z

daemon/export.go

 	}
 	defer func() {
-		if err := rwl.Unmount(); err != nil {
+		if err := rwl.Unmount(ctx); err != nil {


WithoutCancel

vvoland · 2026-02-02T11:47:21Z

daemon/archive_unix.go

 	content = ioutils.NewReadCloserWrapper(data, func() error {
 		err := data.Close()
-		_ = cfs.Close()
+		_ = cfs.Close(ctx)


WithoutCancel

vvoland · 2026-02-02T11:47:39Z

daemon/archive_unix.go

 		return err
 	}
-	defer cfs.Close()
+	defer cfs.Close(ctx)


WithoutCancel

vvoland · 2026-02-02T11:47:57Z

daemon/create_unix.go

 		return err
 	}
-	defer daemon.Unmount(ctr)
+	defer daemon.Unmount(ctx, ctr)


AFAIK the underlying code doesn't actually respect cancellation currently, but I think it's good to be explicit about the intent.

Suggested change

defer daemon.Unmount(ctx, ctr)

defer daemon.Unmount(context.WithoutCancel(ctx), ctr)

vvoland · 2026-02-02T11:51:35Z

daemon/daemon.go

 			logger := log.G(ctx).WithField("container", c.ID)

-			rwlayer, err := daemon.imageService.GetLayerByID(c.ID)
+			rwlayer, err := daemon.imageService.GetLayerByID(context.Background(), c.ID)


We have a ctx here we can use:

Suggested change

rwlayer, err := daemon.imageService.GetLayerByID(context.Background(), c.ID)

rwlayer, err := daemon.imageService.GetLayerByID(ctx, c.ID)

vvoland · 2026-02-02T11:51:42Z

daemon/daemon.go

 					logger(c).WithError(err).Warn("failed to mount container to get BaseFs path")
 				} else {
-					if err := daemon.Unmount(c); err != nil {
+					if err := daemon.Unmount(ctx, c); err != nil {


WithoutCancel

vvoland · 2026-02-02T11:56:31Z

daemon/workdir.go

 		return err
 	}
-	defer daemon.Unmount(container)
+	defer daemon.Unmount(ctx, container)


WithoutCancel

vvoland · 2026-02-02T11:57:18Z

daemon/oci_windows.go

 				return nil, err
 			}
-			defer daemon.Unmount(c)
+			defer daemon.Unmount(ctx, c)


WithoutCancel

vvoland · 2026-02-02T12:12:26Z

daemon/create.go

 	defer func() {
 		if retErr != nil {
-			err = daemon.cleanupContainer(ctr, backend.ContainerRmConfig{
+			err = daemon.cleanupContainer(ctx, ctr, backend.ContainerRmConfig{


WithoutCancel

vvoland · 2026-02-02T12:14:52Z

daemon/delete.go

+	if err := daemon.removeMountPoints(ctx, ctr, config.RemoveVolume); err != nil {
+		log.G(ctx).Error(err)


We don't actually return on error (only log), so we should either:

WithoutCancel

Return error

Looks like all other context-aware calls in this method are actually using WithoutCancel already, perhaps we should just make the whole ctx uncancellable:

// at the top ctx = context.WithoutCancel(ctx)

Or maybe it's fine for the cleanupContainer itself to be able to cancel in the middle, and let the caller decide (AFAIK most of the calls pass an uncancellable context already)?

Not sure where the boundary is tbh.

corhere added 3 commits January 29, 2026 22:15

daemon: propagate context to ingress jobs

ea89155

Signed-off-by: Cory Snider <[email protected]>

d/libnetwork: contextify driver registration

740f9d2

Lots of interesting setup happens during the bridge and overlay driver registration. Rough-in context plumbing for more correlated tracing of driver initialization. Signed-off-by: Cory Snider <[email protected]>

corhere added this to the 29.2.1 milestone Jan 30, 2026

corhere requested review from akerouanton, robmry, thaJeztah and vvoland January 30, 2026 04:53

corhere added kind/enhancement Enhancements are not bugs or new features but can improve usability or performance. area/daemon Core Engine labels Jan 30, 2026

vvoland requested changes Jan 30, 2026

View reviewed changes

corhere added 2 commits January 30, 2026 16:07

daemon: plumb all the contexts

f668420

Draw the rest of the owl. Wire up contexts from the API handlers, Swarmkit callbacks, event handler callbacks, and background goroutines all the way into libnetwork and the containerd client. Signed-off-by: Cory Snider <[email protected]>

daemon: make certain atomic operations uncancelable

14e72ee

Signed-off-by: Cory Snider <[email protected]>

corhere force-pushed the trace-the-rest-of-the-owl branch from 30467c7 to 14e72ee Compare January 30, 2026 21:08

vvoland reviewed Feb 2, 2026

View reviewed changes

vvoland modified the milestones: 29.2.1, 29.3.0 Feb 2, 2026

	defer rwLayer.Release(ctx)
	defer rwLayer.Release(context.WithoutCancel(ctx))


	if err := rl.c.LeasesService().Delete(context.WithoutCancel(ctx), lease); err != nil {

	rwLayer.Release(ctx)
	rwLayer.Release(context.WithoutCancel(ctx))

	_ = rwlayer.Unmount(ctx)
	_ = rwlayer.Unmount(context.WithoutCancel(ctx))

	err := rwlayer.Unmount(ctx)
	err := rwlayer.Unmount(context.WithoutCancel(ctx))

	// shutdownDaemon just wraps daemon.Shutdown() to handle a timeout in case
	// d.Shutdown() is waiting too long to kill container or worst it's
	// blocked there
	func shutdownDaemon(ctx context.Context, d *daemon.Daemon) {
	var cancel context.CancelFunc
	if timeout := d.ShutdownTimeout(); timeout >= 0 {
	ctx, cancel = context.WithTimeout(ctx, time.Duration(timeout)*time.Second)
	} else {
	ctx, cancel = context.WithCancel(ctx)
	}

	go func() {
	defer cancel()
	d.Shutdown(ctx)
	}()

	<-ctx.Done()
	if errors.Is(ctx.Err(), context.DeadlineExceeded) {
	log.G(ctx).Error("Force shutdown daemon")
	} else {
	log.G(ctx).Debug("Clean shutdown succeeded")
	}
	}

	defer daemon.Unmount(ctx, ctr)
	defer daemon.Unmount(context.WithoutCancel(ctx), ctr)

	rwlayer, err := daemon.imageService.GetLayerByID(context.Background(), c.ID)
	rwlayer, err := daemon.imageService.GetLayerByID(ctx, c.ID)

		if err := daemon.removeMountPoints(ctx, ctr, config.RemoveVolume); err != nil {
		log.G(ctx).Error(err)

Conversation

corhere commented Jan 30, 2026

Uh oh!

vvoland left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants