Skip to content

bug: Node.js Lambda Debugging Issue with LocalStack #13480

@jcheenath

Description

@jcheenath

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Actual Behavior (LocalStack 4.10.0 Current)

When debugging Lambda functions:

  • Functions use --inspect-brk mode (breaks on first line)
  • Functions pause immediately, waiting for debugger to connect
  • All invocations block until debugger attaches and continues execution
  • Automated tasks hang indefinitely
  • Port mapping changed to: 9252:9229 (external:internal)
  • Environment variables in Lambda container now include:
    LDM_DEBUG_PORT=9252
    LDM_DEBUG_SERVER_PORT=9229
    

Expected Behavior

Expected Behavior (LocalStack 4.7.0)

When debugging Lambda functions:

  • Functions could use --inspect mode (non-breaking)
  • Functions execute normally until hitting explicit breakpoints
  • Debugger can attach at any time during execution
  • Automated invocations complete successfully
  • Port mapping was consistent: 9252:9252 (external:internal)
  • Environment variable in Lambda container:
    LDM_DEBUG_PORT=9252
    

How are you starting LocalStack?

Custom (please describe below)

Steps To Reproduce

  1. Configure LocalStack with debug mode docker-compose.yml:
    services:
        localstack:
            container_name: "${LOCALSTACK_DOCKER_NAME:-localstack-main}"
            image: "localstack/localstack-pro${LOCALSTACK_BUILD-:latest}"
            labels:
            - "cic.local.dev=true"
            ports:
            - "${LOCALSTACK_PORT_BIND:-127.0.0.1}:4566:4566"
            - "${LOCALSTACK_PORT_BIND:-127.0.0.1}:4510-4559:4510-4559"
            environment:
            - DEBUG=0
            - LS_LOG=warn
            - PERSISTENCE=1
            - SNAPSHOT_SAVE_STRATEGY=ON_SHUTDOWN
            - SNAPSHOT_LOAD_STRATEGY=ON_STARTUP
            - CLOUDFORMATION_RETAIN_STACKS=1
            - LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=60
            - LAMBDA_LIMITS_CONCURRENT_EXECUTIONS=5overall, not per function
            - LAMBDA_DISABLE_JAVA_SDK_V2_CERTIFICATE_VALIDATION=0
            - LAMBDA_REMOVE_CONTAINERS=1
            - LAMBDA_KEEPALIVE=0
            - LAMBDA_DEBUG_MODE=1
            - LAMBDA_DEBUG_MODE_CONFIG_PATH=/tmp/debug_config.yaml
            - SERVICES=s3,sns,sqs,iam,lambda,secretsmanager,sesv2,iot,cloudwatch,events,apigateway,xray,dynamodb,cloudfront,ses,sts,ec2,cognito-idp,logs,cloudformation,ssm,acm,ecr,route53
  1. Create debug_config.yaml mounted on /tmp/debug_config.yaml:
functions:
  arn:aws:lambda:us-west-2:000000000000:function:my-function:
    debug-port: 9252
    enforce-timeouts: false
  1. Deploy a Node.js Lambda function with debug configuration:
exports.handler = async (event) => {
    return { statusCode: 200, body: 'Hello World' };
};
  1. Invoke the function:
aws lambda invoke --function-name my-function output.json

Note: We deploy using Serverless Framework with the following configuration:

functions:
  somefunction:
    handler: .webpack/index.handler
    memorySize: 2048
    timeout: 29 # seconds
    environment:
      QUERY_TIMEOUT: 29 # seconds
      NODE_OPTIONS: --inspect=0.0.0.0:9252

Environment

- OS: Windows -> Running localstack in a docker container
- LocalStack:
  LocalStack version: 4.10.0
  LocalStack Docker image sha: sha256:824b90017959cec940b8ed04e94fc845f2eaa53f48497b88325925eb6b80e7ca
  LocalStack build date:
  LocalStack build git hash:

Anything else?

Documentation that we have been referencing
https://docs.localstack.cloud/aws/tooling/lambda-tools/remote-debugging/#advanced-configuration

Observed Changes

Port Mapping Change

  • LocalStack 4.7.0: Debug port mapping was 9252:9252 (external:internal)
  • LocalStack 4.10.0: Debug port mapping changed to 9252:9229 (external:internal)

Environment Variables in Lambda Container

LocalStack 4.7.0:

LDM_DEBUG_PORT=9252

LocalStack 4.10.0:

docker exec <lambda-container> sh -c "env | grep -E '(NODE_OPTIONS|DEBUG)'"

Output:

AWS_LAMBDA_DEBUG_PORT=9229
LDM_DEBUG_SERVER_PORT=9229
NODE_OPTIONS=--inspect=0.0.0.0:9252  # Note: Uses external port, not internal
LDM_DEBUG_PORT=9252

Container Logs

Debugger listening on ws://0.0.0.0:9229/3dc2c50f-6135-4553-839e-21a8a39903c5
For help, see: https://nodejs.org/en/docs/inspector

Suspected Root Cause

We believe this issue is related to the introduction of the LDM_DEBUG_SERVER_PORT environment variable. This functionality was previously working even with LocalStack 4.10.0 before this environment variable appeared in Lambda containers.

The automatic debugger instrumentation that uses --inspect-brk (which breaks on the first line of code) appears to be tied to the presence and usage of LDM_DEBUG_SERVER_PORT.

Questions

  1. When was LDM_DEBUG_SERVER_PORT introduced?

    • This environment variable wasn't present in LocalStack 4.7.0 and its introduction appears to coincide with the breaking behavior change
  2. Is --inspect-brk the intended default behavior when LDM_DEBUG_SERVER_PORT is present?

    • If so, what is the recommended workflow for scenarios where we want debugging available but don't want functions to block on startup?
  3. Is there a configuration option to use --inspect instead of --inspect-brk?

    • We've tried setting NODE_OPTIONS explicitly in the function environment, but it appears to be overridden when LDM_DEBUG_SERVER_PORT is set
    • Example: LAMBDA_DEBUG_SUSPEND_ON_START=false or similar
  4. Why did the port mapping change from 9252:9252 to 9252:9229?

    • The NODE_OPTIONS still references port 9252, but the actual debugger listens on 9229 inside the container
  5. What is the purpose of LDM_DEBUG_SERVER_PORT vs LDM_DEBUG_PORT?

    • Understanding these internal variables might help us configure the behavior we need

Attempted Workarounds

1. Using LAMBDA_DOCKER_FLAGS (Failed)

environment:
  - LAMBDA_DEBUG_MODE=1
  - LAMBDA_DOCKER_FLAGS=-e NODE_OPTIONS=--inspect=0.0.0.0:9229

Result: Lambda containers fail to start with error:

LDM could not create a debug environment for 'arn:aws:lambda:...:function:my-function:$LATEST'
Exception: Executor environment shutdown during container startup

Conclusion: LAMBDA_DOCKER_FLAGS conflicts with LAMBDA_DEBUG_MODE

2. Disabling LAMBDA_DEBUG_MODE (Partial Success)

environment:
  - LAMBDA_REMOVE_CONTAINERS=0
  - LAMBDA_KEEPALIVE=1
  # - LAMBDA_DEBUG_MODE=1  # Disabled

Set NODE_OPTIONS in Lambda function environment:

environment:
  NODE_OPTIONS: "--inspect=0.0.0.0:9229"

Result:

  • ✅ Functions use --inspect (non-breaking)
  • ✅ Automated tasks complete successfully
  • ❌ Lose automatic port forwarding from LocalStack
  • ❌ Must manually forward ports from Lambda containers

Desired Solution

One of the following would resolve our issue:

  1. Configuration option to control break behavior:

    - LAMBDA_DEBUG_SUSPEND_ON_START=false
  2. Default to --inspect instead of --inspect-brk:

    • Makes debug mode compatible with automated workflows
    • Users can still set breakpoints where needed
  3. Documentation on proper debug configuration:

    • How to use debug mode without blocking execution
    • How to override default --inspect-brk behavior

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions