-
-
Notifications
You must be signed in to change notification settings - Fork 4.5k
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
Actual Behavior (LocalStack 4.10.0 Current)
When debugging Lambda functions:
- Functions use
--inspect-brkmode (breaks on first line) - Functions pause immediately, waiting for debugger to connect
- All invocations block until debugger attaches and continues execution
- Automated tasks hang indefinitely
- Port mapping changed to:
9252:9229(external:internal) - Environment variables in Lambda container now include:
LDM_DEBUG_PORT=9252 LDM_DEBUG_SERVER_PORT=9229
Expected Behavior
Expected Behavior (LocalStack 4.7.0)
When debugging Lambda functions:
- Functions could use
--inspectmode (non-breaking) - Functions execute normally until hitting explicit breakpoints
- Debugger can attach at any time during execution
- Automated invocations complete successfully
- Port mapping was consistent:
9252:9252(external:internal) - Environment variable in Lambda container:
LDM_DEBUG_PORT=9252
How are you starting LocalStack?
Custom (please describe below)
Steps To Reproduce
- Configure LocalStack with debug mode docker-compose.yml:
services:
localstack:
container_name: "${LOCALSTACK_DOCKER_NAME:-localstack-main}"
image: "localstack/localstack-pro${LOCALSTACK_BUILD-:latest}"
labels:
- "cic.local.dev=true"
ports:
- "${LOCALSTACK_PORT_BIND:-127.0.0.1}:4566:4566"
- "${LOCALSTACK_PORT_BIND:-127.0.0.1}:4510-4559:4510-4559"
environment:
- DEBUG=0
- LS_LOG=warn
- PERSISTENCE=1
- SNAPSHOT_SAVE_STRATEGY=ON_SHUTDOWN
- SNAPSHOT_LOAD_STRATEGY=ON_STARTUP
- CLOUDFORMATION_RETAIN_STACKS=1
- LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=60
- LAMBDA_LIMITS_CONCURRENT_EXECUTIONS=5overall, not per function
- LAMBDA_DISABLE_JAVA_SDK_V2_CERTIFICATE_VALIDATION=0
- LAMBDA_REMOVE_CONTAINERS=1
- LAMBDA_KEEPALIVE=0
- LAMBDA_DEBUG_MODE=1
- LAMBDA_DEBUG_MODE_CONFIG_PATH=/tmp/debug_config.yaml
- SERVICES=s3,sns,sqs,iam,lambda,secretsmanager,sesv2,iot,cloudwatch,events,apigateway,xray,dynamodb,cloudfront,ses,sts,ec2,cognito-idp,logs,cloudformation,ssm,acm,ecr,route53- Create
debug_config.yamlmounted on/tmp/debug_config.yaml:
functions:
arn:aws:lambda:us-west-2:000000000000:function:my-function:
debug-port: 9252
enforce-timeouts: false- Deploy a Node.js Lambda function with debug configuration:
exports.handler = async (event) => {
return { statusCode: 200, body: 'Hello World' };
};- Invoke the function:
aws lambda invoke --function-name my-function output.jsonNote: We deploy using Serverless Framework with the following configuration:
functions:
somefunction:
handler: .webpack/index.handler
memorySize: 2048
timeout: 29 # seconds
environment:
QUERY_TIMEOUT: 29 # seconds
NODE_OPTIONS: --inspect=0.0.0.0:9252Environment
- OS: Windows -> Running localstack in a docker container
- LocalStack:
LocalStack version: 4.10.0
LocalStack Docker image sha: sha256:824b90017959cec940b8ed04e94fc845f2eaa53f48497b88325925eb6b80e7ca
LocalStack build date:
LocalStack build git hash:Anything else?
Documentation that we have been referencing
https://docs.localstack.cloud/aws/tooling/lambda-tools/remote-debugging/#advanced-configuration
Observed Changes
Port Mapping Change
- LocalStack 4.7.0: Debug port mapping was
9252:9252(external:internal) - LocalStack 4.10.0: Debug port mapping changed to
9252:9229(external:internal)
Environment Variables in Lambda Container
LocalStack 4.7.0:
LDM_DEBUG_PORT=9252
LocalStack 4.10.0:
docker exec <lambda-container> sh -c "env | grep -E '(NODE_OPTIONS|DEBUG)'"Output:
AWS_LAMBDA_DEBUG_PORT=9229
LDM_DEBUG_SERVER_PORT=9229
NODE_OPTIONS=--inspect=0.0.0.0:9252 # Note: Uses external port, not internal
LDM_DEBUG_PORT=9252
Container Logs
Debugger listening on ws://0.0.0.0:9229/3dc2c50f-6135-4553-839e-21a8a39903c5
For help, see: https://nodejs.org/en/docs/inspector
Suspected Root Cause
We believe this issue is related to the introduction of the LDM_DEBUG_SERVER_PORT environment variable. This functionality was previously working even with LocalStack 4.10.0 before this environment variable appeared in Lambda containers.
The automatic debugger instrumentation that uses --inspect-brk (which breaks on the first line of code) appears to be tied to the presence and usage of LDM_DEBUG_SERVER_PORT.
Questions
-
When was
LDM_DEBUG_SERVER_PORTintroduced?- This environment variable wasn't present in LocalStack 4.7.0 and its introduction appears to coincide with the breaking behavior change
-
Is
--inspect-brkthe intended default behavior whenLDM_DEBUG_SERVER_PORTis present?- If so, what is the recommended workflow for scenarios where we want debugging available but don't want functions to block on startup?
-
Is there a configuration option to use
--inspectinstead of--inspect-brk?- We've tried setting
NODE_OPTIONSexplicitly in the function environment, but it appears to be overridden whenLDM_DEBUG_SERVER_PORTis set - Example:
LAMBDA_DEBUG_SUSPEND_ON_START=falseor similar
- We've tried setting
-
Why did the port mapping change from
9252:9252to9252:9229?- The
NODE_OPTIONSstill references port9252, but the actual debugger listens on9229inside the container
- The
-
What is the purpose of
LDM_DEBUG_SERVER_PORTvsLDM_DEBUG_PORT?- Understanding these internal variables might help us configure the behavior we need
Attempted Workarounds
1. Using LAMBDA_DOCKER_FLAGS (Failed)
environment:
- LAMBDA_DEBUG_MODE=1
- LAMBDA_DOCKER_FLAGS=-e NODE_OPTIONS=--inspect=0.0.0.0:9229Result: Lambda containers fail to start with error:
LDM could not create a debug environment for 'arn:aws:lambda:...:function:my-function:$LATEST'
Exception: Executor environment shutdown during container startup
Conclusion: LAMBDA_DOCKER_FLAGS conflicts with LAMBDA_DEBUG_MODE
2. Disabling LAMBDA_DEBUG_MODE (Partial Success)
environment:
- LAMBDA_REMOVE_CONTAINERS=0
- LAMBDA_KEEPALIVE=1
# - LAMBDA_DEBUG_MODE=1 # DisabledSet NODE_OPTIONS in Lambda function environment:
environment:
NODE_OPTIONS: "--inspect=0.0.0.0:9229"Result:
- ✅ Functions use
--inspect(non-breaking) - ✅ Automated tasks complete successfully
- ❌ Lose automatic port forwarding from LocalStack
- ❌ Must manually forward ports from Lambda containers
Desired Solution
One of the following would resolve our issue:
-
Configuration option to control break behavior:
- LAMBDA_DEBUG_SUSPEND_ON_START=false -
Default to
--inspectinstead of--inspect-brk:- Makes debug mode compatible with automated workflows
- Users can still set breakpoints where needed
-
Documentation on proper debug configuration:
- How to use debug mode without blocking execution
- How to override default
--inspect-brkbehavior