Skip to content

Pod with exitCode 137, The reason has nothing to do with resources。 #125639

Closed
@yxz11

Description

What happened?

After the pod runs for a period of time, it will be killed. Through auditctl tracking, it is found that the container is killed by runc.

Through monitoring, it is found that all resource utilization is very low, requests and limits are within the range, and there is no OOM situation.

What did you expect to happen?

Reason for being killed

How can we reproduce it (as minimally and precisely as possible)?

Because it is sporadic, it is difficult to reproduce accurately

Anything else we need to know?

The probe used is execProbe

Kubernetes version

kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.20", GitCommit:"1f3e19b7beb1cc0110255668c4238ed63dadb7ad", GitTreeState:"clean", BuildDate:"2021-06-16T12:51:17Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

docker info
.....
Server Version: 19.03.15
Storage Driver: overlay2
......
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: v1.1.6-0-g0f48801a
init version: fec3683
......
Kernel Version: 5.4.242-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
......

Cloud provider

Private cloud, self-built VM

OS version

cat /etc/os-release

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"

uname -a

......5.4.242-1.el7.elrepo.x86_64 #1 SMP Tue Apr 25 09:46:07 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Nothing

Container runtime (CRI) and version (if applicable)

docker version

Client: Docker Engine - Community
Version: 19.03.15
API version: 1.40
Go version: go1.13.15
Git commit: 99e3ed8919
Built: Sat Jan 30 03:17:57 2021
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.15
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 99e3ed8919
Built: Sat Jan 30 03:16:33 2021
OS/Arch: linux/amd64
containerd:
Version: 1.2.2
GitCommit: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc:
Version: 1.1.6
GitCommit: v1.1.6-0-g0f48801a
docker-init:
Version: 0.18.0
GitCommit: fec3683

Related plugins (CNI, CSI, ...) and versions (if applicable)

Nothing

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.sig/nodeCategorizes an issue or PR as relevant to SIG Node.triage/needs-informationIndicates an issue needs more information in order to work on it.

    Type

    No type

    Projects

    • Status

      Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions