[Bug]: Tracking issue for unidentified memory leak - Rapid increase of threads

# What happened?

This is more of a tracking issue.... A massive memory leak started on 11th December, and has been plauging ever since.

Which... Does not make sense to me, as I run SaltStack Master within a static pre-built Docker container, and nothing has changed within the last 3 months, yet:

<img width="1592" height="755" alt="Image" src="https://github.com/user-attachments/assets/05337ebb-7d96-4660-b759-8e5f1718041f" />

We can see it is specifically the `MWorker` ndoes leaking memory. All of them rise linearly, until SaltStack exhausts all available RAM on the server, crashes, and repeats.

<img width="1822" height="699" alt="Image" src="https://github.com/user-attachments/assets/397f7280-eefe-4e46-83ca-95cc21d96d12" />

In the last couple of days I have tried upgrading to 3007.9, but that hasn't helped either.

# Type of salt install

Docker Image:

```Dockerfile
# syntax=docker/dockerfile:1

# Use the official Debian image as a base
FROM debian:bookworm-20250811-slim

ARG SALT_VERSION=3007.6
ARG EXPOSE_PORTS=4505 4506 8080

WORKDIR /opt/saltstack/salt
ENV LOG_LEVEL=info
ENV CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1
EXPOSE ${EXPOSE_PORTS}

# Install salt from deb
RUN apt update \
    && apt-get install -y --no-install-recommends ca-certificates patchelf binutils libssl3 wget \
    && wget -q "https://packages.broadcom.com/artifactory/saltproject-deb/pool/salt-common_${SALT_VERSION}_amd64.deb" -O /tmp/salt-common.deb \
    && wget -q "https://packages.broadcom.com/artifactory/saltproject-deb/pool/salt-master_${SALT_VERSION}_amd64.deb" -O /tmp/salt-master.deb \
    && wget -q "https://packages.broadcom.com/artifactory/saltproject-deb/pool/salt-api_${SALT_VERSION}_amd64.deb" -O /tmp/salt-api.deb \
    && apt-get install -y /tmp/salt-master.deb /tmp/salt-api.deb /tmp/salt-common.deb \
    && rm -f /tmp/salt-master.deb /tmp/salt-api.deb /tmp/salt-common.deb \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* \
    && salt-pip install --no-cache-dir pymysql pygit2

CMD ["sh", "-c", "salt-master -l $LOG_LEVEL & salt-api -l $LOG_LEVEL"]
```

### Major version

3007.x

### What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

debian-12

### salt --versions-report output

```shell
root@a47d6e9b8516:/opt/saltstack/salt# salt-master --versions-report
Salt Version:
          Salt: 3007.8
 
Python Version:
        Python: 3.10.18 (main, Sep  5 2025, 22:48:51) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.8.0
  cryptography: 42.0.5
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: 1.9.1
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.18.2
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.20.6
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.4.2
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: debian 12.11 bookworm
        locale: utf-8
       machine: x86_64
       release: 5.4.0-224-generic
        system: Linux
       version: Debian GNU/Linux 12.11 bookworm
```

```shell
root@044f03c9af85:/opt/saltstack/salt# salt-master --versions-report
/opt/saltstack/salt/lib/python3.10/site-packages/cherrypy/__init__.py:60: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
Salt Version:
          Salt: 3007.9
 
Python Version:
        Python: 3.10.19 (main, Oct 30 2025, 04:53:28) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.8.0
  cryptography: 42.0.5
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: 1.9.1
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.18.2
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.21.2
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.5.2
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: debian 12.11 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.12.57+deb13-amd64
        system: Linux
       version: Debian GNU/Linux 12.11 bookworm
```

### Master  Config

```yaml
# ===============
# ===== API =====
# ===============

rest_cherrypy:
  port: 8000
  host: 0.0.0.0
  debug: True
  disable_ssl: True

#external_auth:
  # REDACTED

netapi_enable_clients:
  - local
  - local_async
  - local_batch
  - runner
  - runner_async
  - wheel
  - wheel_async

# ================
# ===== MAIN =====
# ================

# Listen on all IPv4/IPv6 interfaces
interface: '0.0.0.0'
ipv6: True

# Auto-accept clients trying to join
auto_accept: True

# Increase worker threads from 5 to 12
worker_threads: 12

# The level of messages to send to the console.
# One of 'garbage', 'trace', 'debug', info', 'warning', 'error', 'critical'.
#
# The following log levels are considered INSECURE and may log sensitive data:
# ['garbage', 'trace', 'debug']
#
#log_level: warning
log_level: info


# ===================
# ===== Storage =====
# ===================

# Do not follow symlinks on server side
fileserver_followsymlinks: False

# Enables use of remote formulas
fileserver_backend:
  - roots
  - git

gitfs_provider: pygit2
gitfs_base: main
gitfs_remotes:
  - https://git.example.com/linux/config/global.git
  - https://git.example.com/linux/config/redacted.git
  - https://git.example.com/linux/config/redacted.git
  - https://git.example.com/linux/config/redacted.git
  - https://git.example.com/linux/config/redacted.git
  - https://git.example.com/linux/config/redacted.git
  - https://git.example.com/linux/config/redacted.git
  - https://git.example.com/linux/config/redacted.git
  - https://git.example.com/linux/config/redacted.git
  - https://git.example.com/linux/config/redacted.git
  - https://git.example.com/linux/config/redacted.git

# Disable global lock - this is only fine when underlying FS is not shared (e.g. GlusterFS), which it is not.
gitfs_global_lock: False

# Remote Pillar
git_pillar_provider: pygit2
git_pillar_base: main
git_pillar_branch: main
git_pillar_global_lock: False
ext_pillar:
  - git:
    - main https://git.example.com/linux/pillar/global.git
    - main https://git.example.com/linux/pillar/redacted.git
    - main https://git.example.com/linux/pillar/redacted.git
    - main https://git.example.com/linux/pillar/redacted.git
    - main https://git.example.com/linux/pillar/redacted.git
    - main https://git.example.com/linux/pillar/redacted.git
  - http_json:
      url: http://provider:8000/pillar/%s
      username: null
      password: null

# GitFS authentication
gitfs_user: "git"
gitfs_password: "REDACTED"

# Remote Pillar authentication
git_pillar_user: "git"
git_pillar_password: "REDACTED"
```


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Bug]: Tracking issue for unidentified memory leak - Rapid increase of threads #68557

What happened?

Type of salt install

Major version

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

salt --versions-report output

Master Config

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Tracking issue for unidentified memory leak - Rapid increase of threads #68557

Description

What happened?

Type of salt install

Major version

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

salt --versions-report output

Master Config

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions