Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

xray core stoped randomly when sniffing is enabled #3914

Open
4 tasks done
sssagsag opened this issue Oct 15, 2024 · 40 comments
Open
4 tasks done

xray core stoped randomly when sniffing is enabled #3914

sssagsag opened this issue Oct 15, 2024 · 40 comments

Comments

@sssagsag
Copy link

sssagsag commented Oct 15, 2024

Integrity requirements

  • I confirm that I have read the documentation, understand the meaning of all the configuration items I wrote, and did not pile up seemingly useful options or default values.
  • I provided the complete config and logs, rather than just providing the truncated parts based on my own judgment.
  • I searched issues and did not find any similar issues.
  • The problem can be successfully reproduced in the latest Release

Description

There is a big problem in versions 24.9.30, 24.9.19, 24, 9.16 (version 1.8.24 and
24.9.7 are not a problem), when the tls sniffing feature is active on a high-traffic server with about 100 online users with 5000 open connections, the xray core crashes randomly for 5 to 10 minutes and the xray must be run again (my server resources It is good and there is no shortage (8 cores with 16 gigs of RAM), I checked this problem on a low traffic server and it takes hours for the core to stop working. Also, all my users use mux The problem does not occur when Sniff is turned off and the core is active for days.

Client config


{
  "log": {
    "access": "",
    "error": "",
    "loglevel": "warning"
  },
  "dns": {
    "servers": [
      "1.1.1.1",
      "8.8.8.8"
    ],
    "queryStrategy": "UseIPv4"
  },
  "inbounds": [
    {
      "tag": "socks",
      "port": 10808,
      "listen": "127.0.0.1",
      "protocol": "socks",
      "sniffing": {
        "enabled": true,
        "destOverride": [
          "http",
          "tls"
        ],
        "routeOnly": false
      },
      "settings": {
        "auth": "noauth",
        "udp": true,
        "allowTransparent": false
      }
    },
    {
      "tag": "http",
      "port": 10809,
      "listen": "127.0.0.1",
      "protocol": "http",
      "sniffing": {
        "enabled": true,
        "destOverride": [
          "http",
          "tls"
        ],
        "routeOnly": false
      },
      "settings": {
        "auth": "noauth",
        "udp": true,
        "allowTransparent": false
      }
    }
  ],
  "outbounds": [
    {
      "tag": "proxy",
      "protocol": "vless",
      "settings": {
        "vnext": [
          {
            "address": "",
            "port": 443,
            "users": [
              {
                "id": "",
                "alterId": 0,
                "email": "[email protected]",
                "security": "auto",
                "encryption": "none"
              }
            ]
          }
        ]
      },
      "streamSettings": {
        "network": "ws",
        "security": "tls",
        "tlsSettings": {
          "allowInsecure": true,
          "fingerprint": "chrome",
          "serverName": ""
        },
        "wsSettings": {
          "headers": {
            "Host": ""          },
          "path": "/?ed=2560"
        },
        "sockopt": {
          "dialerProxy": "fragment0",
          "tcpKeepAliveIdle": 100,
          "mark": 255,
          "tcpNoDelay": true
        }
      },
      "mux": {
        "concurrency": 8,
        "enabled": true,
        "xudpConcurrency": 8,
        "xudpProxyUDP443": "reject"
      }
    },
    {
      "tag": "fragment0",
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "AsIs",
        "fragment": {
          "packets": "1-1",
          "length": "1-10",
          "interval": "1-10"
        }
      },
      "streamSettings": {
        "sockopt": {
          "tcpNoDelay": true,
          "tcpKeepAliveIdle": 100
        }
      }
    }
  ],
  "routing": {
    "domainStrategy": "IPIfNonMatch"
  },
  "remarks": ""
}

Server config


{
  "inbounds": [
    {
      "listen": null,
      "port": 8080,
      "protocol": "vless",
      "settings": {
        "clients": [
        ],
        "decryption": "none",
        "fallbacks": []
      },
      "sniffing": {
        "destOverride": [
          "tls"
        ],
        "enabled": true,
        "metadataOnly": false,
        "routeOnly": false
      },
      "streamSettings": {
        "network": "ws",
        "security": "none",
        "wsSettings": {
          "acceptProxyProtocol": false,
          "headers": {},
          "host": "",
          "path": "/"
        }
      },
      "tag": "inbound-8080"
    }
  ],
  "log": {
    "access": "none",
    "error": "/root/error.log",
    "loglevel": "debug"
  },
  "observatory": null,
  "outbounds": [
    {
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "ForceIPv4"
      },
      "tag": "direct"
    },
    {
      "protocol": "blackhole",
      "settings": {},
      "tag": "blocked"
    }
     ],
  "policy": {
    "levels": {
      "0": {
        "statsUserDownlink": true,
        "statsUserUplink": true
      }
    },
    "system": {
      "statsInboundDownlink": true,
      "statsInboundUplink": true
    }
  },
  "reverse": null,
  "routing": {
    "domainStrategy": "IPIfNonMatch",
    "rules": [
      {
        "inboundTag": [
          "api"
        ],
        "outboundTag": "api",
        "type": "field"
      },
      {
        "outboundTag": "blocked",
        "protocol": [
          "bittorrent"
        ],
        "type": "field"
      }
    ]
  },
  "stats": {},
  "transport": null
}

Server log

serverroot.txt
xray-debug2.log

@sssagsag
Copy link
Author

please check this problem
@mmmray @yuhan6665 @RPRX

@Fangliding
Copy link
Member

dup #3904

@Fangliding Fangliding closed this as not planned Won't fix, can't repro, duplicate, stale Oct 15, 2024
@sssagsag
Copy link
Author

dup #3904

What exactly is the connection between #3904 and my problem?

@Fangliding
Copy link
Member

dup #3904

What exactly is the connection between #3904 and my problem?

See the pull request comment(if you understand)

@MHSanaei
Copy link
Contributor

@sssagsag Bro, just download that PR build and test it.
https://github.com/XTLS/Xray-core/actions/runs/11339237462

@sssagsag
Copy link
Author

@sssagsag Bro, just download that PR build and test it. https://github.com/XTLS/Xray-core/actions/runs/11339237462

Yes, I will definitely test it now

@sssagsag
Copy link
Author

@sssagsag Bro, just download that PR build and test it. https://github.com/XTLS/Xray-core/actions/runs/11339237462

The problem is not solved, the core is stopped one after another

@sssagsag
Copy link
Author

dup #3904

What exactly is the connection between #3904 and my problem?

See the pull request comment(if you understand)

The problem is not solved, the core is stopped one after another

@Fangliding
Copy link
Member

I need to see new log file

@sssagsag
Copy link
Author

I need to see new log file

OK, I will capture it and send it to you

@sssagsag
Copy link
Author

I need to see new log file

newxray.txt
xray-debugnew.log

@Fangliding
Copy link
Member

I understand, I'll handle it tomorrow

@Fangliding Fangliding reopened this Oct 15, 2024
@sssagsag
Copy link
Author

I understand, I'll handle it tomorrow

thank you sir

@Fangliding
Copy link
Member

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914

@sssagsag
Copy link
Author

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914
Is this version that you builded different from the new 24.10.16?

@Fangliding
Copy link
Member

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914
Is this version that you builded different from the new 24.10.16?

It's a test version, not on the main branch, not included in 24.10.16

@Fangliding
Copy link
Member

log

@sssagsag
Copy link
Author

sssagsag commented Oct 16, 2024

log

I put your modified your xray core on one super high-traffic server and it has been online for about 5 hour now, and apparently the problem has been completely solved.

@sssagsag
Copy link
Author

sssagsag commented Oct 16, 2024

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914

10 hours online without any problem with the worst route and the worst possible settings and the large number of online users, the problem is completely solved, thank you for your attention , please add to main branch

@RPRX
Copy link
Member

RPRX commented Oct 17, 2024

@Fangliding 开个 pr

@realartin
Copy link

realartin commented Oct 18, 2024

i guess i have a similar issue :

xray core v24.9.30

github.com/xtls/xray-core/common/task/task.go:12 +0x22 github.com/xtls/xray-core/common/task.Run.func1(0xc0066067e0?) github.com/xtls/xray-core/common/task/task.go:28 +0x28 created by github.com/xtls/xray-core/common/task.Run in goroutine 23562 github.com/xtls/xray-core/common/task/task.go:27 +0xdb goroutine 24996 [select]: github.com/xtls/xray-core/transport/internet/splithttp.(*requestHandler).ServeHTTP(0xc00118eb40, {0x153af30, 0xc005fa3d88}, 0xc005736140) github.com/xtls/xray-core/transport/internet/splithttp/hub.go:217 +0xdea golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP({{0x152d320?, 0xc00118eb40?}, 0xc001caf140?}, {0x153af30, 0xc005fa3d88}, 0xc005736140) golang.org/x/[email protected]/http2/h2c/h2c.go:125 +0x673 net/http.serverHandler.ServeHTTP({0x1ff?}, {0x153af30?, 0xc005fa3d88?}, 0xc000492000?) net/http/server.go:3210 +0x8e net/http.initALPNRequest.ServeHTTP({{0x153d300?, 0xc0032aeb40?}, 0xc000122e08?, {0xc000134288?}}, {0x153af30, 0xc005fa3d88}, 0xc005736140) net/http/server.go:3819 +0x231 net/http.(*http2serverConn).runHandler(0x445edb?, 0x0?, 0x0?, 0x0?) net/http/h2_bundle.go:6249 +0xf5 created by net/http.(*http2serverConn).scheduleHandler in goroutine 23156 net/http/h2_bundle.go:6183 +0x21d goroutine 21880 [select, 2 minutes]: github.com/xtls/xray-core/common/task.Run({0x153d300, 0xc006155d70}, {0xc005a9bec8, 0x2, 0xc006e79bb0?}) github.com/xtls/xray-core/common/task/task.go:48 +0x265 github.com/xtls/xray-core/proxy/vmess/inbound.(*Handler).Process(0xc00032fa40, {0x153d300, 0xc0062bd3b0}, 0x151dbd0?, {0x1544698, 0xc00015d8a0}, {0x153f720, 0xc000045490}) github.com/xtls/xray-core/proxy/vmess/inbound/inbound.go:298 +0xdef github.com/xtls/xray-core/app/proxyman/inbound.(*tcpWorker).callback(0xc000e8b540, {0x1544698, 0xc00015d8a0}) github.com/xtls/xray-core/app/proxyman/inbound/worker.go:109 +0x5e7 created by github.com/xtls/xray-core/app/proxyman/inbound.(*tcpWorker).Start.func1 in goroutine 11 github.com/xtls/xray-core/app/proxyman/inbound/worker.go:123 +0x7a goroutine 25381 [select]: github.com/xtls/xray-core/transport/pipe.(*pipe).ReadMultiBuffer(0xc004b71b30) github.com/xtls/xray-core/transport/pipe/impl.go:100 +0xeb github.com/xtls/xray-core/transport/pipe.(*Reader).ReadMultiBuffer(0xc004b2a540?) github.com/xtls/xray-core/transport/pipe/reader.go:16 +0x16 github.com/xtls/xray-core/common/buf.copyInternal({0x152b600, 0xc0071af2d8}, {0x152d9c0, 0xc004b2a540}, 0xc00711d848) github.com/xtls/xray-core/common/buf/copy.go:93 +0x5b github.com/xtls/xray-core/common/buf.Copy({0x152b600, 0xc0071af2d8}, {0x152d9c0, 0xc004b2a540}, {0xc00423df58, 0x1, 0xc004b2a4e0?}) github.com/xtls/xray-core/common/buf/copy.go:116 +0x98 github.com/xtls/xray-core/proxy/vless/inbound.(*Handler).Process.func5() github.com/xtls/xray-core/proxy/vless/inbound/inbound.go:567 +0x3ff github.com/xtls/xray-core/common/task.Run.func1(0xc006553f50?) github.com/xtls/xray-core/common/task/task.go:28 +0x28 created by github.com/xtls/xray-core/common/task.Run in goroutine 25332 github.com/xtls/xray-core/common/task/task.go:27 +0xdb goroutine 17308 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7c72e93d27d8, 0x72) runtime/netpoll.go:351 +0x85 internal/poll.(*pollDesc).wait(0xc004d1a900?, 0xc001efe000?, 0x0) internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc004d1a900, {0xc001efe000, 0x2000, 0x2000}) internal/poll/fd_unix.go:165 +0x27a net.(*netFD).Read(0xc004d1a900, {0xc001efe000?, 0x410845?, 0x30?}) net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc002625928, {0xc001efe000?, 0x2000?, 0xc001efe000?}) net/net.go:189 +0x45 github.com/xtls/xray-core/common/buf.(*Buffer).ReadFrom(...) github.com/xtls/xray-core/common/buf/buffer.go:295 github.com/xtls/xray-core/common/buf.ReadBuffer({0x152d820, 0xc002625928}) github.com/xtls/xray-core/common/buf/reader.go:30 +0x66 github.com/xtls/xray-core/common/buf.(*ReadVReader).ReadMultiBuffer(0xc0066b3bd0) github.com/xtls/xray-core/common/buf/readv_reader.go:125 +0x65 github.com/xtls/xray-core/common/buf.copyInternal({0x152da40, 0xc0066b3bd0}, {0x152b660, 0xc001473dc0}, 0xc006d61f20) github.com/xtls/xray-core/common/buf/copy.go:93 +0x5b github.com/xtls/xray-core/common/buf.Copy({0x152da40, 0xc0066b3bd0}, {0x152b660, 0xc001473dc0}, {0xc004cb2f30, 0x1, 0x10c3d80?}) github.com/xtls/xray-core/common/buf/copy.go:116 +0x98 github.com/xtls/xray-core/proxy/wireguard.(*Handler).Process.func3() github.com/xtls/xray-core/proxy/wireguard/client.go:206 +0xf9 github.com/xtls/xray-core/proxy/wireguard.(*Handler).Process.OnSuccess.func7() github.com/xtls/xray-core/common/task/task.go:12 +0x22 github.com/xtls/xray-core/common/task.Run.func1(0x24?) github.com/xtls/xray-core/common/task/task.go:28 +0x28 created by github.com/xtls/xray-core/common/task.Run in goroutine 17301 github.com/xtls/xray-core/common/task/task.go:27 +0xdb goroutine 17295 [select, 4 minutes]: github.com/xtls/xray-core/transport/internet/splithttp.(*requestHandler).ServeHTTP(0xc00118eb40, {0x153af30, 0xc0026257c8}, 0xc0065b9900) github.com/xtls/xray-core/transport/internet/splithttp/hub.go:217 +0xdea golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP({{0x152d320?, 0xc00118eb40?}, 0xc001caf140?}, {0x153af30, 0xc0026257c8}, 0xc0065b9900) golang.org/x/[email protected]/http2/h2c/h2c.go:125 +0x673 net/http.serverHandler.ServeHTTP({0x0?}, {0x153af30?, 0xc0026257c8?}, 0xc0011fa300?) net/http/server.go:3210 +0x8e net/http.initALPNRequest.ServeHTTP({{0x153d300?, 0xc004ce9380?}, 0xc000004708?, {0xc000134288?}}, {0x153af30, 0xc0026257c8}, 0xc0065b9900) net/http/server.go:3819 +0x231 net/http.(*http2serverConn).runHandler(0xaad7e8?, 0xc000e8b540?, 0x1544698?, 0xc000284b00?) net/http/h2_bundle.go:6249 +0xf5 created by net/http.(*http2serverConn).scheduleHandler in goroutine 12548 net/http/h2_bundle.go:6183 +0x21d goroutine 19879 [select, 2 minutes]: github.com/xtls/xray-core/common/task.Run({0x153d338, 0xc006655900}, {0xc003e87cd0, 0x2, 0xc000199480?}) github.com/xtls/xray-core/common/task/task.go:48 +0x265 github.com/xtls/xray-core/proxy/wireguard.(*Handler).Process(0xc0001dcee0, {0x153d300, 0xc003fe48a0}, 0xc00695edc0, {0x153a640, 0xc000199480}) github.com/xtls/xray-core/proxy/wireguard/client.go:230 +0xc85 github.com/xtls/xray-core/app/proxyman/outbound.(*Handler).Dispatch(0xc000199480, {0x153d300, 0xc003fe48a0}, 0xc00695edc0) github.com/xtls/xray-core/app/proxyman/outbound/handler.go:211 +0x388 github.com/xtls/xray-core/app/dispatcher.(*DefaultDispatcher).routedDispatch(0xc0010e6c00, {0x153d300, 0xc003fe48a0}, 0xc00695edc0, {{0x153d450, 0xc00321458c}, 0x35, 0x3})

@sssagsag
Copy link
Author

sssagsag commented Oct 23, 2024

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914

Since I put your modified core on the server a few days ago, the kernel has stopped 3 times. Please see the log file.
output.txt

@Fangliding
Copy link
Member

This log is incomplete and lacks the key information

@sssagsag
Copy link
Author

This log is incomplete and lacks the key information

The log is 25 GB in size and I am unable to extract it.

@Fangliding
Copy link
Member

Logs can be truncated, but they should be truncated at the end rather than the beginning

@sssagsag
Copy link
Author

Logs can be truncated, but they should be truncated at the end rather than the beginning

I put the error data at times, but some of them have disappeared, what mode do you suggest to diagnose the problem and make the log smaller? no i use debug mode

@realartin
Copy link

Does this fixed on latest release ?

@sssagsag
Copy link
Author

Does this fixed on latest release ?

I put the new version on high traffic servers to see if it is fixed or not

@RPRX
Copy link
Member

RPRX commented Oct 31, 2024

Does this fixed on latest release ?

解决了,但还没根治,这个 issue 需要继续保持 open

@realartin
Copy link

Does this fixed on latest release ?

解决了,但还没根治,这个 issue 需要继续保持 open

Can we use mux and snifing on or mux is still unstable the core ?

Tnx for your help

@sssagsag
Copy link
Author

Does this fixed on latest release ?

I put the new version on high traffic servers to see if it is fixed or not

version v24.10.31 stopped after 5 hours with 5 high load server

@sssagsag
Copy link
Author

sssagsag commented Nov 1, 2024

after 24 hours xray running core is stoped
2024/11/01 15:45:38 DEBUG - XRAY: github.com/xtls/xray-core/common/protocol/quic.SniffQUIC({0xc001d96000, 0x4b0, 0x2000})
2024/11/01 15:45:38 DEBUG - XRAY: panic: runtime error: slice bounds out of range [18:17]

@ImMohammad20000
Copy link

ImMohammad20000 commented Nov 1, 2024

after 24 hours xray running core is stoped
2024/11/01 15:45:38 DEBUG - XRAY: github.com/xtls/xray-core/common/protocol/quic.SniffQUIC({0xc001d96000, 0x4b0, 0x2000})
2024/11/01 15:45:38 DEBUG - XRAY: panic: runtime error: slice bounds out of range [18:17]

Do you use quic sniffer?
if the answer is yes disable quick sniffer and test again

@Fangliding
Copy link
Member

Try this version https://github.com/XTLS/Xray-core/actions/runs/11645174316

@sssagsag
Copy link
Author

sssagsag commented Nov 6, 2024

Try this version https://github.com/XTLS/Xray-core/actions/runs/11645174316

I didn't test your version, I tested the new version 24.11.5 directly.

nohup: ignoring input
Xray 24.11.5 (Xray, Penetrates Everything.) afc7ec5 (go1.23.2 linux/amd64)
A unified platform for anti-censorship.
2024/11/06 08:03:07 [Info] infra/conf/serial: Reading config: &{Name:/root/config.json Format:json}
panic: runtime error: slice bounds out of range [90:89]

goroutine 44557 [running]:
github.com/xtls/xray-core/common/protocol/quic.SniffQUIC({0xc001a72000, 0x4e2, 0x2000})
github.com/xtls/xray-core/common/protocol/quic/sniff.go:162 +0x1ed0
github.com/xtls/xray-core/app/dispatcher.NewSniffer.func4({0x80?, 0x13278a0?}, {0xc001a72000?, 0x1?, 0x4e2?})
github.com/xtls/xray-core/app/dispatcher/sniffer.go:41 +0x25
github.com/xtls/xray-core/app/dispatcher.(*Sniffer).Sniff(0xc003a2fe78, {0x1549c28, 0xc003befad0}, {0xc001a72000, 0x4e2, 0x2000}, 0x3)
github.com/xtls/xray-core/app/dispatcher/sniffer.go:65 +0xcc
github.com/xtls/xray-core/app/dispatcher.sniffer.func1({0x1549c28, 0xc003befad0}, 0xc003befb60, 0xc003befb90, 0xc003a2fe78, 0x3)
github.com/xtls/xray-core/app/dispatcher/default.go:383 +0x125
github.com/xtls/xray-core/app/dispatcher.sniffer({0x1549c28, 0xc003befad0}, 0xc003befb60, 0x0, 0x3)
github.com/xtls/xray-core/app/dispatcher/default.go:393 +0x116
github.com/xtls/xray-core/app/dispatcher.(*DefaultDispatcher).Dispatch.func1()
github.com/xtls/xray-core/app/dispatcher/default.go:275 +0x10f
created by github.com/xtls/xray-core/app/dispatcher.(*DefaultDispatcher).Dispatch in goroutine 44556
github.com/xtls/xray-core/app/dispatcher/default.go:270 +0x47f

@sssagsag
Copy link
Author

sssagsag commented Nov 6, 2024

Try this version https://github.com/XTLS/Xray-core/actions/runs/11645174316

I put this version and will keep you updated on the status.

@sssagsag
Copy link
Author

sssagsag commented Nov 6, 2024

Try this version https://github.com/XTLS/Xray-core/actions/runs/11645174316

after 4 hour online your version crash xray core seem sloved will keep you updated on the status.

@sssagsag
Copy link
Author

sssagsag commented Nov 6, 2024

update after 10 hours online without crash
{B90740C6-5DEE-4985-AA13-AF1126F7668F}

@sssagsag
Copy link
Author

sssagsag commented Nov 11, 2024

problem sloved , thanks to @Fangliding and @RPRX , now update 24.11.11 work perfectly

@RPRX
Copy link
Member

RPRX commented Nov 14, 2024

这个 issue 需要保持开启,#3908 #3921 #3978 这三个都只是治标,还没有治本。话说 @Fangliding 写一下 #3867 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants