Memory leak(?) when working with Workers #18414
Comments
|
Possibly related to #18369, but needs a proper investigation since it involves web workers. |
bartlomieju
added
the
needs investigation
|
We've updated to 1.33.0, which seems like a great improvement for Node APIs and other perf concerns 🚀, but, unfortunately, memory growth when booting up and terminating workers seems the same. |
|
Unfortunately, no - still seeing the same memory creep in production and in this isolated test case: |
|
We depend on I don't think we're using workers (but not certain). Still not sure what it might be related to. Useful to hear your results are consistent. |
|
@onelson could you provide a reproduction for your case? If it's limited to |
|
I'll dig into this and see if I can reduce it down 👍 |
|
So, I had cause to work on an entirely different project which called for some deno that appears to leak with In this case we're using Here's the RSS:
Allocations over time for the biggest leaking backtrace: I ran a steady load through this process for around 37 minutes. It does seem like the growth could actually be bounded and started to taper off towards the end of the run. I might repeat this overnight at some point just to see if it does plateau. The backtrace related to this particular graph: pointing back to https://github.com/denoland/deno/blob/v1.33.2/core/bindings.rs#L22 The other "big" items in the profile are in the details 👇 👀: It wouldn't surprise me if we're doing something to promote this growth just from our usage. I do plan to downgrade the version of In the earlier case, an entirely separate project, the leak only showed up when we upgraded from an older version. |
|
I rewrote my stuff in terms of fn run_script_inner(input: &serde_json::Value, script: String) -> Result<JsReturn> {
let mut runtime = JsRuntime::new(Default::default());
let res = runtime.execute_script(
"<anon>",
format!("{script}\nhandler({})", serde_json::to_string(input)?).into(),
);
match res {
Ok(global) => {
let scope = &mut runtime.handle_scope();
let local = v8::Local::new(scope, global);
match serde_v8::from_v8::<JsObject>(scope, local) {
Ok(v) => Ok(JsReturn::Object(v)),
Err(serde_v8::Error::ExpectedObject(_)) => Ok(JsReturn::Invalid),
Err(e) => Err(e)?,
}
}
Err(err) => Err(anyhow::format_err!("Evaling error: {:?}", err)),
}
}While running this in a threadpool, once per request, in my service (where Actually, I spot-checked the highest patch level for each minor Honorable mention to the minor version to follow, After Perhaps the bisection is less helpful. I just found it interesting. Here's the top couple of leaky backtraces from In both cases, the leaks seem to be coming from After these two, the next few are very low allocation counts and stemming from things like My expectation is I should be able to create a new |
|
Here's a modest repro for what I've been seeing. Still unsure if @tmcw and I are seeing the same leak. Using the default iteration count of 10,000 I see this program run in just over 1 minute and steadily grow the RSS in that time. [package]
name = "deno-repro"
version = "0.1.0"
edition = "2021"
[dependencies]
deno_core = "0.186.0"use deno_core::{serde_v8, v8, JsRuntime};
fn run_script() -> i64 {
let mut runtime = JsRuntime::new(Default::default());
let global = runtime.execute_script_static("<anon>", "1 + 1").unwrap();
let scope = &mut runtime.handle_scope();
let local = v8::Local::new(scope, global);
let out = serde_v8::from_v8::<i64>(scope, local).unwrap();
assert_eq!(out, 2);
out
}
fn main() {
let iterations: usize = std::env::args()
.nth(1)
.as_deref()
.unwrap_or("10000")
.parse()
.unwrap();
for _i in 0..iterations {
let _x = run_script();
}
}The cold start memory was around 26Mb, and grew to over 68Mb.
Per bytehound, the top leaking points in the code appear to be: |
|
Tried to run |
|
Here is a minimalistic repro. You can notice that the memory usage grow up to 200+ MB [dependencies]
deno_core = "0.195.0"use deno_core::JsRuntime;
use deno_core::RuntimeOptions;
fn run_js() {
JsRuntime::new(RuntimeOptions::default());
}
fn main() {
for i in 0..50000 {
run_js();
if i % 1000 == 0 {
println!("{}", i);
}
}
} |
|
Our approach so far is to restart intermittently because creating & destroying workers doesn't seem to ever free memory - we go from 450mb to 10gb every hour. |
Maybe this is similar to #18369 and should be folded in? I know that memory leak-like issues are pretty hard to really diagnose.
Example:
index.ts
worker.ts
I'd expect, at some point for GC to kick in and collect old worker instances, but it appears that this code just has ever-increasing memory requirements, with memory-per-worker decreasing but not that much:
This is in
Results with Deno 1.32.1 are the same:
Trying other versions, it does look like there's a jump in memory-per worker at some point - 1.29.0 has much lower numbers:
The text was updated successfully, but these errors were encountered: