Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot enlarge memory arrays - when running memory heavy python program #200

Open
Darker opened this issue Nov 22, 2016 · 5 comments
Open

Cannot enlarge memory arrays - when running memory heavy python program #200

Darker opened this issue Nov 22, 2016 · 5 comments

Comments

@Darker
Copy link

Darker commented Nov 22, 2016 •
edited
Loading

So I decided to try how well does PyPyJS deal with some heavy code. In particular, it was Reversi AI that traverses state space of the game. State space based algorighms are typically CPU and memory demanding.

The program DID run for quite some while. Illustrational output:

Computer (simple) played:
   0 1 2 3 4 5 6 7 
0 |X X X X X X X - 
1 |- O X O O X O O 
2 |X X O X O O O O 
3 |X X X X X X X - 
4 |- - X O X X X X 
5 |- - X X - X X - 
6 |- - X O - X O - 
7 |- - - O - - O - 
Predicting LVL 0 stats:
    [-10, -12, -2, -8, -4, -10, -10, -12, -6, -6, -6]
Predicting LVL 1 stats:
    [-189, -223, -86, -135, -99, -191, -134, -190, -114, -100, -133]
Predicting LVL 2 stats:
    [-1050, -1354, -210, -532, -292, -1004, -562, -968, -448, -322, -636]
Predicting LVL 3 stats:
    [-12588, -15590, -3421, -6176, -3476, -11904, -5845, -9955, -6059, -3386, -8897]

This is where it fell. It already was predicting 4th level of state space. Funny thing this. It actually performs better than on ym home PC.

Anyway, the crash error was this:

pypy.html:108 PyPy failed:  abort("Cannot enlarge memory arrays. Either (1) compile with -s TOTAL_MEMORY=X with X higher than the current value 134217728, (2) compile with ALLOW_MEMORY_GROWTH which adjusts the size at runtime but prevents some optimizations, or (3) set Module.TOTAL_MEMORY before the program runs.") at Error
    at jsStackTrace (blob:http://127.0.0.1:9090/e01bd320-5ef9-4bec-b22f-369f4232552f:1:20766)
    at stackTrace (blob:http://127.0.0.1:9090/e01bd320-5ef9-4bec-b22f-369f4232552f:1:20949)
    at abort (blob:http://127.0.0.1:9090/e01bd320-5ef9-4bec-b22f-369f4232552f:86:31988)
    at enlargeMemory (blob:http://127.0.0.1:9090/e01bd320-5ef9-4bec-b22f-369f4232552f:1:21299)
    at Function.dynamicAlloc [as alloc] (blob:http://127.0.0.1:9090/e01bd320-5ef9-4bec-b22f-369f4232552f:1:6286)
    at _sbrk (blob:http://127.0.0.1:9090/e01bd320-5ef9-4bec-b22f-369f4232552f:1:168522)
    at bGd (blob:http://127.0.0.1:9090/e01bd320-5ef9-4bec-b22f-369f4232552f:81:156160)
    at xad (blob:http://127.0.0.1:9090/e01bd320-5ef9-4bec-b22f-369f4232552f:77:432693)
    at E9c (blob:http://127.0.0.1:9090/e01bd320-5ef9-4bec-b22f-369f4232552f:77:363659)
    at Q9c (blob:http://127.0.0.1:9090/e01bd320-5ef9-4bec-b22f-369f4232552f:77:380051)
If this abort() is unexpected, build with -s ASSERTIONS=1 which can give more information.(anonymous function) @ pypy.html:108

It is my understanding that this error happens in emscripten. What can I do with the PyPyJS release to avoid the error. I don't mind increasing memory...

The main problem is that after this error occurs PyPyJS crashes completely. If I could at least set soft memory limit...

The sources to simulate this:

The reproduction steps:

  1. Launch some HTTP server over the pyweb directory
  2. Open http://127.0.0.1/pypy.html in browser. Use new instance of browser in case of crash
  3. Wait till you see Welcome to PyPyJS!
  4. Select all aforementioned files in the file selection field. If nothing happens it was success
  5. Run execfile("/lib/pypyjs/lib_pypy/player.py") by entering it in big text field and pressing Enter
  6. Your browser will lag, if it doesn't crash you will eventually see the error
@refi64
Copy link

refi64 commented Nov 22, 2016

Try pypyjs.totalMemory = some_memory_count_to_use.

@Darker
Copy link
Author

Darker commented Nov 22, 2016

I used 2 times the value mentioned in the emscripten error. I put that here:

      pypyjs.totalMemory = 268435456;
      pypyjs.ready().then(function () {
          pypyjs.stdout('Welcome to PyPy.js!\n');

But the issue still occurs. I tried 10, 20 and 40 times memory. I don't think increasing the value any more will have some effect.

@rfk
Copy link
Contributor

rfk commented Nov 22, 2016

Hi @Darker, it's possible you're encountering an error in the JIT backend, I've seen these cause out-of-memory errors in the past. You could try using the "nojit" release and see if you get the same problem:

https://github.com/pypyjs/pypyjs/releases/download/v0.4.0/pypyjs-nojit-0.4.0.tar.gz

(It will of course run slower without the JIT)

@Darker
Copy link
Author

Darker commented Nov 23, 2016

Ok, I tried that version. It was indescribably slower but it still crashed. Same error.

I need to emphasize that the algorithm is memory demanding. However, I am convinced it still should run, albeit slowly. I certainly made JS programs that consumed more memory, so the limitation is not in the javascript.

@cloudtracer
Copy link

Just thought I'd chime in on this since I have the same issue with a project I'm working on to parse files. Whats odd about it is that if I parse a smaller file and then attempt to parse a bigger file with the script it throws this error. However, if I parse a big file and then parse smaller files the issue doesn't occur. Its almost as if there is a size for something that is set initially and then locked after the first run.

Not sure if this is helpful or not, I haven't dug into the pypyjs code base much, just a user and fan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rfk @refi64 @Darker @cloudtracer