I love LuaJIT but I am starting to feel seriously uncomfortable about its memory limitations.
64-bit LuaJIT can only use memory allocated within the first 2 GB. Mainstream desktop systems consume more and more memory so by the time you fire up your LuaJIT app there might be (next to) no memory left within that range.
32-bit LuaJIT can use the first 4 GB but again sooner or later desktop systems will leave little there too + 32-bit code is already deprecated technology on the desktop.
Makes me damn sad really. Unfortunately making LuaJIT capable of using the full address space requires a major rewrite.. one which will likely never happen. LuaJIT's author has moved on to other projects and LuaJIT is now in maintenance mode, getting nothing but small fixes.
Seriously if you are a Mike Pall-grade elite low-level hacker (I am talking to all four of you ;) ) would you please continue LuaJIT development?
IIRC, he's said that even if the pointer model gets extended, the garbage collector's not up to scanning more than 2GB efficiently so your programs will run like a slug.
Also, it's worth remembering that the 2GB limit only applies to Lua objects. Objects allocated through the FFI API aren't limited by this. So any pure data structures which don't contain Lua object pointers can be allocated with simple malloc(). It'll be loads faster that way, too.
If I could be sure that LuaJIT has access to all the memory the GC can efficiently handle I would be happy. Again the problem is that LuaJIT may only be able to allocate a few MB or even no memory at all on a desktop system where the user starts your LuaJIT(embedding) app after the first 2GB of address space are eaten up. And that scenario is becoming more and more likely every year because the RAM requirements of mainstream desktop systems certainly are not going down.
Right now I do not feel comfortable offering a LuaJIT app for download because it may crash immediately with an "out of memory" error when the user has some memory heavy app running in the background.
How does system memory usage stop Lua from addressing memory in its own address space? That doesn't at all fit with my understanding of how virtual memory works in modern operating systems.
It appears LuaJIT employs optimizations that assume that its pointers fit in 31 bits. This wouldn't be particularly surprising, fast interpreters often try and steal bits from pointers for lots of reasons, e.g. inlining values (so the indirection is eliminated), encoding the class type in the pointer (some JVMs have done this), etc.
LuaJIT stores non-number objects as NaN floating point values. (A NaN value has 48 bits of payload which can be set to anything.) LuaJIT uses 16 bits of that value for the type and the remaining 32 bits as payload, so pointers are limited to 32 bits.
If you control the application that is embedding LuaJIT, you should be able to reserve (but not commit) that address space early on in the application startup.
Other applications won't affect your own process's address space.
64-bit LuaJIT can only use memory allocated within the first 2 GB. Mainstream desktop systems consume more and more memory so by the time you fire up your LuaJIT app there might be (next to) no memory left within that range.
32-bit LuaJIT can use the first 4 GB but again sooner or later desktop systems will leave little there too + 32-bit code is already deprecated technology on the desktop.
Makes me damn sad really. Unfortunately making LuaJIT capable of using the full address space requires a major rewrite.. one which will likely never happen. LuaJIT's author has moved on to other projects and LuaJIT is now in maintenance mode, getting nothing but small fixes.
Seriously if you are a Mike Pall-grade elite low-level hacker (I am talking to all four of you ;) ) would you please continue LuaJIT development?