Time Nick Message 01:52 bgstack15 Multiplexing serverlist is even easier than contentdb. I've got a working model, but need to clean up code before adding it to the public copy of contentdbproxy. 01:58 bgstack15 I was thinking of renaming my project LuantiWebServicesProxy, lwsp. Because it would be a multiplexer of both ContentDB and Serverlist. A client can use either or both parts. 14:38 bgstack15 I got the LuantiWebServicesProxy repo up: https://codeberg.org/bgstack15/lwsp 14:40 pgimeno LuaJIT's garbage collector is completely unfit for a real-time video game 14:40 pgimeno Not true. First, it's the same as PUC Lua, so it makes no sense to single out LuaJIT here. Second, take a look at Gravity Circuit, an action platformer game made with Love2D. It certainly takes awareness of the issue in order to avoid unnecessary allocations (e.g. not creating functions on the fly, not creating ad-hoc single-use tables for parameters, etc.) and sometimes, manually running the GC or fine-tuning the GC parameters to be more aggressive. 14:43 repetitivestrain pgimeno: i don't know much about PUC Lua, but it is absolutely true. take any minetest game which is free of bottlenecks elsewhere and you will observe that long globalsteps are either dominated by garbage collection or by ABM execution. a good garbage collector such as exist in the JVM can sustain multiple gigabytes of allocations per second in tiny objects while incurring sufficiently little overhead to allow rendering 144 frames 14:43 repetitivestrain per second 14:45 pgimeno I'm horrified every time I look into any mod's code, because the mod programmers are totally unaware of this. That doesn't make LuaJIT unfit for a real-time video game. 14:45 sfan5 "you have to work around several core language features that will break your neck" is not what I'd call "fit for a real-time video game" 14:47 sfan5 but it's true that it's not impossible 14:49 pgimeno Lua is beginner friendly, but you can't expect a beginner to be able to pull a real-time video game. It takes a deeper knowledge of the language's limitations to become able to do that, is my opinion. 14:49 MTDiscord pgimeno: I'm not sure whether it is fair to compare 2d with 3d games. the latter usually require an order of magnitude (or more) more computational resources since many things scale cubically instead of quadratically. 14:49 repetitivestrain https://ibb.co/gbGX2HTG 14:51 repetitivestrain the language runtime on which a certain voxel game which shall remain unnamed is implemented, for instance, can sustain 2 GB/s of allocations and while constantly performing garbage collection without ever falling behind my display's refresh rate 14:51 repetitivestrain i would give almost anything for this degree of flexibility in my code 14:51 repetitivestrain (my lua minetest code, that is) 14:52 repetitivestrain because, when last i benchmarked a server with thousands of mobs and numerous additional objects, most of the garbage collection overhead was incurred by the engine allocating vectors and moveresult tables for on_step callbacks 14:53 MTDiscord Some Java GCs are crazy good, that is true 14:53 pgimeno @luatic there are 3D games made with love2d, you know... 14:54 sfan5 i think that shows more how much work went into the jvm gc and not that java is a good language for a real time game ;) 14:54 sfan5 repetitivestrain: a big problem for engine reuse is that mods can't promise not to modify a given argument 14:54 MTDiscord pgimeno: I'm aware, but how many are actually fast? Is there a single fast voxel game in love 2d? 14:54 sfan5 or well they can but there's no language feature to enforce this 14:55 repetitivestrain yes, exactly, and this is one of the many reasons i don't think lua is optimal for real-time video games 14:55 [MatrxMT] just push an error state back to Lua if they do >:^) 14:56 sfan5 lua is great for light business logic 14:56 pgimeno @luatic and Luanti mods are not a 3D game, the 3D part is the engine's task, leaving the scripting parts to Lua(JIT). And about your question: https://github.com/groverburger/lovecraft (again that's Lua fully dedicated to doing the 3D, which is quite a feat) 14:56 repetitivestrain but how performant is it? 14:57 sfan5 in any case we're kinda stuck with lua right now so if you want it fixed there better be an idea on how 14:57 MTDiscord Don't get me wrong, love2d is a lovely framework. But I do think (and have observed myself) that you'll run into performance limitations when you start doing computationally heavy tasks like mapgen. 14:57 pgimeno the program in that link is quite fast, 60fps in my slow card IIRC 14:57 sfan5 maybe ffi can fix this, somehow 14:58 repetitivestrain yeah if you read the context i mentioned these bottlenecks by way of encouragement for mod authors to be conscious of memory allocation 14:58 MTDiscord We should explore FFI for VoxelManip 14:59 MTDiscord A lazy API could just return nil for non LuaJIT, placing the burden of a reasonable fallback for PUC on the modder 14:59 sfan5 what would personally interest me is an answer to: what is the biggest current Lua/API bottleneck that is caused by the engine and cannot be worked around? 14:59 pgimeno ffi does go a long way with , see https://notabug.org/pgimeno/ffi_accel and #6863 which sadly was rejected 15:00 ShadowBot https://github.com/luanti-org/luanti/issues/6863 -- Add LuaJIT FFI-friendly memory-intensive functions 15:00 repetitivestrain avoiding allocations of temporary objects is not simple (and i speak from experience, having written a map generator which allocates nearly all arrays and data ahead of time, and allocates perhaps two or three dozen tables/mapblock for generation notifications, which is one factor behind its performing so well for a lua mapgen of its complexity) 15:00 pgimeno *with mapgen 15:00 repetitivestrain sfan5: the greatest mapgen bottleneck is not a bottleneck but a limitation that produces excessive memory consumption, and it is the absence of any means of accessing a voxelmanip's mapnode array directly from lua 15:01 repetitivestrain the greatest gameplay bottleneck is the sheer number of tables that are allocated for object on_step callbacks and the fact that numerous apis, such as object:get_pos, return vectors 15:02 sfan5 which of those two would you want fixed first? 15:02 repetitivestrain the mapgen issue 15:03 pgimeno #6863 + FFI would have totally mooted the first one 15:03 ShadowBot https://github.com/luanti-org/luanti/issues/6863 -- Add LuaJIT FFI-friendly memory-intensive functions 15:03 repetitivestrain i have mostly resigned myself to the other issue, though an object:get_pos_raw function which returns object positions on the stack rather than as a vector would be very helpful 15:04 pgimeno btw the code is still here: https://notabug.org/pgimeno/minetest/pulls/2/files 15:04 sfan5 could the engine simply return an ffi array and safely allow mods to work with it? 15:04 sfan5 optimizations that only work without mod security are basically useless 15:04 repetitivestrain sfan5: unfortunately the ffi is not secure at all 15:07 pgimeno sfan5: it already does, sort of, the problem is that the interface is C++ instead of C, which isn't accessible by FFI in practice, hence the PR 15:08 pgimeno yeah, an FFI array is not secure, it's a raw pointer 15:09 pgimeno even the ffi_accel mod, which tried to limit access through functions, is unsafe 15:09 repetitivestrain you could wrap it with a bounds check, but i think the debug.* functions could be abused to manipulate code in the process of performing such tests 15:10 sfan5 ah yeah the docs say so: >The FFI library provides no memory safety, unlike regular Lua code. It will happily allow you to dereference a NULL pointer, to access arrays out of bounds or to misdeclare C functions. If you make a mistake, your application might crash, just like equivalent C code would 15:10 pgimeno debug.* requires an insecure environment, right? 15:10 repetitivestrain i don't know 15:11 pgimeno anyway, I don't remember much but I do remember that I attempted to make ffi_accel as secure as was feasible 15:11 pgimeno but there were still security holes remaining 15:13 repetitivestrain ultimately with lua you have to cut your losses and either compromise on safety or obtain a better computer, and even then without really attaining performance comparable to the JVM (which is luanti's foremost competition i think) or native code with static memory management 15:14 sfan5 anyway that means ffi is off the table 15:14 repetitivestrain and that is why it is not optimal for real-time video games, though it is not that luanti has much of a choice in the matter 15:15 sfan5 next best thing I can think of is integrating wasm and then letting wasm code interact with API using native data types 15:16 repetitivestrain or maybe i'm making a mountain out of a molehill in insisting on <=50 ms globalsteps 15:17 repetitivestrain since the linux forks people have 1 second globalsteps as a matter of routine 15:17 repetitivestrain and they don't appear to have suffered much for it 15:17 [MatrxMT] lol it's true 15:18 repetitivestrain but mineclonia is generally more sensitive to server lag, as we have the lowest dtime target of any luanti game i've seen thus far 15:18 [MatrxMT] orwell had to work out how to make advtrains move reliably under server load. Railwaytime means that even if the server globalsteps are ramping up in length, the trains get a fair share of processing and don't get out of sync 15:18 repetitivestrain 50 ms 15:18 repetitivestrain and with dtimes above 75 ms mob physics begin to malfunction 15:19 repetitivestrain Blockhead256: how are 1 second dtimes tolerable though 15:20 [MatrxMT] well you see, for starters, my RTT to lifo is 330 ms 15:20 repetitivestrain i imagine it should produce plenty of jank when attached to entities and so forth 15:26 pgimeno I don't get why FFI is off the table and built-in mapgens not. FFI code can go out of bounds; so can engine code. If you trust the engine, you can trust a mapgen, right? 15:26 sfan5 no because the Lua mod code is untrusted 15:27 repetitivestrain well insecure-only optimizations would be acceptable for me too 15:28 repetitivestrain considering that i've already implemented several optional ones, more couldn't hurt 15:28 pgimeno well, it is untrusted unless the user trusts it... Android lets you run untrusted code too, you just have to accept permission. 15:29 repetitivestrain e.g. debug.upvaluejoin in combination with dumping and loading functions enables you to copy a luajit GCproto bytecode object and assign each closure a unique GCproto 15:29 [MatrxMT] the myth of consensual trust: User: I trust my OS. OS: I trust my user. Microsoft: Isn't there someone you forgot to ask? 15:29 [MatrxMT] also, isn't Android moving to block unsigned APKs? 15:29 repetitivestrain which prompts luajit consistently to specialize recorded instructions to the values of upvalues in the closure 15:30 pgimeno lol, power users are an endangered species 15:30 repetitivestrain and yields a healthy 10% speed improvement, but requires that mod security be disabled so that functions may be dumped and loaded from strings 15:30 sfan5 it can be put behind secure.trusted_mods for sure but then it's not useful to most users 15:31 sfan5 and we don't want to train users to grant full access to their computer to random mods "because it's too slow otherwise" either 15:31 repetitivestrain i'm a bit skeptical of that, since a map generator is unquestionably a very significant mod which users will display interest in optimizing 15:32 repetitivestrain particularly on large servers or slower computers 15:32 [MatrxMT] adding mods to the trusted list is one thing, but disable security entirely leaves them open to the next thing which they may not even realise needs security off to become malware 15:33 repetitivestrain unfortunately mapgen scripts cannot exploit the trusted list 15:33 repetitivestrain for core.request_insecure_environment just doesn't function in mapgen environments 15:33 repetitivestrain or doesn't exist, but i don't remember which 15:34 sfan5 ah yeah there was that 15:34 pgimeno maybe secure.trusted_mods could be maintained by a menu option to grant permission to a mod that requires it... e.g. declare in mod.conf if it is required, then when a mod requires insecure environment, ask the user whether to grant permission 15:35 pgimeno of course the dialogue to accept permission should be damn scary and encourage non-acceptance 15:35 repetitivestrain i am also _optionally_ addressing a number of luajit bottlenecks with the FFI 15:35 sfan5 pgimeno: yeah the risk of "what is this, let me click 'OK' to get rid of it" is very real 15:36 [MatrxMT] what about double-confirmation? 15:36 sfan5 by not having a consent dialog we're (for better or worse) we're avoiding that risk 15:36 [MatrxMT] or is a scary red dialogue ok 15:36 repetitivestrain if a server operator (such as yours truly) desires, it can be enabled by compiling a C file as a shared object, disabling mod security, dropping ffi.so into the mod folder, and toggling a setting 15:36 repetitivestrain https://codeberg.org/mineclonia/mineclonia/src/branch/mineclonia_mapgen/mods/MAPGEN/mcl_levelgen/ffi.c 15:37 sfan5 what are those __GNUC__ checks for? looks fishy 15:37 repetitivestrain GCC guarantees arithmetic rshifts 15:37 repetitivestrain while certain other C compilers don't 15:39 repetitivestrain the ffi itself is also used to eliminate array bounds checks and indirection while accessing the gradient list and indexing permutation arrays: https://codeberg.org/mineclonia/mineclonia/src/branch/mineclonia_mapgen/mods/MAPGEN/mcl_levelgen/noise.lua#L34 15:39 repetitivestrain which, when combined with the C module, nearly doubles performance 15:41 repetitivestrain also, gcc guarantees that lshifts on signed values operate on their binary representation 15:41 repetitivestrain and that overflows are well-defined as such 15:42 repetitivestrain "Bitwise operators act on the representation of the value including 15:42 repetitivestrain both the sign and value bits, where the sign bit is considered 15:42 repetitivestrain immediately above the highest-value value bit. Signed ‘>>’ acts on 15:42 repetitivestrain negative numbers by sign extension. 15:42 repetitivestrain As an extension to the C language, GCC does not use the latitude 15:42 repetitivestrain given in C99 and C11 only to treat certain aspects of signed ‘<<’ 15:42 repetitivestrain as undefined." 15:42 identity should have removed the newlines 15:42 repetitivestrain ugh, it didn't wrap properly 15:42 sfan5 I'm sure there are more portable and obvious ways to go about that 15:42 repetitivestrain i tried 15:43 repetitivestrain the portable approach is in the !__GNUC__ conditional 15:44 sfan5 casting to uint? 15:44 repetitivestrain not if i require sign extension 15:46 repetitivestrain *lx = (int) (hash << 22) >> 22; *ly = (int) (hash << 12) >> 22; *lz = (int) (hash << 2) >> 22; is equivalent to: arshift (lshift (pos, 22), 22), arshift (lshift (pos, 12), 22), arshift (lshift (pos, 2), 22) 15:46 repetitivestrain where pos is an aquifer position hash encoding a 10-bit signed offset from a base position 18:24 MinetestBot 02[git] 04SmallJoker -> 03luanti-org/luanti: CGUITTFont: Clean up, unify and comment draw code (#16380) 13b6a23b1 https://github.com/luanti-org/luanti/commit/b6a23b1bcccccafa7f124310398c0094cd313d0e (152025-09-10T18:23:45Z) 18:24 MinetestBot 02[git] 04sfan5 -> 03luanti-org/luanti: Add exclude_player to particle spawners 13f714ac0 https://github.com/luanti-org/luanti/commit/f714ac06116b1725c656aa226688432811cb6f04 (152025-09-10T18:23:55Z) 18:24 MinetestBot 02[git] 04sfan5 -> 03luanti-org/luanti: Send node dig particles to all other players 13fcd9c73 https://github.com/luanti-org/luanti/commit/fcd9c73f2f63f93e8d7a0f07ae605fa3d0d467b3 (152025-09-10T18:23:55Z)