Time Nick Message 01:09 [MatrxMT] It would be useful if custom mapgens could implement get_biome_data and provide biome maps from Lua 01:09 [MatrxMT] Not necessarily in advance 05:14 MTDiscord yeah true 08:32 Krock turns out that mip mapping is disabled on RTTs. After enabling, the nametag scaling looks a lot better: https://i.postimg.cc/gjwbDzSG/out.png 08:32 Krock this is about the best result that I could achieve so far 08:53 Farooq Hello. I just realized something and I wanted to talk about my concern. I realized now we've got core.find_path. I became both happy and worry. I am happy because in CTF, we needed such a thing. And this might be useful. But I'm worried because I think Luanti is going the wrong direction. I have no experience working on the engine development. But I thought the goal is dehardcoding things. But adding such these features goes right in the opposite 08:53 Farooq direction IMO. I might be wrong. But a small engine which you can add modules and plugins to it IMO is much better than having a big engine with tons of features. Because usually each game uses only a small subset of these features. 08:55 Farooq In the last 2 roadmap brainstorms, dehardcoding has been recommended multiple times in each. Also a feature to load dynamic modules has been suggested several times. 08:55 Farooq For instance extending the Lua API with custom Rust/C/C++ modules or libraries. 08:57 Farooq maybe this should be discussed in a Github issue. Let me know and I will open an issue or contribute to an already existing one 09:00 Farooq As a reminder, many big and popular servers are already using their custom fork of Luanti 09:02 Krock Farooq: custom modules are either impractical (needs compiling), or insecure (cannot easily be sandboxed to prevent malicious code) 09:02 Farooq server admins are already using their custom fork 09:02 Farooq some server owners 09:03 repetitivestrain And? 09:03 Krock yes. what's the problem with that? 09:03 repetitivestrain if mine is any indication, i doubt they've diverged much 09:03 repetitivestrain you can already extend minetest with C other native modules anyway by means of lua's standard module interface or the LuaJIT FFI 09:03 repetitivestrain see here: https://codeberg.org/mineclonia/mineclonia/src/branch/mineclonia_mapgen/mods/MAPGEN/mcl_levelgen/ffi.c 09:03 Farooq not anything fits within Lua 09:03 Krock Look at the Wine project. I don't think there's many people who use the official builds, and instead add patches to improve compatibility with certain games. 09:05 Krock the goal of Luanti should be the same: to provide a generic API that fits well enough to cover most use-cases. If server owners still need more, they are free to suggest certain API, implement them in Lua, or use the insecure environment 09:05 Krock to import a C/C++ library with Lua exports 09:05 Farooq repeating myself, not anything fits as a mod 09:05 Farooq otherwise server owners wouldn't go for patching the engine 09:05 repetitivestrain well to take your example of core.find_path, mineclonia has had a very efficacious lua pathfinder for nearly half a year 09:06 Krock Farooq: http://lua-users.org/wiki/BuildingModules 09:06 repetitivestrain and there's been absolutely no need to patch the engine 09:06 Krock alternative link: https://piembsystech.com/integrating-lua-modules-into-c-programs/ 09:06 Krock oops scratch the 2nd one. it's the wrong direction. 09:08 Krock code/library example for a C library exposed to Lua, which can be used within Luanti: https://github.com/lunarmodules/luasocket 09:09 Krock repetitivestrain: I regularly patch it to get the 3D buttons back 09:09 repetitivestrain yes and i've a series of patches that implement apis enabling CSMs to override client object and local player physics 09:09 Krock and some servers tweak their anticheat code for various reasons 09:10 repetitivestrain which are used by this quasi-official mineclonia CSM: https://codeberg.org/halon/mcl_localplayer/ 09:10 Farooq Krock, repeating myself for the third time, not anything fits as a Lua mod for Luanti. I know all of them are possible. But there are certain cases in which the server owners need to patch the engine 09:10 Krock Farooq: so what? 09:11 repetitivestrain generally in my experience it's the client that must be patched 09:11 Krock repetitivestrain: hehe I've also got a set of patches ready to test mods against vulnerabilities :3 09:12 Farooq Krock, In light of my idealism, the engine shouldn't need to be patched at all. If that "ideal" is not possible. I think Luanti should go for the way which requires least number of patches 09:12 repetitivestrain Farooq: what are your proposals? 09:12 repetitivestrain why don't you submit them individually 09:13 Farooq wdym? 09:13 repetitivestrain there is not and cannot be a one size fits all C++ API, you know 09:13 Farooq I don't follow. What doy you mean by all C++ API? 09:13 Farooq s/doy/do 09:14 [MatrxMT] 100 prs on the luanti repo yet again 09:14 repetitivestrain you were proposing to introduce facilities for loading native modules, but the material question is what APIs do you need which are not currently available to Lua 09:14 Krock birdlover32767: review them 8) 09:14 Farooq repetitivestrain, nope that's only half of my proposal 09:14 repetitivestrain What's the rest 09:14 Farooq the other half is that the engine should provide the least 09:15 Farooq that is there shouldn't be core.find_path 09:15 Farooq unless it is loaded by a dynamic module on demands of server owners 09:15 repetitivestrain How does core.find_path interfere with your (very general) proposal 09:15 repetitivestrain you are under no obligation to use it and serious Luanti mods do not exercise the built-in pathfinder anyway 09:15 Farooq my proposal is a very small engine which can be extended to whatever you want using native or non native modules 09:16 repetitivestrain and how precisely would you set about realizing such an engine 09:16 repetitivestrain the devil is in the details 09:16 Krock ABI compatibility is annoying to deal with 09:16 repetitivestrain what apis would you expose to lua and native modules? 09:17 Farooq repetitivestrain, it seems I need to really write a detailed proposal which doesn't fit in chat 09:18 Farooq for example there shouldn't be APIs in core namespace about "trees" and "ores" 09:18 Farooq I understand my idea is not yet well formed. But instead of immediately rejecting my idea, please iterate over it and improve it 09:18 repetitivestrain look i've just one question, how would you render the engine sufficiently extensible for these prospective modules to implement https://github.com/luanti-org/luanti/pull/16429 09:19 repetitivestrain which is the only reason why i continue to patch my server 09:19 Farooq lemme open it. I don't have any experience with Graphics 09:20 Farooq the only potentional drawback I see in my idea is what Krock said about ABI compatability. It's something I have never worked with outside Linux and x86_64 09:20 repetitivestrain as regards the rest, the map generator on my server does not exercise the built in decoration system, l-system trees, the built-in ore generator, any of the built-in ValueNoise facilities, etc. 09:20 repetitivestrain in a word, it is completely indifferent to the existence of the facilities you propose to remove 09:21 Farooq exactly! 09:21 repetitivestrain why then is it necessary to _remove_ existing facilities the better to support lua map generators? 09:21 repetitivestrain or other programs which do not require them? 09:21 repetitivestrain i don't see the imperative here 09:23 Krock also consider that every module would come with a significant overhead of imported symbols (function/variables), either from the C/C++ standard library or the Luanti engine 09:24 Farooq nope I'm not proposing "removing" them. I'm proposing that ideally, the engine is modular. And these are modules like other modules in the game. The engine can come with a starterpack of these modules if most games require it. But there should be also a possibility for people like you to not have them when they don't need them. 09:24 repetitivestrain why not 09:24 repetitivestrain it's plenty of complexity for questionable or little gain 09:24 Krock and if C++ is used, then most of the functions get baked into the module - again. this includes the std::vector, std::map and whatnot implementations. especially stringstream can bloat it considerably. 09:25 Farooq Aren't these part of standard C++ library? 09:25 Krock the C++ standard library is mostly headers-only. 09:25 repetitivestrain and such facilities as would actually require server or mod implementers to patch the engine couldn't possibly be accommodated by a general purpose and stable module API anyway 09:26 Farooq I mean isn't that a dynamic library. If my map generator X requires standard C++ library and so does the engine, it is loaded once. Isn't it? 09:26 repetitivestrain see https://github.com/luanti-org/luanti/pull/16429 09:26 Farooq Unless you are using a static library 09:26 repetitivestrain no, the C++ stl is inlined into its consumers 09:27 Farooq Note that I haven't got almost any experience with C++ 09:27 repetitivestrain a copy of a substantial amount of code generally exists in every object file which includes it 09:27 Farooq That's a problem. But also consider that your custom map generator module might not be written in C++ 09:28 Krock Farooq: consider decompiling Luanti, preferably with debug symbols. A lot will be generated code from the C++ headers 09:29 Krock s/A lot/A portion/ 09:30 Farooq Something here I don't understand. I haven't worked with C/C++ for so long. But as far as I remember, headers are (mostly) not the code to be executed. Am I right? 09:31 Krock https://stackoverflow.com/a/3482966 09:32 Farooq yeah exactly. except I didn't know the difference is cultural 09:35 Farooq Do you think core developers would welcome a PR or at least a suggestion for experimentally adding, loading and executing dynamic modules for the patches of JMA server 09:35 Farooq s/server/server?/ 09:35 Farooq my mind is now fried and at the same time I need food 09:35 Farooq bbl 09:38 MTDiscord Farooq, in C++, header files also contain a substantial amount of source code sometimes. 09:43 MTDiscord However, for your purposes, I do not think that is necessarily a problem. You might want to remove unnecessary code from the binary, but it looks like that is doable. 09:45 MTDiscord Loading dynamic modules in the engine might be perceived as a risk for general users. Perhaps general users aren't responsible enough to avoid loading up dynamic modules that happen to be malware. 10:42 MTDiscord sfan5: specifying humidity/heat map would certainly be useful... even more useful would be ability to specify a general heightmap that would offset the noise 10:49 Farooq josiah_wi, I agree about the risk. I was more thinking to servers than clients. Another option might be WebAssembly which is said to have only 10% overhead comparing to native. And also so many languages can be compiled into it. 10:56 Farooq If I limit scope of what is needed, we are able to judge things better. For instance let's consider any module which needs to do read information from Luanti like map, players, chat and so on, compute and return something. For let's consider core.find_path(). If it can be implemented in Lua without trouble and hassle and with good enough performance, then it shouldn't have been added to the engine. Now for the sake of thought experiment, let's consider 10:56 Farooq it's not possible. The engine should have some mechanism of loading an external native or WASM code, give it the information it wants and read the information it has computed from it. One way for sure is dynamic libraries. Another way is TCP/UDP sockets. Another way is standard stdout and PIPE. 10:57 Farooq For now I want to add this thing to my TODO list. In future, I would check why JMA server requires patches to the engine. And how a modular mechanism could be implemented. 11:00 Farooq I want to get my hands dirt with engine code for a very long time. And now I have the excuse to do it. 11:10 sfan5 03:09 <[MatrxMT]> It would be useful if custom mapgens could implement get_biome_data and provide biome maps from Lua 11:11 sfan5 it sure would but pretty much anyone at any time can call core.get 11:11 sfan5 _heat or get_humidity and funelling those calls into callback in a single thread(?) is annoying and also a bottleneck 11:11 sfan5 hence the suggestion to specify everything ahead of time 11:11 repetitivestrain well my biome system has proved efficient enough to support all uses of get_biome_data that exist in mineclonia 11:12 repetitivestrain and it does run independently in mapgen, async, and server environments 11:12 repetitivestrain just my two cents as someone who has recently been implementing custom biome systems 11:13 sfan5 which resolution does your biome system need to work nicely? as in "I can decide the biome every N nodes" 11:13 sfan5 Do you think core developers would welcome a PR or at least a suggestion for experimentally adding, loading and executing dynamic modules for the patches of JMA server 11:14 repetitivestrain sfan5: biome positions are quantized to multiples of 4 11:14 repetitivestrain and it's the same in minecraft for what it's worth 11:14 sfan5 without knowing what exactly JMA needs this sounds prone to ending up with a bunch of hooks that are useful to them but nobody else 11:15 sfan5 repetitivestrain: sure but what is the least resolution that is acceptable? 11:15 Farooq sfan5, I should really write a detailed proposal 11:15 Farooq It seems people in chat don't understand my point 11:16 repetitivestrain sfan5: i'd have to say 4x4, perhaps 6x6, since biomes must follow the contours of rivers and other water bodies, and 4x4 already cuts it rather fine in that regard 11:18 sfan5 6x6 resolution would require 230 MB to specify it ahead of time for the entire world 11:19 sfan5 which is not clear if it's practical for you anyway 11:19 repetitivestrain i suspect it would take much longer to generate that much data 11:19 repetitivestrain i can actually attempt to measure this, one moment 11:20 repetitivestrain (and note that biomes are partially 3d too, which get_biome_data can account for but the current biomemap system cannot) 11:22 Krock is it necessary to define ahead of time? 11:23 Krock I mean the entire map. chunk-wise could work too. 11:25 sfan5 core.get_heat() can be called at any time and that was my idea to fix this in an uncomplicated way 11:26 sfan5 but I guess that doesn't work 13:06 luatic !tell Farooq (1) core.find_path has existed for what, a decade? and barely been touched. it's not a new addition. (2) new additions *are* adding lower level, general purpose APIs. we *are* dehardcoding. (3) esp. minor forks are fine. (4) if modsec is off, you can load dynamic libraries from lua. 13:07 sfan5 use ~tell here 13:07 luatic ~tell Farooq (1) core.find_path has existed for what, a decade? and barely been touched. it's not a new addition. (2) new additions *are* adding lower level, general purpose APIs. we *are* dehardcoding. (3) esp. minor forks are fine. (4) if modsec is off, you can load dynamic libraries from lua. 13:07 ShadowBot luatic: OK. 14:10 repetitivestrain sfan5: generating biomes for all of one map took 1 hr and 14 minutes, but only the 2d version 14:10 repetitivestrain on a single thread, which is all that's feasible in lua 14:11 repetitivestrain however, since the depth value varies by the base terrain offset noise, it would be necessary to sample this noise vertically at increments of 4/6 nodes too for each column, to handle underground biomes and suchlike 14:32 Farooq luatic, thanks for letting me know. I never saw it before 14:55 MTDiscord For the record I don't personally hold the opinion that users should be prohibited from downloading and running C++ modules for their own protection. 15:06 luatic Ultimately they are not - if they trust a mod, or disable mod security, they can load shared libraries from Lua, for example - but sandboxing is a legitimate feature of Lua. 15:06 repetitivestrain i think szutilpack is on contentdb and also provides native modules 15:07 repetitivestrain although users are required to compile them themselves 15:07 repetitivestrain also core.find_path hasn't been touched because it's never worked 15:07 repetitivestrain do your users a favor and avoid it like the plague