| Time |
Nick |
Message |
| 00:28 |
|
Sharpman joined #luanti-dev |
| 03:34 |
|
imi joined #luanti-dev |
| 03:59 |
|
crazylad joined #luanti-dev |
| 04:05 |
|
SFENCE_arch joined #luanti-dev |
| 04:21 |
|
Alias joined #luanti-dev |
| 04:25 |
|
SFENCE joined #luanti-dev |
| 04:48 |
|
vampirefrog joined #luanti-dev |
| 05:02 |
|
MTDiscord joined #luanti-dev |
| 05:43 |
|
SFENCE joined #luanti-dev |
| 06:07 |
|
YuGiOhJCJ joined #luanti-dev |
| 06:29 |
|
BuckarooBanzai joined #luanti-dev |
| 08:03 |
|
alek13 joined #luanti-dev |
| 08:11 |
|
alek13_ joined #luanti-dev |
| 10:26 |
|
cheapie joined #luanti-dev |
| 10:58 |
|
pgimeno joined #luanti-dev |
| 11:52 |
|
wsor4035 joined #luanti-dev |
| 12:19 |
|
AliasStillTaken joined #luanti-dev |
| 12:21 |
MTDiscord |
<rudzik8> I think it's reasonable to take either the Blender or the Fedora one and add that no media file should be AI-generated (that is, images/textures, audio files, etc), only code, docs and other text. I do not, however, think it's reasonable to allow any AI assistance in reviewing contributions, as I believe I'm not alone in thinking that it's a kind of last line of defense (as a passing review generally means approval). IMHO that last line |
| 12:21 |
MTDiscord |
of defense should be done by humans, as LLMs (esp ones not configured properly, but even then there are going to be issues) are prone to hallucinate problems where there are none and miss the ones that are less prominent in their data sets |
| 12:23 |
MTDiscord |
<rudzik8> by 'reviewing contributions' I mean the classical Gerrit-esque code review process on platforms like GitHub, Gitlab, Forgejo & descendants, etc. I don't think automatic labelling/assignment bots are what the Fedora AI policy refers to in its Contribution & Community Evaluation clause |
| 12:28 |
celeron55 |
i don't think it's sustainable to generate code using an LLM and review it by humans without the help of LLMs |
| 12:29 |
celeron55 |
luanti has been bottlenecked by reviews for 15 years already |
| 12:29 |
celeron55 |
and that's with human generated code |
| 12:29 |
|
AliasAlreadyTake joined #luanti-dev |
| 12:32 |
celeron55 |
i mean, soon humans will give up if they don't get help, and then there will not only be LLMs reviewing the code, but additionally no humans at all |
| 12:34 |
MTDiscord |
<rudzik8> I'm just afraid that all LLMs fall to garbage-in-garbage-out and that we may get situations like with this famous post, but for code contribution/review: https://www.threads.com/@woodekobina/post/DU3zO90DkrG/media |
| 12:34 |
MTDiscord |
<rudzik8> :juanchi_face: oh I wish xdddd |
| 12:35 |
sfan5 |
just like how AIs can help with coding but not replace it, they can help with reviews but not replace them |
| 12:37 |
MTDiscord |
<luatic> Agreed |
| 12:38 |
MTDiscord |
<rudzik8> so for coding, we'll be welcoming AI coding with human supervision, and for code review, we'll be welcoming AI supervision with human supervision..? |
| 12:38 |
MTDiscord |
<luatic> Or human supervision with AI as a fallback |
| 12:39 |
MTDiscord |
<rudzik8> AI-assisted coding sessions already are following the typical code review process, just more instant, with an LLM coder and a human reviewer pointing out errors |
| 12:39 |
MTDiscord |
<luatic> We could presumably use AI to catch mistakes in addition to comprehensive human review |
| 12:44 |
celeron55 |
how i think of it is you do a partial review by LLM and partial review by human, each focusing on their strengths, and as a result the human has less work to do and can focus on the things that aren't too inhumane (like kipping trying to look for typos, which the LLM will easily catch) |
| 12:44 |
celeron55 |
skipping* |
| 12:45 |
MTDiscord |
<rudzik8> I can tolerate having AI as a fallback, as long as its suggestions are taken with a heavy grain of salt, no matter how much they satisfy the token predictors in the human brain, and are checked against what's in the actual contribution first |
| 12:46 |
MTDiscord |
<rudzik8> grammar and spelling in docs/comments/naming -- sure, that's all about patterns, a thing that LLMs are reasonably good at |
| 12:47 |
celeron55 |
code is often a matter of patterns |
| 12:47 |
celeron55 |
it's the architecture of the code that the LLM is bad at reviewing |
| 12:48 |
celeron55 |
and things like potential licensing issues, and other things like that that really aren't internal to the code and require being aware of a wider context |
| 12:51 |
MTDiscord |
<rudzik8> agreed on architecture. LLMs will also quite frequently decide to make more branchfuck, monkey-patch changes to code unless instructed to build on architectural/design specs, which will (at least for now) require a clever human to come up with |
| 12:53 |
celeron55 |
also code duplication will happen without human oversight, especially if the code isn't organized very well |
| 12:53 |
MTDiscord |
<rudzik8> there are patterns in code, but I guess there's just too much shitty code and bad commits out there for LLMs to be trained on, not to mention that they still have all the natural language stuff and will hallucinate variable/function names, esp in a narrow use case such as Luanti |
| 12:54 |
MTDiscord |
<rudzik8> though strict integrated linting/tests can combat some of that |
| 12:54 |
whosit |
are there any public examples of projects using AI reviews successfully? To me it feels like LLMs generate walls of text and you'll need another AI to review the reviews, but I haven't looked too deep into this :p |
| 12:54 |
celeron55 |
yeah the problem with luanti for LLM coding is that the LLM has a really hard time testing its work |
| 12:55 |
celeron55 |
whosit: i use github copilot for speeding up code reviews at work. but it's propreitary code so i have nothing to show |
| 12:56 |
MTDiscord |
<rudzik8> waiter, waiter! more markdown headings and lists please 🍽️ |
| 12:57 |
celeron55 |
anyway, on the generation side, you can instruct an LLM to be less verbose, or use specific style, or target a specific audience and so on. it's all about defining the task |
| 12:58 |
celeron55 |
an LLM can generate content even without being asked to, but you shouldn't let it do that. always have it generate what you want |
| 13:00 |
whosit |
looking into it, I found stuff like this: https://github.com/pingdotgg/uploadthing/pull/1247 |
| 13:00 |
whosit |
(found among links here https://github.com/coderabbitai/awesome-coderabbit?tab=readme-ov-file#projects-using-coderabbit ) |
| 13:00 |
whosit |
is it good? I can't tell X) |
| 13:01 |
whosit |
after my encounter with the thing here: https://github.com/C-C-Minetest-Server/player_settings/pull/3 |
| 13:01 |
celeron55 |
i think i could request a review to a selected Luanti PR, if we want to see how it looks like |
| 13:02 |
celeron55 |
from copilot, that is |
| 13:03 |
MTDiscord |
<rudzik8> as an isolated test, sure, but I'm against getting dependent on products like GitHub Copilot. shouldn't need to elaborate why that is |
| 13:04 |
celeron55 |
it's not like it would be written as a requirement to any proecss. it would just be an allowed tool, to be explicitly called by a human reviewer |
| 13:07 |
MTDiscord |
<rudzik8> it's not like Luanti has that many human reviewers (the bottleneck huh?). if you bring crack cocaine to a tribe that has tried none, it's natural that with time they'll all get addicted to it, and then you're suddenly dependent on how the crack cocaine company feels about your project |
| 13:08 |
MTDiscord |
<luatic> meh it's not like anyone will forget how to review code because they invoked copilot or whatever |
| 13:09 |
MTDiscord |
<luatic> what I'm more worried about is annoying contributors with AI slop |
| 13:10 |
MTDiscord |
<luatic> also, we'd probably need explicit permission to feed PRs into AI at all |
| 13:10 |
sfan5 |
in order for the AI to properly review engine PRs I assume we need to write more and better documentation |
| 13:10 |
sfan5 |
because there's too much knowledge on the right way to do stuff only contained inside the coredev's heads |
| 13:11 |
MTDiscord |
<luatic> true |
| 13:11 |
MTDiscord |
<rudzik8> they won't, but it's an opaque box that emits colorful gobledygooque designed specifically to make you feel like good stuff is happening because of it. I'd be more comfortable with such boxes at least being transparent (open-source), if requiring some community-funded LLM compute/API access to run, as long as it doesn't risk the PR process getting dependent on Microsoft or some shitty bubble startup that may fail in half a year without |
| 13:11 |
MTDiscord |
revenue and sabotage stuff |
| 13:11 |
MTDiscord |
<luatic> lets AI generate docs so we can feed the slop machine it's own slop! what could go wrong |
| 13:12 |
MTDiscord |
<rudzik8> this is all of course future thinking, but isn't that what this discussion is about? |
| 13:12 |
sfan5 |
re AI policy: the big question is what is the problem that we want to solve with the policy? |
| 13:14 |
MTDiscord |
<luatic> Make our expectations for AI assisted contributions clear |
| 13:14 |
MTDiscord |
<bla8722> like I said before the numbers of AI generated PRs, no matter if slop or just with assistance, will increase anyway. core devs need to decide on a level that is acceptable for them and if they are also ok with using a LLM as review tool(no matter which) and do a testrun. Everyone else can simply fork and use anything they like on their fork |
| 13:14 |
MTDiscord |
<rudzik8> the problem that we don't have a specific enough contributor policy rn on generative AI further than "don't hide it pls" |
| 13:14 |
MTDiscord |
<luatic> You could argue that this ought to be implicitly clear, but explicit is better than implicit. |
| 13:15 |
sfan5 |
https://github.com/mpv-player/mpv/blob/master/DOCS/contribute.md#ai-assisted-contributions here's another data point for a policy |
| 13:16 |
sfan5 |
but the fedora one looked good too |
| 13:16 |
rubenwardy |
I think the bigger issue with using LLMs on our code is the lack of architecture or testing discipline |
| 13:17 |
MTDiscord |
<rudzik8> > This also includes confirming that the code can be submitted under the same license as the relevant files (usually LGPLv2 or GPLv2). I guess it's reasonable that the submitter should first check for blatant plagiarism from the LLM via things like GitHub global code search. we might want to shine light at this too |
| 13:20 |
sfan5 |
while not having a policy is something we should address, "LLM PRs will lead to broken code" or "there are too many slop PRs" are examples of problems I meant in my question |
| 13:26 |
MTDiscord |
<rudzik8> on plagiarism: it is possible to prompt LLMs to extract "memorized" copyrighted works like the first book of Harry Potter -- https://arxiv.org/abs/2505.12546 (2025!). I also recall seeing examples of coding-specific LLMs reciting parts of the Quake source code 1:1 without even being asked to not that long ago |
| 13:28 |
MTDiscord |
<rudzik8> and by 'reciting' source code I mean including verbatim copies of it in the code the LLM generated |
| 13:29 |
Sheriff_U3 |
I wonder when they'll teach the AI's to respect Copyright law... |
| 13:31 |
MTDiscord |
<rudzik8> although it does appear that there are solutions to memorization in LLMs: https://arxiv.org/abs/2505.13171 but this is already going a bit offtopic as (a) I don't know how many LLMs widely available right now may already implement this (most certainly not a lot), and (b) it's unclear how much time it will take for every other model to catch up lol |
| 13:31 |
sfan5 |
are there examples of FOSS projects that ban LLM code due to copyright concerns? |
| 13:32 |
|
Juri joined #luanti-dev |
| 13:34 |
MTDiscord |
<rudzik8> on my last message: that's not what I wanted to link but still relevant, that study focuses on evaluating those risks, not resolving them |
| 13:37 |
|
SFENCE_arch joined #luanti-dev |
| 13:39 |
MTDiscord |
<rudzik8> sfan5: at least gentoo linux (https://lwn.net/Articles/970072/) and netbsd (https://mastodon.sdf.org/@netbsd/112446618914747900) out of the big ones |
| 13:43 |
MTDiscord |
<rudzik8> some news articles like https://www.theregister.com/2025/09/03/freebsd_project_update_no_ai/ claim that FreeBSD banned genAI outside of docs/translations citing their report (https://www.freebsd.org/status/report-2025-04-2025-06/#_freebsd_core_team), but it only actually states that the core team is considering coming up with such a policy (notice the stretch?), and the contributors' guide |
| 13:43 |
MTDiscord |
(https://docs.freebsd.org/en/articles/contributing/) does not have any details on AI/LLMs usage as far as I can tell |
| 13:45 |
MTDiscord |
<rudzik8> (ironically enough, this stretch may come from the authors of said articles using LLMs themselves for research and fact-checking; ChatGPT when asked about this also first states that the FreeBSD project enforces such a ban, citing articles as well as the contrib guidelines :juanchi_face:) |
| 13:49 |
|
SFENCE joined #luanti-dev |
| 13:50 |
MTDiscord |
<rudzik8> as for fellow game engines, Bevy also enforces a copyright-driven ban on genAI: https://bevy.org/learn/contribute/policies/ai/ > Until there are well established answers to [questions about legal status of AI-generated/assisted works], the use and/or distribution of AI-generated code and assets may constitute copyright infringement or may be subject to licensing terms incompatible with the FOSS licenses used by the Bevy Organization. |
| 13:50 |
MTDiscord |
they also cite the US Copyright Office stating that "human authorship is a prerequisite to copyright protection" (https://www.copyright.gov/rulings-filings/review-board/docs/a-recent-entrance-to-paradise.pdf) |
| 13:51 |
|
SFENCE joined #luanti-dev |
| 14:33 |
|
SFENCE joined #luanti-dev |
| 14:59 |
|
SFENCE joined #luanti-dev |
| 15:53 |
|
crazylad joined #luanti-dev |
| 15:53 |
|
crazylad joined #luanti-dev |
| 15:53 |
|
AliasAlreadyTake joined #luanti-dev |
| 16:22 |
|
sfan5 joined #luanti-dev |
| 17:25 |
|
Desour joined #luanti-dev |
| 21:00 |
ShadowNinja |
rubenwardy: I like the idea of having something like an api_version in mod.conf. You could also have a require_trusted_env setting api v2 mods would get their own lua_State and trusted ones would just get the insecure environmet loaded as their global environment without any need for request_insecure_environment, we could then deprecate request_insecure_environment and require api v2 for trusted mods. |
| 21:05 |
ShadowNinja |
Handling the insecure environment is tricky, I more recently took a look at the IRC mod and found a couple exploits where the env could be stolen by an untrusted mod, and if even the author of the mechanism can't get it right while being reasonably careful we can't really expect third party mods to be careful enough about the insecure environment. |
| 22:02 |
MTDiscord |
<luatic> related: #12182 |
| 22:02 |
ShadowBot |
https://github.com/luanti-org/luanti/issues/12182 -- Security: Load trusted mods first to provide them with a clean environment |
| 22:02 |
MTDiscord |
<luatic> API levels + truly separate environments would probably be the cleanest solution. Also goes well with require :) |
| 22:04 |
sfan5 |
the insecure env was a historical mistake |
| 22:10 |
ShadowNinja |
luatic: That would help a lot. I'm looking at the IRC mod now and it's tricky to patch out the issue when you're requiring libraries that are requiring other libraries and you have to wory about any global function that you call being potentially overridden with something that could use require to escape the sandbox. Kind of just another hack to keep the security hack working though, we really need separate lua_States for a clean solution. |
| 22:38 |
|
SFENCE joined #luanti-dev |
| 23:32 |
|
panwolfram joined #luanti-dev |
| 23:42 |
|
YuGiOhJCJ joined #luanti-dev |
