Time Nick Message 02:50 MTDiscord @ExeVirus is https://pad.riseup.net/p/Luanti-Docs entirely from yesterday or am I only seeing that because I first opened it yesterday? 03:25 MTDiscord No it's from like 15 days ago 03:26 MTDiscord Though Acorp has been active throughout 03:33 MTDiscord OK, good to know 🙂 here are my thoughts for now, I'll probably keep chipping away for the next few hours too: https://github.com/luanti-org/docs.luanti.org/issues/296#issuecomment-3704389640 TLDR: Markup languages rejected, they're not the right canonical form for a user guide. Markdown continues to be my leading approach. I think it will always have the best ease of use. Even if we were to magically fork LuaLS and LuaCATS and 03:33 MTDiscord add every feature we wanted, it'd be weird for doc writers to have to start every line with a --- or deal with "comment" sections having different syntax highlighting colors in IDEs. In short, if I'm going to write a comprehensive guide on how to use a system, I'm going to use Markdown for at least most of it, and then a mature doc-generation system for the "core" of it (documenting types of args, etc.) 03:35 MTDiscord rust isnt an option? heresy 03:36 MTDiscord jokes aside, thanks for making the overview. 03:45 MTDiscord https://github.com/luanti-org/docs.luanti.org/issues/296#issuecomment-3704401989: At this point I'm very comfortable saying we should go with Markdown. The more I think about it, the more I realize this API documentation is primarily natural language and should be written in a way that best supports long sections of natural language. - Markup languages: They can do this, but often have syntactic overhead that decreases ease of 03:45 MTDiscord use, especially deep nesting and repetitive boilerplate. - Symbol typing systems (LuaLS addon, Luau, TypeScript, etc.): These are less conducive to long paragraphs (manual line breaks + IDEs often color all comments as fully green, not showing e.g. lists or code fences separately) and can be especially annoying when trying to add in characters that are reserved by the typing system itself (e.g. adding **/ as end-user text in a TypeScript type, or 03:45 MTDiscord ---@class in LuaCATS) That only leaves Markdown 🙂 it's the industry standard for doc sites like learn.microsoft.com, docs.luanti.org, readme files across the internet, and many, many more. The only downside is that we'll have to define a schema and write our own tools. However, work on defining the schema has already begun with #297 and we have https://github.com/remarkjs/remark to help us out 🙂 I really don't have any reservations about 03:45 MTDiscord our ability to deliver an extremely high-quality doc writing experience with this toolchain, and I haven't seen much resistance to this approach in all our conversation in Discord/IRC or other comments on GitHub. Please let me know your thoughts! 03:46 MTDiscord could abusin mdx perhaps help? 03:47 MTDiscord basically custom comments for important bits like what function it is, params, etc. but inside of them can be markdown 03:47 MTDiscord dunno, just a quick thought, i might be stupid 03:47 MTDiscord haha dang you reminded me of MDX. It's the only contender 😄 fortunately it's a superset of Markdown. I'll think about it, but let's say yeah, if we go with Markdown, we can gradually add in MDX feature later. The people behind remark are called the unified collective, and they're also the maintainers of MDX, so all the tooling will transfer 🙂 03:51 MTDiscord IRC: s/MDX feature/MDX features 03:58 MTDiscord Hurm, anything MDX provides I suspect we can provide in code blocks. Like what specifically do we need from essentially a full programming language in our docs? 03:58 MTDiscord MDX or no, a schema is a must to get things in order 03:58 MTDiscord lol, no 03:58 MTDiscord It's second to format. If we can decide on formatting, making a parser and schema to enforce is pretty trivial 03:59 MTDiscord I'll accept the lol, and the no, but explain what we use it for?? 03:59 MTDiscord https://discord.com/channels/369122544273588224/926231483155378176/1456494039259152404 04:00 MTDiscord mdx is for components in markdown. code blocks are code non functional in markdown 04:00 MTDiscord Oh, gotcha, yeah I'm basically expecting us to make a DSL format within markdown that's parsable 04:00 MTDiscord You are expecting that might take the form of something like MDX, and I'm thinking it may not 04:01 MTDiscord the idea is so your could do something like core.luanti_bikeshedding something *luanti* issue tracker is all about ~~bike~~shedding 04:02 MTDiscord Right, lemme grab something from the rise up that matches that 04:03 MTDiscord for the irc plebs, ive edited the example, and im far to lazy to keep updating you on your ancient stuck in the past platform 04:04 MTDiscord #### `node = core.get_node(pos)` Returns the node at the given position. - **args** - pos : {Vector} : position of the node on the map - **return** - node : {MapNode} : name is "ignore" for unloaded chunks - **env** - server-main - server-async - client - server-mapgen 04:05 MTDiscord Same kinds of ideas here, I'm just leaning towards using pure markdown for the solution instead of a system. Though obviously our types are using a { tag } in this example, so it's not quite clean 04:06 MTDiscord the nice thing about mdx is if we are rendering this and showing it, we can add in components for whatever, while the schema ones exist for ast compiling to extensions, and other jazz 04:06 MTDiscord say we want to add interactive demos or some jazz 04:07 MTDiscord oh, we want that kind of formatting embedded in our docs? I was thinking it would be pure human readable in plaintext, good rendering when viewed in html (markdown render), good machine parsability (markdown can directly translate to xhtml, which can become json, etc... ||with the added bonus minimal tokens for LLMs|| 04:08 MTDiscord dunno if we want that, but it leaves us with open options, im just throwing it out there 04:09 MTDiscord fair, just making sure that's all you were after - it's a good point - what are the root goals? 1. Machine parsable 2. Human parsable 3. Writer friendly (schema + easy to write) 4. Optionally beautiful website rendering? 5. ? 04:09 MTDiscord overall i would summerize the idea as having formatted docs that can "compile" to whatever platform, so in theory if you want pure text, we could have a "compiled" output for that 04:10 MTDiscord ah, my goal was for the docs to be one thing, and all machine parsable stages derive from it, not the other way around. 04:11 MTDiscord i mean, unless ive misunderstood greenxenith, they want docs they can compile into there extension rather than the current hacking on lua api.md 04:11 MTDiscord though I mean if we had asserts in our C++ code, and lua code, we could "compile" it from the raw source of luanti - don't think we'll be pulling that off effectively though 04:11 MTDiscord from my understanding, they just want it to be very clean, sane, with a schema to enforce rigity so that it's easy to parse and stays easy to parse 04:12 MTDiscord i dont disagree with that, however in https://discord.com/channels/369122544273588224/926231483155378176/1456500021922107434 im pointing out why they want that i believe 04:13 MTDiscord So @ Xenith, would you want something more natively object based or okay with something that parses easily to a useable data object (JSON, XML, etc.) 04:16 MTDiscord also i think mark is in this because he wants to "compile" whatever information to typescript at some point. at least that was mentioned way back when 04:16 MTDiscord correct me if im wrong 04:25 MTDiscord I don't think we should look at this as Markdown vs MDX: MDX is a superset of Markdown by the same people who make enterprise-grade Markdown parsers. We'll be able to gradually incorporate MDX features as needed. It's kinda like starting with JavaScript and gradually adding types 🙂 04:27 MTDiscord RE generating docs from in-source comments: no, I don't want to do this. It would be a lot of work and ultimately we wouldn't have ergonomic space to write long-form guidance like we do with standalone Markdown. That said, maybe eventually, we can get the basic types described in-source and extract. But that is high effort, low reward at our current state. "Maybe" and "eventually" are doing a lot of work here 04:28 MTDiscord Yes my original goal was (and still is) to write a TypeScript API for Luanti 🙂 but that's for 2027, I think! 04:29 MTDiscord yeah, this is stupid. not to mention being constrained by the engine, its processes, etc 04:29 MTDiscord stupid is a strong word, but I otherwise agree 04:30 MTDiscord I mean, that's... Tough. You'd have to go look at how Bun integrated the apple JS engine. Lua is designed for C++ integration. JavaScript a lot less so 04:30 MTDiscord im not sure why your still going on this tangent 04:31 MTDiscord Sorry, the typescript one is something I've thought about for a while, I was replying to marks 2027 dream, since I respect those with dreams 04:31 MTDiscord ah, sorry 04:32 MTDiscord iirc its typescripts types from the docs so typescript -> lua can be used, rather than typescript in the engine 04:32 MTDiscord yeah I use a project called TypeScriptToLua, it works pretty great 🙂 https://github.com/mark-wiemer/hello-hello/tree/main/packages/luanti 04:33 MTDiscord anyways, if we do want mdx, i think its better to start off with it as our schema, then to try and bake it in later 04:35 MTDiscord I'd disagree there. I'm not sure how much it'd need to be "baked in" vs "gradually allowed as an alternative" in places. I foresee a relatively open "notes" section for each function that allows doc writers to describe the function with multiple paragraphs, lists, code blocks, etc. The schema is primarily for ensuring each symbol signature and properties (environment, etc.) is documented consistently. MDX can easily be added into 04:35 MTDiscord the notes section, and could relatively easily be added as an alternative option for parts of the signature section later 04:36 MTDiscord im suggestion using mdx for the schema 04:36 MTDiscord https://discord.com/channels/369122544273588224/926231483155378176/1456497572930650243 04:36 MTDiscord But I agree that we should move forward with this in mind and consider MDX when writing our schema 🙂 the bestest part about a schema, though, is that once we implement it, we can also implement an automated system to transform into any other schema, including, say, one that better incorporates MDX 04:37 MTDiscord I see MDX as the same as Markdown at this point, to be clear. Until I know why we need MDX, it feels premature to add MDX-specific features 04:37 MTDiscord Do you think it's reasonable to start with Markdown and design with the forethought that we'll incorporate MDX into the schema later? 04:38 MTDiscord i think its absolutely insane for the schema to say where just doing random markdown to start with and then transfer it all to a mdx based solution 04:38 MTDiscord either decided on https://discord.com/channels/369122544273588224/926231483155378176/1456497572930650243 or https://discord.com/channels/369122544273588224/926231483155378176/1456498268073758924 or some other solution for the schema itself 04:40 MTDiscord probably a bad anology it like saying im going to build a house out of wood, and then later deciding to use concrete while keeping the room layout and furniture the same 04:40 MTDiscord if im understanding you correctly 04:44 MTDiscord exevirus: do you happen to have a link to this riseup handy if you dont mind? 04:48 MTDiscord I think it's more like saying I'm going to use oak wood, and then changing to spruce + some iron bolts later. MDX is built by the same people who wrote the industry-standard Markdown parser. It will be the same toolchain. MDX is a superset of Markdown. Yes there will be a small lift and shift when we add in the initial MDX tooling but we should be able to take the core new_lua_api.md doc, rename it new_lua_api.mdx and everything 04:48 MTDiscord will just work. It'll be the same core toolchain by the same people, MDX is specifically designed for gradual adoption like this. 04:49 MTDiscord maybe im just completely stupid, but we are trying to decide on a solution here, a schema, something fresh. why would we say where doing thing and then plan to completely change it up later 04:52 MTDiscord YAGNI? 80/20? Scope creep? There are a lot of ways to describe it. My goal is to start with something lightweight that we know will work, but is also extensible to potential future needs. Markdown accomplishes that. Incorporating MDX-specific features at the very beginning of the design sounds like we're overcomplicating things. Again, I very much agree that we should design with MDX in mind. But I don't think that means moving 04:52 MTDiscord forward with the mindset that "everything we do must be MDX-native". Rather, I think it means "design something extensible, where we can alter it later if we want MDX or some other tool to serve this minor role (e.g. being used to describe a function signature)". 04:53 MTDiscord I also want to reiterate that adding MDX will not be a complete change-up. It'll be like adding Christmas lights to the outside of the house. It might take a bit of work to find an outlet that works, and some nails to hold the lights, but once that's done, it's done. The interior of the house, the 95% that we care about, won't change at all. 04:54 MTDiscord i think i just fundimentally disagree at this point. i will note that i just threw out mdx as a suggestion, not that im actually rooting for it. what im concretely saying here that we should pick something and go with it, not do half measures. ofc if we find out that it sucks (like for example asciidocs), and need to change it later, thats a different story, not trying to lock us into something 04:55 MTDiscord That's totally fine. I think the best thing to move forward is for both of us to propose some visions with examples. I may very well be missing something! 04:56 MTDiscord > we should pick something and go with it, not do half measures. On this we definitely agree 🙂 04:56 MTDiscord honestly i dont really care to much what is picked as long as it is sensible, exevirus schema very well maybe the solution, or something completely different not touched on here. just that we need to pick 04:56 MTDiscord I propose = Markdown 🙂 04:57 MTDiscord yet in your previous messages it conveys wanting thing to be markdown now, but maybe mdx in the future .... 04:58 MTDiscord we can very much go with Markdown indefinitely. MDX is something we can incrementally adopt as needed later, yes. Kinda like saying "Let's go with JavaScript, but maybe TypeScript in the future" 04:59 MTDiscord sorry, we are talking in circles at this point and I am tired and hungry. Gonna sign off for the night, no disrespect. I'm curious to hear more about your objections to my approach, just not tonight! 04:59 MTDiscord night 👋 05:49 MTDiscord Ok, futher thought about mdx. Skipping over a lot of the in-between. But what if instead of mdx we just did custom html elements? As then we can still have the sample for example, but we don't need at mdx baggage to render it in the docs project. Custom html elements are supported in the browser, syllable, et. Additionally html is supported in the markdown spec natively 05:51 MTDiscord . As compared to exevirus example, one reason I do like html elements is it's a schema people are used to rather than needing to understand some bespoke dsl. 07:35 MTDiscord i'm cool with using MDX within markdown to make parsing easy. HTML and XML are much more expressive than YAML and much of concepts i introduced lend easily to that approach; it would normalize entries and sub-entries into simply tag blocks we all know and love. it really shouldn't degrade the writing and reading experience that much. heck, this could be easier since there's even less to teach syntactically. surely devs here are 07:35 MTDiscord familiar enough with MDX more than the "headings as entries blocks, and lists as sub-entries blocks. learn this bespoke DSL markdown syntax". 07:37 MTDiscord we lose out on github previews (but honestly i don't encourage that) 07:46 MTDiscord oh MDX has JSX integrated inside its syntax. surely we don't need JS features at all lol 12:15 MTDiscord btw, something to keep in mind: things like operator overloading can not be expressed in typescript, right? 12:28 MTDiscord This is a fair point. The assumption is that any editor will look at nearby stuff and mirror the style exactly, and it would be intuitive, and then if they mess up, the schema catches it and gives a sane error. That's not obvious it will work out of the box, gotta agree there. Though it sounds like those here are done bike shedding on the first attempt solution, which will just be formatted markdown. Which means we can try format 12:28 MTDiscord discussions - Mark if you read the rise up, we were working out how we'd like to specify a function, can you take a crack at writing how you'd like that to look? 12:28 MTDiscord lua variadics are kinda iffy to try and express in typescript. Maybe someone smarter figures it out 🤷 12:29 MTDiscord I've written an xml+markdown style variant of the format i wrote in riseup. I'll post it after dinner 12:30 MTDiscord So the only "DSL part" left would be the type system 12:32 MTDiscord Correct, since I couldn't think of anything but { type } and { bound } 12:34 MTDiscord Goal was to be a light and embedded as possible in my design, so it's maximum human readable, but it could backfire too if you need like 8 "tags" 14:05 MTDiscord posted the xml+markdown variant at the end of the riseup pad. i expressed how it could also just be yaml+markdown, lua DSL+markdown or any desirable data format that's nice for humans to write. personally, i'd prefer any of these rather than a custom markdown DSL like the one exe_virus initially proposed (and i expanded). 14:07 MTDiscord i mean technically all of them are kinda DSLs, but adopting an existing data description language is a bit better than markdown 14:15 MTDiscord .s/data description/data document 14:53 MTDiscord I did start reading the rise up, I'll check it again now that there are updates. And I haven't forgotten to check the chat either 🙂 14:54 MTDiscord I didn't confirm this, but I would say operator overloading is very difficult to express in TypeScript 15:16 MTDiscord well, the problem is lack of a 1:1 mapping between "luaisms" and typescript things. operator overloading is just the first thing that comes to mind (and somewhat important e.g. for vectors). ACorp also mentioned varargs. then there's also stuff like mixed tables, or maps with integer keys. the way OOP and such work is also subtly different. 16:15 MTDiscord There is certainly a lot of expressiveness to capture 🙂 fortunately Markdown can do anything pretty well if we build our own tooling! It'll take time but my current goal is a tiny vertical slice: one function, described in Markdown, with its docs surfaced in an IDE via hover text, IntelliSense, etc. Might take a bit as I've never done a lot of this work, but should have something nice well by the end of the month (hopefully much 16:16 MTDiscord sooner) 17:38 MTDiscord what if the preferred way to make code for luanti was just replaced with a typed language/lua dialect (joking, but would be interesting if it was) 21:12 MTDiscord As some input, Xenith already maintains a solid Luanti plugin for vscode/codium. It would be a good existing target to try to integrate 21:27 MTDiscord what if what if we split off the documentation into engine documentation and engine lua stuff documentation 21:27 MTDiscord oo lua dialect would be cool for vector stuff 21:35 MTDiscord I mean, this discussion has been almost entirely about lua_api.md only