Time Nick Message 03:40 MTDiscord Hi everyone, I'm back πŸ˜„ my new project will be the Luanti TypeScript API, and I'm curious for thoughts on how the core team would feel about changes to the lua_api.md file to make it more consistently parseable. The workflow I imagine is: 1. Parse lua_api.md into an AST 1. Extract the core namespace reference section 1. Identify all the functions, their arguments, and their descriptions 1. Generate a TypeScript API from that 03:40 MTDiscord Obviously, the actual types of the arguments is going to be a bit more complicated. Right now there isn't a consistent way to programmatically determine the type of each argument, some are documented and some are implied, e.g. "if x is set to true, then ..." means x is a boolean, but it's never explicitly stated I'm thinking of adding some consistent typing to the docs, but of course Lua doesn't have that and I want to propose a change that the 03:40 MTDiscord team is willing to accept. Some ideas are simple: change core.mvdir(source, destination) to core.mvdir(source: Vector3, destination: Vector3) but that might be too weird for Lua. There's also core.mvdir(source, destination) - source (Vector3): the start location - destination (Vector3): the end location But this encourages https://blog.codinghorror.com/avoiding-undocumentation/ which nobody likes. Thoughts? Ideas? Hope you've all been well 03:40 MTDiscord πŸ™‚ 03:50 MTDiscord > Luanti TypeScript API 03:50 MTDiscord https://tenor.com/view/dj-khaled-another-one-one-gif-5057861 03:52 MTDiscord sounds like https://github.com/luanti-org/luanti/issues/13926 is sorta what your looking for 05:09 MTDiscord Nah I want TS, Lua has so little tooling available. Formatters, linters, types, package managers, all much better in TS from what I've seen. I only know of one other TS API and it's very incomplete. And I like to think I have a good track record of actually getting things done πŸ˜‰ 05:11 MTDiscord your missing the point. you want docs that can be used programtically, lua headers would (at least somewhat) 05:11 MTDiscord Yes agreed, I know that much πŸ™‚ 05:11 MTDiscord also there is like 4 typescript projects 05:11 MTDiscord Ah, happy to follow links, I only found the one that was reasonable 05:13 MTDiscord there is the one by whatever there face is that is in a tower defense like game, supports jsx to formspecs. iirc not open source there is one that whoever made + then forked off by fgax 05:13 MTDiscord https://leanrada.com/wares/miniforts/ first one is buried in here 05:13 MTDiscord Haha not much to follow there, closed source and "a fork of fgax" isn't very encouraging. Surely you agree more tooling would be better? 05:14 MTDiscord mostly just dumping the ones i can think of off the top of my head 05:14 MTDiscord https://github.com/RepComm/mt-api 05:17 MTDiscord Doesn't look like mini forts has been updated in 4 years, might be something there tho :/ 05:18 MTDiscord https://github.com/jordan4ibanez/crafter/blob/23def382ab8acd30e8bdd840705439dda1d894fc/minetest-api.d.ts 05:20 MTDiscord anyways, handfuls of them around, none has ever "won" 05:20 MTDiscord My project is a fork of mt-api yes 05:21 MTDiscord I'm built different πŸ™‚ there were lots of doc website proposals around for a while too! 05:23 MTDiscord ....im just not going to engage on that 05:24 MTDiscord anyways, back to machine readable/parsable docs, should annoy green, they want similar. one of there ♾️ projects that will be done in 2069 05:25 MTDiscord Sorry, I didn't meant to take credit for others' work. But I helped, didn't I? We got a lot done late last year, a lot of stuff that folks said wouldn't happen. I think the negativity around a TS API is very similar. 05:26 MTDiscord Crafter looks pretty good, lots of updates in the past six months πŸ™‚ 05:26 MTDiscord maybe im coming off as negative, but just saying there has been a handful of them before that may be useful for you to steal borrow πŸ˜‰ from 05:27 MTDiscord Fair enough, appreciate the links πŸ™‚ 05:27 MTDiscord i dont really give a shit what you program in, more so am just annoyed when a project that uses typescript doesnt have good lua api docs to use it 05:28 MTDiscord since you cant just read self documenting code 05:28 MTDiscord I'm confused if you're annoyed at my proposal or at the core devs for not documenting their code programmatically 05:29 MTDiscord im annoyed at people who make mods/games with typescript and dont provide lua api docs 05:29 MTDiscord Fair enough, I plan on contributing back upstream for sure, hence my original proposals πŸ™‚ 05:30 MTDiscord mt annoyance is unrelated to your goal here. as to your goal seems legit as long as it can be useful to others and doesnt become a pain/maintaince burden 05:30 MTDiscord Sounds good πŸ™‚ 05:31 MTDiscord (also repcom is on matrix if you want to see if they are still around) 05:31 MTDiscord Might be worth trying to set that up πŸ™‚ 05:32 MTDiscord > I'm curious for thoughts on how the core team would feel about changes to the lua_api.md file to make it more consistently parseable Did you forget the projects you were (are?) a part of? A major part of organizing the API team was to convert lua_api.md to a machine- and human-readable format 05:33 MTDiscord I have forgotten, I suppose πŸ™ 05:33 MTDiscord Like, thats the main reason I barely maintain luanti-tools is id rather focus on making the API parseable to generate official API snippets 05:33 MTDiscord When I have time to dedicate, that is 05:34 MTDiscord topological things getting pushed off the top after jam? πŸ˜‰ 05:34 MTDiscord In fact, the API team exists because Benrob, Luatic, and I wanted the lua_api to be parseable 05:34 MTDiscord that should be vague enough 05:35 MTDiscord Well, let's just say that's my goal now too 05:35 MTDiscord It got sidelined because we were able to gain more traction moving the wikis into a unified doc page, but most of the short term goals are complete by now. That original goal is still in the roadmap for the docs project 05:36 MTDiscord https://github.com/luanti-org/docs.luanti.org/issues/47 05:36 MTDiscord (you actually reacted to comments this issue, mark ;p ) 05:36 MTDiscord Hey it's been a long year! 05:37 MTDiscord The original project was to convert the monolith file to AsciiDoc, but I dont think that was the correct choice now that we tried it in an extended capacity 05:37 MTDiscord asciidoc can burn in hell 05:37 MTDiscord AsciiDoc, while , is very cumbersome to write. 05:37 MTDiscord I think Markdown with an automated check system should be sufficient, I've been using remark and it's really good 05:37 MTDiscord while parseable* 05:38 MTDiscord The problem is markdown is missing a lot of very useful elements 05:38 MTDiscord Like what? Anything that couldn't be handled by an external parser/checker? 05:39 MTDiscord I think it was mostly admonitions, includes, and macros 05:39 MTDiscord I had a proposed alternative and I dont remember what it was 05:40 MTDiscord MDX is close, but I'm thinking a simple JS script could expand macros, etc 05:40 MTDiscord admonitions is solved now in markdown 05:40 MTDiscord I think it was one of the markdown flavors with admonitions + a preprocessor 05:40 MTDiscord Yeah I like preprocessors nowadays πŸ™‚ 05:41 MTDiscord Pandoc 05:41 MTDiscord https://discord.com/channels/369122544273588224/926231483155378176/1300317894848020501 05:42 MTDiscord Not mark joining literally 3 days later https://discord.com/channels/369122544273588224/926231483155378176/1301755758047662132 05:42 MTDiscord I've been rethinking Pandoc, but I never gave it a fair shot 05:43 MTDiscord I briefly tested it 05:43 MTDiscord I am not attached to any particular tool 05:43 MTDiscord Pandoc just had a lot of advantages 05:43 MTDiscord Two major ones being: We dont have to maintain it ourselves, and it supports Lua macros 05:44 MTDiscord How useful are Lua macros? 05:44 MTDiscord Whether or not you like Lua as a tooling language, it means we can build snippets using what Luanti comes bundled with instead of, say, an entire JS engine 05:44 MTDiscord "Lua macros" = you can script pandoc to preprocess your markdown using Lua 05:45 MTDiscord Specifically I think I was writing a {{mustache}} macro replacer for pandoc 05:45 MTDiscord except you need pandoc which is external anyways rendering your point moot 05:45 MTDiscord I don't see significant downside in onboarding JS. Yes it's a new lang, but I imagine it'd have a lot more tools than Lua 05:46 MTDiscord This is true 05:46 MTDiscord something like ldoc would be more inline with your comments 05:46 MTDiscord (if we added it in) 05:46 MTDiscord That is correct, and I very much do not like ldoc 05:46 MTDiscord We tried ldoc 05:46 MTDiscord it was painful 05:46 MTDiscord I think step one is identifying what features we need, then we can browse tools 05:46 MTDiscord the group never did, you might have 05:46 MTDiscord Done and done 05:47 MTDiscord "done and done" to what? 05:47 MTDiscord at least iirc anyways 05:47 MTDiscord "We" meaning a couple of us have used ldoc 05:47 MTDiscord Luatic, myself, and I believe Ben 05:47 MTDiscord anyways, personally i would pick mdx over pandoc these days 05:49 MTDiscord We need a markdown preprocessor/templating engine that ideally can export to an easier to read data format (yes you can strictly format and read markdown, but it would be better if I could export to XML or JSON or something) 05:49 MTDiscord Rendering the markdown is a separate issue that is basically already solved 05:49 MTDiscord remark turns Markdown into an AST 05:51 MTDiscord Then we can turn that into whatever, of course. The issue is that we'll need a more strict guideline than "syntactically valid Markdown", we'll need a description of what properties need to be documented (args, types, descriptions) and how they should be syntactically structured for consistent parsing 05:51 MTDiscord And that is the next issue, which has also kind of been tackled already 05:51 MTDiscord can enforce that via ci 05:51 MTDiscord Should enforce during editing 05:51 MTDiscord Yep πŸ™‚ 05:52 MTDiscord personally i think its better to keep stuff in markdown everyone understands rather than raising a barrier of understand whatever tooling one feels like 05:52 MTDiscord speaking of ci, do need to fix cspell 05:52 MTDiscord Also possible πŸ™‚ I want to focus on critical requirements first, CI is critical, editing less so 05:52 MTDiscord This is in response to? 05:52 MTDiscord in general to both of you 05:52 MTDiscord I didnt think anyone disagreed that we wanted markdown 05:53 MTDiscord Never hurts to clarify πŸ™‚ 05:53 MTDiscord But it does hurt to move one if someone thinks the intent is something else 05:53 MTDiscord more so why do some flavor of imports, other rules, etc when you can just have the standard in a format and use remark for example to do the transforms 05:53 MTDiscord move on* 05:54 MTDiscord We will still absolutely have templating ("imports") 05:54 MTDiscord Thats what remark is for 05:54 MTDiscord why tho? make a markdown schema and roll with it 05:54 MTDiscord What? We need to be able to re-use content consistently 05:55 MTDiscord Like type annotation. All types should backlink, and we are NOT manually filling those out 05:55 MTDiscord remark can export markdown to an ast. if its in a specified schema then they will all be the same 05:55 MTDiscord Im not even talking about exporting 05:55 MTDiscord at least running with what mark said, havent looked at remark in depth so could be full of shit 05:55 MTDiscord I am talking about in the documentation formatting 05:56 MTDiscord Like I am not going to write Type: Position or whatever every time I annotate a Pos type. That is going to break if the section changes or someone is going to forget one. Sure you could enforce via formatting but why put the effort into the enforcement if you could template it for cheaper? 05:58 MTDiscord The same concept goes for version tagging, deprecation notices, page headers, etc 05:58 MTDiscord I am not saying we need anything crazy. Just the ability to template the markdown. Which remark should provide? 05:59 MTDiscord Templating was the one feature from ADoc that was genuinely important 06:01 MTDiscord If there is a Markdown flavor that already does this, great, but I wasnt aware of any mainstream ones 06:02 MTDiscord I mean I guess MDX technically, but thats cheating 06:05 MTDiscord To wsor's point, better to keep as close as possible to commonmark, with the necessary feature to stay as organized and stable as possible 06:05 MTDiscord Putting JS inside a markdown file feels wrong 06:06 MTDiscord remark can take any Markdown and turn it into anything else. If we have a strict schema we don't have to handwrite back links, we can write a remark plugin to recognize where a back link should be added. So it's both a parser and a templating engine in that sense yes 06:07 MTDiscord Does recognizing where the link should be added equate to adding it? I dont want to write it myself 06:08 MTDiscord Yes, we write a plugin to say where a link should be, and in that same plugin write the template expansion for the links. Nobody would have to handwrite any back links 06:09 MTDiscord I will work on some schema proposals with sample input output for reference 06:10 MTDiscord There are also some things that need to be wrapped in consistent elements. For examples, parameters often get thrown in tables that have a type, value, and note column. Tables are a pain to write usually, so converting a simple comma-separated list (likely similar to how it would be written in code) would get converted to a table 06:10 MTDiscord Yes, we should have the schema enforce all that πŸ™‚ 06:10 MTDiscord Sounds good to me then 06:11 MTDiscord (I was already satisfied with the solution, I am just here to defend my templates) 06:11 MTDiscord Yes, I see the value of templates now πŸ™‚ 06:11 MTDiscord There is something to be said about documentation per-method/object vs trying to categorize them ourselves, but that can be bikeshed another day. 06:12 MTDiscord FWIW we should probably have a preliminary extractor that dumps as much function/object data as it can from the engine itself 06:16 MTDiscord And future note: I am sure there is some way to do optional schema or something, with which we might eventually have one which adds some flags for the HTML export to pretify some things (like icons, borders, colors, etc) 06:17 MTDiscord Either way, would be stoked if this works well. Have been wanting officially-generated data for language servers for a long time 06:19 MTDiscord This is also coming within a week of API autocomplete being brought up in a different channel 06:21 MTDiscord Created https://github.com/luanti-org/docs.luanti.org/issues/296: Make API programmatically parseable 06:21 MTDiscord Apologies for harsh tones, I didn't mean to push so much. Im just glad Mark isnt dead and we have actionable movement. 06:23 MTDiscord Another note from prior discussions: I dont actually remember what concensus was, but there was lengthy debate about how to balance developers writing documentation with their code (in-code or on the side) and writers writing/editing docs. Unfortunately probably not going to land a perfect solution 06:24 MTDiscord The important part here is for devs to see the value of good docs. If there is value, low friction, and quick automated enforcement, it shouldn't be too hard to push for 06:24 MTDiscord ... and I think you were part of that discussion because early on you were concerned about in-file code docs 06:25 MTDiscord Yeah I vaguely remember. Honestly it's been a very long year though!! Most important would be having discussions with the devs themselves to see how they feel, what concerns they have, etc 06:26 MTDiscord My 2 cents for now: Well-structured docs in-code requires well-structured code, which Luanti is not. I would avoid relying on engine structure to keep track of what needs documentation and where it belongs 06:26 MTDiscord My focus will definitely be "take lua_api.md and make it better" more than looking at engine internals 06:28 MTDiscord Their input is important, but keep in mind: Most features are written by people that arent the core devs. Whats important is a good workflow for the common developer + docs writers. We will end up doing most of the grunt work to convert anyway. 06:32 MTDiscord v1 of transforming current lua_api.md to TS docs, for anyone curious what remark can do for a noob: https://github.com/mark-wiemer/hello-hello/blob/main/packages/remark/lua_api_transformed.md 06:47 MTDiscord https://github.com/luanti-org/docs.luanti.org/issues/297 has my initial thoughts. Pretty late for me here and I'm on-call tmrw, then travelling for Thanksgiving. I'll keep chipping away at this though, the idea of making a Markdown parser, then a lil language server and a VS Code extension sounds fun πŸ™‚ current work is defining a schema though 14:14 MTDiscord mark, how familiar are you with emmylua? the API spec should provide enough information to generate definition files for existing language servers. emmylua's the only one i think worth pursuing since luals development stalled. there's another idea that sfan5 proposed using yaml instead for less parsing work. 14:34 MTDiscord Any way to write down types and associated documentation works, doesn't have to be YAML. Personally I'd be gravitating towards a Lua DSL. In either case it seems to me we're rehashing a lot of the discussion in https://github.com/luanti-org/luanti/issues/13926 here. 15:17 MTDiscord its linked above πŸ˜‰ 16:57 MTDiscord Generally I want to avoid Lua-based tools just because of the very weak Lua ecosystem. Luarocks isn't a great package manager, code linters and formatters are weak, and overall the language just doesn't have a lot of usage compared to JavaScript. I'm happy to look at proofs-of-concept but just naming tools will result in a flood of options without clarity as to which ones can actually do the job we need. 16:59 MTDiscord I plan to read over that incident as well. Ultimately I think the need has been clear for some time, but a reasonable solution hasn't been written. We've spent time on tools but not on details or proofs-of-concept. A POC would push this work a lot further than proposing yet another tool without any further details. Again, very open to new tools if they are backed by a proof-of-concept 17:00 MTDiscord Lol I sent the message "s/incident/GitHub issue" and Discord actually edited the message! Sorry IRC folks lol 17:23 rubenwardy 23qwa 18:22 MTDiscord Well, as I've written in the use, LΓ–VR has demonstrated that a Lua-based DSL can work well for this kind of thing: > See e.g. LΓ–VR for an example of a custom Lua-based DSL format that works fine: https://github.com/bjornbytes/lovr-docs/blob/v0.18.0/api/lovr/math/Mat4/rotate.lua (rendered output: https://lovr.org/docs/Mat4:rotate). 19:56 MTDiscord It would probably be a good idea to leverage the existing standards for lua language servers. Reinventing the wheel is generally a bad idea. 20:00 MTDiscord That discussion should really be moved to the docs repo 20:04 MTDiscord Its nice to talk about options but we only really get anywhere by trying things. 23:49 MTDiscord should i just yolo it over and see who screams? 23:49 MTDiscord also why do we have two issues for this 23:55 user333_ i did not expect #298 to get merged within 2 minutes of me opening it 23:56 MTDiscord literally the most trival pr ever 23:56 user333_ yea 23:56 user333_ 1 letter changed 23:56 user333_ added*