Time Nick Message 02:52 MTDiscord XSD? 03:29 MTDiscord I went with json-schema draft 7 for better readability, but yeah same idea. Here's the first working version: https://github.com/luanti-org/luanti/compare/master...ExeVirus:luanti:feature/api_ast 1. Outputs a somewhat hierarchical lua_api.json (file .gitignore/not committed, recommend testing locally, it's right at 1MB) 2. Uses the initial schema (that validates really nothing yet) to validate that output json 3. Requires only 3 03:29 MTDiscord well supported python libs: pip install markdown pygments jsonschema 4. Pygments is used to read the lua codeblocks and turn that into lexed code in json for further parsing/schema validation, for now I have just removed all whitespace tokens from that, but kept comments as that should be validated 5. Timing to convert on my mid-range CPU/DDR4 system: $ time python parseLuaApi.py Validation: PASSED Output saved to: lua_api.json real 03:29 MTDiscord 0m2.688s user 0m0.000s sys 0m0.031s 6. To run, do (3) pip install, nav to doc/, and run python parseLuaApi.py Don't have a good recommended online viewer, probably just change it to output pretty print indents and self-inspect. Thing that jumps out at me: yeah we'd need to do some solid parsing ourselves to make most of these sections more machine readable (most being a special snowflake), and likely still need to rewrite many of the 03:29 MTDiscord section formatting, if we keep this exact file format 03:31 MTDiscord https://cdn.discordapp.com/attachments/926231483155378176/1447792686949073026/lua_api.json?ex=6938e983&is=69379803&hm=2e90773b1f8a80c5c00c761a0510451f23d3d4d35167c32f27d37fa4192cb0ac& 03:32 MTDiscord Fun fact, we have a stray I had to fix before we could render to fully valid xhtml, but that was it - pretty impressive code quality honestly. 03:34 MTDiscord why pygments? 03:34 MTDiscord It's got lua lexing and outputs to json 03:37 MTDiscord ehh, its output is a good start i guess 03:38 MTDiscord yeah my feelings exactly, not warm fuzzies haha 03:38 MTDiscord (side note: don't wanna imagine the IRC logs of this lol) 03:38 MTDiscord But to be fair, we have crap like: Table of resulting digging times: crumbly 0 1 2 3 4 <- level -> 0 - - - - - 1 0.80 1.60 1.60 - - 2 0.60 1.20 1.20 - - 3 0.40 0.80 0.80 - - level diff: 2 1 0 -1 -2 really? gfm would be nice 03:39 MTDiscord is gfm actually fully supported? 03:39 MTDiscord by the current doc builder 03:39 MTDiscord I mean, I'm sure markdown supports it, just a sec 04:41 MTDiscord What's the bonus of GFM vs CommonMark? Overall I think we will need to update the canonical form (lua_api.md) to be a lot stricter, not just focus on something that gets the AST of the canonical form and tries to work with that. The core issue here is that lua_api.md was not written with machine-readability in mind 04:48 MTDiscord Absolutely, I just wanted to get us to a baseline AST builder/parser, and that piece I'd say is set up now. And now we can roll up our sleeves and address areas we want to improve, and know that we have a schema that can validate those areas, so that as we make changes to the schema, we can ensure everything is updated at once. Also not restricted really, cause we can parse code blocks of Lua, as well as normal prose. 04:48 MTDiscord What would you like to tackle first? 04:49 MTDiscord Functions? Methods? (Are they the same?) 04:49 MTDiscord Hmm I will say I don't fully understand what's been done with your AST branch or the value it provides. What would be a sample bad change that would be caught with your parser? Can you provide an example? 04:50 MTDiscord Basically you've described a JSON schema and written a program to make sure lua_api.md meets that schema, is that correct? 04:50 MTDiscord IRC: s/described/defined 04:51 MTDiscord Sure, it fails and errors out of we put any non code block or `fenced` tag That's one already haha. But yeah we have a thing that can be machine parsed, provides header hierarchy, and a setup that lets us use a production schema format for validation, and a flexible system that can lex Lua and markdown at the same time 04:52 MTDiscord Nice 🙂 I didn't read it in detail 😄 a bit tired as it's the end of a long day for me rn. Appreciate the effort of course 🙂 04:53 MTDiscord I think we'll need to start drafting a stricter schema that considers what info we actually need, as lua_api.md is missing lots of info as it stands 04:53 MTDiscord No worries, took under an hour since I had thought about it for 3 months. You should have no issues reading and understanding the 100 lines of code to do it, and the like 50 lines of schema 04:53 MTDiscord And yeah, the thinking was loose schema at first and then we drill down 04:54 MTDiscord I think we should operate under the assumption that the entire doc will be at least restructured, tons of changes are needed for something meaningfully machine readable 12:35 MTDiscord As for drilling down, I think it best to start with the basic "function". What things make up a "function"? Args Return Function name Description Usage? What about variants of the same function? Bounds for the arguments/return? 12:59 MTDiscord - args, return, fn name are fine. - descriptions may cover usage, but it's expected that supplementary documentation cover things in more detail. - need an example of function variants, because i'm unsure you're suggesting overloads/'many function specs sharing name' or simply "see also this function" - however, each function needs a way to specify which environment it can be used. i've separated it into 4: Load/Startup, 12:59 MTDiscord Main/Usual, Async and Async Mapgen. this can be lumped in with bounds below. - bounds would probably need to be more than just types, including value and engine state restrictions. for instance, state-wise, core.override_item expects an existing registered item name. type-wise it's just a string. value-wise the string must follow the nodeID scheme (mod:item_name). back to state-wise, the function must be invoked during Load/Startup. 13:03 MTDiscord i think setting bounds on things will be the hardest part, so a goal may be to gradually make bounds machine parseable 13:08 MTDiscord Agree on description/usage Forgot about environment. Are we merging client_lua_api and main_menu_api into this? And yeah everything should be machine readable, especially bounds 13:12 MTDiscord oh yeah, i forgot CSM exist 13:12 MTDiscord well, in the future SSCSM too 13:12 MTDiscord that's a lot of environments 13:14 MTDiscord re:bounds i don't think all bounds can be machine readable. but ultimately all the hard data should be provided so implementations avoid magic values. kinda like how unicode just provides data but implementations are not machine parseable 13:14 MTDiscord Mapgen environment too 13:16 MTDiscord I mean, there's strings, objref, vector, numbers, tables, function ptr... Anything else type wise? 13:16 MTDiscord Sounds like environment is an array of enums 13:17 MTDiscord we have all basic lua primitives, luanti tables with metatable, luanti (light)userdata with metatable, and uhh that's it i think 13:17 MTDiscord luanti x with metatable covering stuff like objectref, areastore, vectors and others like it 13:18 MTDiscord What's a good spot to work on fleshing out such a function format? We'll need to revise it and work it, ideally come and go editing as we please with tracking 13:20 MTDiscord Maybe a fiddle? I'd say Google doc, but there's no markdown Google doc editor... Is overleaf able to do markdown? 13:20 MTDiscord google doc doesn't have markdown yet? 13:22 MTDiscord It does but apparently limited 13:24 MTDiscord Hurm, this might work, gotta see if here's a hosted version: https://hedgedoc.org/ 13:25 MTDiscord i found this https://stackedit.io 13:27 MTDiscord That might work, just gotta sync it with gdocs or other cloud. Will give it go later today 13:31 MTDiscord environments should not be a property of functions. rather, an environment should list what it contains. 13:34 MTDiscord i'm grouping it all together as the boundaries/contracts of a thing instead of having restrictions describes everything that's affected 13:40 MTDiscord I think it's best to start with types. * All types should support annotations. * Primitives: nil, boolean, number, integer, string, possibly with e.g. bounds but such specifics can come later. * Annotated type aliases, e.g. player name being an alias for (a special kind of restricted) string. * Composite types: Function type as argument types + return types. For documentation purposes, parameters and return values should be named, and 13:40 MTDiscord should support annotations. * Union types: A or B. This is also how you represent variants of the same function. * Record (struct) types: Field name maps to some type. Each field should support an annotation. Namespaces are just structs of mostly function types. * Lists of some type. * Maps with some key types and some value types. * Userdata / classes should be typed solely by its interface. This can be done using struct-like types. * Some sort 13:40 MTDiscord of type parameters / generics is necessary, e.g. core.after(fn(A... as) -> any..., A... as) (made up syntax, but you get the idea) 13:40 MTDiscord hmm, i suppose you also need environments to list what it has for it to be defined too 13:43 MTDiscord I can agree with that, but my tiny brain can't do that work without an example I'm trying to represent, haha. Bounds will likely not be tied to the type, unless we alias every single argument 13:45 MTDiscord Also what format of language supports annotations and aligned with markdown, that's where I'm at. Might just be a code block, but what goes in the codeblock? 13:45 MTDiscord Hmm, maybe I'm misunderstanding you, but with what I have in mind tying bounds to types does not necessitate aliasing every argument? 13:46 MTDiscord For integers for example, it'd just be parameterizing the type, e.g. int(min, max) instead of just a vague int 13:46 MTDiscord personally i like int satisfies range(min, max) better 13:47 MTDiscord Ah, gotcha I think I know what you are going for. I guess I've run into more than just min, and max, for integers a lot in my work. 13:48 MTDiscord Sure, of course, type systems can be very sophisticated to the point where you can prove mathematical statements in them 13:48 MTDiscord Right.... What do we want, haha 13:50 MTDiscord could all boundaries just be expressed like satisfy ? (different syntax is fine, but basically just put it side by side instead of mingling together with types) 13:51 MTDiscord I'm thinking type systems can be very fine grained, but you quickly get diminishing returns. There is a major step up from "people have to guess types, don't get editor type checking or autocomplete" to "people have some decent types that tell them what fields there are, what Lua type they have and what to do with them". 13:53 MTDiscord That's what I think a first step should be. Our approach should be flexible enough that we can always add more precise constraints later, but to get going, we should deliberately mostly ignore them at the moment. 13:54 MTDiscord Things like "specify range bounds for all integer types" are valuable, but it's more work (a lot of checking the widths of underlying engine types, for example) for reduced reward, e.g. as many type systems don't support "range" types well (or if they do, tend to be fairly permissive), and you quickly run into "dependent types" where you have something like "must be an integer from 1 to number of joints" which often can not be recorded 13:54 MTDiscord well. 13:55 MTDiscord Hmm, what is the benefit of this supposed to be? 13:58 MTDiscord it allows defining lower class citizen of types, letting inexpressible restrictions just be specified and deferring the actual definition to human words. 13:58 MTDiscord the important part is that it provides reference data 13:59 MTDiscord Well we should definitely have "interface" kind of type restrictions which need not depend on the implementing type 14:00 MTDiscord e.g. a particular "class" type could be implemented either by userdata or by a table, as long as it is guaranteed to support some operations (e.g. some arithmetic, some methods, etc.) 14:00 MTDiscord update on stack edit, I have it tied to a google drive account (personal), and the only way to collab is f you also use a google drive account to access - is that workable or should I try something else before pushing that? 14:01 MTDiscord personally i'm fine either way, but others might care 14:02 MTDiscord hmm, i'm fine with it, but it would be nice if it wasn't 14:02 MTDiscord just a sec, trying overleaf 14:03 MTDiscord https://tenor.com/view/war-flashback-meme-gif-11373632052098267276 14:04 MTDiscord this is how i feel about latex and overleaf 😆 14:04 MTDiscord but i will survive 🫡 14:05 MTDiscord i feel you, latex is NOT fun 14:05 MTDiscord uhh don't take that out of context 14:06 MTDiscord It's not bad, but overleaf only allows 2 people to work on the same file on the free tier... \documentclass{article} \usepackage{markdown} \begin{document} \begin{markdown} # My Markdown Title ## My Second Title ### My Third Title #### Test This is a paragraph of **bold** text and *italic* text. - First bullet point - Second bullet point \end{markdown} \end{document} Alrighty, what else can I try.... 14:07 MTDiscord it's still latex, just with markdown inside it XD 14:12 MTDiscord HackMD supports 3 users, that might be enough... 14:13 MTDiscord Aha, if you make the "note" public, we can do unlimited editors, that works. just a sec and I'll throw up a link 14:18 MTDiscord Okay, this is my favorite solution: https://hackmd.io/team/luantidocs Still have to create a throw-away account (email, password) with them to edit, but any one can edit, we have a public workspace, can create as many markdown docs as we want under that workspace. If you just want to view, no account creation needed. Someone else try it 14:21 MTDiscord 403 forbidden hmm 14:24 MTDiscord corporate? 14:28 MTDiscord i can get to hackcmd and even register a throwaway account 14:28 MTDiscord strange 14:42 MTDiscord https://hackmd.io/@luantidocs Try this 14:49 MTDiscord Spoke too soon, even easier service: https://pad.riseup.net/p/TestLuanti-tmp Can make single docs that last for a year after inactivity, no login, come and go. This link is to a 1-day one. Maybe that is the best solution 15:29 MTDiscord There: first stab at a rewritten function format in markdown https://pad.riseup.net/p/Luanti-Docs #### core.get_node(pos) Returns the node at the given position as a {NodeTable} - **args** - pos : {vector} : position of the node on the map - **return** - {NodeTable} : name is "ignore" for unloaded chunks - **env** - server-main - async - client - mapgen #### player.set_fov(fov, is_multiplier, 15:29 MTDiscord transition_time) Sets the player's Field of View - **args** - fov : {number} {min=0} {max=180} : Field of View to set, 0 = clear any overrides - is_multiplier : {bool} {optional=false} : If the FOV is a multiplier or not - transition_time : {number} {optional=0} : Smooth FOV transition time in seconds - **return** - {nil} - **env** - server-main - async - client 15:32 MTDiscord could have node = core.get_node(pos) 15:33 MTDiscord ? example usage? 15:33 MTDiscord no 15:33 MTDiscord but wait, yes, example usage would be great, but also just having the returns named would be also great 15:33 MTDiscord the returns aren't actually named, heck neither are the args, but we have that so I kept it 15:34 MTDiscord but you can name them with lsp, and i think it would be less confusing 15:34 MTDiscord lsp - meaning name it for other parsers? 15:36 MTDiscord i meant like uhh, lua_ls supports: [see screenshots] 15:36 MTDiscord https://cdn.discordapp.com/attachments/926231483155378176/1447975073494466744/image.png?ex=6939935f&is=693841df&hm=b785a90bb52ec37dee2a6006f8913bbf9cc041e9f0fa26735a64ac7d8cc274b5& 15:36 MTDiscord gotcha, so named returns, for other parsers, sure! 15:37 MTDiscord also i dont know if it would be a good idea to do it in the #### part too 15:37 MTDiscord Updated the above 15:37 MTDiscord Also, small nitpick but a function returning nil is not the same as a function returning nothing 15:37 MTDiscord I'd like to keep the header just the function name and args for now. parsers will have that info in the below data on - **returns** Uh in lua that's exactly what a function returning nothing returns 15:39 MTDiscord uh, no, it's different 15:39 MTDiscord https://cdn.discordapp.com/attachments/926231483155378176/1447975842486550582/image.png?ex=69399416&is=69384296&hm=043798f592f5574443fab852a5d8d3080f1ee66b11c5a1c05e8e3763d9202504& 15:39 [MatrxMT] exe_virus: The difference is when you have e.g. t = {f()} 15:39 MTDiscord weird, yeah that's a solid nitpick 15:39 MTDiscord and also looks ugly, that's the main reason behind the nitpick 15:39 MTDiscord Alrighty, return gone for that example, if omitted, no returns 15:40 MTDiscord also, i think you can format core.get_node(pos) with like, actually surrounding it with `` 15:40 MTDiscord Why bother? 15:40 MTDiscord idk 15:40 MTDiscord To make it more human readable, assuming that's a goal 15:41 MTDiscord I'd like to just get rid of most of those things do they make it less human readable? 15:41 MTDiscord because you already format args, return, env, etc 15:41 MTDiscord So just the `functionName.name()` or what else would you like to see `fenced`? 15:41 [MatrxMT] Oops, I didn't mean {f()}. I was thinking about passing to varargs (select("#", ...)) 15:42 MTDiscord yeah, i think that would be better 15:42 MTDiscord changed, you can open that link and make edits yourself - i have to sign off, don't care if people make edits willy nilly, that's the point for us to find a good common baseline 15:42 MTDiscord okay