| Time |
Nick |
Message |
| 02:52 |
MTDiscord |
<mark.wiemer> XSD? |
| 03:29 |
MTDiscord |
<exe_virus> I went with json-schema draft 7 for better readability, but yeah same idea. Here's the first working version: https://github.com/luanti-org/luanti/compare/master...ExeVirus:luanti:feature/api_ast 1. Outputs a somewhat hierarchical lua_api.json (file .gitignore/not committed, recommend testing locally, it's right at 1MB) 2. Uses the initial schema (that validates really nothing yet) to validate that output json 3. Requires only 3 |
| 03:29 |
MTDiscord |
well supported python libs: pip install markdown pygments jsonschema 4. Pygments is used to read the lua codeblocks and turn that into lexed code in json for further parsing/schema validation, for now I have just removed all whitespace tokens from that, but kept comments as that should be validated 5. Timing to convert on my mid-range CPU/DDR4 system: $ time python parseLuaApi.py Validation: PASSED Output saved to: lua_api.json real |
| 03:29 |
MTDiscord |
0m2.688s user 0m0.000s sys 0m0.031s 6. To run, do (3) pip install, nav to doc/, and run python parseLuaApi.py Don't have a good recommended online viewer, probably just change it to output pretty print indents and self-inspect. Thing that jumps out at me: yeah we'd need to do some solid parsing ourselves to make most of these sections more machine readable (most being a special snowflake), and likely still need to rewrite many of the |
| 03:29 |
MTDiscord |
section formatting, if we keep this exact file format |
| 03:31 |
MTDiscord |
<exe_virus> https://cdn.discordapp.com/attachments/926231483155378176/1447792686949073026/lua_api.json?ex=6938e983&is=69379803&hm=2e90773b1f8a80c5c00c761a0510451f23d3d4d35167c32f27d37fa4192cb0ac& |
| 03:32 |
MTDiscord |
<exe_virus> Fun fact, we have a stray <action> I had to fix before we could render to fully valid xhtml, but that was it - pretty impressive code quality honestly. |
| 03:34 |
MTDiscord |
<a.corp.serot> why pygments? |
| 03:34 |
MTDiscord |
<exe_virus> It's got lua lexing and outputs to json |
| 03:37 |
MTDiscord |
<a.corp.serot> ehh, its output is a good start i guess |
| 03:38 |
MTDiscord |
<exe_virus> yeah my feelings exactly, not warm fuzzies haha |
| 03:38 |
MTDiscord |
<a.corp.serot> (side note: don't wanna imagine the IRC logs of this lol) |
| 03:38 |
MTDiscord |
<exe_virus> But to be fair, we have crap like: Table of resulting digging times: crumbly 0 1 2 3 4 <- level -> 0 - - - - - 1 0.80 1.60 1.60 - - 2 0.60 1.20 1.20 - - 3 0.40 0.80 0.80 - - level diff: 2 1 0 -1 -2 really? gfm would be nice |
| 03:39 |
MTDiscord |
<a.corp.serot> is gfm actually fully supported? |
| 03:39 |
MTDiscord |
<a.corp.serot> by the current doc builder |
| 03:39 |
MTDiscord |
<exe_virus> I mean, I'm sure markdown supports it, just a sec |
| 04:41 |
MTDiscord |
<mark.wiemer> What's the bonus of GFM vs CommonMark? Overall I think we will need to update the canonical form (lua_api.md) to be a lot stricter, not just focus on something that gets the AST of the canonical form and tries to work with that. The core issue here is that lua_api.md was not written with machine-readability in mind |
| 04:48 |
MTDiscord |
<exe_virus> Absolutely, I just wanted to get us to a baseline AST builder/parser, and that piece I'd say is set up now. And now we can roll up our sleeves and address areas we want to improve, and know that we have a schema that can validate those areas, so that as we make changes to the schema, we can ensure everything is updated at once. Also not restricted really, cause we can parse code blocks of Lua, as well as normal prose. |
| 04:48 |
MTDiscord |
<exe_virus> What would you like to tackle first? |
| 04:49 |
MTDiscord |
<exe_virus> Functions? Methods? (Are they the same?) |
| 04:49 |
MTDiscord |
<mark.wiemer> Hmm I will say I don't fully understand what's been done with your AST branch or the value it provides. What would be a sample bad change that would be caught with your parser? Can you provide an example? |
| 04:50 |
MTDiscord |
<mark.wiemer> Basically you've described a JSON schema and written a program to make sure lua_api.md meets that schema, is that correct? |
| 04:50 |
MTDiscord |
<mark.wiemer> IRC: s/described/defined |
| 04:51 |
MTDiscord |
<exe_virus> Sure, it fails and errors out of we put any non code block or `fenced` <html> tag That's one already haha. But yeah we have a thing that can be machine parsed, provides header hierarchy, and a setup that lets us use a production schema format for validation, and a flexible system that can lex Lua and markdown at the same time |
| 04:52 |
MTDiscord |
<mark.wiemer> Nice 🙂 I didn't read it in detail 😄 a bit tired as it's the end of a long day for me rn. Appreciate the effort of course 🙂 |
| 04:53 |
MTDiscord |
<mark.wiemer> I think we'll need to start drafting a stricter schema that considers what info we actually need, as lua_api.md is missing lots of info as it stands |
| 04:53 |
MTDiscord |
<exe_virus> No worries, took under an hour since I had thought about it for 3 months. You should have no issues reading and understanding the 100 lines of code to do it, and the like 50 lines of schema |
| 04:53 |
MTDiscord |
<exe_virus> And yeah, the thinking was loose schema at first and then we drill down |
| 04:54 |
MTDiscord |
<mark.wiemer> I think we should operate under the assumption that the entire doc will be at least restructured, tons of changes are needed for something meaningfully machine readable |
| 05:00 |
|
MTDiscord joined #luanti-docs |
| 12:35 |
MTDiscord |
<exe_virus> As for drilling down, I think it best to start with the basic "function". What things make up a "function"? Args Return Function name Description Usage? What about variants of the same function? Bounds for the arguments/return? |
| 12:59 |
MTDiscord |
<a.corp.serot> - args, return, fn name are fine. - descriptions may cover usage, but it's expected that supplementary documentation cover things in more detail. - need an example of function variants, because i'm unsure you're suggesting overloads/'many function specs sharing name' or simply "see also this function" - however, each function needs a way to specify which environment it can be used. i've separated it into 4: Load/Startup, |
| 12:59 |
MTDiscord |
Main/Usual, Async and Async Mapgen. this can be lumped in with bounds below. - bounds would probably need to be more than just types, including value and engine state restrictions. for instance, state-wise, core.override_item expects an existing registered item name. type-wise it's just a string. value-wise the string must follow the nodeID scheme (mod:item_name). back to state-wise, the function must be invoked during Load/Startup. |
| 13:03 |
MTDiscord |
<a.corp.serot> i think setting bounds on things will be the hardest part, so a goal may be to gradually make bounds machine parseable |
| 13:08 |
MTDiscord |
<exe_virus> Agree on description/usage Forgot about environment. Are we merging client_lua_api and main_menu_api into this? And yeah everything should be machine readable, especially bounds |
| 13:12 |
MTDiscord |
<a.corp.serot> oh yeah, i forgot CSM exist |
| 13:12 |
MTDiscord |
<a.corp.serot> well, in the future SSCSM too |
| 13:12 |
MTDiscord |
<a.corp.serot> that's a lot of environments |
| 13:14 |
MTDiscord |
<a.corp.serot> re:bounds i don't think all bounds can be machine readable. but ultimately all the hard data should be provided so implementations avoid magic values. kinda like how unicode just provides data but implementations are not machine parseable |
| 13:14 |
MTDiscord |
<exe_virus> Mapgen environment too |
| 13:16 |
MTDiscord |
<exe_virus> I mean, there's strings, objref, vector, numbers, tables, function ptr... Anything else type wise? |
| 13:16 |
MTDiscord |
<exe_virus> Sounds like environment is an array of enums |
| 13:17 |
MTDiscord |
<a.corp.serot> we have all basic lua primitives, luanti tables with metatable, luanti (light)userdata with metatable, and uhh that's it i think |
| 13:17 |
MTDiscord |
<a.corp.serot> luanti x with metatable covering stuff like objectref, areastore, vectors and others like it |
| 13:18 |
MTDiscord |
<exe_virus> What's a good spot to work on fleshing out such a function format? We'll need to revise it and work it, ideally come and go editing as we please with tracking |
| 13:20 |
MTDiscord |
<exe_virus> Maybe a fiddle? I'd say Google doc, but there's no markdown Google doc editor... Is overleaf able to do markdown? |
| 13:20 |
MTDiscord |
<a.corp.serot> google doc doesn't have markdown yet? |
| 13:22 |
MTDiscord |
<exe_virus> It does but apparently limited |
| 13:24 |
MTDiscord |
<exe_virus> Hurm, this might work, gotta see if here's a hosted version: https://hedgedoc.org/ |
| 13:25 |
MTDiscord |
<a.corp.serot> i found this https://stackedit.io |
| 13:27 |
MTDiscord |
<exe_virus> That might work, just gotta sync it with gdocs or other cloud. Will give it go later today |
| 13:31 |
MTDiscord |
<luatic> environments should not be a property of functions. rather, an environment should list what it contains. |
| 13:34 |
MTDiscord |
<a.corp.serot> i'm grouping it all together as the boundaries/contracts of a thing instead of having restrictions describes everything that's affected |
| 13:40 |
MTDiscord |
<luatic> I think it's best to start with types. * All types should support annotations. * Primitives: nil, boolean, number, integer, string, possibly with e.g. bounds but such specifics can come later. * Annotated type aliases, e.g. player name being an alias for (a special kind of restricted) string. * Composite types: Function type as argument types + return types. For documentation purposes, parameters and return values should be named, and |
| 13:40 |
MTDiscord |
should support annotations. * Union types: A or B. This is also how you represent variants of the same function. * Record (struct) types: Field name maps to some type. Each field should support an annotation. Namespaces are just structs of mostly function types. * Lists of some type. * Maps with some key types and some value types. * Userdata / classes should be typed solely by its interface. This can be done using struct-like types. * Some sort |
| 13:40 |
MTDiscord |
of type parameters / generics is necessary, e.g. core.after<A...>(fn(A... as) -> any..., A... as) (made up syntax, but you get the idea) |
| 13:40 |
MTDiscord |
<a.corp.serot> hmm, i suppose you also need environments to list what it has for it to be defined too |
| 13:43 |
MTDiscord |
<exe_virus> I can agree with that, but my tiny brain can't do that work without an example I'm trying to represent, haha. Bounds will likely not be tied to the type, unless we alias every single argument |
| 13:45 |
MTDiscord |
<exe_virus> Also what format of language supports annotations and aligned with markdown, that's where I'm at. Might just be a code block, but what goes in the codeblock? |
| 13:45 |
MTDiscord |
<luatic> Hmm, maybe I'm misunderstanding you, but with what I have in mind tying bounds to types does not necessitate aliasing every argument? |
| 13:46 |
MTDiscord |
<luatic> For integers for example, it'd just be parameterizing the type, e.g. int(min, max) instead of just a vague int |
| 13:46 |
MTDiscord |
<a.corp.serot> personally i like int satisfies range(min, max) better |
| 13:47 |
MTDiscord |
<exe_virus> Ah, gotcha I think I know what you are going for. I guess I've run into more than just min, and max, for integers a lot in my work. |
| 13:48 |
MTDiscord |
<luatic> Sure, of course, type systems can be very sophisticated to the point where you can prove mathematical statements in them |
| 13:48 |
MTDiscord |
<exe_virus> Right.... What do we want, haha |
| 13:50 |
MTDiscord |
<a.corp.serot> could all boundaries just be expressed like <type> satisfy <contracts>? (different syntax is fine, but basically just put it side by side instead of mingling together with types) |
| 13:51 |
MTDiscord |
<luatic> I'm thinking type systems can be very fine grained, but you quickly get diminishing returns. There is a major step up from "people have to guess types, don't get editor type checking or autocomplete" to "people have some decent types that tell them what fields there are, what Lua type they have and what to do with them". |
| 13:53 |
MTDiscord |
<luatic> That's what I think a first step should be. Our approach should be flexible enough that we can always add more precise constraints later, but to get going, we should deliberately mostly ignore them at the moment. |
| 13:54 |
MTDiscord |
<luatic> Things like "specify range bounds for all integer types" are valuable, but it's more work (a lot of checking the widths of underlying engine types, for example) for reduced reward, e.g. as many type systems don't support "range" types well (or if they do, tend to be fairly permissive), and you quickly run into "dependent types" where you have something like "must be an integer from 1 to number of joints" which often can not be recorded |
| 13:54 |
MTDiscord |
well. |
| 13:55 |
MTDiscord |
<luatic> Hmm, what is the benefit of this supposed to be? |
| 13:58 |
MTDiscord |
<a.corp.serot> it allows defining lower class citizen of types, letting inexpressible restrictions just be specified and deferring the actual definition to human words. |
| 13:58 |
MTDiscord |
<a.corp.serot> the important part is that it provides reference data |
| 13:59 |
MTDiscord |
<luatic> Well we should definitely have "interface" kind of type restrictions which need not depend on the implementing type |
| 14:00 |
MTDiscord |
<luatic> e.g. a particular "class" type could be implemented either by userdata or by a table, as long as it is guaranteed to support some operations (e.g. some arithmetic, some methods, etc.) |
| 14:00 |
MTDiscord |
<exe_virus> update on stack edit, I have it tied to a google drive account (personal), and the only way to collab is f you also use a google drive account to access - is that workable or should I try something else before pushing that? |
| 14:01 |
MTDiscord |
<luatic> personally i'm fine either way, but others might care |
| 14:02 |
MTDiscord |
<a.corp.serot> hmm, i'm fine with it, but it would be nice if it wasn't |
| 14:02 |
MTDiscord |
<exe_virus> just a sec, trying overleaf |
| 14:03 |
MTDiscord |
<luatic> https://tenor.com/view/war-flashback-meme-gif-11373632052098267276 |
| 14:04 |
MTDiscord |
<luatic> this is how i feel about latex and overleaf 😆 |
| 14:04 |
MTDiscord |
<luatic> but i will survive 🫡 |
| 14:05 |
MTDiscord |
<a.corp.serot> i feel you, latex is NOT fun |
| 14:05 |
MTDiscord |
<a.corp.serot> uhh don't take that out of context |
| 14:06 |
MTDiscord |
<exe_virus> It's not bad, but overleaf only allows 2 people to work on the same file on the free tier... \documentclass{article} \usepackage{markdown} \begin{document} \begin{markdown} # My Markdown Title ## My Second Title ### My Third Title #### Test This is a paragraph of **bold** text and *italic* text. - First bullet point - Second bullet point \end{markdown} \end{document} Alrighty, what else can I try.... |
| 14:07 |
MTDiscord |
<a.corp.serot> it's still latex, just with markdown inside it XD |
| 14:12 |
MTDiscord |
<exe_virus> HackMD supports 3 users, that might be enough... |
| 14:13 |
MTDiscord |
<exe_virus> Aha, if you make the "note" public, we can do unlimited editors, that works. just a sec and I'll throw up a link |
| 14:18 |
MTDiscord |
<exe_virus> Okay, this is my favorite solution: https://hackmd.io/team/luantidocs Still have to create a throw-away account (email, password) with them to edit, but any one can edit, we have a public workspace, can create as many markdown docs as we want under that workspace. If you just want to view, no account creation needed. Someone else try it |
| 14:21 |
MTDiscord |
<a.corp.serot> 403 forbidden hmm |
| 14:24 |
MTDiscord |
<exe_virus> corporate? |
| 14:28 |
MTDiscord |
<a.corp.serot> i can get to hackcmd and even register a throwaway account |
| 14:28 |
MTDiscord |
<a.corp.serot> strange |
| 14:42 |
MTDiscord |
<exe_virus> https://hackmd.io/@luantidocs Try this |
| 14:49 |
MTDiscord |
<exe_virus> Spoke too soon, even easier service: https://pad.riseup.net/p/TestLuanti-tmp Can make single docs that last for a year after inactivity, no login, come and go. This link is to a 1-day one. Maybe that is the best solution |
| 15:29 |
MTDiscord |
<exe_virus> There: first stab at a rewritten function format in markdown https://pad.riseup.net/p/Luanti-Docs #### core.get_node(pos) Returns the node at the given position as a {NodeTable} - **args** - pos : {vector} : position of the node on the map - **return** - {NodeTable} : name is "ignore" for unloaded chunks - **env** - server-main - async - client - mapgen #### player.set_fov(fov, is_multiplier, |
| 15:29 |
MTDiscord |
transition_time) Sets the player's Field of View - **args** - fov : {number} {min=0} {max=180} : Field of View to set, 0 = clear any overrides - is_multiplier : {bool} {optional=false} : If the FOV is a multiplier or not - transition_time : {number} {optional=0} : Smooth FOV transition time in seconds - **return** - {nil} - **env** - server-main - async - client |
| 15:32 |
MTDiscord |
<et086> could have node = core.get_node(pos) |
| 15:33 |
MTDiscord |
<exe_virus> ? example usage? |
| 15:33 |
MTDiscord |
<et086> no |
| 15:33 |
MTDiscord |
<et086> but wait, yes, example usage would be great, but also just having the returns named would be also great |
| 15:33 |
MTDiscord |
<exe_virus> the returns aren't actually named, heck neither are the args, but we have that so I kept it |
| 15:34 |
MTDiscord |
<et086> but you can name them with lsp, and i think it would be less confusing |
| 15:34 |
MTDiscord |
<exe_virus> lsp - meaning name it for other parsers? |
| 15:36 |
MTDiscord |
<et086> i meant like uhh, lua_ls supports: [see screenshots] |
| 15:36 |
MTDiscord |
<et086> https://cdn.discordapp.com/attachments/926231483155378176/1447975073494466744/image.png?ex=6939935f&is=693841df&hm=b785a90bb52ec37dee2a6006f8913bbf9cc041e9f0fa26735a64ac7d8cc274b5& |
| 15:36 |
MTDiscord |
<exe_virus> gotcha, so named returns, for other parsers, sure! |
| 15:37 |
MTDiscord |
<et086> also i dont know if it would be a good idea to do it in the #### part too |
| 15:37 |
MTDiscord |
<exe_virus> Updated the above |
| 15:37 |
MTDiscord |
<et086> Also, small nitpick but a function returning nil is not the same as a function returning nothing |
| 15:37 |
MTDiscord |
<exe_virus> I'd like to keep the header just the function name and args for now. parsers will have that info in the below data on - **returns** Uh in lua that's exactly what a function returning nothing returns |
| 15:39 |
MTDiscord |
<et086> uh, no, it's different |
| 15:39 |
MTDiscord |
<et086> https://cdn.discordapp.com/attachments/926231483155378176/1447975842486550582/image.png?ex=69399416&is=69384296&hm=043798f592f5574443fab852a5d8d3080f1ee66b11c5a1c05e8e3763d9202504& |
| 15:39 |
[MatrxMT] |
<y5nw> exe_virus: The difference is when you have e.g. t = {f()} |
| 15:39 |
MTDiscord |
<exe_virus> weird, yeah that's a solid nitpick |
| 15:39 |
MTDiscord |
<et086> and also looks ugly, that's the main reason behind the nitpick |
| 15:39 |
MTDiscord |
<exe_virus> Alrighty, return gone for that example, if omitted, no returns |
| 15:40 |
MTDiscord |
<et086> also, i think you can format core.get_node(pos) with like, actually surrounding it with `` |
| 15:40 |
MTDiscord |
<exe_virus> Why bother? |
| 15:40 |
MTDiscord |
<et086> idk |
| 15:40 |
MTDiscord |
<et086> To make it more human readable, assuming that's a goal |
| 15:41 |
MTDiscord |
<exe_virus> I'd like to just get rid of most of those things do they make it less human readable? |
| 15:41 |
MTDiscord |
<et086> because you already format args, return, env, etc |
| 15:41 |
MTDiscord |
<exe_virus> So just the `functionName.name()` or what else would you like to see `fenced`? |
| 15:41 |
[MatrxMT] |
<y5nw> Oops, I didn't mean {f()}. I was thinking about passing to varargs (select("#", ...)) |
| 15:42 |
MTDiscord |
<et086> yeah, i think that would be better |
| 15:42 |
MTDiscord |
<exe_virus> changed, you can open that link and make edits yourself - i have to sign off, don't care if people make edits willy nilly, that's the point for us to find a good common baseline |
| 15:42 |
MTDiscord |
<et086> okay |
| 20:50 |
|
mrcheese joined #luanti-docs |
