Time Nick Message 20:50 MTDiscord I mean, I love good 'ol Json schema, and there's also XSD. Are we trying to bake a schema together with documentation like lua_api.md? 20:55 MTDiscord Cause if we are, then I think a reasonable solution is: 1. Disallow any embedded html in lua_api.md 2. Convert Lua_api.md to xhtml (trivial) 3. Validate the resulting xhtml as XML using an XSD. 4. Enforce the XSD via CI/CD As for enumerating everything in the engine, I mean sounds like we must dump global G at various stages, and compare that to a parsed version of the XHTML/XML, verifying everything we can. 21:02 MTDiscord And start with a simple XSD that is super accepting and slowly drill down into lua_api.md making it more restricted and fixing irregularities. But who knows, maybe I'm wildly off here. Would me showing a full example benefit us? 21:13 MTDiscord Well, we want primary source of Lua API documentation, including types (e.g. function signatures) and descriptions of what functions do, what parameters are for, what a function returns, etc. 21:14 MTDiscord We want this in a (truly) machine-readable format so we can generate proper interlinked docs pages as well as typedefs (e.g. LuaLS, Teal, TS) from this 21:16 MTDiscord Right, what other restrictions do we have? Must the enforcement system be kept with our source? What about deps for CI? Can it rely on outside python or js libraries? 21:19 MTDiscord Looks like we currently use python and mkdocs for deployment, so I suspect something python based is desired 21:22 MTDiscord I'm not aware of any restrictions necessarily on deps, so long as it's easy enough to set up locally and the deps are solid 21:31 MTDiscord Sounds good. Mostly the hard part will be bringing a reasonable hierarchy to what is a flat AST. Obviously headers are a good first start, and luckily we already do h1, h2, h3 correctly. So I can use those to nest the output. The harder part will be parsing the function definitions as is, or proposing reasonable adjustments to them. 21:34 MTDiscord I think it's pretty clear that lua_api.md doesn't have the information we want it to have (e.g. types are often just implied or specified inconsistently and vaguely in prose) and even if it did, it's hardly in a consistent format 21:34 MTDiscord So we will need to define some kind of doc source format 21:35 MTDiscord Well, I'm comfortable with working that into lua_api.md honestly 21:36 MTDiscord I don't think it is so straightforward 21:37 MTDiscord Gotcha, well markdown can hold any format in code blocks, so I'll wait to pass judgement until I get there 21:37 MTDiscord well yes, but then you just push the question of format to what you use inside the code blocks 🙃 21:37 MTDiscord for example, we would have to define a sort of mini type system and associated notation 21:37 MTDiscord Correct, but we keep the human readable benefit of lua_api.md along the way. How else would we combine API with tutorial 21:38 MTDiscord Unless we try to separate those two things (which has its own issues) 21:39 MTDiscord well, whatever format we go with, can still embed markdown 21:39 MTDiscord and that format can then be turned into an interlinked doc site 21:39 MTDiscord Correct, do we embed types in markdown, or markdown into types, haha 21:40 MTDiscord I'm leaning markdown cause it's what we have, we'll see soon if it's an issue 21:40 MTDiscord Both will have the same AST or nearly the same AST at the end of the day 21:41 MTDiscord with enough processing, sure 21:41 MTDiscord Right, I don't think processing is a bottleneck once it's read in? 21:42 MTDiscord i don't really care what the source format is so long as it is sufficiently editable, and the AST is workable 21:42 MTDiscord i used to think lua might be a good choice - and it could make the parsing part next to trivial, and give us a lua AST right away - but i must admit that i don't think there's a good way to use lua for this without quite some verbosity 21:43 MTDiscord lua AST as in a bunch of nested lua tables, not a tree of lua source :p 21:44 MTDiscord Why do we want an AST in Lua? I figured an interchange format of XML or Json would be better, but I'm open to other thoughts 21:46 MTDiscord we don't really need to interchange the AST at any point? 21:46 MTDiscord whatever we do, it should just read the source format, and then have the AST in memory, and there we can work with it directly and produce the output formats we want 21:46 MTDiscord Well, it needs to be machine readable, by libraries, I assume that means JS plugins, Lua, and maybe something else? 21:47 MTDiscord Ah, you want a workflow that builds it into an AST, I was just going straight to interchange format to be validated and read in later 21:48 MTDiscord i think the validation should happen in the reader; needn't be a shotgun parser, but you never want to read something that is invalid, and the reader has information such as line numbers to give useful error messages 21:48 MTDiscord Ah I was going to give line numbers via grep, which I'll admit is silly but effective 21:49 MTDiscord there's nothing wrong with a two-stage process, it's just that i'd go for the MVP now; we can always add a simple JSON AST dump once we have that 21:51 MTDiscord Gotcha, well in my terrible world, sadly the 2 stage is easier for me. I'll see what I do tonight and tomorrow 22:19 MTDiscord API vs tutorial is actually a conversation we need to have. Most docs these days paginate their docs so including tutorials doesn't get in the way. Long-term lua_api.md should likely be turned into a paginate tree from which a monolithic reference could be generated. Using one file as source of truth is bad practice for source control and navigation 22:38 MTDiscord So, break Lua_api.md into a folder structure (tree) for easier to nav, and for source control benefits? 22:38 MTDiscord Though easier to nav may not be true, hard to say 22:39 MTDiscord One file is really hard to f up in terms of doc rot 23:25 MTDiscord agree, usually there would be at least 2 areas: the API spec and features spec. sometimes the features spec is looser, turning more into a tutorial of some sorts. 23:26 MTDiscord nonetheless, the API spec is the ultimate source of truth and the complement helps to explain concepts and features the API exposes 23:28 MTDiscord i think at least two files should be considered, if it is ever decided to avoid splitting up the docs 23:33 MTDiscord I'm not against that idea, let's see just how ridiculous this looks after I take a stab, and then I'll at least have an informed Luanti opinion. Cause I've been on projects with all sorts of different specialized schemas and docs, as well as mono projects. Luanti may be either in my book 23:38 MTDiscord personally, i can see benefits of both as i got involved in annotation projects with one or the other's approach.