Addition of WorldEdit help strings

[Luashine]

How should the official GUI trigger/native description be implemented and added? I haven't looked into the format of these files yet, I will do that later.

Technical note: Apparently all of these files are stored in Game MPQ/CASC under UI/

(1) I thought if WE descriptions added to the docs source as-is, it'd violate some loicences in this repo (but there's no loicence at all). Plan: add a custom @tag to distinguish between official descriptions and ours.

(2) Same as above but instead of adding new tags to the docs source, the build system fetches them from an intermediate format (json?) - Easier than (1) because there's no need to find and inject tags at the right places in files.

(3) Handled in a separate repo(?) and merged into the .db for the front-end to process and display differently.

[lep]

The best solution would imo be a non-technical one, that is, to read the WE entries and possible rewrite them to fit into jassdoc. Now of course that would also be the most effort. That being said, i don't know how those tips are stored. Doing an automated conversion i would probably go for a specific tag, and then maybe over time remove the tags and work them into the normal comment. I don't expect that file/those files to change that often from now on. Maybe if we ever get new natives. Currently the build system writes some .sql files and uses the sqlite3 binary to add them to the final .db. If you want to generate a separate .json file that's fine but you can also work directly on the .db file; every language should have sqlite3 bindings. I don't know what kinda work you're doing/wanna do, but if you have such a WE file lying around i could also take a look and see if/how i could write a parser for that.

…

On 26.05.22 01:55, Luashine wrote: How should the official GUI trigger/native description be implemented and added? I haven't looked into the format of these files yet, I will do that later. /Technical note:/ Apparently all of these files are stored in Game MPQ/CASC under |UI/| (1) I thought if WE descriptions added to the docs source as-is, it'd /violate some loicences/ in this repo (but there's no loicence at all). Plan: add a custom @tag <https://github.com/tag> to distinguish between official descriptions and ours. (2) Same as above but instead of adding new tags to the docs source, the build system fetches them from an intermediate format (json?) - Easier than (1) because there's no need to find and inject tags at the right places in files. (3) Handled in a separate repo(?) and merged into the .db for the front-end to process and display differently. — Reply to this email directly, view it on GitHub <#47>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA3SWODDXV5KOELQOQJ3MLVL24WPANCNFSM5W7DY26A>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Luashine]

The best solution would imo be a non-technical one, that is, to read the WE entries and possible rewrite them to fit into jassdoc.

This is #1 then with custom tags as I imagined. I don't expect either that they'll be changed, but I wouldn't want to mix them with our own descriptions, for one to keep them separate, and not to "poison" previous contributions with (technically) Blizzard stuff(?)

I will figure out how the format works and comment later and write a parser/converter.

[BribeFromTheHive]

worldedit docs.txt

I've parsed this in the following format:

/*
@worldedit world editor help text from triggerstrings.txt
*/
FunctionName

[Luashine]

Looks good! @BribeFromTheHive:

The best solution would imo be a non-technical one, that is, to read the
WE entries and possible rewrite them to fit into jassdoc.

Considering this, the current output format is unnecessarily complex (jassdoc style) imho - although technically, totally structured and so it is regex-able. Do all trigger strings fit on a single line? If the descriptions don't contain a = equality sign, I'd prefer an .ini output format: funcname=localized description, it can be read out easily with any tool. Sorry we haven't talked about it before.

Also release your code in a usable fashion so it can be modified and reused later. If it wasn't code then describe your steps properly and what have you.

@lep Incorporating into current jassdoc will require a program to parse jass+jassdoc comments and inject/update the @worldedit ones. Do you have any parser readily available or do we have to look elsewhere? (see above quote)

My suggestion for the official docs:

1. They will technically taint the license-purity of the repository but at the same time a) nobody cares b) can be easily erased
2. Due to localizations I suggest the following format: @worldedit localeCode Description where localeCode=enUS etc. so the descriptions can be output in multiple languages where available.

EDIT: Something like updatedocs <docs.j file> <localeCode> <bribe's output file>

[BribeFromTheHive]

The process is quite simple. The FindAll Regex that you want is:

(\w+)Hint=(".+")

ReplaceAll regex you want:

/*\n@worldedit enUS $2\n*/\n$1

There are "=" signs in some of the quotes.

The below are actually not real functions, but rather GUI-specific stuff:

TriggerRegisterCommandEvent="Triggers when a specific ability command is issued."
TriggerRegisterBuildCommandEventBJ="Triggers when a specific build command is issued."
TriggerRegisterTrainCommandEventBJ="Triggers when a specific Train command is issued."
TriggerRegisterUpgradeCommandEventBJ="Triggers when a specific Research command is issued."
TriggerRegisterCommonCommandEventBJ="Triggers when a specific common command is issued."

I'm attaching the input file for reference.
triggerstrings.txt

@Luashine if you show me the format the JSON needs to use to merge the records, I can coordinate it differently.

[Luashine]

These are all real functions. First is from common.j and the rest from blizzard.j

[BribeFromTheHive]

They have the following disclaimer:
// Special events not defined in common.j,
// handled internally by the editor

They aren't found in common.j nor Blizzard.j

[BribeFromTheHive]

That is odd, because I am finding TriggerRegisterTrainCommandEventBJ on war3modding.info, but I do not find it in my Blizzard.j and Common.j files. When were these added, do you know?

[lep]

@lep Incorporating into current jassdoc will require a program to parse jass+jassdoc comments and inject/update the @worldedit ones. Do you have any parser readily available or do we have to look elsewhere? (see above quote)

Yes, astute observation. We could theoretically re-construct the .j files just from the sqlite DB. But I don't know if that's worth the effort in the long run or if just hacking together a script to get the data in here at once is enough.

My suggestion for the official docs:

1. They will technically taint the license-purity of the repository but at the same time a) nobody cares b) can be easily erased

I'm quite happy that so see the same as I do, but unfortunately this repo doesn't have any license attached anyways atm…

2. Due to localizations I suggest the following format: @worldedit localeCode Description where localeCode=enUS etc. so the descriptions can be output in multiple languages where available.

In contrast to the possibility of a tool to update the .j files in place an alternative would be to simply generate some sql or work with the DB directly with that data from WE. We can structure that any way we want, e.g. have script to generate this data from some binary of blizzards origin and/or have an intermediate artefact which could be checked into git.
And we're adding to the DB/sql from many different scripts already (right tool for the right job) so I think just adding a new script for these type of strings could be worthwhile. It would even kind of solve the localisation issue when we could simply provide another blob to the script.

[Luashine]

Having the Bliz descriptions in current docs would be helpful to derive yet undocumented functions (a separate lookup is tedious during documentation work). While some funcs are obvious and where to start with them, others not so much imho. So I would prefer them in the text form in current doc. Whether that'll be a regenerated file or not doesn't matter much:

1. The official structure isn't kept already, files are split
2. The official comments (and what else will be lost) are useless

It's your call. If you want to do WE -> SQLiteWE and then SQLiteWE + SQLiteJassdoc -> .jass generation, I'd say the second step is yours and I would prepare the first step :)

If we decide to work with current .j files textually: WE -> (intermediate form) -> enhance existing .jass then I'll work on it.

[lep]

I see what you mean. For me having an automated way to get this data into jass.db would suggest to me that another script would be the way to go, but your comment about user experience and having to look things up is of course very valid as well.
The good thing is that I wrote this using a real parser so I might see what I can do. But I'm not sure when. If you want to hack something together that would be fine by me as well.

[Luashine]

Sorry Bribe, I looked at it again and decided to write a parser. A very elaborate parser, but I don't want to digress here (yet).

Meantime a lot has changed, injecting data directly to the database is viable now.

• Want: 100% fully automatic triggerdata.txt parsing - DONE, the file is not without errors
  • makes possible to load up 3rd party definitions too
  • extract any of the data we want
• Want: localization support and per-language entries - TODO but rather simple

Below is an example JSON output, I did not do any major transformations but that's possible too. For now it looks like this, very close to the spirit of triggerdata.txt, and it'll be a standalone tool. Looking for suggestions for the format, on the other hand I don't want to deviate too much from the original format.

{
    "Category": {
      "value": "TC_LEADERBOARD"
    },
    "Defaults": {
      "arg1Default": "Player00",
      "arg2Default": "GetLastCreatedLeaderboard"
    },
    "DisplayName": {
      "value": "Leaderboard Position"
    },
    "Parameters": {
      "paramformat1": "Position of ",
      "paramformat2": "~Player",
      "paramformat3": " in ",
      "paramformat4": "~Leaderboard"
    },
    "arg1Type": "player",
    "arg2Type": "leaderboard",
    "minGameVersion": "roc",
    "name": "LeaderboardGetPlayerIndexBJ",
    "returnType": "integer",
    "usableInEvents": false
  }

{
    "Category": {
      "value": "TC_HERO"
    },
    "Defaults": {
      "arg1Default": "GetTriggerUnit",
      "arg2Default": "2",
      "arg3Default": "ShowHideHide"
    },
    "DisplayName": {
      "value": "Set Level"
    },
    "Limits": {
      "arg1MinLimit": "_",
      "arg2MaxLimit": "_",
      "arg3MinLimit": "1",
      "arg4MaxLimit": "_",
      "arg5MinLimit": "_",
      "arg6MaxLimit": "_"
    },
    "Parameters": {
      "paramformat1": "Set ",
      "paramformat2": "~Hero",
      "paramformat3": " Hero-level to ",
      "paramformat4": "~Level",
      "paramformat5": "",
      "paramformat6": " \"",
      "paramformat7": "~Show/Hide",
      "paramformat8": " level-up graphics"
    },
    "ScriptName": {
      "value": "SetHeroLevelBJ"
    },
    "arg1Type": "unit",
    "arg2Type": "integer",
    "arg3Type": "showhideoption",
    "minGameVersion": "roc",
    "name": "SetHeroLevel"
  }

For jassdoc tags I imagined @worldedit lang-Region followed by all plain text. Then if we wanted to format the output some other way, simply regenerate all those worldedit entries.

@lep if I read it right then your current script generate a huge piece of SQL to generate the database? Anything special?

[lep]

if I read it right then your current script generate a huge piece of SQL to generate the database? Anything special?

I mostly generated a big SQL file because in the beggining (as you know) i split the common.j into many files and then i would generate SQL only for the changed files. If you want to you can also talk directly to the DB.

If those annotations are independent from the .j file you need to do proper updates, so insert or replace into or delete; insert, because otherwise you end up with multiple annotations in the DB.

[Luashine]

@lep I finalized the structure to be this:

{
    "ArgTypes": [ "player", "leaderboard" ],
    "Category": [ "TC_LEADERBOARD" ],
    "Defaults": [ "Player00", "GetLastCreatedLeaderboard" ],
    "DisplayName": "Leaderboard Position",
    "Parameters": [ "Position of ", "~Player", " in ", "~Leaderboard" ],
    "minGameVersion": "roc",
    "name": "LeaderboardGetPlayerIndexBJ",
    "returnType": "integer",
    "usableInEvents": false
  }

Of course the full file contains every piece of the WE trigger definition, way more than we would need for annotations. The tool is here https://github.com/Luashine/convert-triggerdata

How to integrate this? It'd be fine by me if I just inserted these as opaque text, but that doesn't take into account per-language data. It looks like we have COMPLETE translations for English, Chinese (zh-CN, official), Russian (community) and Korean (partial, official). This is asking for a "lang" field.

@editor zhcn <text> and then I again treat text as an opaque blob that I format to my heart's content before export. Updates made easy by deleting all rows of the language and inserting anew. Table metadata_we, columns: fnname, lang, value. Sounds OK?

If we inserted these annotations in current files, diffs to formatting would be huge due to changing the doc text. I don't think you'd welcome the alternative - doing rendering on the Jassbot side? A lot of work to properly split each data type for SQL and then recombine it to only have a cleaner architecture. If any better integration is wanted, it can be done piece by piece where needed. One such example could be "pseudo types" -- types only in use by the editor.

{
    "codeText": "'AOvd'",
    "minGameVersion": "tft",
    "name": "HeroSkillVoodoo",
    "prettyStringId": "Orc Shadow Hunter - Big Bad Voodoo",
    "variableType": "heroskillcode"
  }

Here it gives the AOvd rawcode a fancy name, somewhere deeper the type heroskillcode will have the "baseType": "integer". There's no use for this until Jassbot were to give a relevant listing of these values for functions specified accepting this pseudo-type. Same story with another variable type:

{
    "codeText": "`robogoblin`",
    "minGameVersion": "tft",
    "name": "UnitOrderRoboGoblinEx",
    "prettyStringId": "WESTRING_unitorderEx_ROBOGOBLIN",
    "variableType": "unitorderEx"
  }

TLDR: I'd just pack everything in the metadata_we table (see few paragraphs above) and the current changes on your end will remain minimal, just to show the corresponding entry. Probably show/hide automatically client-side depending on Accept-Language. 3-4 Languages.

Now that I think of it, maybe we don't need to clutter the original common.j etc at all, just export the WE descriptions to their own file? If anything, just for editors' ease of reference.

---

1. What SQL structure?
2. Where to keep these explanations in plain sight for editors?
3. Fancy integration for later, if ever.

PS: After this we'd reach feature-parity with WE except for inability to search and browse types, global vars.

[lep]

Great work! SQLite has json support so in theory we could dump the whole json – maybe even with hashmap key by language – into the database.

There are two ways this data can be used, which were both mentioned in this thread already:

1. Annotate the actual jass files to have some default value and help the editors while writing documentation because it's just there, inline, easy to see, etc.
2. Have it consumed by other programs. Currentl that is probably only jassbot. But that was always my idea at least that people would take the sqlite file and do something with it.

Re 1: by now we have the tools do db -> .j file so we could at least try it and see how we like it.
Although technically that would need some adjustments on the schema and code side. Currently annotations are just key-value pairs and only parameters have that extra dimension. So maybe we need another table (which wouldn't be a big problem, see below).

Re 2: For jassbot the question would be what to be displayed and how to display it. My first instinct would be that DisplayName and Parameters are the most helpfull ones and maybe MinGameVersion. But how to display it? Just like every other annotation we do so far (keyed by Accept-Language)? Maybe that's more of a discussion for jassbot-bp.

Personally i would dump the json à la

insert into annotations (fnname, anname, value)
values ("<function name>", "westrings.json", json("<parsed json>"))

and then one can use either the sqlite json functions or create specific views, etc.
Again, this is more on the developer/backend side and not really visible in the jass files.

I just don't really know how to treat this: for me it's auto-generated data and should not be put into the human written jass files. But i also get the discovery aspect.

[Luashine]

@lep Uploaded the converted files. The Russian translation broke the original strings in some places by translating type names by mistake. But viewed as just text data it doesn't matter for the JSON structure.

triggerdata-1.36.zip

Your comment has a lot to go through, I will think about it later. Happy New Year!