Hierarchical version matching.

[Victov]

Hey @gtker! Today I was doing some testing to see if I want to take the leap to convert my entire project (wrath-rs) to use the wow_messages crate.

As an experiment I've been fiddling around with the wowm language and ran into a little issue that restricts how usable it is (or perhaps I'm misunderstanding).

When I enter a versions = "3" tag, I expect that object to match for all Wotlk versions. However, when I create a message for versions = "3.3.5", during parsing it complains that there is no version of the first object for 3.3.5. Are versions not matching in a hierarchical way? I expect a message for 3.3.5 to be satisfied with an object for all versions of 3.

I've attached a screenshot with my code and the resulting error.

[gtker]

Hey, cool that you're wanting to use it. I gotta be honest right now it's mostly optimized completely for 1.12 and I even removed some code that made it more capable of handling versions that weren't 1.12.

As far as I can see what you're attempting to do should definitely work, I'll try to recreate locally and see if I can figure out what's wrong.

Just for your information:
I've been considering supporting Wrath/other versions than Vanilla for a while but I didn't think anybody would actually use it. If you intend on using it/checking it out I'll definitely get it ready to create a library for Wrath, but there are a few issues that I'll need to solve first:

1. wow_srp needs to be changed to accommodate the Wrath header encryption. This will be a breaking change for those who use the Vanilla HeaderCrypto (probably just myself).
2. The helper types need to be defined for Wrath. Possibly the only things that need to change are the UpdateMask and AuraMask. I only really thought this would ever be used for Vanilla so I'm fudging a few things and I will likely need to rethink how I'm generating the UpdateMask setters/getters.
3. A new wow_wrath_messages must be created in the repo to house all the Wrath messages.
4. (Maybe) a new wow_wrath_base must be created. I have this for Vanilla because I want to be able to share types with another crate I'm working on called wow_vanilla_common that has things that are easy to put in a crate but tedious to figure out like stats per level, teleport locations and other things that can be implemented as free functions or consts.

[gtker]

After recreating it seems I can't even add the Class type with versions = "3" since I removed anything but the code for X.Y versions.

Currently I only really support X.Y versions like 1.12 because it makes crate paths easier and there aren't any differences for 1.12.X, do you know if there are any changes for 3.3.X or would it be fine to lump all messages into just 3.3 @Victov?

[Victov]

wow_srp needs to be changed to accommodate the Wrath header encryption. This will be a breaking change for those who use the Vanilla HeaderCrypto (probably just myself).

From my brief analysis I thought I could avoid this by just writing the packet to a vec and having my "legacy" code do the encryption instead of using the write_encrypted methods directly on the stream. Perhaps it would be sufficient to put the wow_srp dependency behind a feature? I don't expect I'll be using it soon since I have a framework and encryption setup already functional right now.

A new wow_wrath_messages must be created in the repo to house all the Wrath messages.

Doesn't it make sense to just put them in the same folder as the vanilla messages? Many (especially simple) messages are unchanged since vanilla, so that code could be re-used instead of duplicated in another folder. So the wowm files have all the definitions for all the versions, and then they export to a single folder (like world_messages) and then in the server I can just use world_messages::version_3_3_5::*. And then the generated code can mod version_3_3_5 { pub use version_1_1_12 } whenever messages have not changed since vanilla.

do you know if there are any changes for 3.3.X or would it be fine to lump all messages into just 3.3?

I think in reality nobody would ever make a server for wrath that isn't 3.3.5a, so I could just input 3.3.5 everywhere. However, from the perspective of having the information "make sense" (since it's also exported to documentation) it would be neat to see "this thing is stable for all of wrath" in one glance because the version is 3 and not 3.3.5. The problem is really not a practical one, but more from an information point of view.

[gtker]

From my brief analysis I thought I could avoid this by just writing the packet to a vec and having my "legacy" code do the encryption instead of using the write_encrypted methods directly on the stream.

That could work.

Perhaps it would be sufficient to put the wow_srp dependency behind a feature?

I have already considered doing this but just didn't have the time.

I don't expect I'll be using it soon since I have a framework and encryption setup already functional right now.

That's fair, although I think for completeness sake I want to get the Wrath header encryption into wow_srp before bumping it to 1.0.0. Do you know if TBC uses the Wrath encryption or the Vanilla encryption?

Doesn't it make sense to just put them in the same folder as the vanilla messages?

That is definitely a solution. It was actually the solution I wanted to go with at the start but compile times just became massive and I wanted the crate scope to be more focused since I didnt't intend on filling out anything other than 1.12 for my own purposes. Obviously it can just be put behind features.

And then the generated code can [...]

It actually already works like that for example for types that already have a lower version. The only problem with this is that I can't cfg out the 1.2 version since then the 1.12 version doesn't have anything to reference. I'm considering just creating a "base" struct that's always active that 1.2 and 1.12 just then use instead, but this will also have to be implemented.

I think in reality nobody would ever make a server for wrath that isn't 3.3.5a, so I could just input 3.3.5 everywhere. However, from the perspective of having the information "make sense" (since it's also exported to documentation) it would be neat to see "this thing is stable for all of wrath" in one glance because the version is 3 and not 3.3.5. The problem is really not a practical one, but more from an information point of view.

I totally get that and I'm glad you're worried about having the correct information, but my question was more of a usability question from the POV of the rust crate. Like you say, nobody is probably going to create a non-3.3.5 Wrath server.

3.3.5a

Also just realized that my versioning only supports integers and not a/b. Do you know if this matters for messages? As far as I recall 3.3.5 is the only verison that has non integer fields in it and it would be a little annoying to handle the special case if it didn't matter.

---

If you want to get to prototyping I think everything should work if you just enter the version for everything as 3.3.

Let me know what you think and if you're interested in using it I'll get to working on the sticking points. To reiterate:

All WoW versions in the same crate

My main concerns with this are:

• compile times, which can be countered with feature cfgs. I don't think this is actually an issue with cfgs.
• less "focused" crate topic. This will mean things like documentation will have to be more general instead of only being Vanilla/Wrath focused. I don't think this will actually matter all that much either.
• the helper types. This will obviously also just have to be put into appropriate modules and cfgd out. My main worry is the UpdateMask type which generating for 1.12 through a long array of offsets into this. It's still experimental but it seems to work fine. This is an example of it in action on an auto generated test. This will probably also be fine and I'm just dreading the work. :)

No wow_srp support for Wrath

This is fine. Eventually I'll get Wrath support into wow_srp and then into the messages, but you can just keep using your own system. I'll add a cfg at some point when I get the time.

Versioning and the Rust crate

Initially I had it so that objects would be placed in crate::vX::vY::vZ::OBJ, but this got really annoying when I only had 1.12. What do you think is the solution here? Analyzing the objects and if there are no X.Y.Z but only X.Y messages it should just create a version_X_Y like currently and otherwise it would create the full version_X_Y_Z and version_X_Y_Z+1?

[Victov]

Do you know if TBC uses the Wrath encryption or the Vanilla encryption?

I believe they chanced encryption in TBC, but don't quote me on that. I have not researched TBC nearly as much as Wotlk.

my question was more of a usability question from the POV of the rust crate.

The usability is fine if I just create all the wowm data with 3.3.5. For most messages I can set versions to both 1.12 and 3.3.5 since they are unchanged. Perhaps the docs could clarity a bit now the tag-all and the message-specific versions interact? Does a message-specific version overwrite a tag-all or does it get appended?

As far as I recall 3.3.5 is the only verison that has non integer fields in it

As far as I know the a in 3.3.5a is just a "human readable" version name. Your system is solid enough, and does not need alphanumeric support.

Let me know what you think and if you're interested in using

Absolutely interested, since it improves readability a ton. Don't expect I'll have time soon to do the full refractor, and there's some humps in the road to solve, but refactoring to use this crate has been on my list for a long while now. I'll definitely keep you posted on my thoughts and progress

Initially I had it so that objects would be placed in crate::vX::vY::vZ::OBJ, but this got really annoying when I only had 1.12. What do you think is the solution here?

Now that I'm thinking about this more: the absolutely most perfect solution from the crate end-user perspective would be to specify the version as a dependency feature. So in my server toml: wow_messages = {version="1.0", features=["messages_3_3_5"]} and then the server code becomes super clean use wow_messages::world::CMSG_ATTACK without caring about the version in the use statements.

[gtker]

Perhaps the docs could clarity a bit now the tag-all and the message-specific versions interact? Does a message-specific version overwrite a tag-all or does it get appended?

Sure, I'll update it to be more understandable. tag_all just appends.

Absolutely interested, since it improves readability a ton.

Cool. I'll get on making it more suitable for Wrath. As I said you can prototype by just giving everything version 3.3.

Now that I'm thinking about this more: the absolutely most perfect solution from the crate end-user perspective would be to specify the version as a dependency feature.

Sadly this can't really be done. Cargo features must be additive as described here and here.

For example, if you add both feature messages_3_3_5 and feature messages_1_12, which version should actually be used?

Since the dependency resolution only compiles one version of a crate with all feature flags that are added by all dependencies this could easily break builds.

[Victov]

Sadly this can't really be done. Cargo features must be additive

Ah. That makes sense, my bad. Then as a second best option one could argue that of every expansion, only the latest patch is interesting. So you could (still also using features per expansion) make it so you can do use wow_messages::wrath::world::CMSG_ATTACK and use wow_messages::vanilla::world::CMSG_ATTACK. Corresponding to 3.3.5 and 1.12, disregarding the versions in between. You will lose a tiny bit of flexibility, but have a huge gain in readability. And how many people are interested in making a server for 1.6.0 from scratch? I doubt anybody would. Internally you could still use the version numbers, just expose readable shortcuts for each end-of-expansion patch?

Edit: or swallow the code duplication (since its mostly generated anyway) and indeed just make crates per expansion. But again, if I'm already including a wotlk-specific crate then I would like to avoid having to write the version with every use statement. It's still fair to assume the end-of-expansion patch imho

[gtker]

Then as a second best option one could argue that of every expansion, only the latest patch is interesting.

I think you're right. Realistically I think vmangos is the only source of vanilla messages that aren't 1.12, and I don't think there's any info at all on non-max version which means that I would have to manually reverse them which isn't happening.

IMO even when your objective is to create a rolling version server like vmangos it's a mistake to also use older version clients since it just massively complicates almost everything.

What are you plans in regards to using the crate? Like are you going to transcribe all messages first and then use them, or gradually transcribe as you need them? Are there any things that you would need from me or from the crate before you can do something else?

[Victov]

even when your objective is to create a rolling version server like vmangos it's a mistake to also use older version clients

Yes absolutely!

What are you plans in regards to using the crate?

I was thinking to make changes as I need them. Realistically a full server is waaaay to much work for one person, it will never be done. If I have to verify and/or update each message for wrath before I start refactoring my server, I will be bald and dying before I can even start on server logic. It would be a lot of work to bring messages up to date for a server that will likely never use them since I'm all alone.

First I refractor all the messages that I currently have working in my server, add 3.3 to the wowm as I go, and repeat that process as I add more server functionality. Probably do pull requests to this repo in chunks as I go.

[gtker]

If I have to verify and/or update each message for wrath before I start refactoring my server, I will be bald and dying before I can even start on server logic.

Haha, true. :) I did all the 1.12 messages at the same time as writing the parser/compiler and it didn't take that long, but I also haven't verified all of them. I'm currently writing a server and as I get verified messages for them I add a test for that message. It would be appreciated if you could do the same, but I understand if you don't. Depending on time available I might help you out since you have helpfully narrowed the scope of messages I need to collect to only 1.12/2.4.3/3.3.5a.

I think to help out it might be useful to add the possiblity unimplemented or such statement to the body of messages in the wowm to signify that they exist but that they don't have a verified body. In rust it would just return a Vec<u8> so that it's still relatively useful. It should be relatively easy since it would just be an alias for an endless u8 array (u8[-]).

Probably do pull requests to this repo in chunks as I go.

That would be very appreciated.

Just to reiterate for myself, the changes would need to:

• Rename wow_vanilla_messages to wow_world_messages
• Rename wow_vanilla_base to wow_world_base.
• Move only 1.12 messages to crate::vanilla and not add any other vanilla versions to the crate.
• Add support for versions other than X.Y.
• Add support for 3.x.y.z messages in crate::wrath.
• Split wow_message_base into wrath, tbc, and vanilla.
• Add helper types for 3.3.5 (and possibly 2.4.3).
  • Guid
  • AuraMask
  • UpdateMask
  • Expected functions
• Cfgs for vanilla/wrath/etc.
• Add Wrath support to wow_srp.
• Add Wrath header support to messages.

[Victov]

Yup, that list sounds about right.

In the meantime I will clean up my server in preparation for implementing this crate. I have been so eager to add features that it has gotten quite messy. Don't feel pressured to rush these things, I got plenty of cleanup to keep me busy ;)

[gtker]

I consider this closed for now since I fixed versions basically being non-functional, as well as adding complete support for Wrath and TBC.

The only thing missing are the AuraMask types for Wrath and TBC, but they are relatively trivial to implement and will be implemented when we encounter messages that need them.