03-28-2018, 02:03 PM (This post was last modified: 03-30-2018, 01:28 AM by TwiliChaos.)
So I put in quite a bit of work trying to get animations ripped. For the most part, I've figured out the format, but some bytes are encrypted and I only have 1/5 of them solved, as of writing this. Basically the animation chunk goes as followed.
PLM Chunk:
4 bytes String "PLM$"
2 bytes > unknown_size_A anim_track_count
20 bytes > some unknown value
anim_track_count * 7 bytes > the 6th byte 6th and 7th byte are a short for the maximum size of an animation track, which I put in track_size array
And right after bone data
I don't know how to explain the format without a bit of indentation, so here you go
Track (amount of tracks equal to anim_track_count)
24 bytes unknown, possibly position values for UI elements
Bone data, amount equal to bone_count
Dynamic bytes > encrypted values that contain the amount of Quaternion keyframes.
quaternion_keyframe_count * 8 > quaternion values
Dynamic bytes > encrypted values that contain the amount of Position keyframes.
position_keyframe_count * 12 > position values
Dynamic bytes > encrypted values that contain the amount of Scale keyframes.
scale_keyframe_count * 12 > scale values
Dynamic bytes is equal to (int(ceil (track_size[current_track] / 8 + 1))), and if the first byte is 0x00 then dynamic bytes' size is 1 and keyframe count is 1.
The values of dynamic bytes are plugged into a 256 length array that outputs values 0 through 8, and these values are added together to get the final value for that bone's quaternion, position, or scale. (Scale seems to always be empty with the models I've checked)
Code:
FF FF FF FF FF FF FF FF FF FF 01 -- Quaternion keyframe count, 10 bytes of 8 and 1 byte of 1, adds to 81
EF 0C A6 6C C6 C9 A6 D9 -- Quaternion values (81 * 8 bytes)
EE 0C A7 6C C7 C9 A7 D9
EB 0C AB 6C CC C9 AA D9
...
EF 0C A6 6C C6 C9 A6 D9
00 -- position keyframe count (empty)
C7 4F 00 00 F4 40 00 00 4A 1D 00 00 -- position value
00 -- scale keyframe count (empty)
00 00 01 00 00 00 01 00 00 00 01 00 -- scale value
Unfortunately I'm starting to find that some of these values are likely incorrect, and with this being such a tedious and very error-prone process where any mistake, if not caught early, will undo a ton of work, I need to know if there is a quicker way to get this array filled out with 100% accuracy.
Also, some of the values seem like they work for newer models, but not work for older ones, such as 0x1F as 5 works for newer ones, but 0x1F as 4 works for older ones. It's difficult to say if the newer models add 1 to the final number, the older ones subtract 1, or if there's a flag somewhere that tells the game to add or subtract... I don't know.
Edit: In regards to the existence of a flag that may add or subtract from the keyframe count, that might be true. With a couple files I've looked at that have been a pain in the neck in that regard, it looks like in the PLM$ area with the track info, the 1st byte's 2nd nibble is 0 for unaffected keyframe counts, and 8 for keyframe counts that subtract 1 to get the final result. I don't know yet if the first nibble has much relevance, as it's mostly C and in one example 4. This byte is also always the same for each track in the file.
Edit2: Okay, if there is a flag, what I pointed out is almost certainly not it.
Edit3: Confirmed because of Valkyrja transmog 3's idle animation having 361 frames, the 6th and 7th byte are 2 bytes frame count rather than one.
An update to my findings, it's starting to look like there IS a correlation between the 20 bytes unknown after anim_track_count, and whether the keyframe value is modified after counting. Looking through a bunch of files, the ones with keyframe value change after count has what seems like a bunch of random byte values, whereas the ones that do not have those 20 bytes all zeroed out, as well as a small part of the 55 bytes further up in the file zeroed out too. I think this is a rather solid correlation right now, with a small possibility of being wrong. Checking through a dozen other files for comparison, this looks like the only "flag" that can be checked against. This change appears to have been added in the same update that brought the Frankenstein transmog. I do not remember when it was added, but it looks like every model made after that has zeroed bytes, and every model before it has random bytes.
So, to be more on the point, if those 20 bytes seem random, add the keyframes together and then subtract by one to get the final value. If those 20 bytes are zeroed out, then they're only added together. Sheesh c2u you made this quite a ride, but I'm getting closer.
04-22-2018, 09:27 AM (This post was last modified: 07-19-2018, 11:45 PM by TwiliChaos.)
Animations can now be ripped. In the past couple days I've learned quite a bit about the format, and just how far off I was.
Tested the script on a variety of older and newer models. The MaxScript file I included is not optimized at all so after a few uses Max will return a USE. If anyone cares to optimize it they can.
I'll probably regret sharing this, since around a month ago I shared my rip (back when I had to hardcode it) to Reddit of some unused animations for a model (RTA Panda transmog before rewards were given) and basically immediately after, C2U decided to replace all the unused animations in the next model (RTA Sky Dancer) with walk cycles just to spite me. From what I can tell, unused animations were never removed before.
Welp, onward to buildings and environments, and eventually particle effects. Not that anyone would care for the latter
Edit: There was some unneeded code that I put in a while ago while trying to fix an issue. Regardless, it didn't work, and I forgot to remove it. It copied the first frame in a track to the last frame in a track, which is a problem for non-looping animations (i.e. World Boss). I updated the script.
Late edit 2018/07/19: I gave up on the script a lil while back. The iteration I uploaded, if I remember it correctly, is mostly correct, but has issues with the last frame of bones in animations. I once tried to fix that by copying the first frame of a track to the last frame, but that does not work on animations that do not loop, for obvious reasons.
09-13-2020, 08:07 AM (This post was last modified: 09-15-2020, 05:18 AM by RedBear.)
Hi, TwiliChaos.
Thanks for sharing your awesome script.
Can you update your script as it doesn't work with models from the latest version of the game?
Textures cannot be opened at all, and the model is gray.
Thanks.
PS: It seems that they have changed the texture format. Now the file signature is 4A6F6B657200 (Joker.), rather than 89504E47 (�PNG).
PPS: I was able to figure out the textures. Now textures are stored in JPEG File Interchange Format (JFIF) format, instead of PNG format.
Textures chunk:
--header (24 bytes)
4 bytes > texture ID
4 bytes joker_size > size of joker chunk data (if 0, no texture???)
6 bytes magic > null terminated string "Joker."
2 bytes unknown > F101 => 497
4 bytes diffuse_size > size of the diffuse texture data
4 bytes alpha_size > size of alpha texture data (if 0, no texture)
(09-13-2020, 08:07 AM)RedBear Wrote: Hi, TwiliChaos.
Thanks for sharing your awesome script.
Can you update your script as it doesn't work with models from the latest version of the game?
Textures cannot be opened at all, and the model is gray.
Thanks.
PS: It seems that they have changed the texture format. Now the file signature is 4A6F6B657200 (Joker.), rather than 89504E47 (�PNG).
PPS: I was able to figure out the textures. Now textures are stored in JPEG File Interchange Format (JFIF) format, instead of PNG format.
Textures chunk:
--header (24 bytes)
4 bytes > texture ID
4 bytes joker_size > size of joker chunk data (if 0, no texture???)
6 bytes magic > null terminated string "Joker."
2 bytes unknown > F101 => 497
4 bytes diffuse_size > size of the diffuse texture data
4 bytes alpha_size > size of alpha texture data (if 0, no texture)
(05-27-2015, 08:50 AM)TGE Wrote: If you're just interested in ripping the meshes, you can use the MaxScript I attached.
I didn't manage to figure out the bones though, so that's kinda sad.
It exports the pngs as well.
05-15-2023, 04:17 PM (This post was last modified: 05-15-2023, 04:19 PM by Neostatos4.
Edit Reason: I didn't wrote one of my sentence properly. English is not my native language.
)
Hey RedBear, do you have an updated script of the "SummonerWar_DAT_Final.txt" working with 3dsmax or blender with the adjustement that you explained? The old script seem to work with few recent model, such as Shadow Claw and the second awakening of the wandering knight, but many other unit (collab unit or very new one) and the costume does have textures not working, probably a story of jfif format.
As far as I know, the script SummonerWar_DAT_Final have 517 lines of code and start talking about textures on line 207.
It talk about textures on line 207-210-212-216-217-219-220-221-223-227-231-263-264-265-266-267-270-513.
I don't think replacing all the "png" words by "jfif" will work... I don't know. Maybe you were trying to say that the code need to be adapted to each ".dat" file that don't work with the finale script?
I took the data of few model I tried to get, if someone can try to extract them and upload them here that would be great, or just maybe help me understand them with the hex editor(I don't care about the bones, just the model and textures would be great).
(05-15-2023, 04:17 PM)Neostatos4 Wrote: Hey RedBear, do you have an updated script of the "SummonerWar_DAT_Final.txt" working with 3dsmax or blender with the adjustement that you explained? The old script seem to work with few recent model, such as Shadow Claw and the second awakening of the wandering knight, but many other unit (collab unit or very new one) and the costume does have textures not working, probably a story of jfif format.
As far as I know, the script SummonerWar_DAT_Final have 517 lines of code and start talking about textures on line 207.
It talk about textures on line 207-210-212-216-217-219-220-221-223-227-231-263-264-265-266-267-270-513.
I don't think replacing all the "png" words by "jfif" will work... I don't know. Maybe you were trying to say that the code need to be adapted to each ".dat" file that don't work with the finale script?
I took the data of few model I tried to get, if someone can try to extract them and upload them here that would be great, or just maybe help me understand them with the hex editor(I don't care about the bones, just the model and textures would be great).
(05-15-2023, 04:17 PM)Neostatos4 Wrote: Hey RedBear, do you have an updated script of the "SummonerWar_DAT_Final.txt" working with 3dsmax or blender with the adjustement that you explained? The old script seem to work with few recent model, such as Shadow Claw and the second awakening of the wandering knight, but many other unit (collab unit or very new one) and the costume does have textures not working, probably a story of jfif format.
As far as I know, the script SummonerWar_DAT_Final have 517 lines of code and start talking about textures on line 207.
It talk about textures on line 207-210-212-216-217-219-220-221-223-227-231-263-264-265-266-267-270-513.
I don't think replacing all the "png" words by "jfif" will work... I don't know. Maybe you were trying to say that the code need to be adapted to each ".dat" file that don't work with the finale script?
I took the data of few model I tried to get, if someone can try to extract them and upload them here that would be great, or just maybe help me understand them with the hex editor(I don't care about the bones, just the model and textures would be great).
Is there any easy way for me to turn this .dat file into an OBJ or STL? my knowledge is very limited on this subject
Well, it really depend on which one you want to get? In the link I have put in my previous message, in the folder zfailures, you can find some .fbx file that you can easily import into Blender to export them into .obj or .stl, Noesis can also do it for you.
There are the 3 costumes of the beastrider, 2 costumes of the archangel, the normal and awakened shadowcaster and vampire lord.
The game files are about 5Gb, I can put a mediafire link if you want to search a monster you want, but I might delete the link after one or two week because 5Gb is heavy. For example, the third costume of the archangel (japanese schoolboy) can't be imported(object.transform problem),
but some work, such as the chakram/boomerang girl third costumes.
