[PS2] Need help figuring out what to do next!

[Samevi]

Hello,

I'm trying to rip from the game Zero no Tsukaima - Maigo no Period to Ikusen no Symphony, something that's been a passion project of mine for the past while. Problem is, I'm stumped. I've got a file called SCENEDAT.bin which I know contains TIM2 files thanks to opening it in a hex editor, but I'm not sure how to turn this into a format from which I can readily grab said TIM2s.

File here, in case you want to check for yourself.

[Raccoon Sam]

Usually when dealing with this exact issue, TextEr (https://wiki.vg-resource.com/PlayStation...8TextER.29) is the best bet. However, in this case I got the error:

Code:

Looking for TIM2 and TIM3 file(s):

Sorry pal. No TIM2/TIM3 files have been detected

Which unfortunately means that the textures are compressed.
Naive compression analysis is possible and I've done it before but it requires a raw sample from a save state, so I don't know if I can do much without one.

If you can run the game in PCSX2, please make a save state from somewhere with graphics and we can try extracting the sample from memory. Bear in mind the scene has to contain legitimate graphics, so a scene with an FMV playing doesn't count.

[Samevi]

Usually when dealing with this exact issue, TextEr (https://wiki.vg-resource.com/PlayStation...8TextER.29) is the best bet. However, in this case I got the error:

Code:

Looking for TIM2 and TIM3 file(s):

Sorry pal. No TIM2/TIM3 files have been detected

Which unfortunately means that the textures are compressed.
Naive compression analysis is possible and I've done it before but it requires a raw sample from a save state, so I don't know if I can do much without one.

If you can run the game in PCSX2, please make a save state from somewhere with graphics and we can try extracting the sample from memory. Bear in mind the scene has to contain legitimate graphics, so a scene with an FMV playing doesn't count.

I guess compression should've been my first thought. Anyways, as requested, here's a PCSX2 save state that contains gameplay graphics.
Just to make sure I understand correctly, it's possible to rip all of the game's assets at once through this method?

[Raccoon Sam]

Excellent. After unzipping the save state I ran texter again through eeMemory.bin and this time got

Code:

Looking for TIM2 and TIM3 file(s):

Found 13 TIM2 and 0 TIM3 in eeMemory.bin.

Extraction in progress

TIM2: [100%]

Which makes sense because the game had decompressed a few TIMs into memory at runtime, and because a save state is a representation of the game state, texter was able to snag the decompressed TIM from there.
Here's the loot: https://drive.google.com/drive/folders/1...sp=sharing

Now it is time to analyse the compressed TIMs from SCENEDAT.bin and understand what the function is that transforms the compressed data into the uncompressed data.

I am analysing the differences now and it looks doable. Just need a bit of time to work it out, should be done soon hopefully.

Just to make sure I understand correctly, it's possible to rip all of the game's assets at once through this method?

yes. When done, my tool will decompress everything from the scenedat, and I'm assuming it's all the graphics in the game

[Samevi]

Excellent. After unzipping the save state I ran texter again through eeMemory.bin and this time got

Code:

Looking for TIM2 and TIM3 file(s):

Found 13 TIM2 and 0 TIM3 in eeMemory.bin.

Extraction in progress

TIM2: [100%]

Which makes sense because the game had decompressed a few TIMs into memory at runtime, and because a save state is a representation of the game state, texter was able to snag the decompressed TIM from there.
Here's the loot: https://drive.google.com/drive/folders/1...sp=sharing

Now it is time to analyse the compressed TIMs from SCENEDAT.bin and understand what the function is that transforms the compressed data into the uncompressed data.

I am analysing the differences now and it looks doable. Just need a bit of time to work it out, should be done soon hopefully.

Just to make sure I understand correctly, it's possible to rip all of the game's assets at once through this method?

yes. When done, my tool will decompress everything from the scenedat, and I'm assuming it's all the graphics in the game

That's beautiful. Thank you very much!

[Raccoon Sam]

Just giving a quick update: There are more compressed TIM2s for other graphics than that those of the character portraits and whatnot in the other DAT file on the disc. The tool should handle those as well.
And then the bad news: after some deeper analysis I've found the compression, like I said, only looks doable. In reality it looks deceivingly simple but in actually appears to be very complex, even nonsensical at times.
I was expecting to whip out a simple script that just churns through the TIMs and decompresses everything in one evening as a fun programming exercise, but now that I actually understand that the rabbit hole goes a bit deeper than I anticipated, I'll need way more time with this. Sorry!

In short: I'm still interested in reverse engineering the compression but it wasn't as simple as I thought.

[Samevi]

Just giving a quick update: There are more compressed TIM2s for other graphics than that those of the character portraits and whatnot in the other DAT file on the disc. The tool should handle those as well.
And then the bad news: after some deeper analysis I've found the compression, like I said, only looks doable. In reality it looks deceivingly simple but in actually appears to be very complex, even nonsensical at times.
I was expecting to whip out a simple script that just churns through the TIMs and decompresses everything in one evening as a fun programming exercise, but now that I actually understand that the rabbit hole goes a bit deeper than I anticipated, I'll need way more time with this. Sorry!

In short: I'm still interested in reverse engineering the compression but it wasn't as simple as I thought.

That's fine! I'm more than willing to wait for this - and with any luck, it could be useful for understanding other PS2 games with similar compression methods as well.
On that note, I'm more than willing to compensate you for your time (at least a little. I'm afraid my budget isn't too big right now) once all is said and done, just let me know.

[Samevi]

Hey, just wanted to bump and check if there was any progress on that decompression. Not trying to rush or anything though so no sweat if you don't have it yet.

[Raccoon Sam]

Hey there!
Bad news – I really can't wrap my head around it, I'm afraid :C

I've got samples for a compressed TIM2 file ("paktim" is just my nomenclature) extracted from the SCENEDAT and a legitimate, uncompressed counterpart TIM2 for it extracted from your save state. You can download them here.

The first eight bytes in the paktim are a header; 40 64 04 00 01 00 00 00 which should be read as two words, 00046440 00000001 which in turn just define the uncompressed size (0x46440 or 287808 bytes, which unsurprisingly is the file size of the legitimate TIM2 file) and the amount of files in the entry (1, which unsurprisingly is the ... uh, amount of files)


After that, you read a byte as the variable instruction.
Logical AND instruction with 0x1F and add 2 to it to get the variable amount.
Shift instruction right five times to get the variable method.

According to the method and amount, read bytes from the bytestream:
Method 7 is just "read amount bytes and write them to output".
Method 6 is "read 1 byte as surrogate and then repeat that byte exactly amount times to the output".
Method 0 is "repeat FF exactly amount times to the output".

The other methods are.... yeah, I dunno. Some methods look like they make sense but then the same method used later in the file does something completely different. I really can't figure out what the trick here is. Really hope someone else takes a look at this, it's driving me nuts!!

This is the TIM2 we're dealing with, for what it's worth.

[Samevi]

Hmm. I've done more snooping around in the game's files - NORMAL.BIN contains a repeated ASCII phrase CFILORUX. Usually this phrase is then followed by something along the lines of [^adg.lorux] or [^adgjlorux] or some variation of [^*lorux].
       

Anyways, point is I found it a bit odd. I did some digging and it turns out there's a very nice PDF on coding and decoding that mentions CFILORUX and EJOTY - a phrase I didn't find in NORMAL.BIN, but it WAS inside our file of interest, SCENEDAT.BIN, a grand total of twice.
   

From what I understand, CFILORUX and EJOTY are a method one can use to easily remember the numerical location of letters in the alphabet. It can also be used for specific codes, an example given being that "PICTURE" would become "QHDSVQF" by alternating between +1 and -1 (P +1 = Q, I -1 = H, etc.)
       

Now, I'm not sure if this helps in any manner of speaking for the decryption, but maybe someone could attempt to figure out a way to use them? 
I mean, the repetition of the phrases has to mean something, right?

[Samevi]

Quick addition:
CFI > ADG in CFILORUX is
C(3) - 2 = A(1)
F(6) - 2 = D(4)
I(9) - 2 = G(7)
This likely has something to do with the key for whatever CFILORUX/EJOTY style encoding they did wherever it may be.

[rufaswan]

This looks interesting! @Raccoon Sam Are you still working on it? Do you mind if I take a look?

- Rufas

[Raccoon Sam]

This looks interesting! @Raccoon Sam Are you still working on it? Do you mind if I take a look?

- Rufas

I have given up.
Please do take a look (and if you figure it out, please let us know how )

[rufaswan]

I can see why this game trip you up. I don't think I saw this kind of compression algorithm before. But I have a good understanding of it now.

Let's use an example for a start, you want to duplicate 0xC9 for 0x40 times. First you use method 6 for 0x1f+2 bytes, so you'll get "DF C9". For the remaining 0x1f bytes, you repeat the last command, or method 0, but with length 0x1d+2 instead, so you'll get "1D".

Hence, the compressed data to duplicate 0xC9 for 0x40 times is "DF C9 1D".

And the game keep the last 6 commands as history, or method 0 to 5.

The whole decompression goes:

read one byte for METHOD and LEN

Code:

METHOD = BYTE >> 5
LEN = BYTE & 0x1f

if the METHOD is 6, read next byte as BYTE, and then duplicate it for LEN + 2 times to output.

if the METHOD is 7, read the next LEN + 1 bytes as STRING and output it.

For both, add it to HISTORY[0] and push the queue down.

Code:

HISTORY[5] = HISTORY[4]
HISTORY[4] = HISTORY[3]
HISTORY[3] = HISTORY[2]
HISTORY[2] = HISTORY[1]
HISTORY[1] = HISTORY[0]
HISTORY[0] = current

For METHOD 0 to 5, set that as current, and re-queue everything before it. For example, METHOD 3 would

Code:

HISTORY[5] // do nothing
HISTORY[4] // do nothing
current = HISTORY[3]
HISTORY[3] = HISTORY[2] // re-queue
HISTORY[2] = HISTORY[1] // re-queue
HISTORY[1] = HISTORY[0] // re-queue
HISTORY[0] = current // re-queue

If the repeat METHOD is 6, duplicate the same BYTE but with new LEN + 2 times.

If the repeat METHOD is 7, then get SUB_POS and SUB_LEN from LEN

Code:

SUB_POS = LEN >> 2
SUB_LEN = LEN & 3

referring to the same STRING, output SUB_LEN + 1 bytes, starting from position SUB_POS.

That's about it.

The code has already uploaded to my github. You can run them by follow the Setup section of tutorial on my signature.

Then run

Code:

php  psxtools/ps2_zero_hd-bin.php  SCENEDAT.hd

I also noticed there are 3 Zero no Tsukaima games from the same company on PS2, and they are all using the same SCENEDAT hd+bin set. So I updated the script to support all 3 of them.

Sample




Have fun!
- Rufas

[Raccoon Sam]

You're a genius!! I figured there has to be some sort of "history" thing going on but I would've never in a million years figured that out.
Glad to see it in working condition. Gotta check out the other files too, compressed data is also present elsewhere besides SCENEDAT.

Elegant work, rufas!