[PS2] Need help ripping Chobits - Chiidake no Hito

[DeeDee68]

Recently, I finally found a rare game called Chobits - Chiidake no Hito for the PS2. A visual novel, worth about 2 GB of content. The thing is, it has way more sprites than the GBA game and with good reason. Now, I don't really want to resort to just using the emulator and take screenshots (who does), but no guide online has helped me thus far.

I extracted the iso's contents and I'm greeted with... this.
The modules folder has this.
While the "A" folders have... this.
What do I do with this. Or. What program do I use? I need to rip these sprites and give the favor to the world. There's literally no other guide that has helped me, and I need some insight here. Or else, I'll just have to take screenshots and suffer. Thanks for your attention, and please help.

[Raccoon Sam]

The first thing I'd do is scan everything with TextER and see if there's any TIM2s there.
Usage:
TextER -e Chobits_or_whatever.iso -tm

[DeeDee68]

The first thing I'd do is scan everything with TextER and see if there's any TIM2s there.
Usage:
TextER -e Chobits_or_whatever.iso -tm

Maybe I'm really really really stupid, but when I open the program, even as admin, it just immediately closes. Am I doing something wrong?

[Raccoon Sam]

The first thing I'd do is scan everything with TextER and see if there's any TIM2s there.
Usage:
TextER -e Chobits_or_whatever.iso -tm

Maybe I'm really really really stupid, but when I open the program, even as admin, it just immediately closes. Am I doing something wrong?

Generally speaking, there are two kinds of programs: Command Line Applications and Graphical User Interface Applications. Most stuff you use is the latter, but there are cases where GUI is not needed/not worth to implement so you have to work with these Command Line programs.
I'm a Mac User so I don't know how Windows does that stuff but I'm guessing you should google "How to Run Command Line Programs on Windows" or "understanding the command prompt" or something.

Alternatively, make a new text file that contains ONLY this:
TextER -e Chobits_or_whatever.iso -tm
And save it as "run.bat" to the same folder where both the ISO and the TextER program reside. Then open (or, run) the run.bat file.

[DeeDee68]

The first thing I'd do is scan everything with TextER and see if there's any TIM2s there.
Usage:
TextER -e Chobits_or_whatever.iso -tm

Maybe I'm really really really stupid, but when I open the program, even as admin, it just immediately closes. Am I doing something wrong?

Generally speaking, there are two kinds of programs: Command Line Applications and Graphical User Interface Applications. Most stuff you use is the latter, but there are cases where GUI is not needed/not worth to implement so you have to work with these Command Line programs.
I'm a Mac User so I don't know how Windows does that stuff but I'm guessing you should google "How to Run Command Line Programs on Windows" or "understanding the command prompt" or something.

Alternatively, make a new text file that contains ONLY this:
TextER -e Chobits_or_whatever.iso -tm
And save it as "run.bat" to the same folder where both the ISO and the TextER program reside. Then open (or, run) the run.bat file.

Still didn't work for me. I tried about this and still researched for long for ways to try and rip it. No go.

[Raccoon Sam]

Welp, at least now we know that the game does not use TIM2 textures.
I'm taking a look myself, I'll keep you updated.

EDIT: welp, looks like there actually are a lot of TIM2 files inside, they're just compressed. Looking into it.....
EDIT EDIT: The TIM2 files are compressed with an unknown compression but I've got some leads. Trying to write an experimental decompressor....
EDIT EDIT EDIT: Yeeaaaaah the files are standard TIM2 files, packed with a typical LZish compression. <=$7F are raw bytes, anything above that is a compression word (2 bytes) where 6 bits are for amount and 10 bits for distance.
I wrote a decompressor and it yields good results but it fails on some files, not sure why. Investigating....
Here's A001, A002 and A004 files decompressed.

[DeeDee68]

Welp, at least now we know that the game does not use TIM2 textures.
I'm taking a look myself, I'll keep you updated.

EDIT: welp, looks like there actually are a lot of TIM2 files inside, they're just compressed. Looking into it.....
EDIT EDIT: The TIM2 files are compressed with an unknown compression but I've got some leads. Trying to write an experimental decompressor....
EDIT EDIT EDIT: Yeeaaaaah the files are standard TIM2 files, packed with a typical LZish compression. <=$7F are raw bytes, anything above that is a compression word (2 bytes) where 6 bits are for amount and 10 bits for distance.
I wrote a decompressor and it yields good results but it fails on some files, not sure why. Investigating....
Here's A001, A002 and A004 files decompressed.

Well, it is a very rare game, and the version I found was in a spanish blog of an otaku who also uploads PS2 rare visual novels or animes, and it took me a LOT to find it. Nice that you're getting results! I'd like to know how you did it specifically since it didn't work for me. Thanks for the help, by the way!

[Raccoon Sam]

whole lotta research:

Spoiler

Well, I had lots of trouble finding the ISO too but eventually I did find it.
The process is somewhat complex and has high prerequisites: starting to decode this type of problem requires intermediate knowledge of proprietary file formats, data compression, programming and getting your way with a hex editor.

But basically what I did was check out the files with a hex editor to get a grasp on what they might be – their headers said TIM2 so I thought they would be TIM2 files. They didn't open in any TIM2 viewer, so I concluded they're compressed.
The compressed file's first $16 bytes were

Code:

01 2C 80 07 54 49 4D 32 04 00 01 00 94 00 04 70

but according to the TIM2 specification, every TIM2's first $16 bytes should be

Code:

54 49 4D 32 04 00 01 00 00 00 00 00 00 00 00 00

So there were apparently some similarities. Ignoring the first four bytes of the compressed file, they were the same, but the compressed file has 94 00 where the real file should have 00 00 00 00 00 00 00 00. This was my first clue.
As I couldn't figure it out, I booted PCSX2 and ran the game. Immediately when the game starts, you see the "Broccoli" image. I generated a Save State and exited the emulator.
Save states are just snapshots of the console's RAM at runtime, and because I could see the graphics in-game, I knew they were decompressed to RAM. Therefore the RAM (or... save state file) contains an uncompressed TIM2 now.
I opened up the PCSX2 save state in a hex editor only to find it starts with the bytes "PK". I knew that any file that starts with "PK" is a ZIP file, so I renamed the state of a ZIP and decompressed it. There were files and folders inside - it appears that PS2 save states are more complex than say SNES save states.
I didn't know which file I should look into so I opened them all in a hex editor and looked for "TIM2". All returned no results except one, eeMemory.bin. I knew this was my file where the pure TIM2 would reside.
There were many results for TIM2, because the same state holds many other compressed TIM2 files inside it, probably waiting to be decompressed. I searched and searched and amidst all the 94 00-type of TIM2s, I found one 00 00 00 00 00 00 00 00-type of TIM2. This had to be the Broccoli image. I copied the bytes out to a new file and started comparing it to the compressed TIM2.
Essentially, at this point I knew what a compressed file looks like and what a decompressed file looks like. All I was left to figure out was "what happens in-between", as in, how does the game transform the compressed 10kb TIM2 to the decompressed 77kb TIM2?

This is a typical analysis method I use when I'm in a situation like this; I load both files' bytes on top of each other and look for similarities and indent the gaps with spaces. Here's a screenshot of my text editor:

On the top row, you see the compressed bytes. On the bottom row, you see the decompressed bytes. I've indented them so they match as much as possible and started writing my notes below.
The 1-byte commands were immediately obvious. $07 means "read 8 bytes as is" and $04 means "read 5 bytes as is" so it can be safely deducted that one-byte commands N mean "read N+1 bytes and print them to output.

I wrote down things I could deduce of the two-byte commands as much as I could, and started looking at the bytes and what they do. What made me think was the 80 07 command, because it output 2C 01 00, three bytes that were already in the output, eight bytes before.
My hypothesis was that these two bytes have the first byte for amount and the second byte for distance+1 (the second byte was 07 and the data was 8 bytes back, it seemed obvious)
I tested this hypothesis by looking at some other 2-byte commands, too, and they all obeyed this rule. Look at 88 33, then look what's $33(+1) bytes before it. This had to be it.

So I started writing a decompressor in Python and got great output. Some parts were wrong, though, as the hex editor told me when I did a binary diff between my output file and the file I snatched from the save state. It appeared to fail on parts where it needed to retrieve data from very far away.

So, my program can't "reach" back far enough to copy the bytes it needs. Why? I made some fixes to my program so it would be more verbose and fair enough, I noticed that on the 2-byte commands it fails on, the second byte was abnormally small, but the last bit of the first byte was set. It then dawned on me that the two-byte commands need to be read as bits, with a split of 6/10 instead of the typical 8/8. Tried that and boom – it works. I got the Broccoli image out and a binary diff between my output and the original file said "these files are identical"

So I started running the script on other files, and got a few others out, but it's still failing on some files. That's where I am now. Gonna try and figure it out later.

EDIT: I see now. The output files aren't "just" TIM2 files, but clusters of many TIM2 files, just glued together. A001 alone contains 11 TIM2s. Some are still fucked up but I'm getting there....

EDIT EDIT EDIT EDIT: I'm prettttttttty sure I nailed everything down now. As a test, I exhaustively checked a huge 14MB file, A021, and got these 159 images out. Every byte in the compressed file has been read, so I'm certain I'm not missing anything (except maybe CLUTs, so if you see some funny colors, they can probably be fixed super easily but you just need to do it manually...) Check these out:

Spoiler

https://www.dropbox.com/sh/s2r0m0cevgeg7...MRXfa?dl=0

I will get back to this!

yeaaah out of all the files, only A002, A00A, A007, A006, A009, A00E, A00D, A001, A00F, A00C, A004, A011, A00B, A020, and A021 were valid TIM2-only archives. The others? Still unknown, so if you find something's missing from these, please let me know, as they could be hidden in the other files.
Compression ratio is okay-tier. The packed files take 37.8MB, everything uncompressed totals around 79MB.
I did notice that there's a huge amount of MPEG streams, too. I'm guessing the game has a lot of FMVs?

but yeah anyway, i can't convert them to PNG but i asked for help in discord

[DeeDee68]

whole lotta research:

Spoiler

Well, I had lots of trouble finding the ISO too but eventually I did find it.
The process is somewhat complex and has high prerequisites: starting to decode this type of problem requires intermediate knowledge of proprietary file formats, data compression, programming and getting your way with a hex editor.

But basically what I did was check out the files with a hex editor to get a grasp on what they might be – their headers said TIM2 so I thought they would be TIM2 files. They didn't open in any TIM2 viewer, so I concluded they're compressed.
The compressed file's first $16 bytes were

Code:

01 2C 80 07 54 49 4D 32 04 00 01 00 94 00 04 70

but according to the TIM2 specification, every TIM2's first $16 bytes should be

Code:

54 49 4D 32 04 00 01 00 00 00 00 00 00 00 00 00

So there were apparently some similarities. Ignoring the first four bytes of the compressed file, they were the same, but the compressed file has 94 00 where the real file should have 00 00 00 00 00 00 00 00. This was my first clue.
As I couldn't figure it out, I booted PCSX2 and ran the game. Immediately when the game starts, you see the "Broccoli" image. I generated a Save State and exited the emulator.
Save states are just snapshots of the console's RAM at runtime, and because I could see the graphics in-game, I knew they were decompressed to RAM. Therefore the RAM (or... save state file) contains an uncompressed TIM2 now.
I opened up the PCSX2 save state in a hex editor only to find it starts with the bytes "PK". I knew that any file that starts with "PK" is a ZIP file, so I renamed the state of a ZIP and decompressed it. There were files and folders inside - it appears that PS2 save states are more complex than say SNES save states.
I didn't know which file I should look into so I opened them all in a hex editor and looked for "TIM2". All returned no results except one, eeMemory.bin. I knew this was my file where the pure TIM2 would reside.
There were many results for TIM2, because the same state holds many other compressed TIM2 files inside it, probably waiting to be decompressed. I searched and searched and amidst all the 94 00-type of TIM2s, I found one 00 00 00 00 00 00 00 00-type of TIM2. This had to be the Broccoli image. I copied the bytes out to a new file and started comparing it to the compressed TIM2.
Essentially, at this point I knew what a compressed file looks like and what a decompressed file looks like. All I was left to figure out was "what happens in-between", as in, how does the game transform the compressed 10kb TIM2 to the decompressed 77kb TIM2?

This is a typical analysis method I use when I'm in a situation like this; I load both files' bytes on top of each other and look for similarities and indent the gaps with spaces. Here's a screenshot of my text editor:

On the top row, you see the compressed bytes. On the bottom row, you see the decompressed bytes. I've indented them so they match as much as possible and started writing my notes below.
The 1-byte commands were immediately obvious. $07 means "read 8 bytes as is" and $04 means "read 5 bytes as is" so it can be safely deducted that one-byte commands N mean "read N+1 bytes and print them to output.

I wrote down things I could deduce of the two-byte commands as much as I could, and started looking at the bytes and what they do. What made me think was the 80 07 command, because it output 2C 01 00, three bytes that were already in the output, eight bytes before.
My hypothesis was that these two bytes have the first byte for amount and the second byte for distance+1 (the second byte was 07 and the data was 8 bytes back, it seemed obvious)
I tested this hypothesis by looking at some other 2-byte commands, too, and they all obeyed this rule. Look at 88 33, then look what's $33(+1) bytes before it. This had to be it.

So I started writing a decompressor in Python and got great output. Some parts were wrong, though, as the hex editor told me when I did a binary diff between my output file and the file I snatched from the save state. It appeared to fail on parts where it needed to retrieve data from very far away.

So, my program can't "reach" back far enough to copy the bytes it needs. Why? I made some fixes to my program so it would be more verbose and fair enough, I noticed that on the 2-byte commands it fails on, the second byte was abnormally small, but the last bit of the first byte was set. It then dawned on me that the two-byte commands need to be read as bits, with a split of 6/10 instead of the typical 8/8. Tried that and boom – it works. I got the Broccoli image out and a binary diff between my output and the original file said "these files are identical"

So I started running the script on other files, and got a few others out, but it's still failing on some files. That's where I am now. Gonna try and figure it out later.

EDIT: I see now. The output files aren't "just" TIM2 files, but clusters of many TIM2 files, just glued together. A001 alone contains 11 TIM2s. Some are still fucked up but I'm getting there....

EDIT EDIT EDIT EDIT: I'm prettttttttty sure I nailed everything down now. As a test, I exhaustively checked a huge 14MB file, A021, and got these 159 images out. Every byte in the compressed file has been read, so I'm certain I'm not missing anything (except maybe CLUTs, so if you see some funny colors, they can probably be fixed super easily but you just need to do it manually...) Check these out:

Spoiler

https://www.dropbox.com/sh/s2r0m0cevgeg7...MRXfa?dl=0

I will get back to this!

yeaaah out of all the files, only A002, A00A, A007, A006, A009, A00E, A00D, A001, A00F, A00C, A004, A011, A00B, A020, and A021 were valid TIM2-only archives. The others? Still unknown, so if you find something's missing from these, please let me know, as they could be hidden in the other files.
Compression ratio is okay-tier. The packed files take 37.8MB, everything uncompressed totals around 79MB.
I did notice that there's a huge amount of MPEG streams, too. I'm guessing the game has a lot of FMVs?

but yeah anyway, i can't convert them to PNG but i asked for help in discord

I'll see what I can do. Sorry for all the trouble, let's keep at it huh? And again, thanks for all this!

Checked them out. Huh, so far the backgrounds could be ripped. Interesting.

[Raccoon Sam]

Forgot the link

Here is every single TM2 I could get out
https://www.dropbox.com/sh/7id20jmb57mlz...YadCa?dl=0
I can't convert them to PNG though so you need to try Rainbow (https://github.com/marco-calautti) or Tim2View (https://www.romhacking.net/utilities/799/)

If some colours look funny, try another CLUT

[DeeDee68]

Forgot the link

Here is every single TM2 I could get out
https://www.dropbox.com/sh/7id20jmb57mlz...YadCa?dl=0
I can't convert them to PNG though so you need to try Rainbow (https://github.com/marco-calautti) or Tim2View (https://www.romhacking.net/utilities/799/)

If some colours look funny, try another CLUT

Ladies and gents.



It worked. Thank you so much!!!

[Raccoon Sam]

Superb!!

For the benefit of the collective, I would advise you to submit each and every asset that converts successfully in your TIM2 tool of choice to tSR. I know there's a lot of sprites and you gotta dick around with the CLUTs a bit, but just like you had trouble and woe finding these sprites, in time, someone else will lament the same. To ensure these Chobits sprites are archived forever, please consider submitting them!

[DeeDee68]

Superb!!

For the benefit of the collective, I would advise you to submit each and every asset that converts successfully in your TIM2 tool of choice to tSR. I know there's a lot of sprites and you gotta dick around with the CLUTs a bit, but just like you had trouble and woe finding these sprites, in time, someone else will lament the same. To ensure these Chobits sprites are archived forever, please consider submitting them!

Consider? You're saying it to the person who was going to submit them to tSR once they were successfully ripped. Even said it at the beginning of this post! "I need to rip these sprites and give the favor to the world."

It'll take me a lot of time of resizing and fixing since they're squished, but I'll do it.

[Raccoon Sam]

It'll take me a lot of time of resizing and fixing since they're squished, but I'll do it.

Come to think of it, I don't know if tSR has an official policy on sprite resizing. The game assets are stored squished like that but the PS2 resolution stretches them to 200% height I think. Many 3D games do this, as its much more efficient to render a say, 1920 x 540 "squashed" image and stretch it to full HD than to render a full 1080p frame.

Look at this sprite sheet:
https://www.spriters-resource.com/playst...eet/40645/


compare it to this in-game screenshot



See where I'm coming from?

I wouldn't resize the sprites at all to be honest. Let the end-user do the stretching, using their scaling algorithm of choice.

EDIT: asked tsr discord and the Word of God™ is:

1:1 would be the correct ones to submit. No harm in throwing in a second, "corrected" set on the same submission, though, but there are some cases where it won't be accurate to in-game by doing so.

so yeah don't resize them dude

[DeeDee68]

It'll take me a lot of time of resizing and fixing since they're squished, but I'll do it.

Come to think of it, I don't know if tSR has an official policy on sprite resizing. The game assets are stored squished like that but the PS2 resolution stretches them to 200% height I think. Many 3D games do this, as its much more efficient to render a say, 1920 x 540 "squashed" image and stretch it to full HD than to render a full 1080p frame.

Look at this sprite sheet:
https://www.spriters-resource.com/playst...eet/40645/


compare it to this in-game screenshot



See where I'm coming from?

I wouldn't resize the sprites at all to be honest. Let the end-user do the stretching, using their scaling algorithm of choice.

EDIT: asked tsr discord and the Word of God™ is:

1:1 would be the correct ones to submit. No harm in throwing in a second, "corrected" set on the same submission, though, but there are some cases where it won't be accurate to in-game by doing so.

so yeah don't resize them dude

I still need to fix them regardless, since the original files are extremely transparent. Unless TSR wants me to post them that way?