(Another) Zelda Ripping Project!

[link20213D]

Por favor, olhe acima de você .. https://github.com/MeltyPlayer/OoT3D-Importer
Esta versão aprimorada por Meltyplayer suporta até animação.

Hello again
So i would like to thank you from my heart for making this file available.
Many years trying to fix these files and never got it.
So i´m here to thank everyone who helped me.

[nikespike]

Hello.

I am looking for the models for the BOTW - Soldier armour; I can find the helmet , but not the models for torso and legs. 
Does anyone have them?

[Starkium]

I'm back again, looking to get models from OOT 3d and MM 3d, what programs should I use? I have no problem doing textures by hand, but I need the vertex colors on the models intact. Google search results are a mess.
Currently got the stuff from models resource loaded up into a UE4 oculus quest project. Whole thing is working with vertex lighting.

 You got it almost 100% correct.





You're missing out on something important regarding the vertex colors. You need a shader that multiplies the vertex color output by 2.
This way, the colors get more contrast to really make the scene pop, similar to fake lighting baked in.
As well as have the scenes extracted with a tool and Blender version that doesn't botch values f.e. uv accuracy and missing alpha.

https://drive.google.com/file/d/1y-WyflS...sp=sharing //The game files (Some rips available online had damaged cmb files. This one is correct)
https://drive.google.com/file/d/12hjyGDA...sp=sharing //The importer for Blender which exports characters with rig.
https://drive.google.com/file/d/1rUvO95N...sp=sharing //The special Blender version that support vertex color alpha channel.
https://drive.google.com/file/d/1uEUb5Jf...sp=sharing //The game's soundtrack accurately converted to midi with correct soundfont. (Sounds like the real thing)
https://drive.google.com/file/d/1VXFWNYC...sp=sharing //Wanna get those eye and mouth textures from characters.



P.S. Sometimes you need to disassemble the cmb from the zsi. You can easily do this in HxD. All you need to do is cut out the first part of the file until 'cmb'.



After that, just rename the file extension to cmb.

EDIT - Whoops, I forgot to make links accessible for everyone. Now downloads should work.
          Also, huge thanks to M-1, as he's the one who created the importer.
          https://github.com/M-1-RLG

EDIT 12/11/21 - I suggest following Meltyplayer's model utility as well. It extracts the game's model library in it's entirety to both fbx and gltf2 and can be easily navigated through Noesis.
https://github.com/MeltyPlayer/FinModelUtility

OMG Thank you! Bro I've been porting OOT and MM to oculus quest as art research. This is going to be a tremendous help. 

Getting this as glTF2 is even more interesting. I wonder what I can do with that

*EDIT* Is there a list of what file is what somewhere? Opening all of them one by one to find stuff is annoying.

[Twilight Weaver]

Holy cow guys!! You've made some amazing progress since I checked this forum last!! Way to go!! Y'all have no idea how happy this makes me, keep up the great work everyone!!

[Twilight Weaver]

I'm back again, looking to get models from OOT 3d and MM 3d, what programs should I use? I have no problem doing textures by hand, but I need the vertex colors on the models intact. Google search results are a mess.
Currently got the stuff from models resource loaded up into a UE4 oculus quest project. Whole thing is working with vertex lighting.

 You got it almost 100% correct.





You're missing out on something important regarding the vertex colors. You need a shader that multiplies the vertex color output by 2.
This way, the colors get more contrast to really make the scene pop, similar to fake lighting baked in.
As well as have the scenes extracted with a tool and Blender version that doesn't botch values f.e. uv accuracy and missing alpha.

https://drive.google.com/file/d/1y-WyflS...sp=sharing //The game files (Some rips available online had damaged cmb files. This one is correct)
https://drive.google.com/file/d/12hjyGDA...sp=sharing //The importer for Blender which exports characters with rig.
https://drive.google.com/file/d/1rUvO95N...sp=sharing //The special Blender version that support vertex color alpha channel.
https://drive.google.com/file/d/1uEUb5Jf...sp=sharing //The game's soundtrack accurately converted to midi with correct soundfont. (Sounds like the real thing)
https://drive.google.com/file/d/1VXFWNYC...sp=sharing //Wanna get those eye and mouth textures from characters.



P.S. Sometimes you need to disassemble the cmb from the zsi. You can easily do this in HxD. All you need to do is cut out the first part of the file until 'cmb'.



After that, just rename the file extension to cmb.

EDIT - Whoops, I forgot to make links accessible for everyone. Now downloads should work.
          Also, huge thanks to M-1, as he's the one who created the importer.
          https://github.com/M-1-RLG

EDIT 12/11/21 - I suggest following Meltyplayer's model utility as well. It extracts the game's model library in it's entirety to both fbx and gltf2 and can be easily navigated through Noesis.
https://github.com/MeltyPlayer/FinModelUtility

So why should I use this method over the cmbviewer? like what are the benefits of this vs cmbviewer? it seems like more steps with the same end result. Especially since you can just dump the .dae file with cmbviewer and open that in blender and not have to worry about possible coding issues while manually converting the .zsi into a .cmb... so am I missing something? Does this "special" blender version automatically apply the alpha textures in with the material so you don't have to do it manually? Are a lot of files frequently missing the alpha when imported? I've not had a problem with that.. 
You said the IO scene master exports characters with rigs but so does cmb viewer? Explanations would be GREATLY appreciated.

Edit: I just did some tests and realized that using the cmbviewer the skeleton does not exported with the dae file, which I thought it did. The question I still have is this the only benefit? Granted it's a good benefit but only useful if you're trying to export characters, and also I did still use the cmb viewer, I just extracted the cmb instead of exported a dae file.

Edit 2: Importer works fine with the special version of blender but blender won't export a .dae or .fbx with textures and in the case of the .dae it won't even export the model correctly. Not sure why this is happening, I dumped the .dae file because that's the only way I could get it to dump the textures with it and I applied those textures to the model with the skeleton and the UVs were correct, however I tested the skeleton and it wasn't rigged to the model at all, so unless I'm doing everything wrong it appears that the current method still requires manual rigging.

[JamesO2]

Yeah, I'm not sure why some people are insisting that we need some ancient Blender version. I've seen that mentioned in other places too. Blender 2.8+ is perfectly capable of rendering and mixing vertex colors. Here's a screenshot comparing my scene (bottom) rendered in Blender 3.1 -- but I'm pretty sure the Vertex Color node has been there since 2.8 (or maybe 2.9).



My render is nearly identical. Not perfect, but good enough -- it gets the "feel" right, and it took minimal effort. The only issue I had was that the river texture was missing on the DAE export, so I had to assign that texture manually.

But yeah, I even applied the shader math the same as Starkium (oops, I meant Alvare) described:

Code:

vertexColor * 2 * baseColor

The Vertex Color Node is going into a multiply node where the RGB is multiplied by (2,2,2). The color sliders only go from 0.0 to 1.0, but you can actually type in a value of 2 manually and the math works out fine.

Blender does botch the color appearance a bit **at first glance**, but the actual underlying color vertex data shouldn't be destroyed I don't think -- we just need to adjust the output. It was an easy fix: go to "Render Settings > Color Management" and bump up "Exposure" from 1.0 to 1.2. Now the colors are like 99% correct. Oh, also make sure the other Color Management Settings are on "sRGB", "Standard," and "Medium High Contrast."

Alpha transparency works fine too, look at these trees:


You just have to plug the ALPHA channel of the image into the ALPHA slot of the BSDF shader:



Then make sure that Blend Mode is set to "Alpha Clip", with optional "Back Face Culling".


I setup some python scripts that automate this whole process.

• Select all the objects that need alpha transparency and run this script:
  
• https://github.com/JamesO2x/Useful-Blend...t_alpha.py
  
• Then select all the objects in the scene, and run this script to apply the OoT3D vertex color shading effect:
  
• https://github.com/JamesO2x/Useful-Blend...Shading.py
  

You will still have to set the "Color Management" settings manually for now. I might be able to add that to the script at a later date.

---

So that's what I figured out about these vertex colors. I do have one issue though: If anyone could help me out, I'd really appreciate it.

I can't figure out how to get the decompressed (is that the right word?) files for Majora's Mask. I was able to figure it out a few months ago for OoT3D and have a folder with all the ZAR files and such, which is how I was able to convert Hyrule Field and make these renders. But I forgot to write myself notes on how I did it, and I don't remember how.

I really want the maps from Majora, because I'm working on a Majora's Mask animation project, and I'd love to good clean map rips with all the vertex color data intact like this Hyrule Field test.

I have a few different CMB/ZAR model viewer programs already, but I didn't save whatever script converts the rom into a directory of model sources.

Here's a sneak peak at my project. I'm working on recreating the official Majora art effect in Blender:



And here's a video where I did some animation tests:


Does anyone have a "Majoras Mask Romfs.zip" or a method on how I can extract the Romfs files myself?

[scurest]

About vertex color formula vertexColor * 2 * baseColor:

That's also the formula the PS1 used (it let's you also brighten textures instead of only darkening them). The neutral point where there's no effect on the texture is vertexColor=0.5 since 2*0.5=1. But if it's like the PS1 it's supposed to be in sRGB space. Starting in Blender 2.8, colors are passed around in linear space, so 0.5 (sRGB) will be passed into the node as 0.22 (linear) and 2*0.22=0.44 which makes things too dark. So you can try using a factor of about 4.6 instead of 2 to keep the neutral point neutral. I found this worked pretty well for PS1 models I tried. Unfortunately it loses accuracy as you move away from vertexColor=0.5, but maybe you can try it and see if it's any better than changing the global exposure.

[JamesO2]

About vertex color formula vertexColor * 2 * baseColor:

That's also the formula the PS1 used (it let's you also brighten textures instead of only darkening them). The neutral point where there's no effect on the texture is vertexColor=0.5 since 2*0.5=1. But if it's like the PS1 it's supposed to be in sRGB space. Starting in Blender 2.8, colors are passed around in linear space, so 0.5 (sRGB) will be passed into the node as 0.22 (linear) and 2*0.22=0.44 which makes things too dark. So you can try using a factor of about 4.6 instead of 2 to keep the neutral point neutral. I found this worked pretty well for PS1 models I tried. Unfortunately it loses accuracy as you move away from vertexColor=0.5, but maybe you can try it and see if it's any better than changing the global exposure.

Oh cool, that's good to know. I did some earlier experimentation with Vertex Coloring and noticed the same thing you mentioned: that the mid colors (0.5) seemed neutral, while the darks were darker and the lights were lighter. But I couldn't get it to look quite right in Blender 2.9 at the time because I wasn't familiar enough with the program.

I did read from somewhere that Blender made a change into using "linear space" for color and lighting calculations, which I guess is generally better for realistic 3D work, or "how humans would interpret colors" more so. But in this particular case we aren't trying to recreate a realistic 3D space, but a very specific sRGB color space, so yeah that does cause some issue.

I wasn't aware of the exact math behind linear/sRGB calculations, but I was able to correct the 0.44 color imbalance by adjusting the scene's Color Exposure to 1.2. Which, if my math is correct, may have pushed that 0.44 into being perceptually closer to 0.528, which would make it look closer to correct. Adjusting the scene exposure does work, but it also has the effect of over-brightening any object that isn't using this specific shader too.

I'll experiment with the numbers you mentioned here later though and see what I can get. You seem much more knowledgeable about the exact math of it all -- I just approach the issue from an artist perspective, so I'm eyeballing it to get colors at approximate correct values.

I don't know much python, but I skimmed over that link you posted. Is that an entire automation script to import Vagrant Story models? It looks like it even automates the "File > Import" process too, if I'm correct. I'll have to look at it closer.

Do you have a sample model I can borrow to test out your script (or point me to a link to which models you used)?

Here's an interesting thought: I know that you can change the rendering engine in a Blend file, the usual options being "Eevee, Workbench, or Cycles". And I know there used to be an older Blender Render which was scrapped to focus on improving Eevee. Would it be possible to write a 4th render engine that is specialized to render in a videogame sRGB manner? I wouldn't expect it written from scratch, more so a tweaked Eevee renderer that just adjusts some of the internal math values to be more in line with sRGB color space.

[JamesO2]

I did some tests, and it seems like 4.6 is definitely the magic number! It looks pretty close to me, though the darks aren't quite as dark, its acceptable by my standards.

For comparison:
the top-left: Alvare's original image (which I believe is a 100% accurate color representation from OoT3D).
top-right: My original image, using v * 2 * c formula, and adjusting exposure to 1.2.
bottom-right: New formula using v * 4.6 * c, and exposure reset to 1.0
bottom-left: bonus image, view from top of Death Mountain.



And here's a comparison of the Gerudo path entrance to the same spot on Noclip.website (which I also believe is 100% recreation):
https://cdn.knightlab.com/libs/juxtapose...95d9b17862

You can see the differences, but they're pretty minor. And part of it has to do with Eevee's lighting engine, which will always have some effect on the scene, unless you set everything to use a Diffuse material.

I also made some updates to my OoT3D script that make it easier to use. Now the script checks whether the nodes exist first, and only creates new ones if they don't -- this way you can run the script multiple times and adjust values. I also changed the "Vertex Color Multiplier" to be a variable (its x at the top of the script). So you just have to change this one number to mess with the vert color multiplier.

I'm just gonna ask again while I'm here: does anyone have the "romfs" folder for Majora's Mask files, or could help me find some instructions on how to generate my own? Googling has either led me here, or to many broken links.

There was a post a few pages back mentioning HackingToolkit9DS https://github.com/Asia81/HackingToolkit9DS-Deprecated-

But when I tried that method, I got a bunch of .bin files, but wasn't able to extract anything from them. Running the tool further just generated empty folders, so I'm not sure what I'm doing wrong.

EDIT: I eventually did figure out how to get the romfs folder for Majora. The issue I was having having was something to do with my files being corrupted from a botched decompress or something. I have no clue.

EDIT2: Ripping Majora maps coming along nice. This method seems to preserve the vertex colors/alphas well. Pics from Blender 3.1:

[Alvare]

Nice job on obtaining Majora's Mask's files and getting the colors to look and work correctly.
You'll notice how some of MM's environments look more detailed than Oot3D's.

Btw, I didn't have the MM files for a while. Had it some time ago after searching for hours and found it somewhere in a YouTube comment.
Anyway, I have them again as well now, using the tools you mentioned. 
You need the original cia files without decryption and don't have weird folder / file names. Some extractors can't read " ' " symbols too.

[JamesO2]

Nice job on obtaining Majora's Mask's files and getting the colors to look and work correctly.
You'll notice how some of MM's environments look more detailed than Oot3D's.

Btw, I didn't have the MM files for a while. Had it some time ago after searching for hours and found it somewhere in a YouTube comment.
Anyway, I have them again as well now, using the tools you mentioned. 
You need the original cia files without decryption and don't have weird folder / file names. Some extractors can't read " ' " symbols too.

I'm working on a pretty huge MM ripping project.

So far I've ripped just about every "exterior" area in the game and given them plain English names to make them easier to work with. I haven't started on interiors like shops and dungeons yet.



There are 318 "scene" models in the game. Some of them are just empty dummy files with a single triangle, and most of them are the game's dungeons (each room in a dungeon is exported to a separate DAE model).

I couldn't find any solid information for translations of all the filenames, so I had to load them all one-by-one myself to figure it out. I wrote it all down here for anyone else's convenience so they don't have to go through all that work:

• https://docs.google.com/spreadsheets/d/1...sp=sharing
  

Its a spreadsheet list of every area in the game, along with English names, grouped into categories roughly in the order you encounter these areas in the game.

I've set up my own naming convention and been tracking progress on what I have/haven't ripped, as well as notes about issues from ripping (like certain areas are missing textures, etc). The lines with green checkmarks are areas that I've dumped into DAE so far, and the yellow lines are ones that I haven't gotten to yet.

I also included a list of all the useful links, programs, etc that I collected from various threads. There's some notes about the ripping/converting process, but some of the notes are only useful to me specifically (or reference custom scripts I wrote). I'll make these scripts public eventually, but I frequently modify them so often that I wouldn't be able to keep track of changes.

I've written python scripts (for both inside Blender and outside of Blender) to help better organize things.

1. One is an auto-importer that lets you "one-click" import multiple DAE files all at once (such as all of the parts of a dungeon, or Clock Town, etc) and it even auto-sorts them into their own Collection and parents to an Empty.
   
2. A script to fix the "bones" (any objects that are out of place) positions using the CMB Viewer's TXT dump data.
   
3. A bunch of scripts for fixing the vertex color, applying water shaders, alpha textures for trees, etc.
   

If anyone wants a copy of a script, just message me.

EDIT: @Alvare - I did notice that the vertex alpha blending doesn't seem to be preserved when importing into Blender 3.0 -- which is really strange because this version of Blender definitely 100% does support vertex color alpha channel (I've used it in other projects before). So I'm not sure what is going wrong there, I'll have to investigate it further later. Its possible the data is still there in the Blend file, but I just need to figure out some specific node arrangement -- or its possible Blender 3.0 is just ignoring the alpha and not importing it from the start. Not sure.

[scurest]

@Alvare - I did notice that the vertex alpha blending doesn't seem to be preserved when importing into Blender 3.0

Blender doesn't import them. There's a patch to fix it but it's not moving very quickly.

[JamesO2]

@Alvare - I did notice that the vertex alpha blending doesn't seem to be preserved when importing into Blender 3.0

Blender doesn't import them. There's a patch to fix it but it's not moving very quickly.

Ah, so it is an issue with the collada importer then. From what I do know of Blender, basically every feature, importer, etc, is internally handled as an "add-on" written in python. So it shouldn't be too difficult to modify the collada importer to support one extra vertex channel. I'll look into the patch you linked and see what I can figure out from it.

If its anything like any other add-ons I've modified before, patching is as simple as copy/pasting the relevant code snippet into the right py file.

EDIT: Dang, it looks like the collada mesh importer is code is written in c++, so it would have to be compiled into the main Blender program I assume. That's a bit beyond my scope of knowledge.

Its weird because most of the other importers are written as python add-ons (obj, fbx, stl, gltf, etc), and can even be disabled in preferences.

EDIT 2: Ok, so the collada importer is actually a strange case -- it basically is the only exporter/importer not written in python, and there's a whole article written on the Blender website on why this is a special case, and also why collada is weird in Blender, and why this c++ importer is a clunky approach. And apparently they've been wanting to replace the collada code entirely for some years now, but just no ones gotten around to doing it: https://code.blender.org/2016/10/the-collada-case/

EDIT 3: I looked a little further, and there was a python collada importer in development years ago here: https://github.com/skrat/bpycollada
And it looks like a relatively recent continuation (2.8+ compatible) was continued here: https://github.com/ldo/blender_pycollada_importexport

I haven't tested it yet, but I'm posting it here partly as a note to myself for later, and for anyone interested in testing it.

[Alvare]

I agree with the naming convention change. The one of Oot3D is very solid and easy to understand, while Majora's Mask's one is a little convoluted.

Just wanted to mention this quickly before you continue any further. Make sure you rip the CMB file directly from the tools I included, not converting/exporting them to dae.
Because if you do, you're gonna get inaccurate uv's with Cmbviewer's .dae exports. It's been known to me that in multiple occasions, it overshoots the bounding box. (f.e. in Ganondorf's Castle ganon_7_info, the round carpet)
It's not noticeable at first, but when zooming in on the uv edges and comparing both, I saw seams on the dae counterparts that weren't there before in both noclip.website and on the newer tool exports.

As for the Blender not importing vertex color alpha, I simply suggest doing the cmb import work with 2.79a build and then loading the saved file from another Blender to continue working on it from there.

[JamesO2]

Just wanted to mention this quickly before you continue any further. Make sure you rip the CMB file directly from the tools I included, not converting/exporting them to dae.
Because if you do, you're gonna get inaccurate uv's with Cmbviewer's .dae exports. It's been known to me that in multiple occasions, it overshoots the bounding box. (f.e. in Ganondorf's Castle ganon_7_info, the round carpet)

Noted. I checked the ganon file you mentioned, and I see the UV seam bleeding. Using the Blender 2.79 CMB importer, I see that that bleeding issue isn't there, and the UVs are slightly different than the DAE exported ones I was using before.

But I already have an issue: Is there a reliable ZSI to CMB converter? I tried the method you mentioned in a previous post (use HxD to delete the ZSI header part, before the CMB part), but so far I've noticed this doesn't work for all the files.

I was able to import the CMB for the ganon_7_info OoT file using this method. But when I attempted to do this for MM's Termina Field "z2_00keikoku_0_info.zsi" file, the Blender 2.79 importer failed to load it, with a bunch of errors. Specifically:

Code:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xd4 in position 8: ordinal not in range(128)

Oddly, I was able to import the other "Termina Field" models (z2_02keikoku_0_info.zsi, which appears to be a culled version used for cutscenes or something). Just not the main Termina Field file.

From the error, and also just looking at the file in HxD, the main Field map (z2_00keikoku_0_info.zsi) appears to be encoded differently. Beyond that, I don't know enough to move forward.

And I think this might be exactly why I just went with using "N3DSCmbViewer-026-unstable" because that's the only tool I was able to get to actually work, consistently. All my attempts at using other methods produced errors.

As for the Blender not importing vertex color alpha, I simply suggest doing the cmb import work with 2.79a build and then loading the saved file from another Blender to continue working on it from there.

This is really smart, thanks for the tip. My only issue is that I wasn't able to get the CMB importer to work consistently for all ZSI files I tried. If I can't import the Termina Field CMB, then I'm just kinda shit out of luck.

I just tried using the HxD method to convert a few more MM3D ZSI files to CMB, and about half of them won't load in Blender. Again, just looking at the files in HxD, they appear to be encoded differently.

For example, a file that looks like this seems to load fine:

Code:

cmb €k�
�������Z2_02keikoku_00�œ��L���@��x��à$��,'��¸J��äK��»��€È������skl ô
����������ÿÿ��€?��€?��€?������������������������¦-¬â
�����€?��€?��€?�������������XXÅ@¤°B*�¼AíPùÙ�
���€?��€?��€?������������������������%òH÷�
���€?��€?��€?������������������������DòH÷�
���€?��€?��€?��������������������

But files that look like this, won't load:

Code:

cmb €ó´%êò2_00keikokuÔ G�rzp�ëðº!ü�¿pm��„tô Úÿ��ÀÛ��`¤Ï�€™÷ëðsk'l x�w��¯#¢
ã€?"7/?êñwU]á
Û�ëð$?Þý˜è;ñL¯#P?éòXXÅ@ÿ¤°B*�¼AíPùÙ2|?Þý%òH÷£?à?ëðD"Ë0Ï?OëðâË0ª|þ?Œ;�`SÄŸ0ñ©,OÞýøhH÷QSOOëðÙ{@    O„¼Oëð?{@Ê#°OÞýjǼr³ÿ�ÙO_�•E³_@_ëð¶Çê;§71_n_�»ÿQQåqtrs4üØä÷\*ÆEÿÄâB&ÆeÿFªrEGyïóEÿ²TêñmatsßÌ]��<Ø
�ø·RŽU
'
&
÷)
)êñš™™¿�¹R÷_ò_æõJ1d_èó8(o:oä÷ma¶Uð2~o„bMjÀ„ÈbÊêñ°©`À©bÙ�ÿ †©`€?º

I'm sure its possible to convert the encoding, but I have no idea how. I can tell that the patterns are different, but that's all I know. I don't know what to do next.