Blender Add-On for direct BFRES import (WIP!)

[Ray Koopa]

I started an add-on to directly import BFRES files into Blender some days ago. There's some stuff to see in follow-up posts by now. I'll write the missing parts next, maybe with the help of some geniuses here, stay tuned!

You can alpha test the add-on, if you are brave enough! The installation details and the files are all on the GitHub repository here.

If you are a developer:
By now it completely loads the FMDL sections into well defined structures and creates the resulting models and materials.

I could've just jumped from offset to offset to fetch the parts I need for displaying model data, like RTB's (really cool, no flame intended!) MaxScript does; but I want the Blender add-on to...

• prefer correctness over speed (still fast, loading track file in 3-4),
  
• be more flexible to extend,
  
• be less error-prone and easier to fix,
  
• decompress Yaz0 on the fly in the future, done! as fast as Python can (12 seconds for a track file on my PC)
  
• directly extract and convert FTEX textures too if possible (and if not, delegate the work to TexConv2 or call into its TexUtils DLL with a path set up in the options), done! (using TexConv2 at the moment)
  
• become (and now I'm looking really far into the future) part of the core of a complete Mario Kart 8 track importer add-on.
  

It is less a port of my C# library I once wrote for loading BFRES, more a complete rewrite of it to make it easier to export BFRES neversomewhen, the offset bigotry being the biggest problem in a managed language like Python (e.g., I put the headers, which are kept separate in BFRES files in their own structs, rather than that weird Internals approach I used in C#).

Need a quick overview for getting started developing with it? There are some helpful comment sections here and there.

• Make sure to check bfres_file.py, it contains a huge comment visualizing the BFRES format, something the MK8 wiki doesn't deliver so well.
  
• If you want to know more about the way you have to go to get the vertex data for a model, I wrote a comment about it in bfres_fmdl.py.
  

[Ray Koopa]

I'm not as far as I wanted to be by now, as I was mostly dicking around with simple mods of Mario Kart 8, but I'm slowly getting somewhere...

(yes, BFRES has Y as the Z axis and vice-versa; yes, there will be an auto-on option to correct that ;P)

I still have several unknown attribute formats. What that means is that I cannot interprete the shape / looks of some objects on tracks, maybe I can just skip those as they're often associated with unimportant attributes (the 3DS Max-Scripts don't have support for those too as it seems); especially in track files.

If you feel experimental, you can already try the add-on, installing it in the same way as my rip-importer (I don't give detailled instructions here yet as it's still pretty experimental). If you get some errors, most are caused by the missing attributes (the error message is pretty clear about that), for any other errors, please report them already; also try directly decompressing SZS files with it =)

[Ray Koopa]

Took me some time to understand how to do that with Blender's new BMesh API, but simple UV mapping now works too!


Materials are not yet imported. That will be one of the next steps. Vertices are not merged yet to implement UV seams by loops (BFRES splits them so that it can be sent directly to the GPU and save computation time), but that can be easily added.

This also works for lower-resolution LoD models:


(This import took 0.9 seconds! =] )

[Ray Koopa]

Sorry for the long break. I spent some time on writing an extended Cafiine server and did some MK8 mods... xD

The add-on now imports materials and converts the textures embedded in the BFRES on the fly. It delegates the work to TexConv for that, the path has to be set up in the add-on options (I added an installation tutorial in the first post now as it's ready for some tests).

TexConv has a stupid bug that it exports DDS textures without specifying they have an alpha channel, so I had to fix that manually for diffuse textures, which is a bit hackish. Good job Nintendo...

Here are some results:

[Ray Koopa]

The addon now correctly imports all* tracks; and fixes all the texture shenanigans (wrong texture attributes in the files, non-exported alpha from TexConv):



I only have to fix some smaller things now (one unknown UV format, support multiple UV layers), then I can move on to the skeleton / rig.
I'm not sure if I give a break on that as I want to write a track info visualizer first to allow some track modding.

*If that are really all depends on some testing ;P

[Struggleton!]

Dude this is amazing work you're doing here. Great job man!

[Ray Koopa]

Thanks really much! It means a lot to hear that from you.

I paused the development just a little bit yesterday to create a KCL importer for the Mario Kart 8 collision models, if anyone is interested:

[iyenal]

Very good work Ray Koopa, now it's very easy to import a BFRES file with specular and normal maps. Thanks !
But can Blender Cycle render system be supported ?

[Struggleton!]

Thanks really much! It means a lot to hear that from you.

I paused the development just a little bit yesterday to create a KCL importer for the Mario Kart 8 collision models, if anyone is interested:

Definitely! I'd love to make courses for Mario Kart 8!

[Ray Koopa]

Exporting BFRES will still take a lot of time sadly, but we're getting to it!

Cycles is not supported at the moment, it is of lower priority. I just import to typical Blender data blocks, and the correct meaning of all material parameters is unknown anyway yet.

Just as a side note: You can also load the corresponding sky box and surroundings together with the track, after loading a course model (simply import the VR* object afterwards, like VRFirst for Mario Kart Stadium here). Results in some nice screenshots like this:

[Struggleton!]

How long does each import take, on average? Using Random Talking Bush's script, it takes about 20 minutes to import a stage. How much faster/slower is this script?

[Ray Koopa]

Even with my PC having a quite fast i7 6700K, RTB's 3DS Maxscript had similar times ranging from 5-10 minutes. I think that's just because MaxScript is not really performantly executed, it partly works on the slow 3DS Max GUI.

With my Blender add-on, working purely in memory, but being Python interpreted at runtime, I experience the following times (It depends on the input, since you can also decompress SZS compressed BFRES files on the fly):
- Uncompressed BFRES character: 2 seconds
- Uncompressed BFRES track: 20 seconds (texture decompression with TexConv is the slowest part of this)
- Compressed SZS character: 10 seconds
- Compressed SZS track: 1 minute

As you can see, SZS is quite a bottle neck. I can't do it faster with raw Python, and already optimized it together with some professionals. If you need it faster, you should decompress SZS with an external tool (to which in some future version the work can probably be delegated to to speed it up for Blender too).

[iyenal]

Do you can support the emission maps ?

[Ray Koopa]

Yeah, but AFAIK I didn't set up the slot 100% correctly. Maybe you can tell me the Blender Internal settings for that.

Right now it's

Code:

# Emmissive map.
# TODO: Slot settings might be wrong (s. Wild Woods' glowing circles).
slot.use_map_color_diffuse = False
slot.use_map_emit = True

[Struggleton!]

Even with my PC having a quite fast i7 6700K, RTB's 3DS Maxscript had similar times ranging from 5-10 minutes. I think that's just because MaxScript is not really performantly executed, it partly works on the slow 3DS Max GUI.

With my Blender add-on, working purely in memory, but being Python interpreted at runtime, I experience the following times (It depends on the input, since you can also decompress SZS compressed BFRES files on the fly):
- Uncompressed BFRES character: 2 seconds
- Uncompressed BFRES track: 20 seconds (texture decompression with TexConv is the slowest part of this)
- Compressed SZS character: 10 seconds
- Compressed SZS track: 1 minute

As you can see, SZS is quite a bottle neck. I can't do it faster with raw Python, and already optimized it together with some professionals. If you need it faster, you should decompress SZS with an external tool (to which in some future version the work can probably be delegated to to speed it up for Blender too).

Damn man, I can't say this enough: you are amazing in my eyes.