[Mobile] Sky; Children of the Light

[longbyte1]

Hi all,

Someone contacted me who was also wanting to extract mesh data for some personal research, and indeed we were met with success!

Here is Python code for Blender 3.2.2:

Code:

import ctypes
import struct
import io
from ctypes import *

f = open('/tmp/UnitCube.mesh', 'rb')

lz4 = CDLL('liblz4.so.1')

# read uncompressed size
f.seek(0x52)
uncompressed_size = struct.unpack('i', f.read(4))[0]

# read compressed size
f.seek(0x4e)
compressed_size = struct.unpack('i', f.read(4))[0]

# read num lods
f.seek(0x44)
num_lods = struct.unpack('i', f.read(4))[0]

print('compressed_size', compressed_size)
print('uncompressed_size', uncompressed_size)
print('num_lods', num_lods)

# get compressed content
f.seek(0x56)
src = f.read(compressed_size)

# get decompressed content
dest = ctypes.create_string_buffer(uncompressed_size)
ret = lz4.LZ4_decompress_safe(src, dest, compressed_size, uncompressed_size)
if ret <= 0:
    raise IOError('error decompressing mesh - file may not be valid')

buf = io.BytesIO(dest.raw)
buf.seek(0x74)
shared_vertex_count = struct.unpack('i', buf.read1(4))[0]
buf.seek(0x78)
total_vertex_count = struct.unpack('i', buf.read1(4))[0]
buf.seek(0x80)
point_count = struct.unpack('i', buf.read1(4))[0]
buf.seek(0x74)
uv_count = struct.unpack('i', buf.read1(4))[0]

print('shared_vertex_count', shared_vertex_count)
print('total_vertex_count', total_vertex_count)
print('point_count', point_count)
print('uv_count', uv_count)

# build vertex buffer
vertex_buffer = []
vertex_buffer_start = 0xb3
buf.seek(vertex_buffer_start)

for i in range(shared_vertex_count):
    # 3 floats
    x, y, z = struct.unpack('<fff4x', buf.read(16))
    vertex_buffer.append((x, y, z))

# build uv buffer
uv_buffer = []
uv_header_size = uv_count * 4 - 4
buf.read1(uv_header_size) # move caret to uv start
for i in range(uv_count):
    # 2 half-precision floats
    u, v = struct.unpack('<4xee8x', buf.read(16))
    uv_buffer.append((u, v))

# build index buffer
index_buffer = []
face_count = total_vertex_count // 3
buf.read1(4) # advance 4 bytes of padding
for i in range(face_count):
    # 3 shorts (each indexing into the vertex buffer)
    v1, v2, v3 = struct.unpack('<HHH', buf.read(6))
    index_buffer.append((v1, v2, v3))

f.close()
buf.close()

print(vertex_buffer)
print(index_buffer)

# build geometry from buffers
vertices = []
edges = []
faces = []
for face in index_buffer:
    v1i, v2i, v3i = face
    edges += [(v1i, v2i), (v2i, v3i), (v3i, v1i)]
    faces.append(face)

import bpy
mesh = bpy.data.meshes.new('created_mesh')
mesh.from_pydata(vertex_buffer, edges, faces)
mesh.update()

uvl = mesh.uv_layers.new()
uvl.data.foreach_set('uv', [uv for pair in [uv_buffer[l.vertex_index] for l in mesh.loops] for uv in pair])
mesh.uv_layers.active = uvl

obj = bpy.data.objects.new('created_object', mesh)
collection = bpy.data.collections.new('created_collection')
bpy.context.scene.collection.children.link(collection)
collection.objects.link(obj)

Just change the open() line to load the file you want. Note that this is only going to read the first LOD it finds, which should be the highest quality one.
For Windows users, you need to change the path in the CDLL() line to a path to a 64-bit LZ4 DLL (or 32-bit, depending on whether your Blender is 64- or 32-bit). You can find a working copy of LZ4 for Windows here, and you're looking for a DLL called either "msys-lz4-1.dll" or "liblz4.dll".

Many thanks to cobrakyle for their help.

[DancingTwix]

Brilliant work!! Thank you so much!!
Next step is to crack into the .level files as I'm sure theres some tasty goodies in there

I'll have a looksie about matching up models with their textures if possible

---

Hi all,

Someone contacted me who was also wanting to extract mesh data for some personal research, and indeed we were met with success!

Here is Python code for Blender 3.2.2:

Code:

import ctypes
import struct
import io
from ctypes import *

f = open('/tmp/UnitCube.mesh', 'rb')

lz4 = CDLL('liblz4.so.1')

# read uncompressed size
f.seek(0x52)
uncompressed_size = struct.unpack('i', f.read(4))[0]

# read compressed size
f.seek(0x4e)
compressed_size = struct.unpack('i', f.read(4))[0]

# read num lods
f.seek(0x44)
num_lods = struct.unpack('i', f.read(4))[0]

print('compressed_size', compressed_size)
print('uncompressed_size', uncompressed_size)
print('num_lods', num_lods)

# get compressed content
f.seek(0x56)
src = f.read(compressed_size)

# get decompressed content
dest = ctypes.create_string_buffer(uncompressed_size)
ret = lz4.LZ4_decompress_safe(src, dest, compressed_size, uncompressed_size)
if ret <= 0:
    raise IOError('error decompressing mesh - file may not be valid')

buf = io.BytesIO(dest.raw)
buf.seek(0x74)
shared_vertex_count = struct.unpack('i', buf.read1(4))[0]
buf.seek(0x78)
total_vertex_count = struct.unpack('i', buf.read1(4))[0]
buf.seek(0x80)
point_count = struct.unpack('i', buf.read1(4))[0]
buf.seek(0x74)
uv_count = struct.unpack('i', buf.read1(4))[0]

print('shared_vertex_count', shared_vertex_count)
print('total_vertex_count', total_vertex_count)
print('point_count', point_count)
print('uv_count', uv_count)

# build vertex buffer
vertex_buffer = []
vertex_buffer_start = 0xb3
buf.seek(vertex_buffer_start)

for i in range(shared_vertex_count):
    # 3 floats
    x, y, z = struct.unpack('<fff4x', buf.read(16))
    vertex_buffer.append((x, y, z))

# build uv buffer
uv_buffer = []
uv_header_size = uv_count * 4 - 4
buf.read1(uv_header_size) # move caret to uv start
for i in range(uv_count):
    # 2 half-precision floats
    u, v = struct.unpack('<4xee8x', buf.read(16))
    uv_buffer.append((u, v))

# build index buffer
index_buffer = []
face_count = total_vertex_count // 3
buf.read1(4) # advance 4 bytes of padding
for i in range(face_count):
    # 3 shorts (each indexing into the vertex buffer)
    v1, v2, v3 = struct.unpack('<HHH', buf.read(6))
    index_buffer.append((v1, v2, v3))

f.close()
buf.close()

print(vertex_buffer)
print(index_buffer)

# build geometry from buffers
vertices = []
edges = []
faces = []
for face in index_buffer:
    v1i, v2i, v3i = face
    edges += [(v1i, v2i), (v2i, v3i), (v3i, v1i)]
    faces.append(face)

import bpy
mesh = bpy.data.meshes.new('created_mesh')
mesh.from_pydata(vertex_buffer, edges, faces)
mesh.update()

uvl = mesh.uv_layers.new()
uvl.data.foreach_set('uv', [uv for pair in [uv_buffer[l.vertex_index] for l in mesh.loops] for uv in pair])
mesh.uv_layers.active = uvl

obj = bpy.data.objects.new('created_object', mesh)
collection = bpy.data.collections.new('created_collection')
bpy.context.scene.collection.children.link(collection)
collection.objects.link(obj)

Just change the open() line to load the file you want. Note that this is only going to read the first LOD it finds, which should be the highest quality one.
For Windows users, you need to change the path in the CDLL() line to a path to a 64-bit LZ4 DLL (or 32-bit, depending on whether your Blender is 64- or 32-bit). You can find a working copy of LZ4 for Windows here, and you're looking for a DLL called either "msys-lz4-1.dll" or "liblz4.dll".

Many thanks to cobrakyle for their help.

Just another q, am I alright to add this work onto the first post for any others looking for this?

[longbyte1]

Brilliant work!! Thank you so much!!
Next step is to crack into the .level files as I'm sure theres some tasty goodies in there

I'll have a looksie about matching up models with their textures if possible

---

Hi all,

Someone contacted me who was also wanting to extract mesh data for some personal research, and indeed we were met with success!

Here is Python code for Blender 3.2.2:

Code:

import ctypes
import struct
import io
from ctypes import *

f = open('/tmp/UnitCube.mesh', 'rb')

lz4 = CDLL('liblz4.so.1')

# read uncompressed size
f.seek(0x52)
uncompressed_size = struct.unpack('i', f.read(4))[0]

# read compressed size
f.seek(0x4e)
compressed_size = struct.unpack('i', f.read(4))[0]

# read num lods
f.seek(0x44)
num_lods = struct.unpack('i', f.read(4))[0]

print('compressed_size', compressed_size)
print('uncompressed_size', uncompressed_size)
print('num_lods', num_lods)

# get compressed content
f.seek(0x56)
src = f.read(compressed_size)

# get decompressed content
dest = ctypes.create_string_buffer(uncompressed_size)
ret = lz4.LZ4_decompress_safe(src, dest, compressed_size, uncompressed_size)
if ret <= 0:
    raise IOError('error decompressing mesh - file may not be valid')

buf = io.BytesIO(dest.raw)
buf.seek(0x74)
shared_vertex_count = struct.unpack('i', buf.read1(4))[0]
buf.seek(0x78)
total_vertex_count = struct.unpack('i', buf.read1(4))[0]
buf.seek(0x80)
point_count = struct.unpack('i', buf.read1(4))[0]
buf.seek(0x74)
uv_count = struct.unpack('i', buf.read1(4))[0]

print('shared_vertex_count', shared_vertex_count)
print('total_vertex_count', total_vertex_count)
print('point_count', point_count)
print('uv_count', uv_count)

# build vertex buffer
vertex_buffer = []
vertex_buffer_start = 0xb3
buf.seek(vertex_buffer_start)

for i in range(shared_vertex_count):
    # 3 floats
    x, y, z = struct.unpack('<fff4x', buf.read(16))
    vertex_buffer.append((x, y, z))

# build uv buffer
uv_buffer = []
uv_header_size = uv_count * 4 - 4
buf.read1(uv_header_size) # move caret to uv start
for i in range(uv_count):
    # 2 half-precision floats
    u, v = struct.unpack('<4xee8x', buf.read(16))
    uv_buffer.append((u, v))

# build index buffer
index_buffer = []
face_count = total_vertex_count // 3
buf.read1(4) # advance 4 bytes of padding
for i in range(face_count):
    # 3 shorts (each indexing into the vertex buffer)
    v1, v2, v3 = struct.unpack('<HHH', buf.read(6))
    index_buffer.append((v1, v2, v3))

f.close()
buf.close()

print(vertex_buffer)
print(index_buffer)

# build geometry from buffers
vertices = []
edges = []
faces = []
for face in index_buffer:
    v1i, v2i, v3i = face
    edges += [(v1i, v2i), (v2i, v3i), (v3i, v1i)]
    faces.append(face)

import bpy
mesh = bpy.data.meshes.new('created_mesh')
mesh.from_pydata(vertex_buffer, edges, faces)
mesh.update()

uvl = mesh.uv_layers.new()
uvl.data.foreach_set('uv', [uv for pair in [uv_buffer[l.vertex_index] for l in mesh.loops] for uv in pair])
mesh.uv_layers.active = uvl

obj = bpy.data.objects.new('created_object', mesh)
collection = bpy.data.collections.new('created_collection')
bpy.context.scene.collection.children.link(collection)
collection.objects.link(obj)

Just change the open() line to load the file you want. Note that this is only going to read the first LOD it finds, which should be the highest quality one.
For Windows users, you need to change the path in the CDLL() line to a path to a 64-bit LZ4 DLL (or 32-bit, depending on whether your Blender is 64- or 32-bit). You can find a working copy of LZ4 for Windows here, and you're looking for a DLL called either "msys-lz4-1.dll" or "liblz4.dll".

Many thanks to cobrakyle for their help.

Just another q, am I alright to add this work onto the first post for any others looking for this?

Sure! Once I have enough scripts, I will publish a repo on GitHub as well.

[Vlistuvious]

Hey there!
I've been silently using guest mode to watch this project got through as me myself have been trying to access the sky files for several months.
I'd greatly appreciate it if you can explain to me in detail how i can import sky .mesh into blender using your code and export it into OBJ
because im not really experienced in Python, blender.  (PS. my blender is 3.2.1 but it keeps crashing and im not too sure why i've tried to fix it Multiple times)
Thanks!

[Vlistuvious]

Update: I've got my blender working now! However some spirit models such as "CharNPCAnimEmotesButterFly.mesh" Do not load!

ERROR:
`Python: Traceback (most recent call last):
  File "\Text", line 100, in <module>
  File "\Text", line 100, in <listcomp>
IndexError: list index out of range`

[DancingTwix]

Howdy!!

Looks like you've found an error in the script there.
Afraid I'm away from home atm so havent had a chance to add the current method to the top post. But glad to hear you got it working in the end!!

Make sure to clean up the collections the script will make in the layout window before exporting, OR just tick the export selected in the export window (running off the top of my head so apologies if I got something wrong)

In terms of the issue with the anim.mesh file is it any files with the name "anim" in that you're having trouble with?
My guess would be the scripts cant handle the animation data at the moment but the other file with "anim" in should have the original mesh that you can export.

Additionally, I have a sinking feeling we might need to rebuild the textures for models unless their baked versions are sitting in the .bin files. Going through the art of sky video discussed in a previous post, all the meshes just refer to specific values in the "ramp" textures instead of any specific textures.

Some good news, both the DayGate and DawnGate meshes do store their textures and normal maps in the games folders so we can piece them together as I had started a bit of work on before heading off on holiday.

[Vlistuvious]

Heya!!!
The error changes when it's a cosmetic .mesh file
It should look a little like
"CharSkyNPC_Wing_Director_StripAnim_CompOcc_ZipPos_ZipUvs_StripNorm.mesh" These are the files that refuse to open
the error is way different than the one i've stated above
it mentions bytes or something like that i don't quite remember however
answering your .anim files there's no .anim what i meant is
.animpack my bad haha been trying to snippity snoop the contents but i fail to do so
it seems these .animpack files store the "skeleton" for spirits such as the file below:
"CharSkyNPC_Skeleton_Director.animpack"
About the textures, i see a TEX3D folder in the game but the files are encrypted with a .VOL file apparently standing for VOLUME i assume? we're gonna have to figure out together i suppose.
Lastly, happy holidays Twix!

[TrixxedHeart]

It looks like there's an error whenever trying to open a .mesh that contains an armature I'm guessing? That's what I've figured. This stuff is beyond me, I'm not a coder, but I can troubleshoot to the best of my abilities, and the rigs seem to be the issue in terms of being opened. Other .mesh files that do not have rigs have worked just fine.

Python: Traceback (most recent call last):
  File "D:\Users\-\Desktop\COTL\riptest.blend\CotlToBlender.py", line 76, in <module>
struct.error: unpack requires a buffer of 6 bytes

Y'all are doing great work though, I can't wait to get my hands on the player models at some point to see how they're made.

[Vlistuvious]

It looks like there's an error whenever trying to open a .mesh that contains an armature I'm guessing? That's what I've figured. This stuff is beyond me, I'm not a coder, but I can troubleshoot to the best of my abilities, and the rigs seem to be the issue in terms of being opened. Other .mesh files that do not have rigs have worked just fine.

Python: Traceback (most recent call last):
  File "\Users\-\Desktop\COTL\riptest.blend\CotlToBlender.py", line 76, in <module>
struct.error: unpack requires a buffer of 6 bytes

Y'all are doing great work though, I can't wait to get my hands on the player models at some point to see how they're made.

Hey there!
I'm currently facing the same issue as you apologies for my poor wording of the problem! But you guys are very talented i'm sure we'll find a fix one way or another
If we keep trying hard enough.

[longbyte1]

Heya!!!
The error changes when it's a cosmetic .mesh file
It should look a little like
"CharSkyNPC_Wing_Director_StripAnim_CompOcc_ZipPos_ZipUvs_StripNorm.mesh" These are the files that refuse to open
the error is way different than the one i've stated above
it mentions bytes or something like that i don't quite remember however

Ah, so I see a number of you have been using the script! That's nice to hear.

There are certain flags, which, when activated, change the filename during the model loading process. They are:

• StripGeo (strip geometry)
  
• StripAnim (strip animations)
  
• UncompAnim (uncompressed animation)
  
• CompOcc (compute occlusions)
  
• CompEdges (compute edges)
  
• CompAdj (compute adjacency)
  
• ZipPos (packed positions?)
  
• ZipUvs (packed UVs?)
  
• StripUv13 (?)
  
• StripNorm (strip normals)
  
• ForceIdx32 (use 32-bit indexes instead of 16-bit indexes)
  

So, you can tell which model has which flags activated. Each flag changes the format slightly, which may cause the script to fail as it doesn't support these cases yet. Specifically, I still haven't investigated what the ZipPos and ZipUvs flag do during mesh loading.

According to the disassembly the game also recognizes meshes ending in fbx, though I have not seen the code for loading them.

About the textures, i see a TEX3D folder in the game but the files are encrypted with a .VOL file apparently standing for VOLUME i assume? we're gonna have to figure out together i suppose.

Those are volumetric textures (e.g. clouds). Not sure what the format is.

[Vlistuvious]

Hello!
Thanks for your detailed explaination for this issue and i hope we can somehow brute force it in the future
As for textures been sneeping around the KTX, It has some textures but not all, as the cosmetics seem to be stored somewhere else, perhaps the .mesh itself.
Cheers.

[Knowdice]

Any progress on the script? When I try to open meshes with "StripAnim" in the name blender crashes with an access violation. I'm happy to help with anything but I can't wrap my head around this.

Cheers,

Knowdice

[Vlistuvious]

It seems that things still seem unaccessable regarding .Animpack files Tried any tool i could find to unlock them none worked

[Jenny_1367]

Thank you for the script! 
Just curious, will there be any future updates on this? 
Such as working on extracting the textures or other models even without animation or armature >-< 
regards & thanks in advance

[bruh_equinox]

so far it's possible to extract ost and sound effects (quickbms for .banks and fmod pack tools for fsb to .wav), images and textures (pvrtextool is ok, i think every single texture is just in Images folder) and simple 3d meshes with the script by longbyte1.
is there a community or a discord server or something of sort for this kind of thing?