Problem with ripping etc1(a4) and transparency

[HelpTheWretched]

Hey guys, this is sort of a repost from GBAtemp because I've had no replies there. The link is here but I'll describe it again.

I'm using the program Ohana3DS Rebirth to rip textures from the character models for Zelda: Tri Force Heroes, and it seems they're all etc1(a4) encoded. The weird thing is, the alpha channel (transparency) seems tied to each pixel's brightness, so the darker spots are more transparent; a bunch of spots are completely transparent.

On the left is an extracted .png,
On the right is that .png flattened to 100% opacity, showing the "gaps".


So the obvious question is "What can I do about this", but more specifically, is this a feature/problem with etc1(a4) encoding or is it the program I'm using?

I've edited some of them by selecting portions and changing the hues, then reinserted them into the game. It works okay, except some of these gaps are showing up as black. You can see them on Link's palms, and a few little bits around the brim of his cap:


Any help would be appreciated.

[Pingus!]

Could you try exporting the texture with Every File Explorer? I don't know/recall if it's supported, but Ohana3DS MAY cause some transparency issues.(Don't quote me on this, I will probably get roasted by Hallowpape because I'm not a very good 3DS ripper ).

[HelpTheWretched]

That's a fine idea and I just tried it; the .bch loads, but it's just a blank window. Can't tell if it's a compatibility issue or something wrong with EFE, because that program has given me lots of problems before.

[Pingus!]

Could possibly be the *.bch file isn't supported by EFE. (I don't know it that well so ).

[HelpTheWretched]

So.... my new theory is that the textures are actually ETC1 encoded, but Ohana3DS is recognising them as ETC1(A4). There's no need for any of them to have transparency, and some of the .bch models include black & white alpha textures.

If this is the case, I might be able to extract the textures properly with a hex editor, but I'll need to learn about the ETC1 structure so that I know WTF to edit.

I could go look for some myself, but does anybody know where I might find some plain old ETC1 encoded image files to study? Or any games that use them, that I could rip them from?

[Bomb Bloke]

A few weeks ago I wrote an ETC1 decoder, using this specification. The format is a lossy one which doesn't actually support transparency, as such - attempts to make it do so are often designed to be compatible with decoders that won't recognise them (so that you'll always get an otherwise correct representation of the image back, even if the output lacks alpha). Because of this, generic decoders which do support those methods have to likewise guess if they're in use (or be told), because there aren't any specific flags to indicate that they're there...

Editing the files manually via a hex editor may be tricky, as encoded images are divided into chunks which are each represented by multi-byte records. Significant changes targeted at even single pixels within these chunks may likewise require significant changes to their records, which is 1) tricky to figure out in your head and 2) may force you to degrade the quality of the pixels surrounding the ones you want to change.

I'm afraid I can't quickly provide you with sample files directly from my end. There are a small handful of ETC1 texture images in King of Fighters-A 2012's OBMs (eg mstage_bigben_1.obm, buried within its OBB and extractable with this), and Metal Slug Revolution has tons of them within its Unity CABs.

If you really get stuck then I think it might be interesting to write some software for your particular files, but I'd need to get some samples of the encoded originals so's I can code around their headers.

[HelpTheWretched]

Thanks for the info, Bomb Bloke. It doesn't look like I could edited any particular section of an ETC1 texture in a hex editor, but the hope is to be able to completely replace them.

Since that last post, I got a texture compression tool from ARM and made some samples to look at. The byte structure strongly resembles what I'm seeing in the textures embedded in the model files, but those textures are lacking headers.

I also tried comparing the original model file to several with modified textures to see which parts had changed, and I found the big-endian values for the width/height of the four largest textures (128x256) near the start of where each of those textures is stored, but I'm not sure yet if that's just coincidence, since I couldn't find the width/height values for the smaller mip-levels. Not sure if they even have headers; that specification is slightly over my head.

If you're interested, I uploaded the standard Link model (shown in OP) here.

[gdkchan]

It's not a bug, you just need to remove the alpha channel entirely. I think you actually blended the image with a black background where the alpha still affects the blending.

If you have GIMP you can "fix" the texture doing this:
Open the texture, then go to Colors > Curves. On the dialog that will open select "Alpha" on the Channel drop down box, then click on the left black spot on the graph and drag it all the way to the top-left corner, and click on OK. You should now have the texture the way you want.

I've been working on a tool that should be a improvement to the original Ohana3ds, even through it's just bare bones for now, if you want to take a look: https://github.com/gdkchan/SPICA/commits/master

The format is a lossy one which doesn't actually support transparency, as such - attempts to make it do so are often designed to be compatible with decoders that won't recognise them (so that you'll always get an otherwise correct representation of the image back, even if the output lacks alpha). Because of this, generic decoders which do support those methods have to likewise guess if they're in use (or be told), because there aren't any specific flags to indicate that they're there...

The original etc1 format doesn't support transparency yea. But the 3DS GPU does support through. On etc1 textures with alpha information, each 4x4 tile will use 8 bytes for alpha information (4 bits per tile, which is a 0-15 alpha range extended to 0-255), then it have 8 more bytes of the etc1 data. Also IIRC etc1 data on the 3DS in little endian, while the original etc1 tool uses big endian, and there's some tile swizzling, so the original tools aren't able to decode those. The GPU also have separate format for etc1 (without alpha) and etca4 (with alpha) textures, it isn't stored on the texture data itself but on a gpu command buffer (GPUREG_TEXUNITi_TYPE reg) on the bch file.

Example implementation in C: https://gist.github.com/smealum/8897237

[HelpTheWretched]

If you have GIMP you can "fix" the texture doing this:
Open the texture, then go to Colors > Curves. On the dialog that will open select "Alpha" on the Channel drop down box, then click on the left black spot on the graph and drag it all the way to the top-left corner, and click on OK. You should now have the texture the way you want.

Just to confirm: This doesn't work with the textures extracted with Ohana3DS (same problem as before), but it did the trick for those extracted with a BCH2PNG.exe that I found, author unknown. Thank you so much! My regular image editor would still say that the "back" colour in the transparent spots was 0x000000, so I guess that environment just discards the info.

I've been working on a tool that should be a improvement to the original Ohana3ds, even through it's just bare bones for now, if you want to take a look: https://github.com/gdkchan/SPICA/commits/master

I don't understand OpenTK, nor how to build the GLControl from the source. Kinda noob, here. Github's only really useful to me when there are .exe releases or Python scripts that I can save raw... I wish I knew more about working with source code so I could take a look at your work there. Oddly, I'm very adept with scripting languages and making the most of intro-level tools, but actually building software? 

[HelpTheWretched]

Oh, this is interesting...

It turns out those were ETC1(A4) textures after all. I thought they were being misread as such, because there was no need for partial transparency and a few models had alpha textures as well. When I replaced some textures, I left them completely opaque, thinking it wouldn't matter. It seems the alpha channel was tied to their specularity as when I tried them in the game, the replaced textures were very shiny!

Well, mystery solved.