New assets system (#2481)

* wip: New assets system tm

Builds gc-eu-mq-dbg OK from clean after
1) make setup
2) python3 -m tools.assets.extract -j
3) replace 0x80A8E610 with sShadowTex in extracted/gc-eu-mq-dbg/assets/overlays/ovl_En_Jsjutan/sShadowMaterialDL.inc.c
4) make various symbols in extracted data like sTex static

* use variables from config.yml for gMtxClear and sShadowTex addresses

* Write source with static for overlays using `HACK_IS_STATIC_ON` hack

* gc-eu-mq-dbg OK from clean with `make setup && make`

* implement more skeleton-related types, cleanups, fixups

* fix extracted data to no longer produce compilation warnings

* implement more of RoomShapeImage types

* yeet XmlPath from ExternalFile usage

* Implement PlayerAnimationDataResource (link_animetion data)

* fix csdis CS_TIME extra arg

* dmadata file names no longer hardcoded for gc-eu-mq-dbg

* ntsc-1.0 OK

* xml fixes

* slightly improve standard output

* rm extract_assets.py

* generate and use Limb enums (TODO: check Skin skels and implement for Curve skels)

* handle dependencies between xmls

* introduce RawPointers xml attribute to ignore specific pointers and keep them raw

* add tools/extract_assets.sh

* fixups

* only extract if xmls changed or if -f (force) is used

* fixups, gc-eu OK

* all versions OK

* check attributes of xml resources elements

* Implement legacy skelanime resources

* fix ASSET_FILES_BIN_EXTRACTED/COMMITTED: look for .u8.bin specifically instead of just .bin

* implement JFIFResource

* fix png/jpg wildcards: look specifically for .u64.png .u32.png .u64.jpg

* Makefile: Add rules to build .png, .bin and .jpg in assets/ too

* start writing actual docs

* extract sTransCircleDL and sTransWipeDL

* misc cleanup/fixes, pygfxd 1.0.3

* refactor CDataExt.set_write callback args to use a dataclass

* Move {} to in-source

* misc

* more progress on spec

* fix missing braces in n64dd_error_textures.c

* finish xml spec doc

* assets xmls fixes

* some cleanup, use `gNameTex_WIDTH/HEIGHT` macros in dlists

* handle hackmode_syotes_room, fix compile

* C build_from_png

* rm tools/assets/bin2c

* rm ZAPD

* format

* remove rule to generate dmadata_table.py

* CC0 license (and some import cleanup)

* dont try to build zapd (rmd)

* simplify palettes with single user (ci images with a non-shared palette)

* add docs on how images are handled

* bss

* allow -j N

* fix n64texconv python bindings memory management

* move -j at the end of calling extraction script

* with -j, update last_extracts.json as each job completes rather than only if all complete

* make interrupting less jank by making child processes ignore sigint

* use enum names in `SCENE_CMD_SKYBOX_SETTINGS`

* `multiprocessing.get_context("fork")`

* import rich, except ImportError s

* fix optional rich usage

* .bss

* .bss

* .bss

* assets extraction: -j -> -j$(N_THREADS)

* .bss

* change LIMB_NONE/MAX defaults to be FILE_OFFSET instead of SKELNAME

* 0XHEX -> 0xHEX

* fix bss

* Proper includes for assets

mostly proper, some includes like dlists resources always causing a sys_matrix.h include (when not every dlist references gIdentityMtx) could be done better

* rm z64.h

* rm z64.h take two

* bss

* Make .u64 suffix for pngs optional

* fixup: rm .u64 suffix from n64dd image paths

* Remove elemtype suffixes from .bin and .jpg files

* Update images.md

* some build_from_png cleanup, more error handling, comments

* Handle skybox textures

Introduce "sub-format" suffix for pngs, with sub-formats split_lo and split_hi being used for skybox textures

* fixup for older python

* improve collision output some

* fully use SURFACETYPE[01] macros in writing extracted surface types

* use WATERBOX_PROPERTIES in extracted waterboxes

* some SceneCommandsResource cleanup

* format EnvLightSettingsList output

2 files changed, 90 insertions, 0 deletions

diff --git a/docs/assets/extraction.md b/docs/assets/extraction.md
new file mode 100644
index 000000000..a4c342db3
--- /dev/null
+++ b/docs/assets/extraction.md
@@ -0,0 +1,18 @@
+Assets are not committed to the repo; instead, they are extracted from the ROM files as part of `make setup`.
+
+Assets are extracted to `extracted/VERSION/assets` (for example `extracted/ntsc-1.0/assets` for the `ntsc-1.0` version), based on the descriptions stored in xml files in `assets/xml/`.
+
+For details on the xml files contents, see [the assets xml specification file](../../tools/assets/descriptor/spec.md).
+
+The extraction tool can use [rich](https://github.com/Textualize/rich) if installed to make output prettier.
+If you are looking at output or errors from during extraction, consider installing rich for a better experience: `.venv/bin/python3 -m pip install rich`
+
+To run the extraction outside of `make setup`, use `./tools/extract_assets.sh VERSION`.
+- Pass `-f` to force extraction: otherwise only assets for which xmls were modified will be extracted.
+- Pass `-j` to use multiprocessing, making extraction quicker. Note that this makes for less readable errors if any error happens.
+- Pass `-s name` to extract assets using baserom file `name`.
+- Pass `-r -s 'name.*'` to extract assets using baserom files whose name match regular expression `name.*`.
+
+There currently are various hacks in place in the extraction tool source code to make extraction of some corner cases possible, or to silence extraction warnings.
+Some of these hacks check for the name of resources, so renaming a few specific resources may need updating the extraction tool's source too.
+The plan is to eventually remove those hardcoded checks from the source and use a `HackMode` attribute in the xmls to trigger the hacks code paths.
diff --git a/docs/assets/images.md b/docs/assets/images.md
new file mode 100644
index 000000000..e35e16201
--- /dev/null
+++ b/docs/assets/images.md
@@ -0,0 +1,72 @@
+# Images
+
+Images in the rom are in N64 image formats.
+On extraction, they are converted to png and written to `extracted/VERSION/`.
+On build, they are converted back to N64 formats and written as C arrays to `.inc.c` files in `build/VERSION/`.
+
+The build system will also pick up images in `assets/`, allowing modders to add or even [replace](#replacing-images) images.
+
+PNG files have suffixes indicating how they are to be converted. For example, a `gDekuStickTex.i8.png` file will be converted to `i8`.
+
+The valid formats are `rgba32`, `rgba16`, `i4`, `i8`, `ia4`, `ia8`, `ia16`, `ci4` and `ci8`.
+
+An optional suffix can be used to indicate the element type of the C array written out, for example `.u32` in `gHylianMan1ShirtTex.i8.u32.png`. The valid array element types are `u64` and `u32`. If omitted, the element type defaults to `u64`. `u32` is only used for unaligned textures.
+
+The tool implementing png->n64 conversion is [build_from_png](../../tools/assets/build_from_png/build_from_png.c), using [n64texconv](../../tools/assets/n64texconv/) as its backbone.
+
+# Replacing images
+
+The contents of `extracted/` are meant to hold the baserom assets and should not be modified. Instead, replacing an image can be done by creating a png file at the same relative path under `assets/` as the image to replace.
+
+For example, replacing `gLinkHairTex` (`extracted/VERSION/assets/objects/gameplay_keep/gLinkHairTex.rgba16.png`) with the texture for `gHylianMan1BeardedEyeOpenTex` (`extracted/VERSION/assets/objects/object_ahg/gHylianMan1BeardedEyeOpenTex.ci8.tlut_gHylianMan1TLUT_u32.u32.png`):
+
+```sh
+# for VERSION=gc-eu-mq-dbg
+
+mkdir -p assets/objects/gameplay_keep/
+cp \
+    extracted/gc-eu-mq-dbg/assets/objects/object_ahg/gHylianMan1BeardedEyeOpenTex.ci8.tlut_gHylianMan1TLUT_u32.u32.png \
+    assets/objects/gameplay_keep/gLinkHairTex.rgba16.png
+
+# Cause make to rebuild gameplay_keep, where gLinkHairTex is
+touch extracted/gc-eu-mq-dbg/assets/objects/gameplay_keep/gameplay_keep.c
+
+make VERSION=gc-eu-mq-dbg
+```
+
+# CI images
+
+CI (Color Indexed) images also have a palette or TLUT (Texture Look-Up Table).
+
+PNG images to be converted to CI formats may have a `.tlut_gNameTLUT[_<u32|u64>]` suffix indicating the name and element type (optional, defaults to u64) of the TLUT `gNameTLUT.tlut.rgba16[.<u32|u64>].inc.c` file to write the palette to.
+
+If this suffix is omitted, the TLUT will be written to a `gNameTex.tlut.rgba16.inc.c` file named after the CI image.
+
+For example without the `.tlut_` suffix, `gGanonHairFringeTex.ci8.png`:
+
+- extracted to `extracted/VERSION/assets/objects/object_ganon2/gGanonHairFringeTex.ci8.png`
+- texture written to `build/VERSION/assets/objects/object_ganon2/gGanonHairFringeTex.ci8.inc.c`
+- palette written to `build/VERSION/assets/objects/object_ganon2/gGanonHairFringeTex.tlut.rgba16.inc.c`
+
+For example with the `.tlut_` suffix, `gCowNoseTex.ci8.tlut_gCowTLUT.png`:
+
+- extracted to `extracted/gc-eu-mq-dbg/assets/objects/object_cow/gCowNoseTex.ci8.tlut_gCowTLUT.png`
+- texture written to `build/gc-eu-mq-dbg/assets/objects/object_cow/gCowNoseTex.ci8.tlut_gCowTLUT.inc.c`
+- palette written to `build/gc-eu-mq-dbg/assets/objects/object_cow/gCowTLUT.tlut.rgba16.bin`
+
+CI images with a `.tlut_` suffix have a shared palette: there are several CI images using the same palette.
+The build system (`build_from_png`) will find images sharing the same palette by looking at the `.tlut_` suffixes of png files in the same folder and in the corresponding `assets/` folder.
+
+In the matching case of shared palettes, all png files have the same palette, which is written out.
+Otherwise the images are automatically co-quantized and the resulting images and palette are written out.
+
+Note the N64 supports CI images with IA16 palettes instead of RGBA16 palettes, but OoT doesn't have such textures.
+For simplicity, CI images with IA16 palettes are not supported in the build system, and all CI images are assumed to use RGBA16 palettes.
+
+# Skybox textures
+
+Skybox textures, located in (`extracted/VERSION/`)`assets/textures/skyboxes`, are ci8 images with the additional specificity of being limited to a palette of 128 colors that can be loaded as either the first or last 128 colors of a 256-colors palette, depending on the skybox.
+
+The build system identifies such textures with the additional "sub-format" suffix `.split_lo` or `.split_hi` right after the `.ci8` suffix. `split_lo` corresponds to textures where the half palette is mapped to the lower half, and `split_hi` to the higher half.
+
+Like for regular ci images, images are automatically converted, quantized and written within those restrictions by the build system.