oot/requirements.txt, branch main

New assets system (#2481)

2025-05-17T23:29:09+00:00

* wip: New assets system tm

Builds gc-eu-mq-dbg OK from clean after
1) make setup
2) python3 -m tools.assets.extract -j
3) replace 0x80A8E610 with sShadowTex in extracted/gc-eu-mq-dbg/assets/overlays/ovl_En_Jsjutan/sShadowMaterialDL.inc.c
4) make various symbols in extracted data like sTex static

* use variables from config.yml for gMtxClear and sShadowTex addresses

* Write source with static for overlays using `HACK_IS_STATIC_ON` hack

* gc-eu-mq-dbg OK from clean with `make setup && make`

* implement more skeleton-related types, cleanups, fixups

* fix extracted data to no longer produce compilation warnings

* implement more of RoomShapeImage types

* yeet XmlPath from ExternalFile usage

* Implement PlayerAnimationDataResource (link_animetion data)

* fix csdis CS_TIME extra arg

* dmadata file names no longer hardcoded for gc-eu-mq-dbg

* ntsc-1.0 OK

* xml fixes

* slightly improve standard output

* rm extract_assets.py

* generate and use Limb enums (TODO: check Skin skels and implement for Curve skels)

* handle dependencies between xmls

* introduce RawPointers xml attribute to ignore specific pointers and keep them raw

* add tools/extract_assets.sh

* fixups

* only extract if xmls changed or if -f (force) is used

* fixups, gc-eu OK

* all versions OK

* check attributes of xml resources elements

* Implement legacy skelanime resources

* fix ASSET_FILES_BIN_EXTRACTED/COMMITTED: look for .u8.bin specifically instead of just .bin

* implement JFIFResource

* fix png/jpg wildcards: look specifically for .u64.png .u32.png .u64.jpg

* Makefile: Add rules to build .png, .bin and .jpg in assets/ too

* start writing actual docs

* extract sTransCircleDL and sTransWipeDL

* misc cleanup/fixes, pygfxd 1.0.3

* refactor CDataExt.set_write callback args to use a dataclass

* Move {} to in-source

* misc

* more progress on spec

* fix missing braces in n64dd_error_textures.c

* finish xml spec doc

* assets xmls fixes

* some cleanup, use `gNameTex_WIDTH/HEIGHT` macros in dlists

* handle hackmode_syotes_room, fix compile

* C build_from_png

* rm tools/assets/bin2c

* rm ZAPD

* format

* remove rule to generate dmadata_table.py

* CC0 license (and some import cleanup)

* dont try to build zapd (rmd)

* simplify palettes with single user (ci images with a non-shared palette)

* add docs on how images are handled

* bss

* allow -j N

* fix n64texconv python bindings memory management

* move -j at the end of calling extraction script

* with -j, update last_extracts.json as each job completes rather than only if all complete

* make interrupting less jank by making child processes ignore sigint

* use enum names in `SCENE_CMD_SKYBOX_SETTINGS`

* `multiprocessing.get_context("fork")`

* import rich, except ImportError s

* fix optional rich usage

* .bss

* .bss

* .bss

* assets extraction: -j -> -j$(N_THREADS)

* .bss

* change LIMB_NONE/MAX defaults to be FILE_OFFSET instead of SKELNAME

* 0XHEX -> 0xHEX

* fix bss

* Proper includes for assets

mostly proper, some includes like dlists resources always causing a sys_matrix.h include (when not every dlist references gIdentityMtx) could be done better

* rm z64.h

* rm z64.h take two

* bss

* Make .u64 suffix for pngs optional

* fixup: rm .u64 suffix from n64dd image paths

* Remove elemtype suffixes from .bin and .jpg files

* Update images.md

* some build_from_png cleanup, more error handling, comments

* Handle skybox textures

Introduce "sub-format" suffix for pngs, with sub-formats split_lo and split_hi being used for skybox textures

* fixup for older python

* improve collision output some

* fully use SURFACETYPE[01] macros in writing extracted surface types

* use WATERBOX_PROPERTIES in extracted waterboxes

* some SceneCommandsResource cleanup

* format EnvLightSettingsList output

Match iQue compression (#2389)

2024-12-27T15:34:45+00:00

* Match iQue compression

* Use a dict

* Use Makefile variables for compression settings

* Put COMPRESS_ARGS last

* Pad iQue ROMs to 16 KiB

* Replace --pad-rom with --pad-to and --fill-padding-bytes

* Clarify --fill-padding-bytes

Add "disasm sym_info.py" script (#2054)

2024-08-19T21:58:41+00:00

* Add "disasm sym_info.py" script for investigating the expected memory layout

* review

* add friendly error if context.csv is missing

* rom as hexadecimal

* require spimdisasm 1.28.1 (technically not *required* but may as well)

* some fixups on symbol lookup logic

Introduce version-specific YAML config (#1957)

2024-06-21T02:26:32+00:00

Add script to report BSS reordering (#1914)

2024-03-04T17:58:36+00:00

* Write script to report BSS reordering

* Pin pyelftools version

* Fail on unknown relocation types

* Add sanity check for shifted ROM

* segment -> mapfile_segment

* Fix inconsistent offset/address naming

* Format negative addends

* Don't attempt to find relocs in .bss sections

* Compare build_value to mapfile VRAM

update mapfile_parser (#1744)

2024-02-15T17:21:03+00:00

Rework disassembly: out of makefile, separate sections, relocs (#1728)

2024-02-13T01:20:25+00:00

* Rework disassembly: out of makefile, separate sections, relocs

* another nice print

* fix type hints for python 3.8 dinosaurs

* reencode from utf-8 to euc-jp (iconv) before assemble

* Add "include guards" to macro.inc

* For producing the expected file object files, assemble sections together instead of linking together the individually assembled sections

* review

* fixup

* require spimdisasm >=1.21.0

* remove assembling sections individually (for now)

* uppercase encoding names (standard)

Set up disassembly for gc-eu-mq (#1613)

2024-01-28T17:09:03+00:00

* Disassemble gc-eu-mq

* Add script to report progress with matching

* Fix whitespace

* Change T|None to typing.Optional[T]

* Use typing.List

* More type annotations fixes for old Python versions

* Fix type errors

* More type annotations

* Use typing.Iterator

* Use $(PYTHON) for disassembly

* Don't use grouped targets to support very old Make versions

* Docs: suggest checking MM, mention virtualenv, clarify about expected/

* Update sym_info.py

Co-authored-by: Derek Hensley 

* Sync functions.txt

* Start banned symbol range at 0x10000000

* Also ban symbols from 0xC0000000-0xFFFFFFFF

* Unban IPL symbols

* Fix first_diff.py

* Sync z_collision_check functions.txt

* Ban 0xA0 symbols too

* Touch .disasm sentinel file

* Copy -jN comment in docs

* diff.py flags: remove -3, add -s

* Update docs/retail_versions.md

Co-authored-by: Yanis42 <35189056+Yanis42@users.noreply.github.com>

* Comment that segments are still from the Debug ROM

* Revert "diff.py flags: remove -3, add -s"

This reverts commit bfaae66c1d02719e706734be9503e78d2b3f104b.

* Apply suggestions from code review

Co-authored-by: Dragorn421 

* Remove #ifdef example

* Reformat Python files with black

* Add copyright notice to new Python files

* Add TODOs to Makefile

---------

Co-authored-by: Derek Hensley 
Co-authored-by: Yanis42 <35189056+Yanis42@users.noreply.github.com>
Co-authored-by: Dragorn421

Add dependencies for tools to requirements.txt (#1621)

2024-01-25T00:03:57+00:00

* Add dependencies for tools to requirements.txt

* Downgrade mapfile-parser

Setup rom decompression and compression (#1614)

2024-01-24T18:00:10+00:00

* decompress baserom

* cleanup

* specific hash check

* rename baserom

* git subrepo clone (merge) --branch=5da3132606e4fd427a8d72b8428e4f921cd6e56f git@github.com:z64tools/z64compress.git tools/z64compress

subrepo:
  subdir:   "tools/z64compress"
  merged:   "5da313260"
upstream:
  origin:   "git@github.com:z64tools/z64compress.git"
  branch:   "5da3132606e4fd427a8d72b8428e4f921cd6e56f"
  commit:   "5da313260"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo.git"
  commit:   "2f68596"

* setup compression

* Add all compressed segments to the spec

* Update md5 files

* readme instructions

* cleanup

* Setup python dependencies on Jenkinsfile

* Update Makefile

Co-authored-by: cadmic 

* review

* . .venv/bin/activate

* update readme

* whoops

* Yeet other versions from decompress_baserom.py

* my bad

* Move everything to baseroms/VERSION/

* Active venv on each command

* jenkinsfile: use multiline syntax

* Put the correct path on the jenkinsfile

* Forgot to call per_version_fixes

* CC0

* Update readme

* Change where baserom segments are put into

* Update Makefile

Co-authored-by: Dragorn421 

* Update crunch64

* Label compressed instead of uncompressed

* Update README.md

Co-authored-by: fig02 

* Fix

* `make rom`

* baserom-uncompressed

* Update README.md

Co-authored-by: Dragorn421 

* Update README.md

Co-authored-by: Dragorn421 

* Update README.md

Co-authored-by: Dragorn421 

* Update README.md

Co-authored-by: Dragorn421 

* Update README.md

Co-authored-by: Dragorn421 

* review

* baserom-decompressed.z64

* ignore baseroms

* rm -rf tools/z64compress

* wip crunch64-based compress.py

* OK compress

* use ipl3checksum sum directly for cic update on compressed rom

* multithreading

* "docs"

* fix MT: move set_sigint_ignored to global level for pickling

* license compress.py

* rm junk

* Fix (or at least sort out) compress_ranges.txt dependencies

* Update tools/overlayhelpers/damage_table.py

Co-authored-by: Dragorn421 

---------

Co-authored-by: cadmic 
Co-authored-by: Dragorn421 
Co-authored-by: fig02