Set up disassembly for gc-eu-mq (#1613)

* Disassemble gc-eu-mq

* Add script to report progress with matching

* Fix whitespace

* Change T|None to typing.Optional[T]

* Use typing.List

* More type annotations fixes for old Python versions

* Fix type errors

* More type annotations

* Use typing.Iterator

* Use $(PYTHON) for disassembly

* Don't use grouped targets to support very old Make versions

* Docs: suggest checking MM, mention virtualenv, clarify about expected/

* Update sym_info.py

Co-authored-by: Derek Hensley <hensley.derek58@gmail.com>

* Sync functions.txt

* Start banned symbol range at 0x10000000

* Also ban symbols from 0xC0000000-0xFFFFFFFF

* Unban IPL symbols

* Fix first_diff.py

* Sync z_collision_check functions.txt

* Ban 0xA0 symbols too

* Touch .disasm sentinel file

* Copy -jN comment in docs

* diff.py flags: remove -3, add -s

* Update docs/retail_versions.md

Co-authored-by: Yanis42 <35189056+Yanis42@users.noreply.github.com>

* Comment that segments are still from the Debug ROM

* Revert "diff.py flags: remove -3, add -s"

This reverts commit bfaae66c1d02719e706734be9503e78d2b3f104b.

* Apply suggestions from code review

Co-authored-by: Dragorn421 <Dragorn421@users.noreply.github.com>

* Remove #ifdef example

* Reformat Python files with black

* Add copyright notice to new Python files

* Add TODOs to Makefile

---------

Co-authored-by: Derek Hensley <hensley.derek58@gmail.com>
Co-authored-by: Yanis42 <35189056+Yanis42@users.noreply.github.com>
Co-authored-by: Dragorn421 <Dragorn421@users.noreply.github.com>

1 files changed, 126 insertions, 0 deletions

diff --git a/docs/retail_versions.md b/docs/retail_versions.md
new file mode 100644
index 000000000..6c34869f9
--- /dev/null
+++ b/docs/retail_versions.md
@@ -0,0 +1,126 @@
+# Decompiling retail versions
+
+The next decompilation target for OOT is the PAL GameCube Master Quest ROM
+(`gc-eu-mq`), because it is the retail version that is most similar to the Debug
+ROM. Unfortunately there are still a lot of differences, many of which are
+register or stack allocation differences because retail ROMs were built with
+different compiler flags. However, once this version is done, future
+retail versions should be much easier, as the changes between retail versions are
+small in comparison.
+
+Instead of `cp`ing a matching build into `expected/`, the target ROM is disassembled as `.s` files then
+reassembled as `.o` files directly into `expected/build/gc-eu-mq` for diff tools.
+This allows us to make progress matching code in parallel with solving other
+problems (such as the build system, ROM organization, and BSS ordering). The
+files in `tools/disasm/gc-eu-mq` say how to split the source files and where the
+functions and variables are in the target ROM, and these may need to be updated
+if there are mistakes or if function names change due to documentation work.
+
+Unfortunately, the disassembly is not perfect, so a "correct" decompilation might
+still show diffs with data symbols. We might improve this later, but these data
+diffs are fine to ignore for now.
+
+For register and stack allocation differences, often the code can be tweaked so
+that it matches both the retail ROM while continuing to match the Debug ROM (for
+example, by reordering assignments or moving a local variable declaration inside
+an `if` block). Since retail MM versions use the same compiler flags as retail
+OOT, checking MM decomp for similar code can help.
+
+We can disable code that was removed in retail builds by adding
+`#ifdef OOT_DEBUG` around these parts of the code. In order to keep the code
+readable, we should try to minimize the amount of `#ifdef` noise whenever
+possible.
+
+## Setup
+
+1.  Copy your target PAL GameCube Master Quest ROM (non-debug) to
+    `baseroms/gc-eu-mq/baserom.z64`
+
+1.  Extract assets and ROM files **from the Debug ROM** by running
+
+    ```sh
+    make setup -jN
+    ```
+
+    if necessary, where `N` is the number of cores on your machine.
+
+1.  Build the non-matching test ROM by running
+
+    ```sh
+    make setup -jN VERSION=gc-eu-mq
+    make -jN VERSION=gc-eu-mq
+    ```
+
+    where `N` is the number of cores on your machine. This will build into
+    `build/gc-eu-mq` and produce `oot-gc-eu-mq.z64`.
+
+    If you later want to delete all output files, run
+
+    ```sh
+    make clean VERSION=gc-eu-mq
+    ```
+
+1.  Disassemble the target ROM by running
+
+    ```sh
+    make disasm -jN VERSION=gc-eu-mq
+    ```
+
+    where `N` is the number of cores on your machine. The outputs will be written to
+    `expected/build/gc-eu-mq`.
+
+    Note that if you need to copy a matching build for the Debug ROM, you can use
+
+    ```sh
+    mkdir -p expected/build
+    cp -r build/gc-eu-mq-dbg expected/build
+    ```
+
+    to avoid clobbering the disassembly.
+
+## Diff Tools
+
+Note that many tools will require activating the Python virtual environment
+in your terminal session. To do this, run:
+
+```sh
+source .venv/bin/activate
+```
+
+### retail_progress.py
+
+Running `./retail_progress.py path/to/file.c` will attempt to figure out which functions
+in a file still need to match for `gc-eu-mq`. To get an overview of diffs for
+all files, run `./retail_progress.py` with no arguments.
+
+### asm-differ / diff.py
+
+To diff assembly for a single function in `gc-eu-mq`, run e.g.
+
+```sh
+./diff.py -mwo3 -v gc-eu-mq Math3D_CylTriVsIntersect
+```
+
+The `-v` flag tells `diff.py` to compare between `build/gc-eu-mq` and
+`expected/build/gc-eu-mq`, and to use `make VERSION=gc-eu-mq` when rebuilding.
+
+You may also want to diff the Debug ROM in another terminal with
+
+```sh
+./diff.py -mwo3 Math3D_CylTriVsIntersect
+```
+
+to ensure any changes still match there.
+
+### Permuter and decomp.me
+
+Disassembly for individual functions is written to
+`expected/build/gc-eu-mq/functions`, so to get a [decomp.me](https://decomp.me/) scratch you can run
+e.g.
+
+```sh
+decomp-permuter/import.py \
+  src/code/sys_math3d.c \
+  expected/build/gc-eu-mq/functions/src/code/sys_math3d/Math3D_CylTriVsIntersect.s \
+  VERSION=gc-eu-mq --decompme
+```