* Add Torch migration plan
Plan for replacing ZAPDTR + OTRExporter with Torch as SoH's only asset
pipeline, following HarbourMasters/Torch#219 ("oot support").
Co-Authored-By: Claude <noreply@anthropic.com>
* Plan: yml repo is source of truth, drop the move-to-assets-repo steps
- Name the actual repo (briaguya0/soh-asset-yml); it holds yml only, and is
edited directly rather than regenerated. zapd_to_torch.py is a one-shot
conversion, not an ongoing workflow, since nobody edits the ZAPD XMLs once
ZAPD is gone.
- Phase 5 is pure deletion. Generator inputs are already preserved by the
harness's shipwright@95d8f7e submodule pin and by git history, so nothing
needs snapshotting or moving out of Shipwright.
- Clarify soh/CMakeLists.txt:107-109 as a configure-soh-directly fallback with
nothing left to fall back to.
- Downgrade the extraction wall-clock risk; perf work landed in Torch#219.
Co-Authored-By: Claude <noreply@anthropic.com>
* Phase 0: flesh out the verification gates
Adds torch-migration/PHASE0.md with the full gate detail: what each gate's
variable is, the driver source, exact commands, cost, and exit criteria.
Two gates the plan was missing:
- Gate C (Release build) - the harness 14/14 is a Debug measurement; CI
and releases build -O3.
- Gate A2 (two extractions in one process) - free once the driver exists,
retires the re-entrancy risk offline instead of by hand in the game.
Also: gates run full-tree + check.sh rather than test_assets.py (which
hardcodes the torch path and copies a filtered yml subset), 19 ROM dumps
not 17, and libgfxd being USE_STANDALONE-only is called out as the real
hazard in Gate A.
* Phase 0: correct the test_assets.py claim
The "filtered subset" was a misread. The log line prints the pre-recursion
yml count (1320); setup_scratch's external_files recursion then pulls in the
remaining 130, so an unfiltered run copies all 1450 - verified by replaying
the logic. So the gates use test_assets.py, the same tool that produced the
existing 14/14, with a TORCH_BIN env override as the only harness patch.
Gate A2 keeps the check.sh path, since it needs two extractions in one
process and test_assets.py execs the binary once per ROM.
* Phase 0: preflight green, 19/19
Baseline Torch build (USE_STANDALONE=ON, all games, Debug, PORT_VERSION_
ENDIANNESS=ON) reproduces every reference archive: 19 ROM dumps, all
0 failed / 0 not generated / 0 not in reference. Counts match the harness
README's 14-target table, and the duplicate dumps agree with each other,
which the 14-target matrix never showed.
Reference manifests also spot-checked by re-derivation from two o2r files
-- the baseline the gates measure against hasn't drifted.
* Phase 0: Gate A green, 19/19
USE_STANDALONE=OFF plus driving Companion by hand produces byte-identical
archives for all 19 ROM dumps.
The real question was libgfxd, which is fetched only if(USE_STANDALONE) --
_deps/ confirms it is absent, so the Binary export path is genuinely
gfxd-free. Reading said so; now it's run.
Progress denominator confirmed exact: SetPhaseCallback fires once per yml
file, matching the on-disk count for all three classes (1449/1450/1480).
Phase 3's bar can be monotonic with no change to OTRGlobals.
Also observed for Phase 2: zlib is fetched even with BUILD_STORMLIB=OFF and
no zlib.h use in torch/src, confirming risk #1; spdlog is not fetched, so
only the global-logger stomping applies there.
* Phase 0: Gate A2 green, and the duplicate-entry finding
Two extractions in one process, vanilla then MQ, both byte-identical to
their references. The second run's phases=1450 matches pal_mq's on-disk
yml count exactly, which is the direct refutation of the gProcessedFiles
risk -- a leak would have skipped files and truncated the archive. A
fresh Companion per extraction is sufficient; PLAN.md risk #6 retired.
Also records what the gate turned up on the way: torch writes 25
duplicate archive entries, benign because libultraship indexes by CRC64
of the path. Filed as Torch#233 item 5. Corrects the Gate A note that
attributed the whole ~343 KB size delta to compression -- 76 KB of it is
these duplicates.
* Phase 0: Gates B and C green
OoT-only (eight BUILD_<game> flags OFF) and Release (-O3) each 19/19
identical, one variable moved apiece -- verified against the option()
defaults so the other flags genuinely match the baseline cache.
Gate C mattered: every parity measurement to date, including the original
14/14, was Debug, while CI and releases build -O3.
Also records extraction wall-clock from torch's own timing line: Release
averages 11.4s per ROM against Debug's 26.7s, and the OoT-only flags cost
nothing. SoH ships Release, so PLAN.md risk #5 looks much smaller than the
Debug numbers suggested -- though ZAPD's time on the same machine is still
unmeasured.
* Phase 0: Gate A' green -- the shipping configuration is the spec
Static lib + OoT-only + Release together: 19/19 identical plus the
vanilla-then-MQ pair. Flags verified from the generated cache rather than
assumed, so Phase 2's CMake block and TorchExtract.cpp can be
transcriptions of this build and RunOnce respectively.
Archive bytes turn out to be configuration-independent -- the pair
produced exactly the sizes the Debug static-lib run did, phases=1450 on
both runs. Extraction 11.8s mean, so the static-lib wrapper costs nothing
over Gate C.
Confirms both Phase 2 dependency risks in the configuration that actually
ships: zlib fetched with BUILD_STORMLIB=OFF and no zlib.h includes,
tinyxml2 fetched with BUILD_NAUDIO=OFF removing its only consumers. Both
FetchContent declarations are unconditional.
* Copy soh.o2r from a checked-in prebuilt archive (TEMPORARY)
Unblocks the rest of the migration. soh.o2r is produced by ZAPD via
OTRExporter/extract_assets.py, both of which this migration deletes; its
replacement is a small in-tree packer that has to reproduce ZAPD's
texture quantisation exactly. Rather than block Phases 2/3/5 on that
packer, check the archive in and make GenerateSohOtr a copy.
The archive doubles as Phase 4's acceptance oracle -- the packer is
correct when its output matches this entry-for-entry -- which is strictly
more useful than Gate E's planned manifest of hashes of it.
Generation turns out to be content-deterministic but NOT byte-
deterministic: two runs give identical payloads for all 1,042 entries but
different file hashes, because zip stores per-entry timestamps. So the
Phase 4 comparison must be entry-wise; whole-file would fail even on a
correct archive. Recorded in prebuilt/README.md.
Adds a configure-time guard on the baked-in portVersion. OTRGlobals.cpp
:283 requires exact major.minor.patch equality and RunExtract exit(1)s
with "soh.o2r is outdated", which is an opaque symptom for a stale
checked-in file; verified the guard fires by building against 9.2.4.
ExtractAssets, CI and copy-existing-otrs.cmake are deliberately untouched
-- ExtractAssets still generates a correct archive while ZAPD exists, and
the CI job keeps working since it just builds this target.
* Extract ROMs with Torch instead of ZAPD
Adds soh/assets/yml as a submodule (briaguya0/soh-asset-yml, 20,353 asset
definitions) and FetchContents torch at 4cae4416, configured exactly as
Gate A' measured it: static lib, OoT only, no UI, no StormLib.
find_package(ZLIB) has to run before torch is declared. Torch declares
zlib with OVERRIDE_FIND_PACKAGE, so it takes over the whole build's
find_package(ZLIB), and its copy provides no ZLIB::ZLIB -- which both
StormLib and CMake's FindPNG link by name. Filed upstream as Torch#233.
CallZapd becomes CallTorch: no chdir, no symlinked assets dir, no 22-entry
argv. Everything that needed the working directory is now a parameter.
TorchExtract.cpp is the only TU that includes Companion.h, since torch
exports its whole lib/ as PUBLIC includes.
Progress uses torch's phase callback, which fires once per yml file, with
the denominator counted off disk. Gate A confirmed the two match exactly
for all three yml counts, so the bar runs 0-100 without the sawtooth the
old per-file counter would have produced.
GetZapdVerStr becomes GetTorchVersionDir, returning the version directory
under the yml tree rather than a ZAPD xml directory.
soh.elf links torch and no longer references zapd_report.
* Record the two-ROM session passing in-game
Vanilla then MQ extracted in one process, both loaded and played. Closes
PLAN.md risk #6 in the game rather than only against the harness driver,
and confirms config.yml-driven archive naming end to end.
* Add soh-torch, rebuild ExtractAssets on it
soh links torch as a static library, which compiles out torch's own CLI,
so build-time extraction needs an entry point. soh-torch supplies one
around the same SohTorch::Extract the game calls -- one implementation,
so the CLI and the in-game path can't drift.
ExtractAssets keeps its old contract: SOH_ROM_PATH accepts roms and/or
directories of roms and defaults to roms/, so dropping a vanilla and a
master quest rom in produces oot.o2r and oot-mq.o2r in one run, and
chaining GenerateSohOtr keeps soh.o2r coming out of it too.
Extract now returns the archive name torch chose rather than deriving it
from IsMasterQuest(), so config.yml is the only thing naming archives.
The name comes from Companion::GetOutputPath(): scanning the destination
for *.o2r picks up an unrelated soh.o2r when extracting into the build
directory, which is only invisible in the game because it extracts into
an empty temp dir.
ExtractAssetHeaders stays as a target that explains why it can't run --
torch registers no OoT header exporter and emits no #define d<sym>, which
soh source references ~24k times. Documented in BUILDING.md alongside it.
This removes the last ZAPD dependency from the build.
* Add the roms directory ExtractAssets defaults to
BUILDING.md points people at roms/, but git can't track an empty
directory, so a fresh clone didn't have one. Its .gitignore keeps
everything but itself and the readme out.
* Delete ZAPDTR and OTRExporter
Nothing drives them any more: ExtractAssets runs soh-torch, the in-game
extractor calls Companion directly, GenerateSohOtr copies a prebuilt
archive, and soh links torch instead of ZAPDLib. Verified with a build
from a fresh tree with both submodules absent from disk.
Removes the submodules, the 7,680 ZAPD xmls under soh/assets/xml, the
extractor inputs (Config_*.xml, TexturePool.xml, filelists, symbols) and
copy-existing-otrs.cmake. The xmls remain in this repo's history and in
the test harness's pinned shipwright submodule, which is where the yaml
was generated from.
Packaging changes shape as well as content: the extractor assets used to
install into the 'extractor' component, which Packaging-2.cmake only
includes for the AppImage generator, so linux zips shipped without them.
torch is linked into soh rather than being a separate binary, so the yml
tree installs into 'ship' on every platform and every package can extract
a rom.
* Keep CI providing tinyxml2 itself
Torch's fetched copy satisfies libultraship today, so the install steps
are redundant -- but that rests on torch declaring tinyxml2 with an
unconditional OVERRIDE_FIND_PACKAGE, which is an implementation detail,
and one we've asked upstream to reconsider in Torch#233. If the fetch
gets gated, find_package needs a system copy again and CI would break
because we removed the steps providing it.
Also notes that the generate-soh-otr slim-down was written before the
prebuilt archive, which makes that job build nothing for now.
* Generate soh.o2r again, replacing the prebuilt archive
soh-o2r-packer builds soh.o2r from soh/assets/custom, reproducing the
ZAPD/OTRExporter archive exactly: 1,042 entries, 0 missing, 0 extra, 0
content mismatch, checked entry-wise because zip stores per-entry
timestamps and whole-file hashes never match between runs.
Most of it is torch's: Companion::Pack walks the directory, zips it and
writes portVersion; BaseExporter::WriteHeader writes the resource header;
TextureType and CalculateTextureSize come from TextureUtils. The packer
stages the assets into the shape the archive should have and hands that
over. Encoding a PNG into an N64 texture is the only piece nothing else
provides -- torch decodes rom data that is already N64 format, never the
reverse -- so PngTexture.cpp is the whole of what had to be written, and
it follows ZAPD's quantisation rather than n64graphics', which scales
where ZAPD shifts.
GenerateSohOtr copies the libultraship shaders into assets/custom again
before packing; assets/custom/shaders is gitignored and nothing else
populates it, so a fresh clone would otherwise pack three files short.
Removes prebuilt/ and its configure-time portVersion guard.
* Move the asset tools under soh/assets
soh-torch and soh-o2r-packer both exist to turn assets into archives, so
they sit better beside the assets they read than at the repo root.
Nothing tangled: both targets are declared in the root CMakeLists with
explicit paths rather than add_subdirectory, and soh/CMakeLists.txt globs
only include/, soh/ and src/, so sources under assets/ aren't swept into
the soh target.
* Build soh.o2r in CI without libultraship or soh
SOH_TOOLS_ONLY returns from the root CMakeLists once the asset tools are
declared, before libultraship and soh are added, so a configure that only
needs soh-o2r-packer never reaches LUS's find_package(SDL2 REQUIRED).
That lets generate-soh-otr drop building SDL2 from source, SDL2_net, and
the deps cache that existed to hold the SDL2 tarball -- eleven steps down
to six. Verified a tools-only build produces the archive with all 1,042
entries matching.
The tool targets move above the sub-projects so the early return can sit
between them; they only ever needed torch. The packer creates its output
directory, which previously came for free from soh's build directory.
* Fix macos and windows builds, keep the tools out of the game build
macos: libultraship declares spdlog with OVERRIDE_FIND_PACKAGE so its own
find_package(spdlog REQUIRED) resolves, but torch declares spdlog without
it, and FetchContent_Declare is first-wins. Configuring torch first made
LUS's declaration a no-op and left find_package with no config to find.
It only showed on macos because that's the platform where neither project
finds an installed spdlog -- it's kept out of macports.yml because its fmt
dependency breaks the universal build -- so both fall through to fetching.
Declaring it ahead of both keeps macos on the same source-built v1.16.0 it
used before. Torch already uses OVERRIDE_FIND_PACKAGE for tinyxml2 and
zlib, so the real fix is upstream.
windows: soh builds with /WX, and TorchExtract.cpp is the one translation
unit reaching yaml-cpp through torch's headers, which trips the
dll-interface warnings. Suppress those two on that file.
EXCLUDE_FROM_ALL on soh-torch and soh-o2r-packer: they are build-time
tools, so building the game shouldn't compile them. ExtractAssets and
GenerateSohOtr still pull them in through DEPENDS.
Drops the .gitattributes rule for the prebuilt archive, which is gone.
* Drop the gitignore negation for the prebuilt archive
Left over with the .gitattributes rule when prebuilt/ was removed.
* TEMPORARY: put soh/assets/xml back so the branch diff is reviewable
Deleting 7,680 xmls puts the diff past what GitHub will render, and they
are 99.8% of it. Restoring them here leaves the ~46 files that actually
changed, which reviews in the compare view.
Revert this before opening the PR.
* Remove the migration planning docs
Scaffolding for the migration itself rather than anything the repo needs
to carry afterwards.
* Fix NTSC master quest roms being rejected as unsupported
verMap's two master quest rows keyed off the GameCube constants instead
of the MQ ones, so with duplicate keys dropped the map never contained
OOT_NTSC_US_MQ or OOT_NTSC_JP_MQ and both roms were filtered out before
extraction. IsMasterQuest already handles both, and the asset yml covers
ntsc_u_mq and ntsc_j_mq, so keying them correctly is all that's needed.
Predates this migration; the torch cli was unaffected because it hashes
the rom rather than consulting verMap.
* Revert "TEMPORARY: put soh/assets/xml back so the branch diff is reviewable"
This reverts commit 28f41138d6e7bc587000703d367203845433e12e.
* Run clang-format
Renaming CallZapd to CallTorch made the call sites one character longer
and left their continuation lines under-indented.
Co-Authored-By: Claude <noreply@anthropic.com>
* Temporarily pin torch to the fork carrying the Windows fixes
HarbourMasters/Torch#234 fixes three Windows-only defects in the OoT factories:
a 64 KiB over-read past the code segment that crashes extraction, backslashes
in exported scene resource names that crash at scene load, and external_files
being keyed differently from the directory walk, which processed every
dependency twice.
Without them a Windows build of this branch cannot extract a rom. Goes back to
a HarbourMasters sha once that PR merges.
Co-Authored-By: Claude <noreply@anthropic.com>
* Point torch back at upstream
HarbourMasters/Torch#234 landed as 65eb11c, squashing the three Windows fixes
this branch was pinned to a fork for. The squashed tree is identical to the
fork commit that was tested, and extraction output is unchanged: 35,386
entries, same CRCs.
Co-Authored-By: Claude <noreply@anthropic.com>
* Vendor the asset yml instead of submoduling it
The submodule existed only so this PR's diff stayed reviewable; squash-merging
it would have put a submodule on develop. This replaces it with the 20,353
files themselves, from briaguya0/soh-asset-yml@523be1d.
The submodule's own README is dropped rather than vendored, since it describes
that repo and was showing up in the assets folder of release downloads.
Co-Authored-By: Claude <noreply@anthropic.com>
* Bump torch to pick up the yaml-cpp 0.9.0 bump
HarbourMasters/Torch#235 landed as e92c210. The pinned yaml-cpp predated
jbeder/yaml-cpp@4fe2fb8, so every scalar conversion constructed a named
std::locale; cheap on glibc, expensive on MSVC. Windows extraction of a PAL GC
rom drops from 49.0s to 13.9s on the same machine, which also puts it ahead of
ZAPDTR's 48.4s rather than fractionally behind.
The merged commit is tree-identical to the fork commit those numbers were
measured on. Extraction output is unchanged: 35,386 entries, same CRCs.
Co-Authored-By: Claude <noreply@anthropic.com>
* Consume torch as a submodule instead of FetchContent
Reviewer feedback on #6989: FetchContent makes iterating on torch itself
awkward, since testing a change means pushing it somewhere the pin can
reach. A submodule is editable in place, which is how libultraship
already works.
Nothing else changes -- FetchContent_MakeAvailable was already doing
add_subdirectory, so the include topology and every build option are the
same. The pin moves from GIT_TAG into the gitlink, at the same sha.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>