Overhaul the build system (#234)

* wip

* fix

* add disassembler

* Disasm builds OK

* Variable addends

* More wip

* Rodata migration implemented

* Cleanup old tools

* Try fix submodule -> subrepo merge

* git subrepo pull --force --remote=https://github.com/zeldaret/ZAPD.git tools/ZAPD

subrepo:
  subdir:   "tools/ZAPD"
  merged:   "602e609"
upstream:
  origin:   "https://github.com/zeldaret/ZAPD.git"
  branch:   "master"
  commit:   "602e609"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* Builds again but assets are totally broken

* git subrepo pull --force tools/asm-processor

subrepo:
  subdir:   "tools/asm-processor"
  merged:   "1ffdb08a"
upstream:
  origin:   "https://github.com/simonlindholm/asm-processor.git"
  branch:   "master"
  commit:   "1ffdb08a"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* More cleanup, move functions.txt and variables.txt to tools/disasm and rm tables

* rm z64compress in preparation for subrepo

* git subrepo clone (merge) https://github.com/z64me/z64compress.git tools/z64compress

subrepo:
  subdir:   "tools/z64compress"
  merged:   "eb11085c"
upstream:
  origin:   "https://github.com/z64me/z64compress.git"
  branch:   "main"
  commit:   "eb11085c"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* Fix asset extraction

* Fix diff-init make rule

* Split code bss

* Split assumed linker bug padding from assembly files

* add filelists for mm.us.rev1

* Maybe working, but I'm not sure

* add overlays to spec

* Add rodata to actos

* Everything compiles

* Make a lot of C files for code

* Add almost every file in code to spec

* whoops

* 3 code files left

* add scenes to spec

* More progress on progress.py

* Fix skelanime in spec

* audio files!

* Fix merge issues

* Fix some C files in code

* Fix remaining code files

* Use existing O1 C files in spec

* reorder boot order in spec

* update spec

* fault.c

* Convert relocs on completed actors, fixbaserom uses current rom name

* more boot files

* Add VT macros and script

* finish already existing boot files

* most of  libultra

* fix 64bits libultra files

* Use C files for libultra, wrap some functions in NON_MATCHING

* Remove duplicate of OS_CLOCK_RATE from fault.c

* C files for fbdemos

* delete dumb files

* bootstrap C files, still need to add them to the spec

* update fixbaserom

* boot OK?

* I forgot to commit the spec

* C for gamestates

* C for kaleido

* Change all includes to ""

* copy actor sizes script from oot

* I forgot to delete those files

* Basic C files for effects

* Add effects initvars names

* Remove mislabelled boot functions from header/txt

* Begin porting bootstrap_fx, some sizes

* Fix <>

* Fix enum

* Fix diff.py

* fix libultra stuff

* update regconvert

* update setup warnings

* add some missing ;

* Fix some makefile stuff and other fixes on some non_matching functions

* add executable flag in extract_baserom and fixbaserom

* fix relative path

* copy assist from oot

* fix map path

* another assist path fix

* Delete C files for handwritten files

* add code_801A51F0 to spec

* add gfxbuffers to spec

* Move rodata to top of each file when possible

* UNK_TYPEs for func_801A51F0

* Remove kaleido rodata from spec

* Update spec and undefined_syms for recent merge

* GCC warnings and fix errors in nonmatchings,

* round percentage numbers

* progress script: format changes

* progress: error on non-existing files

* fix warning in z_scene_table

* Match 2 nonmatchings in z_actor

* Warnings in lightswitch and invadepoh

* Fix warning in z_actor_dlftbls

* I though I fixed this one

* whoops

* Comment out CC_CHECK

* Removed redundant ultra64.h includes

* Update asm_processor, sorted boot_O1 into other folders, completed the fbdemo bootstrap, cleaned up undefined_syms

* Completed gamestates bootstrap

* Split kaleido_scope

* Remove section.h and segment.h, move keep object externs to a common location in variables.h

* Completed effects bootstrap

* Segmented address externs for effects, fbdemos, gamestates and kaleido

* Move actor data externs out of the if 0

* Segmented address externs for actors

* Prepare actionfunc detection

* fix script, how did it even work before

* Fix actionfunc script again, re-introduce some more intermediate prints to the disassembler

* Automated actionFunc detection in actors

* Segmented addresses from player .text

* rm old segment addrs script and fix build

* Move sizes folder to tools

* Make build.py executable

* New Jenkinsfile Prayge

* Remove numpy dependencies

* Add warnings_disasm_current.txt

* my bad

* Update spec and undefined_syms

* Add z_eff_ss_hahen to pametfrog

* git subrepo pull (merge) --force tools/z64compress

subrepo:
  subdir:   "tools/z64compress"
  merged:   "163ca2af"
upstream:
  origin:   "https://github.com/z64me/z64compress.git"
  branch:   "main"
  commit:   "163ca2af"
git-subrepo:
  version:  "0.4.3"
  origin:   "https://github.com/ingydotnet/git-subrepo"
  commit:   "2f68596"

* Make z64compress print to stdout

* sneeky commit to update warnings tooling

* test

* Another test

* Mark fixing overlay reloc generating as a TODO

* Update warnings stuff

* Communicate the return code from running z64compress back to the Makefile through the wrapper

* Run formatter, remove extra commented copy of function

* Re-fix some includes

* Convert atan to hex to conform to decided style

* Some tidying up, remove c for fp and the other two handwritten code files

* BSS in z_collision_check & z_scene_proc

* add static back in

* Fix timerintr bss, add file to spec, some cleanup

* Remove externs

* Newline

* Readd enums

* Typo

* Colours

* Comments for hitmark enum values

Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>

* Improvements and suggestions

* Organize and remove unused imports and use env for python3 scripts, delete unused overlay.py

Co-authored-by: angie <angheloalf95@gmail.com>
Co-authored-by: Elliptic Ellipsis <elliptic.ellipsis@gmail.com>
Co-authored-by: engineer124 <engineer124engineer124@gmail.com>
Co-authored-by: EllipticEllipsis <73679967+EllipticEllipsis@users.noreply.github.com>

1 files changed, 11 insertions, 26 deletions

diff --git a/tools/asm-processor/asm_processor.py b/tools/asm-processor/asm_processor.py
index 5fe0980c4..302668d7d 100644
--- a/tools/asm-processor/asm_processor.py
+++ b/tools/asm-processor/asm_processor.py
@@ -380,7 +380,7 @@ class Failure(Exception):
 
 
 class GlobalState:
-    def __init__(self, min_instr_count, skip_instr_count, use_jtbl_for_rodata, mips1):
+    def __init__(self, min_instr_count, skip_instr_count, use_jtbl_for_rodata):
         # A value that hopefully never appears as a 32-bit rodata constant (or we
         # miscompile late rodata). Increases by 1 in each step.
         self.late_rodata_hex = 0xE0123456
@@ -388,7 +388,6 @@ class GlobalState:
         self.min_instr_count = min_instr_count
         self.skip_instr_count = skip_instr_count
         self.use_jtbl_for_rodata = use_jtbl_for_rodata
-        self.mips1 = mips1
 
     def next_late_rodata_hex(self):
         dummy_bytes = struct.pack('>I', self.late_rodata_hex)
@@ -609,13 +608,12 @@ class GlobalAsmBlock:
             size = self.fn_section_sizes['.late_rodata'] // 4
             skip_next = False
             needs_double = (self.late_rodata_alignment != 0)
-            extra_mips1_nop = False
             for i in range(size):
                 if skip_next:
                     skip_next = False
                     continue
-                # Jump tables give 9 instructions (11 with -mips1) for >= 5 words of rodata,
-                # and should be emitted when:
+                # Jump tables give 9 instructions for >= 5 words of rodata, and should be
+                # emitted when:
                 # - -O2 or -O2 -g3 are used, which give the right codegen
                 # - we have emitted our first .float/.double (to ensure that we find the
                 #   created rodata in the binary)
@@ -625,14 +623,12 @@ class GlobalAsmBlock:
                 #   generate a jump table)
                 # - we have at least 10 more instructions to go in this function (otherwise our
                 #   function size computation will be wrong since the delay slot goes unused)
-                jtbl_size = 11 if state.mips1 else 9
                 if (not needs_double and state.use_jtbl_for_rodata and i >= 1 and
-                        size - i >= 5 and num_instr - len(late_rodata_fn_output) >= jtbl_size + 1):
+                        size - i >= 5 and num_instr - len(late_rodata_fn_output) >= 10):
                     cases = " ".join("case {}:".format(case) for case in range(size - i))
                     late_rodata_fn_output.append("switch (*(volatile int*)0) { " + cases + " ; }")
-                    late_rodata_fn_output.extend([""] * (jtbl_size - 1))
+                    late_rodata_fn_output.extend([""] * 8)
                     jtbl_rodata_size = (size - i) * 4
-                    extra_mips1_nop = True
                     break
                 dummy_bytes = state.next_late_rodata_hex()
                 late_rodata_dummy_bytes.append(dummy_bytes)
@@ -642,20 +638,12 @@ class GlobalAsmBlock:
                     fval, = struct.unpack('>d', dummy_bytes + dummy_bytes2)
                     late_rodata_fn_output.append('*(volatile double*)0 = {};'.format(fval))
                     skip_next = True
-                    needs_double = False
-                    if state.mips1:
-                        # mips1 does not have ldc1/sdc1
-                        late_rodata_fn_output.append('')
-                        late_rodata_fn_output.append('')
-                    extra_mips1_nop = False
+                    needs_double = True
                 else:
                     fval, = struct.unpack('>f', dummy_bytes)
                     late_rodata_fn_output.append('*(volatile float*)0 = {}f;'.format(fval))
-                    extra_mips1_nop = True
                 late_rodata_fn_output.append('')
                 late_rodata_fn_output.append('')
-            if state.mips1 and extra_mips1_nop:
-                late_rodata_fn_output.append('')
 
         text_name = None
         if self.fn_section_sizes['.text'] > 0 or late_rodata_fn_output:
@@ -734,7 +722,7 @@ float_regexpr = re.compile(r"[-+]?[0-9]*\.?[0-9]+([eE][-+]?[0-9]+)?f")
 def repl_float_hex(m):
     return str(struct.unpack(">I", struct.pack(">f", float(m.group(0).strip().rstrip("f"))))[0])
 
-def parse_source(f, opt, framepointer, mips1, input_enc, output_enc, out_dependencies, print_source=None):
+def parse_source(f, opt, framepointer, input_enc, output_enc, out_dependencies, print_source=None):
     if opt in ['O2', 'O1']:
         if framepointer:
             min_instr_count = 6
@@ -763,7 +751,7 @@ def parse_source(f, opt, framepointer, mips1, input_enc, output_enc, out_depende
     if opt in ['O2', 'g3'] and not framepointer:
         use_jtbl_for_rodata = True
 
-    state = GlobalState(min_instr_count, skip_instr_count, use_jtbl_for_rodata, mips1)
+    state = GlobalState(min_instr_count, skip_instr_count, use_jtbl_for_rodata)
 
     global_asm = None
     asm_functions = []
@@ -815,7 +803,7 @@ def parse_source(f, opt, framepointer, mips1, input_enc, output_enc, out_depende
                 out_dependencies.append(fname)
                 include_src = StringIO()
                 with open(fname, encoding=input_enc) as include_file:
-                    parse_source(include_file, opt, framepointer, mips1, input_enc, output_enc, out_dependencies, include_src)
+                    parse_source(include_file, opt, framepointer, input_enc, output_enc, out_dependencies, include_src)
                 include_src.write('#line ' + str(line_no + 1) + ' "' + f.name + '"')
                 output_lines[-1] = include_src.getvalue()
                 include_src.close()
@@ -1191,7 +1179,6 @@ def run_wrapped(argv, outfile, functions):
     parser.add_argument('--input-enc', default='latin1', help="Input encoding (default: latin1)")
     parser.add_argument('--output-enc', default='latin1', help="Output encoding (default: latin1)")
     parser.add_argument('-framepointer', dest='framepointer', action='store_true')
-    parser.add_argument('-mips1', dest='mips1', action='store_true')
     parser.add_argument('-g3', dest='g3', action='store_true')
     group = parser.add_mutually_exclusive_group(required=True)
     group.add_argument('-O1', dest='opt', action='store_const', const='O1')
@@ -1203,20 +1190,18 @@ def run_wrapped(argv, outfile, functions):
         if opt != 'O2':
             raise Failure("-g3 is only supported together with -O2")
         opt = 'g3'
-    if args.mips1 and (opt != 'O2' or args.framepointer):
-        raise Failure("-mips1 is only supported together with -O2")
 
     if args.objfile is None:
         with open(args.filename, encoding=args.input_enc) as f:
             deps = []
-            functions = parse_source(f, opt=opt, framepointer=args.framepointer, mips1=args.mips1, input_enc=args.input_enc, output_enc=args.output_enc, out_dependencies=deps, print_source=outfile)
+            functions = parse_source(f, opt=opt, framepointer=args.framepointer, input_enc=args.input_enc, output_enc=args.output_enc, out_dependencies=deps, print_source=outfile)
             return functions, deps
     else:
         if args.assembler is None:
             raise Failure("must pass assembler command")
         if functions is None:
             with open(args.filename, encoding=args.input_enc) as f:
-                functions = parse_source(f, opt=opt, framepointer=args.framepointer, mips1=args.mips1, input_enc=args.input_enc, out_dependencies=[], output_enc=args.output_enc)
+                functions = parse_source(f, opt=opt, framepointer=args.framepointer, input_enc=args.input_enc, out_dependencies=[], output_enc=args.output_enc)
         if not functions:
             return
         asm_prelude = b''