# Packing ## Game packages ### PFS0 file-name table missing its final NUL - **Symptom:** package identification stops with `ValueError: subsection not found` in `pfs0_names`, or LibHac reports `ResultFsPathNotFound` opening the last file of an NSP. - **Cause:** older NSZ writers omitted the final NUL in the PFS0 string table. The final name is bounded by the table length, but the Python scanner required a terminator and LibHac reads names as NUL-terminated strings. - **Fix:** `nx_extract` supplies LibHac with a virtual header containing the missing NUL followed by the original payload storage. File offsets remain relative to that payload, and the source package and encrypted NCA bytes are unchanged. Truncated tables and out-of-table name offsets still fail. Synthetic package tests cover the reader and payload preservation (`test_pfs0_storage.py`). The Python scanner's own table reader (`pfs0_names`), which accepted the table boundary as the end of the final name, is gone: the scan reads every package through `nx_extract` ([An update is taken for the version in its file name](#an-update-is-taken-for-the-version-in-its-file-name)). - **Found on:** Moving Out 2 (`010065D018172000`), base 1.1.254 NSP and update 1.3.315 NSZ. Package reading was checked, not gameplay. ### A NUP with complete standalone updates stops at "no base game" - **Symptom:** `no base game, only update packages: pack the base game and its update into one NUP`, even when the update carries the complete game's files ([issue #1195](https://github.com/seedds/nxnative/issues/1195)). - **Cause:** `pack_nup.scan` required a `base` record. NUP v1 permits a container without one when every update has the `standalone_update` flag (package flags bit 0, [SPEC §3.1.2](https://github.com/a55uka/nup/blob/meow/SPEC.md#312-package-kinds)). The reference packer writes one for a dump with no `base/` folder and, since its commit `92089de3`, for an empty `base/` too; before that an empty `base/` gave an empty base record, which got through the check. The issue was filed before that commit, so its NUP came from a dump with no `base/` folder or from another packer; the file itself was not seen. - **Fix:** no base record and a base record with no files are one case: `scan` gives `None` for the base, and only the update versions present are offered. Both packing entry points select the newest update by default; requesting version 0 fails. The selected update must have `exefs/main`, `exefs/main.npdm` and its own RomFS; it cannot borrow another update's RomFS. Control data the update lacks comes from the record that holds application metadata: the empty base record, else the first package ([SPEC §5.1](https://github.com/a55uka/nup/blob/meow/SPEC.md#51-localized-records-types-0x0009--0x000a)). A base with files still supplies the RomFS for an update without one. Program-ID, DLC, content-size and SHA-256 checks apply as before. A version that cannot be packed stops at a `[RED]` line: the disk check reads its RomFS folder before extraction, so `cli()` catches `pack_nup.Error`. - **Not known from the file:** whether the update is the whole game. The flag does not promise it (the spec says such an update needs a matching base elsewhere), and an update holding only changed files would pack and then fail at boot, after a full shader build. The scripts print that the update is taken as the whole game. Measured: both NUPs at hand have an update RomFS that holds every path of its base (Age of Imprisonment: 2,065 base files, 2,153 in the update, none missing; Yoshi and the Mysterious Book: 147 and 147, none missing). - **Found on:** [Mario Kart World](../../dm/games/0400C3F00006E000/README.md). Reproduced with its base dump repackaged by the reference tool as a standalone-update fixture; this checks packaging, not the reported 1.8.0's contents or gameplay. Synthetic tests cover both entry points, both forms of no base, multiple versions, control data, DLC and incomplete updates. ### A NUP or a dump of a game with no profile stops before the survey - **Symptom:** `[RED] a NUP packs only a game with a profile, and has none: the survey that makes a provisional profile reads NSPs`, or the same line for a dump folder ([issue #1265](https://github.com/seedds/nxnative/issues/1265)). A folder holding `exefs/` and `romfs/` directly, or only `updates/<version>/`, was not taken for a dump at all: the script asks for `prod.keys`, then stops with `no NSP with a readable title ID`. - **Cause:** the survey and the build ID lookup (`same_build`) ran `nx_extract --manifest`, which reads NSPs. `pack_dump.is_dump` asked for `base/exefs/main.npdm`. - **Fix:** each source leaves the same files for the survey: `survey_nup` reads the newest version from the container (its ExeFS, a RomFS manifest in `nx_extract`'s format: path, size, first 32 bytes, `Nup.head` inflating only those; the NROs whole and the control data), and `survey_dump` from the folder, whose name names the game, as a dump has no control data. `same_build` reads main from the container or the folder. A dump may have no `base/` when an update's folder holds code and a RomFS, and `exefs/` and `romfs/` directly in the folder, as `nup extract` writes them, count as `base/`: version 0, so a profile that registers another version packs them from a derived profile. - **Found on:** Star Fox (`0400E88025B1A000`, update v131072; no profile in the repository then), The Legend of Zelda: Ocarina of Time (`0400486024DEC000`, issue #1265) and [Fire Emblem Fortune's Weave](../../dm/games/04007EF01F8AE000/README.md), whose dump holds `updates/131072/` alone. ### A damaged file in a NUP or a dump has no hash to fail - **Symptom:** the game reloads one asset for ever: a background loader takes about a core and the process several (Ocarina of Time: 412% CPU on the Mac, `OptimizeChangeBanc` and `ParallelDecompThread` busy all session, one model and its texture opened about 22 times a second). Or, from the NSP script: `vN: K file(s) whose bytes do not start as their names promise (a .zs file that is not a zstd stream), so the game files are damaged`, then the files. - **Cause:** the game files themselves. An NSP's files are checked against the NCA's hashes at extraction, so a damaged one stops the build ([`--allow-damaged`](../development.md#the-packing-script-in-detail)). A NUP or a dump folder carries no such hashes: whatever bytes it holds are packed. On one NUP of Ocarina of Time three `.zs` textures began with 12,288 zero bytes, so no zstd frame: the game could not decompress them, and its loader asked for them again on every pass ([PR #1530](https://github.com/seedds/nxnative/pull/1530)). The container's own SHA-256 of each file matched, so the bytes were like this when the NUP was packed. Another NUP of the same version holds all three as zstd streams, and none of its 102,185 `.zs` files fails the check. - **Fix:** the NSP script reads the first bytes of every file whose name promises a format (`romfs_image.FORMAT_HEADS`: `.zs` starts with a zstd frame or a skippable frame) once per extracted image (`check_formats`, after extraction; the result is kept in `RomFS.img.formats` beside the image until the image changes: Ocarina of Time 16 s from a USB SSD) and stops naming the damaged files: pack again from a good copy of the game. `--allow-damaged` lists them and packs them as they are. The host serves a game's files as they are: substituting other files for damaged ones would hide the damage and change what the game draws. - **Found on:** [The Legend of Zelda: Ocarina of Time](../../dm/games/0400486024DEC000/README.md), a contributor's NUP of v131072. ### A provisional profile of a Switch 2 game shows no SDK and a Switch 1 memory pool - **Symptom:** the survey of a Switch 2 game says `SDK: ?` and `no SDK version string found`, and its provisional profile has `app_pool 0xCD500000`. - **Cause:** the SDK version pattern did not take the variant the Switch 2's SDKs name (`NintendoSdk_nnSdk-Generic_21_4_4`), and `guess` had one pool, a 4 GB Switch's. - **Fix:** the pattern takes the variant, and an `sdk` built for the Switch 2 (`spec: Ounce`, as `dm_sdk_ounce` reads it) gets the 9 GB pool every Switch 2 profile has. Open: only the pool changes; a game with system resource keeps the 5 GB block (Star Fox's profile has 21 GB with a 16 GB alias region), so a provisional profile may still not fit and needs the bring-up. - **Found on:** Star Fox and Kirby Air Riders (`040077201BE50000`; no profile), read from their NUP and extraction; first reported with [Fire Emblem Fortune's Weave](../../dm/games/04007EF01F8AE000/README.md). ## Build tools ### Homebrew .NET is on PATH but apphosts cannot find the runtime - **Symptom:** the first extraction stops with `nx_extract exit code 131` and `.NET location: Not found`, even though `dotnet` is on `PATH`. The shader translator can fail the same way. - **Cause:** standalone .NET apphosts search runtime roots and registered installations, not `PATH` ([.NET runtime discovery](https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-environment-variables#dotnet_root-dotnet_rootx86-dotnet_root_x86-dotnet_root_x64)). Homebrew's formula installs the runtime under `libexec` and its `bin/dotnet` can be a wrapper that sets `DOTNET_ROOT` only for its own child ([Homebrew caveat](https://formulae.brew.sh/formula/dotnet)). Resolving that wrapper's symlink gives `bin`, not the runtime root. - **Fix:** `scripts/build_ipa_from_nsp.py` `configure_dotnet` derives the root from the `Microsoft.NETCore.App` entry in `dotnet --list-runtimes`, using the build's environment, before any extraction or survey. It exports `DOTNET_ROOT` both to that environment and to `os.environ`, for tools launched with an explicit environment and those that inherit it. Caller-supplied roots, including architecture-specific overrides, retain precedence. A failed or malformed probe leaves the environment unchanged rather than guessing the wrapper's directory. - **Found on:** [Balatro](../../dm/games/0100CD801CE5E000/README.md), first extraction with Homebrew .NET ([PR #284](https://github.com/seedds/nxnative/pull/284)). Applies to every game's build on that installation setup. ### The shader step stops with an empty `xlate_source` - **Symptom:** the extraction and captures succeed, then the shader step stops with `translation chain tools do not match the lock file: xlate_source: lock … / actual` and nothing after `actual`, which reads like a stale translator. - **Cause:** the translator (`dmsh_xlate_bin/dmsh_xlate`) is a universal framework-dependent apphost. When it finds no .NET 10 runtime for the slice it starts as, it exits 131 with an empty stdout. A Python under Rosetta (an Intel Homebrew's) starts it as x86_64, and it finds only the arm64 .NET; extraction is not affected, because `nx_extract` is built for the SDK's own architecture only. An Intel .NET on Apple silicon, or a `DOTNET_ROOT` that names one, fails the same way. `tools_now` (then in `dmsh_chain.py`) kept only the stdout of `dmsh_xlate --source-sha`, without its exit code or stderr, so the apphost's own message was lost. - **Fix:** the NSP script checks the environment before extracting anything: `check_native_python` stops under Rosetta (`sysctl.proc_translated`), and `check_prereqs` starts the translator once with `--source-sha` and stops with its first message line (`check_translator`). It also checks glslang and, for an iPhone build, the iOS SDK (`check_ios_sdk`, before the Metal check, whose message would blame the Metal toolchain). `own_compilers.py` no longer asks a Mac-only shader build (`DMSH_TARGETS=mac`) for the iOS Metal compiler. `tool_lock.py` records `--source-sha`'s failure instead of its empty stdout (`xlate_embedded`: `does not run (<first message line>)`), so `dmsh_build.py` run directly and `--update-lock` stop with the same message as the NSP script, without the lock and rebuild lines. The tool checks used to be in `dmsh_chain.py`, which is part of the chain version and of every shader's cache key, so that fix rebuilt every library; they now live in `tool_lock.py`, outside both, and a change to them rebuilds nothing. - **Found on:** NX2Apple builds ([LNPANDY/NX2Apple#31](https://github.com/LNPANDY/NX2Apple/issues/31), item 3; [PR #477](https://github.com/seedds/nxnative/pull/477), [PR #487](https://github.com/seedds/nxnative/pull/487)); reproduced with `arch -x86_64`. Applies to every game. ### An Intel Mac cannot load the bundled translator's dll - **Symptom:** on an Intel Mac, the NSP script stops at its translator check with `[RED] the shader translator does not run (Unhandled exception. System.IO.FileLoadException: Could not load file or assembly '…/dmsh_xlate.dll'. The assembly architecture is not compatible with the current process architecture.)`, although the .NET 10 runtime for x86_64 is installed. - **Cause:** the launcher `dmsh_xlate` is universal, but `dmsh_xlate.dll` beside it was arm64-only (`PE32+`, machine `0xAA64`). A .NET build for a runtime ID (`-r osx-arm64`) sets an executable's `PlatformTarget` to that architecture unless the project names one, and the repository had no single command for rebuilding the translator, so the bundled dll flipped between AnyCPU and arm64 from one rebuild to the next. The Ryujinx libraries stayed AnyCPU, because a library does not take its architecture from the runtime ID. - **Fix:** `dmsh_xlate.csproj` pins `<PlatformTarget>AnyCPU</PlatformTarget>`, so a build with any `-r`, or none, gives an AnyCPU dll (`PE32`, machine `0x14C`, IL only). `dm/tools/tests/test_xlate_anycpu.py` reads the PE/CLI headers of every dll in `dmsh_xlate_bin` and the launcher's Mach-O header, and fails when a dll is tied to an architecture or the launcher lacks x86_64 or arm64; it needs no .NET. The rebuilt translator writes the same SPIR-V and reflection files as the arm64-only one (9,556 shaders compared byte for byte). - **Found on:** an Intel MacBook Pro building any game ([GitHub issue #803](https://github.com/seedds/nxnative/issues/803), which bisected the flips). Applies to every game. ### The Vulkan shader step fails with `Argument list too long: 'xcrun'` - **Symptom:** `vk_shaders.py` stops with `OSError: [Errno 7] Argument list too long: 'xcrun'` after the Metal compiles, in the `metallib` link. - **Cause:** `metallib()` passed every `.air` file as an absolute path in one command. The NSP script runs it with `TMPDIR` set to the work folder, so each path holds that folder's name; a game with thousands of generated modules (Super Mario Galaxy: 3,083 captured shaders, more with the dual-source variants) under a long NSP folder name exceeds macOS's 1 MB `ARG_MAX`. - **Fix:** `metallib()` runs from the files' common folder with relative paths and an absolute output path, so the command length no longer depends on the work folder's name. - **Found on:** [Super Mario Galaxy](../../dm/games/010099C022B96000/README.md), with its dump in a folder named after both games of the cartridge. Applies to every Vulkan game's shader step. ### The split shader tables differ from a game profile's - **Symptom:** the shader split stops with `the split shader tables differ from game profile <id>'s`, on the game version the profile registers. - **Cause:** the check compares the split's data rows with the profile's byte for byte, so it stops for either of two reasons: other game files, or a change to `shader_split.py` that reads the same files differently. A profile's tables can only be split again where its game files are, so continuous integration cannot catch the second reason. - **Fix:** the error shows each differing table's row counts and its first row on each side. For a splitter change, check that the new rows are right, then copy the tables from `shader/pkg.partial/tables` in the build folder into the profile. A change to the splitter's output also needs the other profiles it reaches split again. - **Found on:** [Splatoon Raiders](../../dm/games/0400380020056000/README.md) ([issue #1263](https://github.com/seedds/nxnative/issues/1263)): [NVN2 task and mesh shaders](shaders.md#nvn2-task-and-mesh-shaders) moved one row from vertex to mesh. Donkey Kong Bananza, Pokemon Pokopia and Hyrule Warriors: Age of Imprisonment were checked and hold no such shader; Fire Emblem Fortune's Weave was not checked, as no dump was at hand. ## IPA archives ### AltServer cannot install a file of 2 GiB or more - **Symptom:** AltServer stops installing an IPA with "The file “b-<chain>.img” couldn’t be saved in the folder “msl”", although the RomFS stays out of it (`--romfs separate`). Other install tools install the same IPA. - **Cause:** in AltServer, not here. Its `writeFile` (`AltServer/Devices/ALTDeviceManager.mm`) reads a file whole and hands it to one `afc_file_write`, which libimobiledevice sends with one `send()` and returns its result as an `int`, so a file of 2^31 bytes or more fails, and AltServer reports the short write as a file it could not save. A shader library is one file, `msl/b-<chain>.img` ([Shader libraries take a file per shader variant](#shader-libraries-take-a-file-per-shader-variant)); a bundled RomFS image or a DLC image that large fails the same way. - **Fix:** a workaround, as the limit is the install tool's, and opt-in: `SHADER_FILES = True` or `--shader-files` in the NSP script (`DM_SHADER_FILES=1` for `build_ipa.sh`, `pack_ipa.py --shader-files`) packs the shader library as its separate files, `msl/b-<chain>/`, whatever its size. What the app carries of the image (after the build's exclusions) is written into a folder beside the IPA for the length of the pack, and the host reads a library folder before an image (`nvn_mtl.m`, `msl_data`). The game files still need `--romfs separate` when their image is that large. It is off by default: other install tools take large files, and one file installs faster than tens of thousands. The folder is written and deflated again at every pack, as its files are new each time. - **Found on:** [Tears of the Kingdom](../../dm/games/0100F2C0115B6000/README.md#found-on-this-game) 1.4.3, whose library is 2.10 GiB ([issue #1688](https://github.com/seedds/nxnative/issues/1688), [PR #1697](https://github.com/seedds/nxnative/pull/1697)). The cause is read from AltServer's and libimobiledevice's source; an install through AltServer with `--shader-files` has not been run. ### A streamed file changes size and the IPA fails its CRC check - **Symptom:** packing succeeds, but `unzip -t` or Python's `ZipFile.testzip()` reports a bad CRC for a large deflated file. - **Cause:** `pack_ipa.py` wrote the uncompressed size collected before reading. Its streaming check compared the output size only for stored files, so a compressible file that grew or shrank still published an archive with the old size. - **Fix:** count uncompressed bytes as they are read and compare them with the listed size before completing either streamed method. A mismatch stops the pack and removes the partial archive, preserving the previous IPA and its reuse index. Regression tests cover growth and shrinkage with both deflate and store. - **Found on:** synthetic IPA inputs while reviewing [PR #279](https://github.com/seedds/nxnative/pull/279); no game files required. ## RomFS images ### RomFS folder transfers fail or leave stale game files - **Symptom:** `devicectl` copying a large RomFS folder fails with POSIX error 24 (`EMFILE`) or spends most of its time on small files; deleting an old folder during iOS startup hits the launch watchdog. A push after changing extracted files can also keep sending old assets if an image is reused solely because it exists. - **Cause:** a folder transfer performs hundreds of thousands of file operations. The image cache must track its source tree, and magic plus total length cannot establish that an image's index is valid. Accepting that weak check before deleting the working folder loses the fallback when mounting later rejects the index. - **Fix:** `dm/tools/romfs_image.py` packs one `DMROMFS1` file: a 32-byte header, file bytes, then the index and name pool. Each directory has a sorted, contiguous run of child entries; the host binary-searches each path component and reads file ranges with `pread` through one descriptor per mount. Overlay folders keep precedence and directory listings merge both sources. The shared `dm/host/romfs_image.h` loader checks the full index before either mounting or iOS folder cleanup: length, entry and name bounds, file ranges, a rooted breadth-first directory tree, and sorted, unique child names. It supports an empty root too. Invalid images fall back to the folders on iOS; an explicit invalid `DM_ROMFS` image is refused with a `[RED]` line. The index check detects interrupted sequential transfers and structural damage, not same-length corruption inside file contents. The packer publishes by rename only after the source stays stable. Its reuse stamp tracks source names and file metadata including ctime, the packer and image identity; the push helper always checks it, and `--repack` forces a rebuild. Usage: [Quick reinstalls](../development.md#quick-reinstalls). - **Found on:** [Tears of the Kingdom](../../dm/games/0100F2C0115B6000/README.md#files-and-shaders), whose 304,000-file transfer and startup deletion failures are reported in [PR #369](https://github.com/seedds/nxnative/pull/369). [ANIMAL WELL](../../dm/games/010020D01AD24000/README.md#files) provides the empty-image boot case. Synthetic packer/host tests cover actual file reads, case aliases, overlay merging, corrupt indexes and changed sources; the macOS Foundation test executes iOS's locator to check that rejected images preserve its folder. ### A reboot packs a RomFS image again - **Symptom:** the first build after restarting the Mac spends minutes in `unpack` (`[1/5] unpack v131072: 5%, 1.2/~24.4 GB`) although the game, the packer and the image are as they were: 8 minutes for a 24.4 GB image from a NUP on a network share. - **Cause:** the stamp beside an image (`RomFS.img.inputs.json`) held `st_dev` in the image's identity, and macOS numbers a volume when it mounts it. Across one reboot the Data volume went from 16777234 to 16777232 while inode, size and both times stayed, so `reusable()` refused every game's image at once. - **Fix:** `file_identity` leaves `st_dev` out, as the IPA and app packers' stamps already did. Inode, size, modification and change times still catch an image replaced or written over. An old stamp still misses the fast check, so the [NUP packer verifies the existing image](#an-old-nup-cache-stamp-makes-the-build-unpack-again) before repacking. - **Found on:** Ocarina of Time (Switch 2, `0400486024DEC000`, packed from a NUP; PR #1299); the stamps of Star Fox and Mario Kart World show the same two numbers. `dm/tools/tests/test_romfs_image.py` remounts an image under another device number. ### An old NUP cache stamp makes the build unpack again - **Symptom:** the next build spends minutes unpacking a NUP after a packer update even though its complete `RomFS.img` is still present. - **Cause:** the stamp's digest holds the packers' source (`romfs_image.py` and `pack_nup.py`), and its identity has the fields `file_identity` gives; a change to either misses the fast check for every image, though neither says the image's bytes changed. - **Fix:** on a stamp miss, `pack_nup.verified` compares the existing image with what the packer would write now: its size, its header and index as `romfs_image.layout` makes them (the bytes `write` writes, so a change of format still packs again), and each file's bytes against the SHA-256 in the NUP's table of contents. Only the image is read, not the NUP's file data. If the image is the same before and after the check, its stamp is renewed and later builds take the fast check; otherwise it is packed again. The build's unpack line ends in `reused (image checked)`. - **Found on:** [Ocarina of Time](../../dm/games/0400486024DEC000/README.md) v131072 (PR #1630): the 26.25 GB image took 21.6 s to check on an Apple M1 and 0.1 s to reuse on the next build, against minutes to unpack. `dm/tools/tests/test_nsp_nup.py` (`ImageReuse`) covers an old stamp, a damaged file, a wrong index, a cut-off image, an image changed during the check and an older layout. ### A stale `DM_ROMFS` path crashes the game soon after `nnMain` - **Symptom:** a direct `dmhost` run logs `fs mount rom: → <path> (read-only)` for a path that does not exist, then crashes in the game's code a fraction of a second after `nnMain` (Echoes of Wisdom: `CRASH sig=11 addr=0x88` 0.3 s in). The host now stops at the mount instead with `FATAL the game files <path> (rom:) are missing: neither a folder nor a RomFS image`. - **Cause:** an old run command still named the extraction's `RomFS` folder after a rebuild replaced it with `RomFS.img` ([RomFS folder transfers](#romfs-folder-transfers-fail-or-leave-stale-game-files)). The read-only mount accepted any path, so every read failed and the game used the missing data. The iOS app and the Mac app launcher check their game files before they start the host; direct runs had no such check. - **Fix:** `mount_ex` (`dm/host/hle_fs.c`) stops on a read-only, non-save mount whose path is neither a folder nor a regular file, with the same `stat` that tells an image from a folder. Add-on content and system data check their paths before mounting, so only the rom mount (`DM_ROMFS`, or `$DM_ROOT/romfs` when it is unset) reaches it. `dm/host/tests/t_fs.c` checks a missing path and a fifo in a forked child. - **Found on:** [Echoes of Wisdom](../../dm/games/01008CF01BAAC000/README.md#found-on-this-game) 1.1.0 on the Mac ([issue #615](https://github.com/seedds/nxnative/issues/615), [PR #616](https://github.com/seedds/nxnative/pull/616)). ### Separate RomFS deployment loses its image or folder fallback - **Symptom:** a successful `--romfs separate --no-keep-extracted` build prints a push command pointing at deleted game files. On iOS, a complete image for another build can also prevent startup despite a valid `Documents/romfs` folder. - **Cause:** separate mode omitted the extraction-retention check, so normal post-build cleanup deleted the image needed for transfer. The startup identity rejection returned before the existing folder lookup. - **Fix:** reject separate mode without `--keep-extracted` before extraction or building. Reject a mismatching image while continuing to the validated folder fallback; move a folder aside for deletion only after accepting the image. Matching images and the default bundled route retain their behavior. The identity covers image size, header and index, not same-length file-content changes. - **Found on:** synthetic CLI and Foundation startup tests while integrating [PR #493](https://github.com/seedds/nxnative/pull/493), after [PR #478](https://github.com/seedds/nxnative/pull/478); no game or device is needed to reproduce either failure. ### A hand-copied RomFS.img is not found on iOS - **Symptom:** a `--romfs separate` app whose image was copied into its Documents folder by hand (Finder, the Files app, a LiveContainer guest) logs `boot: game files not found (Documents/romfs.img, Documents/romfs or <App>/romfs)`, and a build with `romfs.id` shows "There is no game file image on this device.", though the complete image is there. Renaming it to `romfs.img` makes the game start. - **Cause:** the NSP script writes the image as `extract/<version>/RomFS.img`, and `push_romfs.sh` copies it to `Documents/romfs.img`. A hand copy keeps the source's name. The Mac's file system ignores letter case, so the mismatch never shows there, but iOS's does not, and `main.m` looked up only the exact `romfs.img`. - **Fix:** `main.m` (`romfs_document_images`) lists Documents for `romfs.img` in any letter case, `romfs.img` itself first, and takes the first that is complete and passes the `romfs.id` check; a rejected or incomplete one falls through to the next spelling, then to the folders. The identity report, the backup exclusion and the log lines use the names found. Folder names (`Documents/romfs`) are still matched exactly. - **Found on:** South Park: The Fractured But Whole 1.05 (`01008F2005154000`, no profile here) on an iPhone 15 Pro Max, [issue #582](https://github.com/seedds/nxnative/issues/582). The Foundation startup test runs the locator on a case-sensitive APFS disk image. ### Building creates a full RomFS tree before packing its image - **Symptom:** reducing the iPhone transfer to one image still creates thousands of resource files on the Mac, and build scanners require the expanded folder. - **Cause:** folder extraction followed by a separate image pack duplicates the work; ordinary filesystem APIs cannot address files inside an image. - **Fix:** the NSP pipeline passes `--romfs-image` to `nx_extract`. LibHac layers the update over the base and checks its integrity while the writer streams file bytes into a temporary `DMROMFS1` image, appends the sorted directory index and publishes by rename. ExeFS remains expanded; RomFS and each DLC remain one physical file. `dm/tools/romfs.py` gives the scanners read-only virtual paths, bounded seeks and archive mappings (Python 3.12+), without changing the process's standard filesystem APIs. The IPA packer maps the physical image to `Payload/DmIOS.app/romfs.img`; iOS and the Mac launcher validate and mount it in place. Neither installation nor startup expands the resource files. Documents overrides and legacy folder inputs remain supported. A source image is uncompressed; the IPA can deflate it, so file-count reduction does not imply a smaller installed byte count. Full images for separate game versions cannot share unchanged individual files through APFS clones as legacy folder extraction did. - **Found on:** [ANIMAL WELL](../../dm/games/010020D01AD24000/README.md#files), whose direct image is an empty root (56 bytes); the local Baba Is You provisional build exercises 6,374 files in a 103,473,292-byte image, 131 shaders and an unsigned IPA containing only the single image entry. [Raging Loop](../../dm/games/010020400E1C2000/README.md#files-and-shaders) supplies Unity resources: all 1,473 source manifest entries match the image, and every registered shader remains in the scan. Synthetic integration tests call the actual LibHac writer, read unaligned archive ranges, and exercise IPA mapping, Xcode staging and bundled-image startup. ### A build folder holds hundreds of thousands of small files - **Symptom:** a game's `<id>_build` folder is slow to count, copy or delete: Metroid Dread's held 160,331 files in 48,782 folders, 84,541 of them in the compile cache. - **Cause:** the compile cache (`shader/cache/cas`) keeps a few files per shader, and Xcode's derived data (`ipa/build/ios/pkg/dd`) over a thousand. Both are needed to make the next build fast, and only while a build runs. The cache has since become [one file read in place](#the-compile-cache-was-unpacked-for-each-build), and the tar remains for the derived data. - **Fix:** `dm/tools/tree_tar.py` keeps each as one uncompressed tar beside where the folder was. `build_shaders` unpacks the cache only when `dmsh_build.py` will run, and packs it after pruning; the NSP script does the same for the derived data around `build_ipa.sh`. Keeping real folders during the build leaves `dmsh_build.py`, its chain version and Xcode unchanged. A build folder from before this converts by itself. Restored files keep their times, and Xcode's incremental build holds across the round trip (1 file compiled, against 262 for a full build). A build that stops leaves the folder, and a damaged tar costs only the cache it held. `regress.py`'s chain builds share the cache under the same lock. The cost is time on each build that runs the step, and a cache that no longer shares its libraries' blocks through APFS clones. The shader libraries (`b-<chain>`) stay folders: the host reads them one by one. - **Found on:** [ANIMAL WELL](../../dm/games/010020D01AD24000/README.md#found-on-this-game) (the round trip) and [Metroid Dread](../../dm/games/010093801237C000/README.md#found-on-this-game) (84,541 cache files: 51 s to pack, 22 s to restore on a T7). ### The compile cache was unpacked for each build - **Symptom:** a build that runs the shader step writes the compile cache's files back and removes them again, on Skyward Sword HD 24,131 files and 10,150 folders each time, Tears of the Kingdom over 900,000 entries. Removing old entries took up to half an hour on an external drive. - **Cause:** the cache was a folder of a few files per shader, read and written by path, so a tar could only hold it between builds. - **Fix:** `dm/tools/shader_cache.py` keeps it as one SQLite file, `cas.db`, whose entries go in and out by name, with nothing unpacked. Bytes are stored by their hash, since a library under its MSL's digest and its shader's digest was one APFS clone. `dmsh_build.py` and `dmsh_mtlcache.py` use it directly; a shader's entry is written in one transaction, so a stop leaves all of it or none. Several writers wait for each other: `dmsh_build.py`'s threads share a connection, and `regress.py`'s chain builds and the NSP script share the file. Pruning is a query that gives its pages back. A `cas/` or `cas.tar` from before this moves in by itself at the next shader step, without unpacking a tar: the Metal compile results come along. The per-shader entries do not, as the change of chain they came with means they cannot hit, and neither do the fetch variants, whose older entries hold their source's file name, so that a fresh compile's bytes differ (`regress.py` reports these as changed variants with no change of MSL). A damaged file costs the work it held and is made anew, and a full drive is one `[RED]` line. A revision from before this keeps its own folder, `regress/cas`. - **Measured** on Skyward Sword HD: a 281 MB `cas.tar` became a 226 MB `cas.db`, restoring all 2,470 shaders from the cache took 6 s (9 to 13 s with the tar), and the build folder's `shader/` holds 8 files. It also ran on an exFAT volume, with three processes writing at once. - **Found on:** [Skyward Sword HD](../../dm/games/01002DA013484000/README.md#found-on-this-game). ### Shader libraries take a file per shader variant - **Symptom:** a game's shader libraries are slow to copy or delete: Skyward Sword HD's `b-<chain>/` held 12,064 files for 2,470 shaders, a third of its build folder's files, and each `regress.py` chain build held another set. A tar would not do: the host reads these files at every run. - **Cause:** `dmsh_build.py` writes each shader's library, reflection, source and side libraries as separate files, and the host opened them by path. - **Fix:** `dm/tools/shader_lib.py` packs the folder into one file beside its name, `b-<chain>.img`, in the [RomFS image](#romfs-folder-transfers-fail-or-leave-stale-game-files)'s format, flat, each file's data on 16 bytes. Nothing unpacks it: the host (`dm/gfx/nvn/msl_image.h`) keeps one descriptor and maps only a shader's pages, and the tools read it through `romfs.py`. `dmsh_build.py` still writes a folder, so the chain version does not change. The NSP script packs at the end of the shader step, also on a reused step, so a folder from before this converts without a rebuild, and removes the old image only just before a build that replaces it. `regress.py` packs each chain build once it has read it; a revision from before this keeps a folder, and has a chain key of its own. The Mac app and the IPA carry the one file, without the Mac's sources, as `msl/b-<chain>.img`. An iPhone build neither compiles nor keeps the pulling vertex libraries when nothing in it reads them (`ios_wants_pulling`: no pipeline archive, the profile's `pso_mtl4` off, no `IOS_PULL=1`, no Xcode project; `DMSH_IOS_PULL=0` for `dmsh_build.py`), so `pack_ipa.py` reuses the image between packs. A build that wants them again compiles them, and `pack_ipa.py` refuses an image without them for an app that reads them. Each file comes from a folder beside the image first, so a partial `dmsh_build.py` run (given keys) goes over a packed library, and the next pack takes it in; a host running from a folder goes on when it is packed. Real files remain where a tool needs them: a compile's temporary files, and the libraries `metal-tt` reads, for the length of its run. `shader_lib.py expand` gives an old kept host its folder back. - **Measured** on Skyward Sword HD, on an external drive, warm cache: 10,017 files to one: copy 3.72 s to 0.18 s, delete 0.40 s to 0.02 s, 150 MB to 131 MB on disk. Reading all 2,470 shaders' reflection and libraries 100 ms to 28 ms. The map must be `MAP_SHARED`: on a file this size a private mapping costs about 80 µs, against 0.9 µs. A cold read has not been measured, as the cache cannot be emptied without an administrator. - **Found on:** [Skyward Sword HD](../../dm/games/01002DA013484000/README.md#found-on-this-game). ### A shader build leaves a file per shader output while it runs or after a stop - **Symptom:** a game's build folder holds tens of thousands of small files while its shaders build, and keeps them when the build stops: Octopath Traveler 0's `_build`, stopped in its compile, held 68,230 files (38,376 in `tmp/dmsh_*`, 29,757 in `shader/cache/b-<chain>/`), against 105 once a build finished. Every later build made another `tmp/dmsh_*` beside the old one. - **Cause:** `dmsh_build.py` wrote each shader's outputs into `b-<chain>/` and packed them into the image only at the end ([the entry above](#shader-libraries-take-a-file-per-shader-variant)); its cache check restored every cache hit into that folder before anything was translated; the translator wrote every missed shader's working files (`.bin`, `.spv`, `.res.json`, `.metal`) into one temporary folder before any was finished; and that folder went only when the build ended normally. - **Fix:** `dm/gfx/tools/dmsh_build.py`. A finished shader's files go into the compile cache (`cas.db`, under `stage/<output folder>/`) and leave the disk (`commit`); reads of another shader's outputs (a vertex shader's paired pixel MSL for `vs_pad`, the mesh step) take them from there (`output`). At the end the image is written from the staged outputs (`publish_library`, through `shader_lib.write`), with a partial build's image under it less the keys it built, and the folder and the staged entries go; when the image cannot be written, the staged outputs go back into the folder. Cache hits are restored when their shader is finished, not at the look. The shaders are translated and finished in batches of 4 × `J` (the next batch is translated while this one is finished), each in a folder of its own, and a shader's working files go as soon as it is finished. Each build's working folder holds its process ID; a later build removes a `dmsh_*` folder whose process is gone (one without the ID after a day). The NSP script's progress comes from a small file the build rewrites (`DMSH_PROGRESS`), no longer from counting its files. The outputs are the same: chain `cfcc3a82c78b` against the new one, Minecraft Dungeons II's 61,655 library files and Octopath Traveler 0's 41,880 are byte for byte the same (reflection without its `chain` field), apart from `_b_cas.txt`, which lists the new chain's cache digests. - **Measured** on an M4 Pro, every shader translated again (the chain changed), the Metal libraries from the cache: Octopath Traveler 0, 10,324 shaders, at most 405 files under `shader/` and `tmp/` (12 outside a build), 2:48 for the step (2:55 with batches of 4 × 128); Minecraft Dungeons II, 15,826 shaders, at most 3,279 with batches of 256 removed whole. Stopped with SIGKILL near the end of a compile: 40 files in `b-<chain>/` and 7 in `tmp/dmsh_*`; the next run cleared them and wrote the same library from the cache in 59 s. A partial build of 42 shaders for the Mac and the iPhone leaves one image each, also when run again over them. - **Found on:** [Minecraft Dungeons II](../../dm/games/0100A7C01B792000/README.md#found-on-this-game) and [Octopath Traveler 0](../../dm/games/01005270232F2000/README.md#found-on-this-game) builds, reported by the user. ### A work folder holds millions of small files - **Symptom:** a game's `<id>-work` folder is too slow to count: Tears of the Kingdom's held 8,893,028 files and 509 GB, and its count ran past ten minutes. - **Cause:** `regress.py` kept one whole shader build per chain version in `regress/chain` (45 there; Breath of the Wild held 57), and agents' investigations leave folders nothing reads again (`build` alone over 351,000 files). - **Fix:** `prune_chains` in `scripts/regress.py` removes chain builds after each chain step, beyond the 3 used last, any used within 24 hours and those of the 3 newest stored baselines; a check that needs one builds it again. `tree_tar.pack_quiet` packs each other top-level folder untouched for 14 days, with 100 files or more, into `<name>.tar` at the end of an NSP script or `regress.py` run. These tars keep extended attributes, so a RomFS folder's case-collision names survive, and `tar --xattrs -xf` restores them (plain `tar -xf` does not). A Python virtual environment stays a folder, as scripts run it by its path and nothing writes to it, which made it look untouched. It frees no space beyond the chain builds: a folder of APFS clones can take more as a tar, so each tar must fit with 20 GB to spare. A player's NSP folder has no work folder. - **Found on:** [Tears of the Kingdom](../../dm/games/0100F2C0115B6000/README.md#found-on-this-game). ### The zstd command given a path inside the image - **Symptom:** the NSP script stops at the shader split of a registered profile with `[RED] 13744 file(s) could not be read, their shaders are left out (first: AI/FileEntry/FileEntry.Product.100.byml.zs: CalledProcessError: Command ['zstd', '-d', '-c', '-q', '…/extract/v0/RomFS.img/AI/…'] returned non-zero exit status 1.)`. - **Cause:** `shader_split.py`'s `unzs` read a `.zs` file through `romfs.open`, which reads inside the image, but then handed its path to the `zstd` command, which can only open real files (`Not a directory`). - **Fix:** `unzs` feeds the command the bytes it read (`input=`). No other tool hands an external command a path inside the image. Test: `test_romfs_read.py` `test_zstd_file_inside_an_image`. - **Found on:** [Super Mario Bros. Wonder](../../dm/games/010015100B514000/README.md#shaders): every file of its RomFS is zstd-compressed, 13,744 left out in both versions; the first image-backed build after the folder extraction. ### Under `--work-dir` the programs of one game overwrote each other's RomFS - **Symptom:** an iPhone build of Super Mario 3D All-Stars with `--work-dir` (as NX2Apple runs the script) builds every program, then stops at the app build with `[RED] missing RomFS marker 'MovieData/BgMovieGalaxy.mp4' in <work dir>/extract/v393216/RomFS.img` and `FAIL: no RomFS at …`. - **Cause:** the first program's build packs each of the others from its own profile ([`oe::ExecuteProgram`](services.md#multi-program-applications-oeexecuteprogram)), each in its own build folder, `<NSP folder>/<id>_build`. `--work-dir` named one folder for all of them, so each program's `extract/<version>/RomFS.img` replaced the one before, and the package found Sunshine's image where the launcher's should be. Without `--work-dir` the folders were already apart. - **Fix:** `build_folder` in `scripts/build_ipa_from_nsp.py` gives a program packed with the first `<work dir>/programs/<id>_build` under `--work-dir`; the first program, a game of one program and `--program N` alone keep the folder as named. Test: `test_nsp_build_folder.py`. - **Found on:** [Super Mario 3D All-Stars](../../dm/games/010049900F546000/README.md#found-on-this-game), 1.1.4, built by NX2Apple 1.2.1 as one IPA. ## Add-on content ### DLC in Documents is ignored or removed DLC remains installed - **Symptom:** a game's DLC works only when packed inside its IPA, although its base RomFS can be deployed separately. After pushing a smaller DLC set, removed indices can remain reported as owned and mounted. - **Cause:** the iOS launcher originally checked only `<App>/aoc`. Merely adding `Documents/aoc` discovery is insufficient: `devicectl device copy to` merges directories by default, retaining removed indices, and skipping a build with no DLC leaves the entire previous set behind. The host counts and mounts the numeric entries under `DM_AOC_DIR`, so those leftovers remain active. - **Fix:** `aoc_locate` in `dm/ios/Sources/main.m` preserves an explicit `DM_AOC_DIR` from the launch environment or `env.txt`; otherwise it selects the first directory at `Documents/aoc`, then `<App>/aoc`, and logs `boot: DLC from <path>`. `Documents/aoc` is excluded from backups. Given `resources.env`, `scripts/push_romfs.sh` transfers the complete `DM_RESOURCE_AOC` directory with `--remove-existing-content true`, replacing old `<index>.img` files and legacy `<index>/` trees. An empty or omitted value sends an empty directory, clearing previous content and explicitly selecting no external DLC rather than falling back to bundled content. A nonempty path that is unavailable stops before any transfer. A direct RomFS-image push has no DLC metadata and leaves DLC unchanged. Transfers are not atomic: rerun a failed push before launching the game. Usage: [Quick reinstalls](../development.md#quick-reinstalls). - **Found on:** synthetic startup and transfer fixtures while reviewing [PR #486](https://github.com/seedds/nxnative/pull/486), part of [issue #388](https://github.com/seedds/nxnative/issues/388). The Foundation fixture runs the actual iOS locator; the push-helper fixture models devicectl's documented merge and replacement behavior, including smaller and empty DLC sets and transfer failure. Applies to every game; no device gameplay validation was performed. ## Launcher metadata ### A game packed from a NUP or a dump shows "Ver. ?" for a build its profile registers - **Symptom:** the game shows `?` where it shows its version (Super Mario Bros. Wonder's title screen: `Ver. ?`), and the build's lines name the version as `v327680 (?)`. - **Cause:** a package with no control data (a NUP update, a dump) packed from a derived profile, here because the folder had no base game for the shader tables. `derive_profile` took the display version of the build row it writes from the package's `control.nacp` alone and wrote `?` without one, and the host answers `nn::oe::GetDisplayVersion` from that row by `main`'s build ID (`hle_sdk16.c`). The profile it derives from registered that very build, with its display version. - **Fix:** `derive_profile` takes the package's control data first, then the display version the profile registers for this `main` build, and `?` only for a build no row names. - **Found on:** [Super Mario Bros. Wonder](../../dm/games/010015100B514000/README.md) (1.2.1, v327680, an update-only NUP; `main` ff773e90…, the registered build): `Ver. ?` before, `Ver. 1.2.1` after, one Mac boot each to the title screen. - **Check:** `dm/tools/tests/test_nsp_alias.py` `test_derived_profile_keeps_the_display_version_the_profile_registers_for_the_build`. ### No Game language row in an app packed from a NUP or a dump - **Symptom:** the settings of a Mac app or IPA packed from a NUP or a dump folder have no Game language row, while `DM_LANG` in the environment changes the language. Reported on [Donkey Kong Bananza](../../dm/games/040074A01BF12000/README.md) ([NX2Apple#619](https://github.com/LNPANDY/NX2Apple/issues/619), [#659](https://github.com/LNPANDY/NX2Apple/issues/659); the reporters' packed files were not available). - **Cause:** `settings.py` shows the row when the game's `control.nacp` lists more than one language, and two cases gave it none. A NUP update may carry no control data: the NSP script's language check fell back to the base game's file, but the Mac settings got only the update's path, and `build_ipa.sh`'s fallback (`$W/data/v0`) is a folder the NSP script does not make. And there may be no NACP at all: the Yoshi and the Mysterious Book and [Age of Imprisonment](../../dm/games/04003F40232DE000/README.md) NUPs carry none in any package, and an FTP dump such as Bananza's has no control data. - **Fix:** `languages_control` in `build_ipa_from_nsp.py` picks the file once (the packed version's, else the base game's), for the check, `mac_app` and `build_ipa.sh` (`CONTROL`). With no list, `settings.py` offers every language with a note, as `game_language` and `DM_LANG` already accept any. Age of Imprisonment shows a language it lacks as English (Russian, Portuguese) and has more text languages than its voice folders suggest (Dutch, Simplified Chinese), so a game's list cannot be read from its folders. - **Found on:** code reading and the NUP records of the two games above, with [PR #1182](https://github.com/seedds/nxnative/pull/1182); boots of [Age of Imprisonment](../../dm/games/04003F40232DE000/README.md) to its first settings screen. Not confirmed on Bananza by the reporters. ### A Mac front end's default settings change Metal execution paths - **Symptom:** starting a Mac app through a front end with unchanged settings enables the encode thread, and can disable Metal 4 pipelines and the pipeline archive; starting the same app from Finder keeps the host's defaults. - **Cause:** Mac `launcher.json` reused iOS defaults from `settings.tsv`. `rec_on` in `nvn_rec.h` defaults off on Mac and on on iOS; `pso4_on` in `nvn_mtl.m` defaults on on Mac and follows the profile on iOS. The exported defaults instead enabled the encode thread everywhere and followed the profile for Metal 4 everywhere. A front end applying those values changed the Mac host's behavior without a player override. - **Fix:** a setting's default may be `<ios>|<mac>`, selected before resolving a profile macro or build variable. Metal 4 uses `DM_GAME_PSO_MTL4|1`, the encode thread `1|0`. The generator validates the platform and the pair. Tests cover a profile with Metal 4 on and one with it off, and both platforms; the host and shader chain are unchanged. - **Found on:** synthetic metadata generation for [Super Mario Bros. Wonder](../../dm/games/010015100B514000/README.md) while reviewing [PR #352](https://github.com/seedds/nxnative/pull/352); checked against the native defaults, without a gameplay run. ### The Mac settings window stays on screen after Play - **Symptom:** in a Mac app, the settings window stays on screen after **Play**, not responding (a spinning cursor over it), until the game ends. - **Cause:** the window runs in the launcher's process (`dm/mac/Launcher.swift`). `finish` ordered the window out and stopped `NSApp.run`; then `launcher.c` started the host and blocked in `waitpid`, so the main thread never ran another run-loop pass. A window's order-out is committed only at the end of such a pass, and its default fade-out runs over several passes, so the window never left the screen. With the run loop kept running, the same order-out took effect at once. - **Fix:** `finish` sets the window's `animationBehavior` to `.none` before the order-out, and `dm_mac_launcher` calls `CATransaction.flush()` after `run()` returns, before it hands back to `launcher.c`. Neither alone was enough. Check: a copy of the window with a timed Play and a `CGWindowListCopyWindowInfo` probe while the thread then blocks; the window was still listed 2 and 4 s after Play before the fix, and gone 3 times in 3 after it. `test_launcher_storage.py` and `test_nsp_mac_app.py` pass. - **Found on:** the Mac app of [Breath of the Wild](../../dm/games/01007EF00011E000/README.md); applies to every Mac app. ## External drives ### A game folder on an exFAT drive reads one update as two, or fails to remove a folder - **Symptom:** with the NSP folder on an exFAT or FAT drive, the build leaves out an update whose name has no `[v…]` tag (the folder seems to hold two), or a stage stops with `FileNotFoundError` on a `._<name>` path inside `<id>_build/` and the next run fails on the half-deleted folder. - **Cause:** these file systems have no extended attributes, so macOS keeps a file's attributes (a download's quarantine flag) in an AppleDouble file `._<name>` beside it, and deletes it with the file. `scan_nsps` took `._Update [<title ID>800].nsp`, 4 KB of metadata, for an update: its name carries the title ID. Deleting `x` takes `._x` with it, so `shutil.rmtree`'s later unlink of the listed `._x` found it gone; before Python 3.13 that error ended the removal. - **Fix:** `scripts/build_ipa_from_nsp.py` `scan_nsps` takes a file only when its header is one nx_extract opens (`is_package`: `PFS0` at 0, or an XCI's `HEAD` at 0x100), and says which files it skips. The name tag alone no longer counts, so the same check also rejects any other tagged file that is not a package. Names that start with `._` are skipped without a line. The script's `rmtree` passes ENOENT from `shutil.rmtree` (the entry is gone, which is the goal) and raises every other error. `dm/tools/tests/test_nsp_external_drive.py` and `test_nsp_scan.py` cover both without game files. - **Found on:** the NX2Apple fork's users, who keep games on external drives ([PR #476](https://github.com/seedds/nxnative/pull/476), part of [issue #388](https://github.com/seedds/nxnative/issues/388)); reproduced on an exFAT disk image made with `hdiutil create -fs ExFAT`. Applies to every game. ### The c1 guard stops on an AppleDouble file after the shader build - **Symptom:** with the build folder on an exFAT or FAT drive, the shader step compiles every library and then stops in `c1_guard.py` with `UnicodeDecodeError: 'utf-8' codec can't decode byte …` (dmsh_build.py: `[RED] c1_guard.py failed`), reading a `._<key>_b.metal`. - **Cause:** the same AppleDouble files as above, beside the shader build's outputs. `c1_guard.py` took every name ending in `.metal` for a shader's MSL and read the binary `._` file as text. `cb_extent.py` (every `_b.json`) skipped it only because it ignores what it cannot parse, and `dmsh_mtlcache.py` (every `_b.metal`) took `._<key>_b` for a shader, found no readable reflection for it and so filled the Metal cache from none of that folder's libraries. - **Fix:** the three take only the names the shader build writes, `<16 hex key>_b….metal` / `.json` as `dmsh_build.py` (`OUT_KEY`) and `regress.py` already did: `c1_guard.py` `outputs` (which `cb_extent.py` uses too) and `dmsh_mtlcache.py` `OUTPUT`. None of them is part of the chain version. `dm/tools/tests/test_nsp_chain.py` runs both table tools on a folder with `._` files; the old `c1_guard.py` stops on it. On 13 Sentinels' 751 shaders both tables are unchanged. - **Found on:** a user's iOS shader build on an external drive ([PR #555](https://github.com/seedds/nxnative/pull/555)). Applies to every NVN game. ### The Mac app step stops on AppleDouble files in its inputs, its copies and its signature - **Symptom:** with the build folder or the app on an exFAT or FAT drive, the Mac app step stops in `pack_app.py`. The first form it takes is `no input file at <build>/shader/cache/<chain>/._<key>_b.metallib`, or `no rewriter fixups for <build>/<version>/._main.patched.bin` when the modules come first. When the inputs get past it, the copies stop with `chflags` or `Permission denied`, and the signing step stops with `codesign failed (1): code object is not signed at all` and `In subcomponent: …/Contents/MacOS/._nxnative`, which reads like a broken launcher build. - **Cause:** the same AppleDouble files as above. The module, shader, pipeline and font globs of `selected_inputs` took `._<name>` for a module or a library, and a `._x.patched.bin` has no `x.fixups.tsv` beside it. `shutil.copy2` of the host tried to set the copy's flags on a file system without extended attributes (`chflags: Operation not permitted`), and `cp -R` carried a folder's sidecars into the bundle, where `DMModuleFiles` listed them. `codesign` then read the `._nxnative` beside the launcher as a nested code object that is not signed and refused the whole bundle. Signing also sets an attribute on what it signs, so the sidecar comes back after a successful signature. - **Fix:** `dm/mac/pack_app.py` `detritus()` drops those names from `selected_inputs`, so neither the pack nor `DMModuleFiles` sees them. `copy()` passes `-X` where it copies and not where it clones (`cp` refuses the two together), and the host goes through `shutil.copy` instead of `copy2`, so a copy creates none of them. `strip_detritus()` runs before `codesign` reads the staged bundle, runs again after `xattr -cr` when `codesign` still refuses one, and runs once more after signing. On APFS nothing changes: no sidecar exists there, and the two passes are a stat walk. `dm/tools/tests/test_nsp_mac_app.py` puts a `._` file in every input folder and checks that it is not selected, does not reach the bundle, and is not in `DMModuleFiles`. - **Found on:** the NX2Apple fork's users, packing Mac apps on an external drive ([LNPANDY/NX2Apple#486](https://github.com/LNPANDY/NX2Apple/issues/486)); reproduced on an exFAT disk image made with `hdiutil create -fs ExFAT`, where the same inputs pack, sign and pass `codesign --verify --strict` with the change and stop three ways without it. Applies to every game's Mac app. ### A second Mac app build stops with `[Errno 45]` when it replaces the old app - **Symptom:** with the app on a Mac OS Extended, exFAT or FAT drive, the first build into a folder works and the next one that changes anything stops in `pack_app.py` `replace_app` with `OSError: [Errno 45] Operation not supported: '<app>'`. The old app stays in place. - **Cause:** `replace_app` exchanges the new and the old app with `renamex_np(RENAME_SWAP)`, which only APFS offers: the others return `ENOTSUP`. A first build has no old app to exchange, so it takes a plain rename. - **Fix:** on `ENOTSUP` or `EOPNOTSUPP`, `swap_by_moves` makes the state the swap leaves with three renames (the old app to `<app>.old`, the new one to `<app>`, the old one to `<app>.partial`), so the key write and its rollback do not change. A rename that fails undoes the ones before it. These volumes cannot make it atomic: a process killed between the first two renames leaves no app at `<app>`, with the old one at `<app>.old` until the next replacement clears it. APFS still swaps. Tests: `dm/tools/tests/test_nsp_mac_app.py` `test_volume_without_atomic_swap_replaces_by_moves`, `test_fallback_restores_the_old_app_when_a_move_fails` and `test_fallback_clears_an_old_app_a_killed_build_left_aside`. - **Found on:** a user's rebuild onto an external Mac OS Extended (Journaled) drive ([LNPANDY/NX2Apple#595](https://github.com/LNPANDY/NX2Apple/issues/595), [PR #1190](https://github.com/seedds/nxnative/pull/1190)); reproduced on HFS+ and exFAT disk images made with `hdiutil create`, where the `PackReuse` tests stop with the real errno without the change and reach the fallback with it. Applies to every game's Mac app. ### External shader directories contain AppleDouble metadata - **Symptom:** on an exFAT or FAT drive, shader splitting stops with `ValueError: not a version 2 shader code archive` or `[RED] /glslc_capture/._…: no GLSLC output`. - **Cause:** `shader_split.py` selected captured `.glslc` programs and decoded `.ushaderbytecode` archives by extension alone. macOS AppleDouble sidecars retain that extension, but their `._<name>` files contain metadata rather than shader bytes. - **Fix:** exclude `._` names when listing both external shader directories, including `DM_GLSLC_CAPTURE` and `DM_UE_SHADERS` overrides. Other dot-prefixed names remain valid inputs, and malformed real inputs still fail their normal validation. - **Found on:** an external-drive build reported in [issue #229's follow-up](https://github.com/seedds/nxnative/issues/229#issuecomment-6563450365). That commenter used a different, unnamed game; this fixes the sidecar failure, not the original Metroid Prime 4 port. ## Compressed inputs ### NSZ decompression rejects `--keys` - **Symptom:** `decompress NSZ` stops immediately with `nsz: error: unrecognized arguments: --keys …`. - **Cause:** `nsz` 4.6.1 has no `--keys` argument. It loads `keys.txt` beside `sys.argv[0]`, or falls back to `~/.switch/prod.keys`, while importing its modules, before parsing arguments ([4.6.1 key loader](https://github.com/nicoboss/nsz/blob/4.6.1/nsz/nut/Keys.py)). Version 5.0 supports `--keys` ([argument parser](https://github.com/nicoboss/nsz/blob/5.0.0/nsz/ParseArguments.py)); staging `keys.txt` also supports 4.6.1. - **Fix:** `scripts/build_ipa_from_nsp.py` `decompress_nsz` prefers the current Python's module: it links the file selected by `-k` as `keys.txt` in the conversion's temporary folder and runs `nsz -D` there, with `sys.argv[0]` set to `nsz` before import. Without the module, `nsz_command` probes `nsz` and `nsz-cli-macos-arm64` on the build's PATH and passes `--keys` to the first whose successful help lists that option. An unsupported or failed candidate does not hide the next one; when none works, the error lists the reasons and installation options. Executable, input and output paths are absolute because conversion changes directories. The temporary folder and link are removed on success or failure; the user's default keys are untouched. - **Found on:** [Super Mario Odyssey](../../dm/games/0100000000010000/README.md), converting its v0 NSZ. Applies to NSZ and XCZ inputs for every game. Synthetic tests during [PR #327](https://github.com/seedds/nxnative/pull/327) also reproduce an older console script hiding the release binary, a failed help probe containing `--keys`, and a relative PATH executable failing after the directory change; no game files are needed for those cases. ### ZBIC-compressed NSO segments - **Symptom:** the NSP script or any module tool stops on `main`: `compressed with zstd (NSO flag bit 7), which this loader does not decode yet`; with the decoder but without its package, `ZBIC (zstd) segments need the zbic package (python3 -m pip install zbic)`. - **Cause:** from system 22.0.0, an NSO may store its segments as **ZBIC**, NSO flag bit 7 for the whole module (the per-segment compression and check-hash bits keep their meaning). ZBIC is Nintendo's variant of zstd, not a container around standard frames: its frames start with the magic `ZBIC` and its FSE tables are written with binary interpolative coding, so neither `compression.zstd` nor the `zstd` command can read them. - **Fix:** `dm/tools/nso.py` decodes such segments with the [`zbic`](https://pypi.org/project/zbic/) package (Kinnay's, zstd with Atmosphère's ZBIC changes), imported only for a flag-bit-7 module; every segment is still held to the header's size and SHA-256. zstd has no fixed ratio bound, so a ZBIC segment is held only to its own size, where an LZ4 one is also held to 255 bytes per stored byte. Install it with the other tools (README, step 1). The NSP script checks for it before it extracts anything, as it does for its other modules, because it first reads a module at the rewrite, after the whole RomFS is unpacked. - **Check:** Well Dweller's `main` decodes to 85,370,056 bytes with segments `0x39680d0 / 0x9f7e74 / 0x9b1930`, every SHA-256 matching; `dm/tools/tests/test_nso.py` checks that a flag-bit-7 module whose segments are not ZBIC is an Error naming ZBIC. - **Found on:** [Well Dweller](../../dm/games/0100217023F6C000/README.md) (SDK 22.2.0). ## Versions with different modules ### A version without one of the profile's modules stops the host at boot - **Symptom:** `FATAL open <folder>/subsdk1.fixups.tsv` right after the host starts, or `build_ipa.sh` stops with `no rewritten module …/subsdk1`, for a version that has no such module. - **Cause:** a game's versions can differ in modules: Mario Kart World's base game has `main`, `subsdk0` (a Mono runtime) and `sdk`, its update 1.8.0 `main`, `subsdk0` (the movie player), `subsdk1` (a C library) and `sdk`. The host, the rewrite, the IPA and the shader split all took the one `modules` list of the profile. - **Fix:** `modules` lists all of them in load order, and each `build` row names its own (`game.py` `build_modules`; a row naming one `modules` lacks is an error). The host loads the modules of the row of `main`'s build, read from the rewriter's `main.fixups.tsv` (`main.c` `profile_modules`, from `DM_GAME_BUILD_MODULES`), and all of `modules` for a build with no row. `build_ipa.sh` takes the same list (`game.py modules <id> <folder>`). The NSP script rewrites, sizes `mem_used` from and splits the modules the packed version's ExeFS holds (`Game.modules_in`), and a derived profile's row names those (`derive_profile`). - **Found on:** [Mario Kart World](../../dm/games/0400C3F00006E000/README.md): the base game boots with its three modules from a profile that lists four. Applies to every game. `test_game_profile.py` checks the rows and the list. ## Package identification ### An update is taken for the version in its file name - **Symptom:** `[RED] leaving out update NSPs without a version tag ([v<N>]) in their name`, for every untagged update when the folder holds any other update (`….v720896.UPD.nsp`, [PR #393](https://github.com/seedds/nxnative/pull/393)). A single untagged update is packed as the newest version the profile registers, whatever it is. A wrongly tagged update is packed as the tagged version, and after the whole extraction the build IDs disagree with the profile's row and the build goes on from a derived profile, without that build's fixes. A base game served at a later version is packed as `v0`, which the profile may not register: Castlevania: Belmont's Curse's base is 1.1.0 (v65536) and its profile registers only `v65536`. - **Cause:** `scan_nsps` took each package's version from the `[v<N>]` tag in its file name and its title ID from its ticket or tag; it read a package's content metas only when it held several. A base game was always `v0`. Tags drift in real collections: a Castlevania base NSZ was tagged `[v0]`, while its content meta and `cnmt.xml` say v65536. - **Fix:** `scan_nsps` reads every package's content metas in one run, `nx_extract <prod.keys> --contents <package>...` (`package <index> <file name>`, then one `content <type> <title ID> <version>` line per content meta; only headers and the meta NCAs are read, about 0.05 s for a folder). Each package's title ID and version are its content metas', for the base game, every update and every DLC, so a file's name no longer matters to the build. `nsp_versions` names the base game by its own version (`v0` for most games; `v65536` for that Castlevania base, its profile's registered row). A `[v<N>]` tag that disagrees is said in one line and ignored. Keys that cannot open the meta NCAs stop the scan with `nx_extract`'s line ([Keys older than the game](#keys-older-than-the-game-stop-the-extraction-with-an-unhandled-missingkeyexception)). A package that cannot be read, or that holds no base game, update or DLC (a delta update), is skipped with a `[RED]` line, since it could not be extracted either. The ticket and tag readers (`title_of`, `nx_extract --title`, `is_merged`) and the untagged-update guess are gone. DLC extraction always names the DLC's title ID (`--aoc … <DLC title ID>`). - **Found on:** [Castlevania: Belmont's Curse](../../dm/games/0100BA4026E94000/README.md)'s base game ([issue #1629](https://github.com/seedds/nxnative/issues/1629)). Checked with the real tool on renamed links: Balatro with an untagged update (base v0, update v851968); Mario Kart 8 Deluxe with its update tagged `[v65536]` (taken as v1441792, said), an untagged DLC (v65536) and a truncated package (skipped: `ResultFsOutOfRange`); NieR:Automata's XCI; keys holding only `header_key` stop with `prod.keys has no key_area_key_application_0c`. Applies to every game. ### An NSP without a ticket or a title ID in its name stops with "no NSP with a readable title ID" - **Symptom:** `[RED] no NSP with a readable title ID in <folder>`, although the keys open the package (the launcher's key check passes). Renaming the file to `Game [0100XXXXXXXXXXXX].nsp` makes the build go on. - **Cause:** `title_of` took the title ID only from the ticket's file name (`<rights ID>.tik`) or from a `[title ID]` tag in the package's file name. Many NSPs carry no ticket (standard-crypto dumps, repacks), and a download's file name often has no tag. - **Fix:** with neither, `title_of` asked the package (`nx_extract <prod.keys> --title <package>`: the lowest title ID of its meta NCAs' headers). Since [issue #1629](https://github.com/seedds/nxnative/issues/1629) every package's title ID comes from its content metas, ticket or not, tag or not ([An update is taken for the version in its file name](#an-update-is-taken-for-the-version-in-its-file-name)); `title_of` and `--title` are gone. It works for NSP, NSZ (the meta NCAs are not compressed), XCI and XCZ. `nx_extract` is built into a temporary folder for it, since the build folder is named after the title ID. Key values are never printed. - **Found on:** NX2Apple, [LNPANDY/NX2Apple#177](https://github.com/LNPANDY/NX2Apple/issues/177) ([Tears of the Kingdom](../../dm/games/0100F2C0115B6000/README.md), whose base NSP has no ticket). Applies to every game. ### A game file whose extension is in capitals is not found - **Symptom:** `[RED] no NSP with a readable title ID in <folder>` for a folder holding `Game [0100XXXXXXXXXXXX].NSP`, on any Mac volume; renaming it to `.nsp` makes the build go on. A dump folder beside `Game.NSP` is packed instead of the package, and `Game.NUP` is not found. A `Game.NSZ` stops with `nsz made no Game.nsp`. - **Cause:** the script listed the folder with `Path.glob('*.nsp')`, which compares case-sensitively on every POSIX system, a case-insensitive Mac volume included. nsz has the same limit of its own: `PathTools.isCompressedGame` takes only a lowercase extension, and nsz skips anything else and exits 0 with no output. It resolves a symlink before it looks. - **Fix:** `files_with_suffix` lists a folder's files by their lowercased extension, for every scan of the inputs, and a package beside a compressed one is found in that listing. A compressed file in capitals goes to nsz as a hard link with a lowercase extension in its temporary folder, or as a copy where there can be none (another volume, exFAT), so the player's file stays as it is. - **Found on:** NX2Apple, [LNPANDY/NX2Apple#554](https://github.com/LNPANDY/NX2Apple/issues/554) ([Tears of the Kingdom](../../dm/games/0100F2C0115B6000/README.md)). Applies to every game. ### A merged NSP (base game, update and DLC in one file) packs as one title - **Symptom:** a game folder holding one merged package (`[Base + Update + DLC].nsp`, or a 1G+1U cartridge dump with no update NSP beside it) stops with `no base game NSP found`, or packs the base game alone (v0) without its update and DLC; when the base game's own build cannot be packed (Mario Kart 8 Deluxe 1.0.0's `main` is 32-bit), the build stops there. - **Cause:** `scan_nsps` gave each file one title ID: its first ticket's, else its name's tag. A merged package holds a ticket per title in any order, so it counted as the base game or as the update, never as both, and its DLC was never seen. `nx_extract` then read the update from a separate update NSP only. - **Fix:** a package is asked for its content metas (`nx_extract <prod.keys> --contents`; first only a package whose file table held more than one `*.cnmt.nca`, since [issue #1629](https://github.com/seedds/nxnative/issues/1629) every package: [An update is taken for the version in its file name](#an-update-is-taken-for-the-version-in-its-file-name)), and counts as each Application, Patch and AddOnContent it lists, with the versions from the metas. The one file is then given to `nx_extract` as both the base game and the update: `OpenNsp` keeps the NCAs that the Application content meta lists for the base and the Patch content meta's for the update, both of the application being packed (`--app`), and `--aoc <package> <out> <DLC title ID>` takes one DLC's data NCA by its title ID. The extraction estimate, the progress total and the NSZ conversion count the file once. A DLC NSP that bundles several DLC is read the same way. - **Found on:** Mario Kart 8 Deluxe (0100152000022000, no profile here) from one merged NSP, in NX2Apple ([PR #842](https://github.com/seedds/nxnative/pull/842), [LNPANDY/NX2Apple#376](https://github.com/LNPANDY/NX2Apple/issues/376)). Checked with Balatro's update NSP and base NSP merged into one PFS0: it scans as base v0 and update v851968, and `nx_extract` given it as base and update writes the same ExeFS, control data and RomFS manifest as the two separate NSPs. Applies to every game. ### A derived profile leaves out a module that moved to another slot - **Symptom:** a version a profile was not brought up on crashes at its first movie: `UNIMPL _ZN5movie9ExtractorC1ENS_13ContainerTypeE`, then `CRASH sig=11` ([issue #1211](https://github.com/seedds/nxnative/issues/1211)). The derived profile's `modules` has only `main sdk`. - **Cause:** `derive_profile` could only take modules out of the profile's list, but a module's slot belongs to a build, not to a game: Absolum's movie module is `subsdk2` in 1.1.2, where `subsdk0` is NVIDIA's Vulkan driver, and `subsdk0` in 1.2.1. - **Fix:** `derive_profile` adds what `profile_guess.main_modules` finds: a module the list leaves out that exports something `main` imports, which no listed module and no host hook supplies, and never a GPU driver. It shares this with `nro_modules` (`modules_supplying`). The added module goes in the version's `build` row too, since the host loads what the row names. Checked on 9 games' real modules with their registered lists: nothing is added (Super Mario Galaxy 2's GLSLC and `nn::pcm`, which the host supplies, stay out), and each of the 7 subsdks taken out of a list comes back. A registered build keeps its row: the rule runs only when a profile is derived. - **Found on:** [Absolum](../../dm/games/0100A83022A22000/README.md) 1.2.1, from the reporter's logs: with the module added by hand the game reached gameplay at 60 fps. No 1.2.1 files on this Mac: not run here. ### A derived profile loads only the modules its version ships - **Symptom:** the NSP script stops at its rewrite stage with `FileNotFoundError: … extract/v0/ExeFS/subsdk1` while packing a version the profile was not brought up on; the extraction has every other module. - **Cause:** `derive_profile` in `scripts/build_ipa_from_nsp.py` gives the derived profile this version's `build` row, made only from the modules its ExeFS holds, but copied `modules` unchanged. A version that ships fewer modules than the registered one (an older base game before an update added a module) then names a module the rewrite and the host try to load. - **Fix:** `derive_profile` keeps in `modules` only those the version's ExeFS holds, with a comment naming the ones left out; a version that ships them all keeps the line as it is, and `shader_only` derivations are unchanged. The host is built from the derived profile, so it loads the same list. - **Found on:** Mario Kart World 1.0.0 (`0400C3F00006E000`, a base-game dump: `main`, `subsdk0` and `sdk`), packed from a profile for 1.8.0, whose `subsdk1` is the C library module the update added ([PR #1081](https://github.com/seedds/nxnative/pull/1081)). With it the rewrite, the shader split (11,561 shaders) and the Mac host built, and the game drew its first course on the Mac. - **Check:** `test_nsp_incremental` (`test_a_module_the_version_does_not_ship_is_left_out`: the module left out with its comment and its build row, then kept once the version ships it). ## Keys ### Keys older than the game stop the extraction with an unhandled MissingKeyException - **Symptom:** `nx_extract` stops with exit code 134 and `Unhandled exception. LibHac.Common.Keys.MissingKeyException: Unable to decrypt NCA section.` with `Key Name: key_area_key_application_<NN>` (or `titlekek_<NN>`), after any NSZ conversion, the build ID lookup or the survey. An NSZ converted with such keys is written anyway: `nsz -D` warns `master_key_<NN> missing … This will lead to corrupted output.` and exits 0. - **Cause:** each NCA's header names the key generation its content needs; a prod.keys dumped from a console on older firmware than the game lacks that master key and every key derived from it. Nothing checked the keys against the packages before the first step that decrypts them, and LibHac's exception reached the log as a crash. - **Fix:** `nx_extract <prod.keys> --keys <package>...` opens every NCA of the packages (an NSZ's or XCZ's compressed ones by their header, which nsz keeps as it was) and applies LibHac's own test (`CanOpenSection`). On the newest key missing it prints `[RED] <file>: prod.keys has no <key> (master key <NN>, which came with firmware <version>) …` and exits with code 6; a rights-ID NCA with no common ticket is named the same way. The NSP script runs it once after scanning the folder (`check_keys` in `package_reader`), before `nsz -D`, the survey and the extraction, and stops with that line; any other failure of the check is left to those steps. Extraction, `--aoc` and `--contents` run the same check before they decrypt anything (the content metas first), so `nx_extract` alone also stops with exit code 6 instead of a crash. The firmware table runs to 21.0.0 (master key 0x14); a newer key is named as "a firmware after 21.0.0". The check reads only headers: the NSP script stops on Balatro's two NSPs in about 1.5 s. - **Found on:** NX Library ([Bium23/NxLibrary](https://github.com/Bium23/NxLibrary)), a Mac front end that checks the keys before starting a build, an idea from NX2Apple. Checked with Balatro's NSPs and an NSZ of its base game, with a copy of prod.keys without the keys of master key 0x10 and later (and one without 0x0a and later): every path names `key_area_key_application_10` and firmware 17.0.0, and the complete keys pass and extract as before. Applies to every game. ## Cartridge dumps ### A cartridge dump with its update on it extracts the update as the base game - **Symptom:** the extraction of a game whose base is an XCI that also carries its update (a "1G+1U" dump) stops with `[RED] nx_extract exit code -6`. `extract_v<version>.log` lists the update's build IDs even for the base game alone, then ends in an unhandled exception from the RomFS (`ResultFsNonRealDataVerificationFailed (2002-4604): Hash error!` from `RomFsFileSystem..ctor`, or `ArgumentException` / `InvalidOperationException` from the patch layers). - **Cause:** the XCI's secure partition holds the update's NCAs beside the base game's, and the update's Program NCA has the program's title ID too. `nx_extract` took the first Program NCA of the lowest title ID as the base, which is the update's when the dump lists it first. Its data section is a patch (BKTR) over the base RomFS, so it cannot be read alone or under another patch. The ExeFS still opens, with the update's build IDs, because an update's code section is complete. The Control NCA (display version, icon) was also whichever of the two came last. - **Fix:** `dm/tools/nx_extract/Program.cs` `OpenNsp` reads the base package's content metas (`*.cnmt` in its Meta NCAs) and keeps only the NCAs that its Application content meta lists, and prints how many it left out. When none of them is a Program NCA (NCA files renamed, so their names are not content IDs), it prints a `[RED]` line and reads every NCA as before. An NSP or a base-only XCI keeps every NCA but its meta NCA, so it extracts as before (Balatro's NSPs and Monster Hunter Rise's XCI give identical manifests). The update on the cartridge is packed too since [a merged package counts as each title it holds](#a-merged-nsp-base-game-update-and-dlc-in-one-file-packs-as-one-title). - **Found on:** [The Legend of Zelda: Tears of the Kingdom](../../dm/games/0100F2C0115B6000/README.md), a 1G+1U XCI (base v0 with update 1.4.3, whose 524 MB Program NCA is listed before the base's 17.1 GB one) beside the 1.4.3 update NSP ([PR #311](https://github.com/seedds/nxnative/pull/311)). Reproduced with a package of Balatro's update NSP followed by its base NSP: the base alone and the base with the update now give the same manifests and control data as the separate NSPs. Applies to any game's XCI that carries its update. ### A cartridge dump that holds two games packs only the first - **Symptom:** an XCI with two applications (Super Mario Galaxy + Super Mario Galaxy 2 on one cartridge dump) always packs the first game; `build_ipa_from_nsp.py` never reaches the second. - **Cause:** the scan took one title ID per file (the file name tags both games, and the first tag won), and `nx_extract` kept the NCAs of every Application content meta and took the Program NCA of the lowest title ID. Nothing could name the other application. - **Fix:** the XCI's content metas make it [count as each title it holds](#a-merged-nsp-base-game-update-and-dlc-in-one-file-packs-as-one-title), so the scan finds two games and stops with `more than one game in the folder`, naming both. `--title-id <title ID>` (`TITLE_ID` in Spyder) picks the game to pack from them; the script passes its title ID to every base extraction (`nx_extract … --app <title ID>`), and `OpenNsp` keeps only that title's Application and Patch content metas' NCAs (an `--app` the package has no content meta of stops with the ones it has). Each game gets its own `<title ID>_build/` folder and app. - **Found on:** [Super Mario Galaxy 2](../../dm/games/0100FD8022DAA000/README.md), on the same XCI as [Super Mario Galaxy](../../dm/games/010099C022B96000/README.md) ([PR #701](https://github.com/seedds/nxnative/pull/701), which packed both from the XCI with an earlier form of the option). This form is checked with a test XCI's file table and with Balatro's packages (`--app` of a title the package does not hold stops, naming the one it has); no two-game XCI is on hand. ## Damaged packages ### Extraction of a damaged NSP ends in "The LinkedList is empty" - **Symptom:** `nx_extract` stops with exit code 134 and `Unhandled exception. System.InvalidOperationException: The LinkedList is empty.` from `LibHac.Tools.FsSystem.CachedStorage.GetBlock`, with no `[RED] hash check failed` list, with or without `--allow-damaged`. - **Cause:** a damaged NSP with more than a few damaged files used up LibHac 0.19's hash cache. `CachedStorage.GetBlock` takes a block off its list before reading into it and does not put it back when the read fails, so the 4 blocks of a filesystem last for 4 failed reads and the next read that misses the cache throws. That exception is no hash error, so `nx_extract` did not catch it. A copy of SteamWorld Dig with 2 flipped bytes listed its 2 files; with 40 it crashed. - **Fix:** each hash error gets a new checked filesystem (`Failed` in `dm/tools/nx_extract/Program.cs`, on the image, folder and manifest paths), so a damaged NSP lists every damaged file and stops with exit code 3, or goes on with `--allow-damaged`. The same copy with 40 flipped bytes lists all 36 files. `test_every_damaged_file_is_listed_past_the_hash_cache` in `dm/tools/tests/test_romfs_read.py` puts a real RomFS behind a real `CachedStorage` on all three paths, with no NSP or keys. - **Found on:** [Breath of the Wild](../../dm/games/01007EF00011E000/README.md#found-on-this-game) ([issue #810](https://github.com/seedds/nxnative/issues/810), whose reporter diagnosed the lost cache blocks and carried a local patch that reopens the filesystem). Applies to any game. ### A damaged RomFS's own tables - **Symptom:** `[RED] the NSP is damaged: a filesystem's own tables fail the NCA's hash check`, exit code 3, right after the build IDs; before 2026-11-04, `[RED] nx_extract exit code -6` with an unhandled `ResultFsNonRealDataVerificationFailed (2002-4604)` under `RomfsHeader`. The ExeFS reads, and the build IDs and name are printed, since only the RomFS is hit. - **Cause:** the damage lies in what the RomFS opens from (its header, or the hash levels above it), so no file can be listed, and `--allow-damaged` has no file to write as read. Hashing each NCA tells which one it is: its file name is the first 16 bytes of its SHA-256. - **Fix:** `nx_extract` says so and exits with 3, as for damaged files. A good copy of the NSP is the only cure. A base game that is intact packs alone with `--package <its version>`, once it has a profile. - **Found on:** [Mario & Luigi: Brothership](../../dm/games/01006D0017F7A000/README.md#versions): update 1.0.1's Program NCA `074db5a4…` (its NSP is also a repack: it holds a `.cnmt.xml`). ### A crash from a damaged NSP's file looks like a host bug - **Symptom:** a build packed with `--allow-damaged` crashes or draws wrongly in game code, the same way on each run, at a place unrelated to the host's last work. The log gave no sign of damage; the build log's list (`logs/extract_<version>.log`) is the only record, and it is not part of a bug report's run log. - **Cause:** `--allow-damaged` writes files that failed the NCA's hash check as they were read, and nothing told the running host. A wrong byte in a compressed pack decodes to wrong data far from where the game then fails. A hash block also spans neighbouring small files, so a listed file's bytes may still be right. - **Fix:** the entry's last byte in a `DMROMFS1` index ([format](../../dm/tools/romfs_image.py)) is 1 on a file that failed the check (`img_ent.flags`, `romfs_image.h`; `nx_extract` sets it, and every reader refuses another value, or a flag on a directory). The mark stays with the image in an IPA, a Mac app or a hand-copied `romfs.img`. The host logs `[RED] fs RomFS image <path>: damaged files <N>: ...` at the mount and `[RED] fs rom:/<file> failed the NCA's hash check ...` at each damaged file's first open. A clean image is byte-identical to one from before this mark (checked on SteamWorld Dig), so its identity does not change. Not marked: an image extracted before this change, and folder extraction, which has no index (the NSP script writes images). Search a crash's log for `hash check` before looking for a cause in the host. - **Found on:** [Breath of the Wild](../../dm/games/01007EF00011E000/README.md#found-on-this-game) ([issue #810](https://github.com/seedds/nxnative/issues/810)). `dm/host/tests/t_fs.c` checks the count, the single report at first open and the refused flags; a boot of SteamWorld Dig on a deliberately damaged image logged the count and the files it opened. ### An update's offline HTML pages reported as a damaged NSP - **Symptom:** the NSP script stops at the offline HTML pages (`logs/extract_html_<version>.log`) with `[RED] the NSP is damaged: <n> files fail the NCA's hash check` and a list under `/html-document/`, although the program extracted cleanly and the NSZ tool verified every NCA of the update. - **Cause:** an update's HtmlDocument NCA can be a patch of the base game's. `nx_extract --html` guessed: it read the update's NCA alone first and over each base NCA second, keeping the first read that did not throw. A read that fails the hash check does not throw, though: `Copy` lists each file as damaged and goes on. So the read alone was kept, the patched one was never tried, and the files it listed stopped the build. - **Fix:** `nx_extract --html` (`dm/tools/nx_extract/Program.cs`) reads the NCA's header: a data section with a relocation table (`IsPatchSection`) is a patch, read over the base game's Manual NCAs, and any other is read alone, once, so damage in it is reported as damage. The header does not say which base NCA a patch is over (HtmlDocument and LegalInformation are both Manual), so `Try` reads over each in turn and undoes a read that adds files that fail the hash check (it deletes the folder and takes the entries off the list). A patch that reads over none is reported with `[RED] an HtmlDocument NCA of the update patches none of the base game's`, and the log names each patched read (`html<n>: a patch of the base game's NCA`). - **Found on:** [Super Smash Bros. Ultimate](../../dm/games/01006A800016E000/README.md#found-on-this-game) update 13.0.5 (`v2031616`) ([PR #1215](https://github.com/seedds/nxnative/pull/1215)): 1,641 of the NCA's files failed alone; read through the base, 2,798 files (122.2 MB) and none damaged, and the game packed and played a battle on the Mac at 60 fps. Balatro's update has a full Manual NCA: it is read alone, as before. `IsPatchSection` checked on Balatro's update: true on the Program NCA's RomFS, false on its ExeFS, its Manual NCA and every base section. ## Sparse base games ### The survey's build ID lookup stops on the base game alone with a hash error - **Symptom:** for a game with no profile, the NSP script stops at `build ID lookup · main of the base game` with `[RED] nx_extract exit code -6`; `<title ID>_survey/logs/exefs.log` ends in `ResultFsNonRealDataVerificationFailed (2002-4604): Hash error!` from `PartitionFileSystemMetaCore.QueryMetaDataSize` under `Nca.OpenFileSystem`, although every NCA in both NSPs matches its content ID (the SHA-256 that names it) and the base game with its update extracts cleanly. - **Cause:** the base game's Program NCA is sparse: its code and data sections carry a sparse layer and leave out data that the update supplies, so the base is smaller than its update (1.6 GB against 2.9 GB). Horizon mounts such an NCA only through its patch; read alone, the left-out ranges fail the hash check from the first block of the ExeFS. The lookup (`same_build`) tries main's build ID of the newest update first, then of the base game alone, and `run` stopped the build on the second attempt's failure. - **Fix:** `dm/tools/nx_extract/Program.cs` stops with exit code 4 and `[RED] the base game's program is sparse: it can be read only with its update NSP` when it is given a sparse base without an update (any enabled section of the Program NCA with a sparse layer). `scripts/build_ipa_from_nsp.py` `run` takes the exit codes a caller accepts (`ok`), and `same_build` accepts 4: the base game alone then has no main of its own and the lookup goes on. The base game with its update reads as before (LibHac layers the update over the sparse base; the whole 4.2 GB merged RomFS of the game below reads with every hash check on). - **Found on:** [Minecraft Dungeons II](../../dm/games/0100A7C01B792000/README.md), base v0 (SDK 22.2.0) with update v131072 (1.1.3.0). Applies to any game whose base Program NCA is sparse. ### The offline pages of a sparse base game stop the build as a damaged NSP - **Symptom:** the NSP script stops right after `extract v<version>` with `[RED] the NSP is damaged: a filesystem's own tables fail the NCA's hash check` and `nx_extract exit code 3` from `logs/extract_html_<version>.log`, although every NCA's SHA-256 matches its name. - **Cause:** the base game's HtmlDocument NCA can be sparse too (16 KB, beside the sparse Program NCA). `--html` reads the update's HtmlDocument NCAs, each over a base one when it is a patch, and then every base HtmlDocument NCA on its own; a sparse one read alone fails the hash check while its tables open. - **Fix:** `dm/tools/nx_extract/Program.cs` reads no sparse base HtmlDocument NCA on its own (`a sparse HtmlDocument NCA of the base game (… bytes): read only through the update's patch, not alone`), as Horizon mounts it only through its patch. - **Found on:** [Minecraft Dungeons II](../../dm/games/0100A7C01B792000/README.md), base v0 with update v131072: the update's patch (`accessible-urls`) and its full legal information NCA are read, the base's legal information NCA too.