Summercart, 64Drive, and Wii VC Multiworld Support

[mracsys]

Overview

ROM-side changes to implement midoshouse/ootr-multiworld#53

Companion Mido's House PR: midoshouse/ootr-multiworld#67
(Credit to Savage/NewLunarFire for laying the groundwork)

This replaces the previous Everdrive-specific implementation with UNFLoader's example generic USB library for serial communication with a multiworld client or autotracker. The UNFLoader protocol is extended to include OOTR-specific message types, more reliable timeout on read failures, some memory leak fixes for the Everdrive functions, and Wii VC support. Ideally, this protocol can be further extended for PC emulators with a system similar to the Wii to enable standardized auto-tracking on top of the multiworld functionality.

Protocol Changes

The high-level connection protocol is mostly unchanged, except that initiating a connection starts from the console sending a heartbeat packet instead of the client periodically sending cmdt text. However, the UNFLoader library introduces some overhead to the protocol over the wire that requires changes to the Mido's House client for support.

Part of the protocol changes is adding headers to every message containing the message type and length. The library handles segmenting large transfers transparently, as seen with the refactor for the save context transmission on file load/connecting in-game. Custom message types beyond the UNFLoader-provided types were added for:

• HANDSHAKE
• INGAME_STATE
• SAVE_FILENAME
• RESET
• SEND_ITEM
• ACK_MESSAGE
• DUNGEON_REWARDS
• PLAYER_NAMES
• READ_MEMORY
• WRITE_MEMORY
• UNRECOVERABLE
• ITEM_GIVEN
• PROG_ITEM_STATE

The READ_MEMORY/WRITE_MEMORY types are implemented in the rom but unused by Mido's House. These are intended for feature parity with emulators that have arbitrary read/write access to RAM. I haven't tested that code though.

Another small change to the item send function in the ROM adds a retry on failure. The write buffer was sized to account for a timeout while collecting all items in the child Market pot room (up to 48 items), which is more than enough to dump the full 8-item queue all at once. Actual behavior only sends one item per frame, so it's extremely unlikely for queueing to fail due to a full buffer over 8 consecutive frames (400ms).

Finally, the protocol now requires acknowledging each sent message in both directions. This is achieved with one of two message types with a fallback timeout of 7 seconds:

• ACK_MESSAGE
  • The other side successfully received and processed the current message in queue.
  • The ACK_MESSAGE receiver can move on to the next queued message.
• UNRECOVERABLE
  • The other side either had a problem receiving the message or had an error processing it.
  • The UNRECOVERABLE receiver then re-sends the currently queued message.
• Message receipt timeout (7 seconds)
  • Assume the connection is no longer valid and restart the handshake process.
  • The message queue is maintained through reconnect as handshake messages bypass it.

Previous behavior in the Everdrive plugin wrote directly to USB with no handling for errors.

ROM-side Process Flow

Flashcarts are detected on game boot in c_init of the rando payload. Further processing every frame is skipped if one isn't detected (i.e. running on a PC emulator).

Every frame, the read buffer is checked for new messages. Writes only happen if the read buffer is empty. This is required as flashcarts use the same region at the end of the loaded ROM as a buffer for both reads and writes. UNFLoader sets the lock registers before touching this area, so race conditions between the ROM and connected clients aren't possible. The write function would raise an error instead when it is unable to set the lock.

Reading messages

Received message size must be less than the 1024-byte buffer size, or the message is discarded. This may pose a problem for WRITE_MEMORY messages in the future, but there are no use cases at the moment needing a larger size.

Message types are distinguished via the message header instead of the first byte of the message body. Offsets in retained code from the previous Everdrive implementation are adjusted accordingly where needed.

Message handling is similar to the Everdrive plugin, with the INIT, HANDSHAKE, and MW protocol states. INIT and HANDSHAKE states are mostly unchanged. MW is extended to include the new message types described in the Protocol Changes section.

ACK_MESSAGE and UNRECOVERABLE message types impact write queue flow. The only difference is one advances the queue and the other does not. Both include handling to reset the connection if these message types are received when the ROM did not send anything, indicating some sort of de-sync occurred.

READ_MEMORY and WRITE_MEMORY messages are handled here, but these are untested as there is no direct use case currently. READ_MEMORY could potentially be used as a replacement for INGAME_STATE, but since the existing implementation works, it wasn't changed. The main reason to do this would be to only send the save context if the client cache is invalid as it freezes the ROM for a noticeable time. However, it should be assumed that the cache is invalid on every handshake, so in practice this isn't any different.

Writing messages

Sending from the ROM to the client is centralized in the per-frame handler. Messages are queued as needed from other ROM hooks. The only unqueued messages are:

• Heartbeats
• Handshake messages (excluding INGAME_STATE and SAVE_FILENAME messages)
• Message state messages (ACK_MESSAGE, UNRECOVERABLE, RESET)

After sending a queued message, the queue is locked until the message is acknowledged by the client.

There are currently three sources of queued messages, plus the unqueued heartbeats and other types.

Player finds an item

move_outgoing_queue() in get_items.c will advance its own queue as it adds items to the flashcart queue. If the flashcart queue is full, the item queue does not advance, effectively supporting up to 72 pending items (8 in the item queue, 64 16-byte messages in the 1024-byte message queue) depending on flashcart queue contents. Queue size was chosen to provide a reasonable chance of recovery in the child Market pot room, which currently has the highest item density that could be collected in the 7-second timeout (48 items -> minimum 768 byte queue, rounded up to 1024 for other messages).

Player looks at dungeon reward info

Viewing dungeon info in the pause menu will add a 19-byte message to the queue containing dungeon reward info. Format is unchanged from the Everdrive implementation.

Connection reset requires sending the player name and save context

The player name and save context messages are included in the queue instead of as a part of the handshake process to take advantage of the queue confirmation safeguards. If packets are lost in transmission, the client crashed, or the client otherwise never received the messages, the system will retry until success.

If for some reason the connection reset while these message types are in the queue, they are not re-queued. Since both message types can change between queueing and final transmission, the message body is not included in the queued message, only a pointer to the referenced data.

Heartbeats

Heartbeats have two variants: HANDSHAKE and HEARTBEAT message types. HEARTBEAT packets are only sent if the ROM is currently connected to the client. This division allows for the client to cleanly detect game resets if it receives a HANDSHAKE message.

Heartbeats are sent directly over USB, bypassing the queue, via usb_sendheartbeat() and usb_sendhandshake().

Message state messages

These messages are sent immediately to the client when their corresponding condition is met, bypassing the queue.

• ACK_MESSAGE
  • Sent at the end of processing a read message via usb_sendreadsuccess()
• UNRECOVERABLE
  • Sent if a message fails processing via usb_sendreadfailure(). Current use cases are for USB failures, incorrect message size (i.e. lost packets), null item IDs, and RAM bounds check failure for raw access messages. This only triggers message retransmission, not a full connection reset. If the client message was not lost or mangled, this could trigger infinite re-sends, so it is important that the client message builder be free of bugs.
• RESET
  • Sent if an unexpected message is received via usb_sendreset(), triggering a connection handshake reset.

UNFLoader Port

Some new symbols to required libultra functions already in the OOT ROM were added to ootSymbols.ld, and remaining required functions were sourced from the libultra decomp at https://github.com/ModernN64SDKArchives/libultra_modern.

UNFLoader's library supports Everdrives (both X7 and V3 variants - any official release with a USB port), Summercart64, and 64Drive. A custom module for Wii VC communication is integrated with the flashcart code in the ROM.

UNFLoader also has a debug library for use with gdb that could be useful for extending rando's debug mode in the future. I didn't include it in order to keep the PR scope focused.

Memory Usage

The buffers used for the different communication stages take up around 3KiB of RAM in addition to the payload. I believe we are doing OK on RAM outside of some of RealRob's work like enemizer, but something to consider. The UNFLoader buffers are sized to maximum DMA transfer size supported by the N64 (512 bytes, plus 32 bytes metadata for the Wii, one buffer for read, another for write). The 1KiB write queue buffer is sized for maximum credible item collection in a ~10 second window (7 second timeout plus some time to reconnect or the user to notice PC-side). The secondary read buffer is sized to mirror the write buffer at 1KiB. It only needs to hold one message at a time though, so it could be reduced back to 16 bytes like before. Main reason to increase it would be for WRITE_MEMORY messages with large payloads.

Wii VC Changes

Patches to the Wii VC emulator allow users to connect to a computer over two USB serial adapters connected via null modem adapter, as shown below courtesy of Vidya James in the OOTR Discord.

This is implemented for both supported WAD variants (US and Japanese). The devkitPro Wii homebrew toolchain is required to build the binary (--compile-wii option to the normal /ASM/build.py pipeline). gzinject handles patching the WAD with the compiled binary. Any changes to hook RAM locations are handled by the build script modifying the .gzi scripts. As a side note, the large binary payload for the crash handler and VC crash fix could potentially be converted to readable C or at least assembly using the same pipeline in the future.

gzinject binaries for Windows are updated as the versions currently bundled with OOTR do not support the --dol-inject and --dol-loading options required for patching to work. macOS and Linux binaries already support them.

Core logic

The VC patch works using the same principle as homeboy, creating a virtual storage device (type SerialVirtualDevice) that the ROM can interface with to tell the emulator what to do with incoming/outgoing data.

VC serial communications function only to pass through messages to/from the ROM.

The serial adapter is checked once per emulated frame at the end of frame processing, and the ROM triggers reads/writes to N64 RAM separately via the storage device registers mid-frame. Total impact to the WAD is a ~38KiB payload, plus an additional 16KiB reserved from the MEM1 heap for an internal heap.

Receiving messages from the client is handled independently of the ROM. Pointers to received data is saved in dedicated queues, up to 16 at a time. Individual message size is limited to 8KiB by the read buffer, while the queue is only limited by the heap size (somewhere below 16KiB). This is significantly larger than the ROM buffers, allowing for growth in the future without touching the Wii toolchain or challenging VC RAM usage.

ROM interface

The virtual storage device is accessible via reads/writes to the SerialVirtualDevice struct at address 0xA8060000. VC maps addresses to different hardware devices as follows:

Device	Start	End
CPU	0xA0000000	0xFFFFFFFF
RAM	0xA0000000 0xA3F00000 0xA4700000	0xA3EFFFFF 0xA3FFFFFF 0xA47FFFFF
RSP	0xA4000000	0xA40FFFFF
RDP	0xA4100000 0xA4200000	0xA41FFFFF 0xA42FFFFF
MI	0xA4300000	0xA43FFFFF
VI	0xA4400000	0xA44FFFFF
AI	0xA4500000	0xA45FFFFF
PI	0xA4600000	0xA46FFFFF
SI	0xA4800000	0xA48FFFFF
RDB	0xA4900000	0xA490FFFF
DISK	0xA5000000 0x06000000	0x05FFFFFF 0x06FFFFFF
PAK	0xA8000000	0xA801FFFF
SRAM	0xA8000000	0xA8007FFF
FLASH	0xA8000000	0xA801FFFF
Homeboy*	0xA8050000	0xA8057FFF
Serial*	0xA8060000	0xA8067FFF
ROM	0xB0000000 0xBFF00000	0xBFBFFFFF 0xBFF0FFFF
PIF	0xBFC00000	0xBFC007FF

* Homeboy takes the area between 0xA8050000 and 0xA8057FFF. In case rando ever includes homeboy, the serial device uses the range 0xA8060000 to 0xA8067FFF.

The device initializes with a key field in the struct set to a specific value. The ROM checks this value on boot to determine if it is running on VC, similar to flashcart detection. ROM-side serial communication is disabled if the key is not valid. Note that VC communication with the client does not depend on this check.

Struct values do not have to be static when accessed by the ROM. Storage devices have handler functions defined in VC for every read/write access, separated by size into 8/16/32/64 bit functions. The ROM can control VC serial access through these handlers. Reading the u32 receive_header will give the ROM the message header of the current message in the Wii's queue for the ROM-side usb_poll() function to return. Writing to the struct triggers the following actions:

• transmit_addr: Translate from N64 RAM to Wii RAM address
• transmit_header: Send the bytes at transmit_addr over serial to the client directly, using the header size field for byte length to send.
• receive_addr: Translate from N64 RAM to Wii RAM address
• status: When device->receiving (part of the bitpacked status field) is non-zero, transfer bytes from the Wii read buffer to the ROM read buffer for the current message. This lets the ROM indicate it is ready for its read buffer to be overwritten.

Other get/put functions for the virtual device exist to pass through values only.

Testing

I've tested on both Everdrive variants and a Summercart with the modified Mido's House client. Savage/NewLunarFire also tested an early version of this on a Summercart. There are two quirks I've noticed:

• The Everdrive menu causes client read commands to stall, but they resume successfully on ROM boot.
• Summercarts send a bunch of text over USB when powering on that should be ignored by clients for initial connections from console cold boot as well as console resets. Everdrives are silent unless prompted. My MH changes handle this.

Neither affects ROM functionality. Gameplay is stable on all three flashcarts.

Wii testing has been done by myself as well as shirosoluna, dotzo, Cybrou, and Jaybone. The ROM and client are stable as of the most recent update with fixed message padding, no VC crashes or frozen games.

I do not have a 64drive, do not know anyone with one, and it appears they are no longer manufactured. Existing UNFLoader support for it is included but untested.

[mracsys]

Testing on the latest client release has been stable. The changes here and for Mido's House are ready for review/merge.

[fenhl]

A couple minor things, and I'm not qualified to review the Wii VC part, so that's something that will have to be reviewed by someone else before it goes on main, but I'm okay with merging it to my branch without review. Thank you again for all your work to make this happen!

I'm going to coordinate actually merging this with releasing a new MH MW version that supports this, which you said on Discord will require some changes to midoshouse/ootr-multiworld#61.