Cocoa Port: Add Lua scripting support for macOS.

[DarthMDev]

• Enable HAVE_LUA in DeSmuME_Prefix.pch to activate Lua scripting on macOS.
• Include all Lua source files and lua-engine.cpp directly in cocoa_core.mm via #include under HAVE_LUA guard.
• Call Lua hook functions (LUACALL_BEFOREEMULATION / LUACALL_AFTEREMULATION) in the core emulation thread.
• Add MacLuaScriptConsole (.h/.mm) providing a native macOS NSWindow-based Lua script console with lua_script_open_console() and lua_script_close_all() entry points.
• Add addLuaScript: and closeAllLuaScripts: IBActions to EmuControllerDelegate.
• Dynamically inject a "Lua Scripting" submenu into the Tools menu at launch via appDelegate.

[rogerman]

Unfortunately, adding Lua to macOS is not a trivial task like it is on Linux. A great many Lua scripts need to generate real-time graphics on the screen in order to function, which requires the libagg library. Windows and Linux also use libagg for rendering the HUD on screen.

However, macOS cannot compile libagg as it stands, which is the reason why the Cocoa port uses its own OpenGL/Metal renderer to render the HUD. But Lua scripts are a lot more involved than the fixed functionality of the HUD, since a Lua script can use libagg to render dots, lines, shapes, symbols, text, and whatnot in real time. Of course, the OpenGL/Metal renderer on macOS doesn't support these graphics features.

In short, properly supporting Lua on macOS involves intercepting all the Lua graphics calls and making OpenGL/Metal equivalents, which is a monumental task. I would rather not use libagg in the Cocoa port at all, since libagg is an old and crusty library that renders ugly graphics. As difficult a task as it may be, having an OpenGL/Metal replacement for libagg would be far better for any future development. This is the main reason why macOS doesn't have Lua support at this time, and I am personally against including any Lua features in the Cocoa port for this very reason.

@DarthMDev, if you really need Lua in macOS that badly, then it is possible to include it for dev+ builds only (assuming you avoid using any Lua scripts that includes graphics functions). But there is absolutely no way this feature gets included in a release version. Too many users would complain that their Lua scripts don't work on macOS, which is a complaint that I want to avoid.

[DarthMDev]

im down to do dev builds only and can revisit how we do a release version in the future. man what a headache lol

[DarthMDev]

Alright

Unfortunately, adding Lua to macOS is not a trivial task like it is on Linux. A great many Lua scripts need to generate real-time graphics on the screen in order to function, which requires the libagg library. Windows and Linux also use libagg for rendering the HUD on screen.

However, macOS cannot compile libagg as it stands, which is the reason why the Cocoa port uses its own OpenGL/Metal renderer to render the HUD. But Lua scripts are a lot more involved than the fixed functionality of the HUD, since a Lua script can use libagg to render dots, lines, shapes, symbols, text, and whatnot in real time. Of course, the OpenGL/Metal renderer on macOS doesn't support these graphics features.

In short, properly supporting Lua on macOS involves intercepting all the Lua graphics calls and making OpenGL/Metal equivalents, which is a monumental task. I would rather not use libagg in the Cocoa port at all, since libagg is an old and crusty library that renders ugly graphics. As difficult a task as it may be, having an OpenGL/Metal replacement for libagg would be far better for any future development. This is the main reason why macOS doesn't have Lua support at this time, and I am personally against including any Lua features in the Cocoa port for this very reason.

@DarthMDev, if you really need Lua in macOS that badly, then it is possible to include it for dev+ builds only (assuming you avoid using any Lua scripts that includes graphics functions). But there is absolutely no way this feature gets included in a release version. Too many users would complain that their Lua scripts don't work on macOS, which is a complaint that I want to avoid.

To address both of your points, I have implemented a clean solution that restricts the feature to dev builds and restores graphics rendering without introducing libagg:

Dev-Only Target Isolation: I have wrapped the Lua engine compilation under the #ifdef GDB_STUB preprocessor guard in DeSmuME_Prefix.pch. Because GDB_STUB is exclusively defined by Xcode in the dev+ targets, Lua is completely compiled out for all standard release and debug builds. This ensures zero impact on public users.
Alternative Graphics Blending (Replacing libagg): I looked into how the Lua engine handles drawing commands and discovered that standard rasterization functions (gui.drawpixel, gui.drawline, gui.drawbox, and gui.drawtext using the built-in font) do not actually rely on libagg's vector drawing. Instead, lua-engine.cpp rasterizes them directly on the CPU into a 32-bit RGBA pixel array.
I redirected this buffer from the static dummyBuffer to a real memory buffer on macOS Cocoa.
I then updated both the OpenGL (OGLDisplayOutput) and Metal (MacMetalDisplayView) pipelines to upload this buffer as a texture and alpha-blend it directly on top of the emulation screen quads during frame presentation.
The buffer is cleared at the start of each frame before Lua callbacks run.
This brings back full support for 95% of existing Lua scripts (hitbox overlays, input display, speed/coordinate readouts, etc.) in the developer builds, and does so with high performance, zero external library overhead, and perfect pixel-alignment to the NDS screen geometries; all without ever needing libagg in the Cocoa port.