+ The backbone of the MUI Framework would be supported by the Ferricia Engine, with + Simple DirectMedia Layer (SDL) 3, OpenAL and OpenGL. Different parts play + different roles in MUI for the complete UI. Thus, the interoperation of + the Engine library and the Kotlin interface becomes essential that should be + efficient and effective. In this framework, the Graphical Model System (GMS) + would also be introduced, as the fundamental graphical system for the Engine's utilization. + This Framework is only applicable to client targets but not dedicated server targets. +
++ The GMS is responsible mostly for the GUI, by communicating with SDL and OpenGL. + The AUI communicates with OpenAL for audio processing and output. + If there is a Game Controller connected, the HUI may communicate with it via SDL. + All other KUI responsibilities are also by communicating with SDL, partially + handled by the GMS. +
++ The entire backend of the MUI Framework would be included in the Engine by interoperating + with the native libraries within it. Its responsibilities consist of window management, + rendering, audio processing, input management, user feedback management and + interactive simulations. Rendering would mostly be handled using OpenGL with only + a single canvas on only a single window; audio processing and simulations would mostly + be handled using OpenAL with sources transmitted via networking or resource management; + other elements would mostly be handled using other SDL subsystems. +
++ macOS enforces that all window-related tasks, including windowing, rendering and + input events, to be handled in the main thread. Therefore, the main thread would + do all windowing, rendering and primary raw event handling, with other tasks + delegated to other threads. This should ensure the application could work on every + target platform while only minimally allocating the main thread for all essential tasks. + Also, this does not include any audio-related task. +
++ Primary raw events include any raw pointer and key inputs, all window events + sent by SDL. Most events would then be sent to GMS Canvas for event handling and + processing in the GMS ticking thread. It is important to note that the objects and + states in the GMS must be carefully handled so that they could be rendered in + the main thread asynchronously safely. +
+
+ All the primary raw events are sent by or as a part of the SDL functionalities.
+ It is important to note that there exist several event types that would never be
+ used in the Engine.
+ Also, custom user events would not be handled in SDL, but mostly in the Kotlin interface,
+ so events like SDL_UserEvent and SDL_QuitEvent are not used.
+ Since touchscreen, camera and pen are not the target devices to support,
+ their corresponding events are also not used.
+
+ Since mobile platforms are not targetted, events primarily designed for mobile platforms + are also ignored. However, the mechanism to notify the user through an application warning + notification about the system or the heap is running low on memory is possible. + Still, it is expected that the application is always optimized at a certain degree that + it can still run and proceed properly without overly optimizing any important resources. + When the application is low on memory, the user is responsible to manage memory size + and reduce memory demands as much as possible depending on the customization configurations; + otherwise, there is nothing the application could do to free up memory. +
+
+ The application window could optionally reduce rendering frequency when the window
+ is not active. This includes the conditions when the window is hidden, minimized,
+ or occluded. The maximum number of frames per second (FPS) could be limited at 30 or
+ the half of the display refresh rate. At any time, the window cannot render properly
+ and should stop rendering when the window bounds are out of a single display area;
+ spanning the window to multiple displays is currently an unsupported behavior for
+ behavior correctness and validity.
+ This means functions such as SDL_GetDisplayForWindow are not supported.
+
+ The application does not rely on system theme but its own textures and themes, + so the event of system theme change is not used; the application might use system + locale settings to identify which language to use when the game data directory is + first initialized, but the value would not change afterward, so the event of + system locale change is also not used. +
++ Display events may be used only when the application is in fullscreen, but only + for the events related to the display the window is rendered on. + Otherwise, there is no need to take care of display information. + However, when the window is moved or resized, the display and window states must + be checked to see whether the window can still be rendered. + It is important to note that the controls of rendering are still handled in + the Kotlin interface, so the events must be forwarded to the Kotlin side. +
++ When the window gained focus, it is assumed that all input device states + are all "zero", meaning that all states are cleared or reset to origin. + However, this must not always be true, so when the events of resetting states + are fired but the states were already "zero", the states remain unchanged, + but with warnings (recovered situations) emitted. + It is important to note that the set of states in SDL is not used due to this reason. + Also, when the window loses focus, all the input device states in the application + should be reset to origin and never get updated until the window regains focus. + When the cursor is out of the canvas's boundaries, the cursor is not regarded as + focussing on the canvas, and the mouse events are not processed until the cursor + enters the canvas's region again; the cursor becomes non-existed in the application + and hidden in canvas when the cursor is not in the canvas's region. +
++ Event timestamps are ignored. + When they are polled from the SDL event queue, it is expected that they are sorted + in chronological order, so the timestamps should not be enough to affect + the synchronization states while they are pumped and polled per frame update. +
++ The modifier key states in SDL are not used generally. + For text typing, this should normally already be handled by the input editor, + which is processed in other events such as text editing or text input. + This includes key repeating, which is likely fired by the device or system. + The application itself internally holds a set of key states updated by + all the primary raw events, but not the keyboard states queried directly. + Also, the game is not playable with an external on-screen keyboard tool. +
+
+ Scancodes of the keys would always be used internally, but the keycodes
+ may be used to display the hotkey settings in the application.
+ Therefore, the keycodes in the events are ignored, but for displaying
+ the key to users, a different way would be done.
+ Since scancodes do not change on keyboard layouts, the users have to
+ identify what layouts their keyboards are.
+ This cannot simply be done by SDL functions as some keys may be undistinguishable
+ (for keycodes) or not mapped against layouts between different keyboards (for scancodes).
+ A custom-built layout key mappings could be done to correctly map the keys
+ by user's choice, which is by default QWERTY.
+ It is still a question whether SDL_GetKeyFromScancode could be used
+ safely across different platforms and various devices.
+
+ There would be two modes for mouse or cursor inputs. + By default, system cursor states are tracked and updated to cursor states, + which means the cursor motion states are updated each tick to correspond to + the cursor position retrieved from SDL. + When the game is in gameplay, the cursor mode turns to relative and wraps the cursor + in the window, most likely only at the center of the canvas. + In this mode, the mouse motion events from SDL are used instead of tracing + system cursor position. +
++ It is possible to select keyboard and mouse devices to use to exclusively only + accept the input events from just one device instead of all. + However, by default, all keyboard and mouse devices would be accepted to allow + further customization depending on this. + There is a possible concern about the application may become unusable when + the devices become unusable although other devices could still control the system, + so this feature might be reserved. +
+
+ Joystick and gamepad update completion events (SDL_EVENT_JOYSTICK_UPDATE_COMPLETE
+ and SDL_EVENT_GAMEPAD_UPDATE_COMPLETE) are generally not used,
+ since the status updates are done through polling all the events in the certain
+ frame.
+ If any issue has been encountered, such as unideal non-synchronization scenarios
+ or delayed events, this should be reconsidered.
+
+ The GMS would only be implemented in the Kotlin interface with abstractions. + Under GMS, all Graphical Model Objects (GMOs) serve as graphical wrappers of + the underlying objects they manage, such as data, states, even other GMOs. + The hierarchy of GMOs includes Screens, Menus and Components, + where one includes the next directly without skipping any stratum. + Rendering of them would be backed by the Engine's rendering system, + using the low level Kotlin bindings. + However, audio feedbacks triggered by GMOs cannot be managed by GMOs but + an audio interface manager out of this GMS. +
+
+ The ScreenManager manages Screens stored in a linked list, with one
+ on the top covers the others underneath.
+ The Screen on the top also blocks user interactions via Graphical User Interface (GUI).
+ In the rendering system, the RenderSystem renders the Screens managed
+ by the ScreenManager onto the Canvas backed by the Engine.
+
+ Rendering of content is not affected by system's display scale; the application itself + would include a GUI scale setting to scale the sizes of Menus and Components. + However, this only affects the dimensions but not all the margins. + In the implementation, margins would most likely use percentages or ratios. + Still, the details of implementation about GUI scaling are still subject to confirmation. +
++ It is complex to query the window safe area for rendering when it varies device by device. + Therefore, it should always assume that the safe area would generally be the canvas area + with paddings of 10 pixels, to make sure the rendering behavior is consistent enough + across platforms. + This only applied to important interactable or rendered content, general displayed content + like the background, miscellaneous margins, paddings and borders are not affected by this. +
++ Most graphical elements should be based on textures and the default shaders used with textures. + Only if certain geometric shapes are really required, like dynamic or variable lines and curves + used to show interactions and associate interactive elements on the GUI, the restriction + to use only textures could be bypassed. + However, it is still recommended to use customizable properties like colors via resources. + In this case, the default shaders which take textures would not be used, but the shaders + that take other attributes. +
++ The entire rendering framework is backed by OpenGL, so the details are mostly based on + the features and functionalities provided by OpenGL. + Basically, the framework aligns the rendering pipeline of OpenGL and general rendering + procedures, such as shaders and textures. +
++ On GUI, the camera is fixed, so the view is also fixed and constant. + Therefore, in shaders, the view matrix and transformation are omitted and disregarded. + The projection should always be orthogonal and thus being a uniform value, while + the model transformation and color filter would be done per instance as uniform values + evaluated each frame depending on the specifics, but not per vertex. +
++ There would be several sets of texture atlases implemented in the Engine. + Texture atlases should be used to enhance performance since there would mostly be + a large number of texture units used in the game; otherwise frequent texture bindings + which are likely expensive operations may occur. + In general, one might use + half pixel correction + to solve the issue, but the outcome may not be ideal or fit perfectly with + the rendering results due to the clipped outermost pixels in a tile as + a texture atlas element (later referred as "texel"). + Therefore, it is more recommended to have paddings or border pixels for each texel + in an atlas to keep the outermost pixels rendered well. + Even using "nearest neighbor" filters without paddings may not help solve the issue + completely because of potential numeric rounding errors. + All GUI textures would be managed with an atlas without any mipmapping, while all tile, + item and entity textures would be managed with different atlases with mipmapping. +
+
+ Different textures would have different constrains resulting different configurations.
+ Since GUI textures are not supposed to be scaled down, but scaled up depending on
+ the scaling setting with the base layout set based on the lowest reference scaling,
+ mipmapping is not suitable for this.
+ On the other hand, the gameplay region may be scaled up or down, so mipmapping may be
+ applied to enhance the performance of rendering diminished textures.
+ Regardless of whether the atlas would be mipmapped, paddings would be done by repeating
+ the outermost pixels to the desired extent for each texel.
+ However, when the atlas is mipmapped, the padding size would be the number of pixels by
+ two to the power of the maximum mipmap level with the atlas, so paddings still take effect
+ for each mipmap.
+ When the atlas reached the hardware-limited size (known by GL_MAX_TEXTURE_SIZE),
+ a new atlas for the same configurations should be used, but stored together in a managed set.
+ Also, manual mipmapping for each texel is required to avoid edge bleeding during mipmapping;
+ computations of texture atlases would be performed and designed with reference to
+
+ the rectangle packing algorithm by David Colson
+ or crates like rectangle-pack and crunch-rs; unless
+ the texels are explicitly constrained to only powers of two, this kind of
+ advanced algorithm would still be used.
+
+ In general, since the art style tends to be pixelated, mostly "nearest neighbor" + would be used to magnify the texels, but blending would be applied for diminishing to + simulate the similar colors. + At least, the mipmaps would be weighted using linear interpolation, but color filtering + could be either one of the available options through settings or launching options due + to rendering quality and computational complexity. + The maximum mipmapping level would be determined by the greatest power of two from + the common factors of all texel dimensions in the specific atlas, for which could be + further limited through settings or launching options. + Basically, there would still be the enforced constraint to the maximum value like four; + the dimensions of tile texels and that of entity texels must be a multiple of 8 and 2 + respectively. +
++ Besides, the application would first enter the early loading stage to render the progress + by initializing minimal components required to show the progress bar and the loading screen. + So, the logo used in the loading screen would be embedded in the application, and other + components would be started loading and tracked after the initialization of the screen. + The loading screen may also show during loading of updated resource pack configurations. + Moreover, a short animated splashscreen could show before the loading screen. +
+
+ Any GMO would internally hold an opaque handle managed by the Engine.
+ When a Screen is sent to the active stack, the handle in the Engine would be sent to
+ the internally held state, RenderingState, to be used during rendering.
+ The state internally holds an empty Mutex lock which controls which thread
+ to allow access to only one thread for thread-safety.
+ Ideally, only the GMS ticking thread and the main thread, which performs rendering,
+ would access the state either for modifications or reading for rendering.
+ The state impacts the thread for modifications only when the GMOs are within the state,
+ where the state would internally mark it during transitions.
+ This pipeline could greatly reduce JNI calls during rendering to improve performance
+ with only one call per loop as the general ideas.
+ Note that synchronization must be carefully examined for any data accessed in the Engine.
+
+ In general, rendering of any GMO should avoid reading framebuffer data drawn by + any previous GMO in the rendering loop. + This includes applying visual effects like color inverting or copying; + this is due to parallelization of shader computations in the render devices. + (See the related question.) + Various optimizations are also conducted, like reusing same shader programs as much + as possible when the previous draw call was also using the same program, eliminating + unnecessary OpenGL commands. +
++ As a general idea, the minimal support version is OpenGL 2.0 with some extensions required. + Although at the time this is written, most consumer devices in the last decade should + support OpenGL 3.0 and above, 2.0 is still targeted as the minimal version for most other + potential users. + However, when a newer version or usable extension is detected with the OpenGL context, + advanced features that may improve performance or even other effects might be available. +
++ According to Wikipedia, the GLSL version in OpenGL 2.0 is 1.10, so version 110 would be used + as the base GLSL version to use for all the shaders used in the Engine. + If a more advanced GLSL version is supported while advanced features are used, the target + GLSL version should be specified and the shaders are applied only when the target version + is matched for both GL and GLSL versions. + This is said, some shaders might be used or available only when they are available provided + by the OpenGL context. +
++ In terms of modularization, with the MUI structure, different components could be placed + and organized optimally. Besides, the components less related to UI-oriented functions + could be separated from the MUI modules. In Kotlin, those "modules" may be + Java packages instead. +
++ For example, interfaces of input devices could be put into a KUI module and inner details + of the devices could be put into a HID module under a possibly I/O module. + At the same time, more sophisticated UI components could be put into submodules of + their UI modules, like detailed rendering functions could be put into a graphics (GFX) + module under the GUI module, though they may be too deep into implementation details + to be inside the GUI module. +
+
From ca5f83fa1facd7981bbf5e0e6f1518d6946d3df6 Mon Sep 17 00:00:00 2001
From: Ben Forge <74168521+BenCheung0422@users.noreply.github.com>
Date: Wed, 25 Mar 2026 03:52:59 +0800
Subject: [PATCH 3/3] EFP 9: Rewriting & Rename: Orryx Impl of MUI Fw
Certain parts are still not being rewritten, yet while new contents have already been started drafting
---
efp/efp009/main.xml | 70 ++++++++++++++++++++++++++++++++++++---------
1 file changed, 57 insertions(+), 13 deletions(-)
diff --git a/efp/efp009/main.xml b/efp/efp009/main.xml
index 4b2002f..c2dc53c 100644
--- a/efp/efp009/main.xml
+++ b/efp/efp009/main.xml
@@ -1,7 +1,7 @@
- The GMS is responsible mostly for the GUI, by communicating with SDL and OpenGL. + The AGIM is responsible mostly for the GUI, by communicating with SDL and OpenGL. The AUI communicates with OpenAL for audio processing and output. If there is a Game Controller connected, the HUI may communicate with it via SDL. All other KUI responsibilities are also by communicating with SDL, partially - handled by the GMS. + handled by the AGIM System.
+ In order to communicate with UIDs in MUI with standardized protocols, there must be defined + sets of protocols for interactions and interoperability. This includes having standard protocols + to interact with input devices, for various generic types of input devices. +
++ Input devices include pointing devices, keying devices, gamepads, joysticks, etc. + Basic input control schemes for Interactions on Screens and Menus would be: Baseline, + keying devices only; and, Hybrid, combo of keying devices and pointing devices. + Extensive schemes may configure input devices even further, like using relative or absolute + pointing devices for different types of input controls. Gameplay input control schemes may be + based on the above configurations. In order to ensure certain usability and controllability + when primary input devices are not available, fallback schemes should be configured, or else + users may have to edit configurations outside the application. Also, application should not + assume that a keyboard must be available at any time in the system levels. +
++ Pointing devices are a class of input devices. They include mice, trackballs, touchpads, + touch screens, pointing sticks, pen tablets, etc. Standardized protocols to interact with + certain behavior patterns of input controls shall be defined. However, eligible pointing + equipment with at least two axes (type of axes depends on scheme) and two main buttons must be + available; without eligible equipment, this may be unavailable, though it may be a combination + of multiple devices. Also, for the main buttons, while two main buttons are necessary, + other buttons must be considered assistive or optional, or otherwise such controls may not + be accessible in common setups. +
+@@ -179,23 +209,23 @@
- The GMS would only be implemented in the Kotlin interface with abstractions. - Under GMS, all Graphical Model Objects (GMOs) serve as graphical wrappers of - the underlying objects they manage, such as data, states, even other GMOs. - The hierarchy of GMOs includes Screens, Menus and Components, - where one includes the next directly without skipping any stratum. + The AGIM would only be implemented in the Kryon layer with abstractions. + Under AGIM, all AGIM Objects (AGIMOs) serve as graphical wrappers of + the underlying objects they manage, such as data, states, even other AGIMOs. + The hierarchy of AGIMOs includes Screens, Menus and Components, + where one includes the next directly possibly without skipping any stratum. Rendering of them would be backed by the Engine's rendering system, using the low level Kotlin bindings. - However, audio feedbacks triggered by GMOs cannot be managed by GMOs but - an audio interface manager out of this GMS. + However, audio reactions triggered by AGIMOs cannot be managed by AGIMOs but + an audio interface manager in MUI out of AGIM.
The ScreenManager manages Screens stored in a linked list, with one
on the top covers the others underneath.
- The Screen on the top also blocks user interactions via Graphical User Interface (GUI).
+ The Screen on the top also blocks user interactions via GUI.
In the rendering system, the RenderSystem renders the Screens managed
by the ScreenManager onto the Canvas backed by the Engine.
+ Basically, an audio mixing public interface may be implemented for usages by other modules. + It should also provide various configurations in preferences for the exact controls over mixing, + like using software mixers or native sound cards directly, or even setting equalizers or compressors. + Modules may also add other mixers for other usages, or even using external software. +
++ Across the application, there shall be a top-level generic manager of audio. In the scope of gameplay, + there should be another subsystem of gameplay audio as another manager under the top-level one. +
+