Custom shaders

[eliotbo]

RFC: add custom shaders to iced

[ids1024]

Passing the shader as a string in the primitive will not only require an allocation every time Iced generates a new frame with new primitives, but also require recompiling the shader, right? Probably should use something reference counted that either contains the compiled shader or allows it to be cached by the backend, like image::Handle.

[eliotbo]

Passing a handle to the shader code is the right way to go, but I'm not sure how to implement it. I'm wondering if simply changing the type from String to &str could work, although I am sure we would run into countless lifetime issues.

For reference in the the case of wgpu::quad, the pipeline inserts the shader like so:

        let shader =
            device.create_shader_module(wgpu::ShaderModuleDescriptor {
                label: Some("iced_wgpu::quad::shader"),
                source: wgpu::ShaderSource::Wgsl(std::borrow::Cow::Borrowed(
                    include_str!("shader/quad.wgsl"),
                )),
            });

while the pipeline for custom_shader_quad does this:

        let shader =
            device.create_shader_module(wgpu::ShaderModuleDescriptor {
                label: Some("iced_wgpu::custom shader quad::shader"),
                source: wgpu::ShaderSource::Wgsl(std::borrow::Cow::Borrowed(
                    shader,
                )),
            });

where the shader argument is a &str. Are the shaders not compiled at every frame in both cases?

edit: If they are both compiled at every frame, then we found a problem with wgpu::quad.

[ids1024]

That create_shader_module() call in quad.rs is in Pipeline::new, not Pipeline::draw. So it's not called every draw and only when the backend is created.

[eliotbo]

I've had a look at image::Handle and I think I've come up with a solution to both the memory allocation issue and the shader compilation problem. Here it goes.

There is a new shader.rs module in iced_native with a Handle struct:

pub struct Handle {
    /// A unique identifier for the shader.
    pub id: u64,
    /// The path to the shader code.
    pub path: PathBuf,
}

Instead of passing the shader as a string in the primitive, we pass the absolute path to the shader file. Internally, a iced_native::shader::Handle is created with a hashed id based on said absolute path.

In the wgpu::custom_shader_quad Pipeline, there are two new fields:

pub struct Pipeline {
    ...
    pipeline: wgpu::RenderPipeline,
    shader_modules_cache: HashMap<u64, wgpu::ShaderModule>,
}

where shader_modules_cache is used to cache the compiled ShaderModules. For each ShaderModule, the key to the HashMap is the corresponding shader::Handle id.

In the draw(..) call, the shader_modules_cache and the Pipeline are updated only if the received Handle id is brand new:

            let shader_handle: Handle = instances[i].shader_handle.clone();

            let has_new_shader_module =
                self.insert_shader_module(device, &shader_handle);

            if has_new_shader_module {
                let shader_module = self
                    .shader_modules_cache
                    .get(&shader_handle.id)
                    .unwrap()
                    .clone();

                let new_pipeline = self.make_pipeline(&device, shader_module);

                self.pipeline = new_pipeline;

[ids1024]

With image handles, you can load an image either from a path or for memory. It may make sense to do the same.

[eliotbo]

I've changed the native::shader module:

pub struct Handle {
    /// A unique identifier for the shader.
    pub id: u64,
    /// Either the path to the shader code or a reference to the shader code in memory.
    pub shader_content: ShaderContent,
}

pub enum ShaderContent {
    /// Shader in a file
    Path(PathBuf),
    /// Shader in memory
    Memory(&'static str),
}

[ids1024]

I feel the more flexible solution may ultimately be necessary. It's pretty standard for UI toolkits (and browsers) to offer a way to use custom OpenGL code in a UI element.

Perhaps if this provided an iced::widget::wgsl_shaded_quad widget, that could offer a simple API for this use case whether that wraps a CustomShaderQuad primitive or provides the necessary boilerplate over a more flexible solution. Though if it makes sense to offer both as primitives that could work too.

[eliotbo]

I've added a native::widget::wgsl_shader_quads, with the following fields

pub struct WgslShaderQuad<Message> {
    on_press: Option<Message>,
    on_hover_entering: Option<Message>,
    on_hover_leaving: Option<Message>,
    width: f32,
    height: f32,
    padding: Padding,
    time: Duration,
    handle: shader::Handle,
}

where time is sent to the GPU for animating the shader. The draw(..) method for the new widget calls renderer.make_custom_shader_quad(..).

There's also a new example called shader_widget, which is now significantly shorter than the previous example since there's no need to define a custom widget anymore.

The new widget does not add more flexibility in its current state. In fact, it takes flexibility away since the WgslShaderQuad widget is now fixed. So I'd like to discuss the specific changes that would be required to attain the level of flexibility that is expected here. For example, these are potential functionalities:

Give user control over

1. uniform variables
2. attributes
3. instancing
4. textures
5. compute shaders

[hecrj]

We are focusing on #23, which I believe is a superset of this approach. Feel free to chime in there!

Thanks for the ideas and discussion!