Skip to content

Review fixes: MyST markup, language, and grammar #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rkratky wants to merge 1 commit into
base: spring-ai-tool-calling
Choose a base branch
from
Open

Review fixes: MyST markup, language, and grammar #3

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
354 changes: 200 additions & 154 deletions docs/tutorials/spring-ai-tool-calling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,195 +3,241 @@ myst:
html_meta:
description: "Extend an existing Spring AI chatbot to support tool calling"
---

(spring-ai-tool-calling)=
# Implementing tool calling with Spring AI

The goal of this tutorial is to help appreciate the ease with which tool-calling may be implemented using Spring AI, on the client-side. We will augment the chat-client from our initial Spring AI tutorial to include a tool interaction that helps the LLM answer queries related to toolchains availability on Ubuntu versions.
The goal of this tutorial is to help appreciate the ease of implementing tool-calling using Spring AI, on the client-side. We augment the chat-client from our initial Spring AI tutorial to include a tool interaction that helps the LLM answer queries related to toolchains availability on Ubuntu versions.


## Introduction to LLM tool calling
Large language models cannot answer questions about data they are not trained on. We previously saw how Retrieval Augmented Generation provides additional information to an LLM in the form of documents. Additional information may also be generated, or deduced from existing data sources, through application logic. If an LLM is made aware of a set of existing _tools_ and the kind of information they serve, it can invoke them while answering a user question, and use the received information to generate a response. This is referred to as tool-calling.

Apart from information retrieval, tools may also be configured to take action. For example, we may configure tools for tasks such as sending an email, triggering a workflow, or updating a database, on a prompt to the LLM.
Large language models cannot answer questions about data they are not trained on. We previously saw how Retrieval Augmented Generation provides additional information to an LLM in the form of documents. Additional information is also generated or deduced from existing data sources, through application logic. An LLM made aware of a set of existing _tools_ and the kind of information they serve invokes them while answering a user question, and uses the received information to generate a response. This is referred to as tool-calling.

Apart from information retrieval, tools also support taking action. For example, configure tools for tasks such as sending an email, triggering a workflow, or updating a database, on a prompt to the LLM.

:::{note}
Not all LLMs support tool-calling.
:::


## Tool calling support in Spring AI
Spring AI eases tool interactions with LLMs that support tools. It provides a consistent means of applying tools when submitting a prompt.

Spring AI eases tool interactions with LLMs that support tools. It provides a consistent means of applying tools when submitting a prompt.

Methods annotated using the `@Tool` annotation act as tool-callbacks. The `description` parameter must be used to suitably word the tool description. LLMs depend on these descriptions to decide which tool to invoke for a given prompt.

The `@Tool` methods should belong to a Spring bean class annotated as `@Component`, whose object may be registered with a `ChatClient`. For such a client, Spring AI auto-generates a JSON object with the tool information while making requests to the LLM backend.
The `@Tool` methods belong to a Spring bean class annotated as `@Component`, whose object is registered with a `ChatClient`. For such a client, Spring AI auto-generates a JSON object with the tool information while making requests to the LLM backend.

Here is a basic diagram that depicts the tool-calling protocol with Spring AI.
Here is a basic diagram that depicts the tool-calling protocol with Spring AI.

![spring-ai-tool-calling](../images/springai-tools/spring-ai-tool-calling.png)


## An LLM chat-client with tool calling

This tutorial will reuse the simple chat-client developed in the {ref}`springai-basic` tutorial. However, we will use the gemma4 model instead of Qwen VL 2.5.

1. Install the gemma4 snap:
```bash
sudo snap install gemma4
```

2. Make sure gemma4 is active by issuing the `gemma4 status` command:
```bash
$ gemma4 status
engine: amd-gpu
services:
server: active
server-webui: active
endpoints:
openai: http://127.0.0.1:8336/v1
webui: http://127.0.0.1:8337/
model:
name: gemma4-e4b-q4-k-m
```

3. Clone the simple chat-client demo repository:
```bash
git clone https://github.com/pushkarnk/spring-ai-chat-client-demo && cd spring-ai-chat-client-demo
```

4. Update `src/main/resources/application.properties` to the following values:
```properties
spring.application.name=chat-client
spring.ai.openai.base-url=http://127.0.0.1:8336
spring.ai.openai.api-key=ignored
spring.ai.openai.chat.options.model=gemma4-e4b-q4-k-m
```

5. Make sure you have JDK 21 installed and run the application:
```bash
sudo apt update && sudo apt install openjdk-21-jdk
```

```bash
./gradlew bootRun
```

6. Open `http://localhost:8080` in a browser window and enter the following question:

```none
Is Zig 0.16 available on Ubuntu?
```

The answer doesn't really make sense:
```none
While Zig is generally available for Linux, getting a very specific and potentially older version like **0.16** directly through standard Ubuntu package managers (like `apt`) can be dif ficult because the official repositories often lag behind the latest stable releases.
```
This tutorial reuses the simple chat-client developed in the {ref}`springai-basic` tutorial. However, we use the {pkg}`gemma4` model instead of Qwen VL 2.5.

1. Install the {pkg}`gemma4` snap:

```{terminal}

sudo snap install gemma4
```

1. Make sure gemma4 is active by issuing the {command}`gemma4 status` command:

```{terminal}

gemma4 status

engine: amd-gpu
services:
server: active
server-webui: active
endpoints:
openai: http://127.0.0.1:8336/v1
webui: http://127.0.0.1:8337/
model:
name: gemma4-e4b-q4-k-m
```

1. Clone the simple chat-client demo repository:

```{terminal}

git clone https://github.com/pushkarnk/spring-ai-chat-client-demo && cd spring-ai-chat-client-demo
```

1. Update {file}`src/main/resources/application.properties` to the following values:

```{code-block} properties
:caption: `src/main/resources/application.properties`

spring.application.name=chat-client
spring.ai.openai.base-url=http://127.0.0.1:8336
spring.ai.openai.api-key=ignored
spring.ai.openai.chat.options.model=gemma4-e4b-q4-k-m
```

1. Make sure you have JDK 21 installed and run the application:

```{terminal}

sudo apt update && sudo apt install openjdk-21-jdk
```

```{terminal}

./gradlew bootRun
```

1. Open `http://localhost:8080` in a browser window and enter the following question:

```{code-block} none
Is Zig 0.16 available on Ubuntu?
```

The answer doesn't really make sense:

```{code-block} none
While Zig is generally available for Linux, getting a very
specific and potentially older version like **0.16** directly
through standard Ubuntu package managers (like `apt`) can be
difficult because the official repositories often lag behind
the latest stable releases.
```


### Adding a tool to the Spring AI application

The `rmadison` command line utility fetches archive package data that can help answer such questions. Let's set up a tool that invokes this command.

1. Add a new @Component class named `UbuntuTools.java` with the following implementation:
```java
package demo.chatclient;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
import org.springframework.stereotype.Component;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;

@Component
public class UbuntuTools {
private static final Logger logger = LoggerFactory.getLogger(UbuntuTools.class);

@Tool (description = "Query the Ubuntu archive for package names and their versions across Ubuntu versions.")
public String rmadison(
@ToolParam(description = "Source or binary package name") String packageName
) {
logger.info("Running rmadison for package: {}", packageName);
try {
Process process = new ProcessBuilder("rmadison", packageName)
.redirectErrorStream(true)
.start();

String output;
try (BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream(), StandardCharsets.UTF_8))) {
output = reader.lines().reduce("", (a, b) -> a + b + System.lineSeparator());
}

process.waitFor();
return output;
} catch (IOException e) {
return "Failed to run rmadison: " + e.getMessage();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
return "rmadison command interrupted: " + e.getMessage();
}
}
}
```

The `rmadison()` method invokes the rmadison utility on the package name received from the LLM. This method is a `@Tool`. The result is converted to a String and returned to the LLM along with the user question.

2. Instantiate an object of class `UbuntuTools` and configure it as a tools object while building the chat client. Modify `src/main/java/demo/chatclient/DemoChatService.java` and modify the constructor source.
:::{note}
This is not a full source listing of DemoChatService.java
:::
The {command}`rmadison` command line utility fetches archive package data that helps answer such questions. Let's set up a tool that invokes this command.

1. Add a new @Component class named {file}`UbuntuTools.java` with the following implementation:

```{code-block} java
:caption: `UbuntuTools.java`

package demo.chatclient;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
import org.springframework.stereotype.Component;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;

@Component
public class UbuntuTools {
private static final Logger logger = LoggerFactory.getLogger(UbuntuTools.class);

@Tool (description = "Query the Ubuntu archive for package names and their versions across Ubuntu versions.")
public String rmadison(
@ToolParam(description = "Source or binary package name") String packageName
) {
logger.info("Running rmadison for package: {}", packageName);
try {
Process process = new ProcessBuilder("rmadison", packageName)
.redirectErrorStream(true)
.start();

String output;
try (BufferedReader reader = new BufferedReader(
new InputStreamReader(process.getInputStream(), StandardCharsets.UTF_8))) {
output = reader.lines().reduce("", (a, b) -> a + b + System.lineSeparator());
}

process.waitFor();
return output;
} catch (IOException e) {
return "Failed to run rmadison: " + e.getMessage();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
return "rmadison command interrupted: " + e.getMessage();
}
}
}
```

The `rmadison()` method invokes the rmadison utility on the package name received from the LLM. This method is a `@Tool`. The result is converted to a String and returned to the LLM along with the user question.

1. Instantiate an object of class `UbuntuTools` and configure it as a tools object while building the chat client. Modify {file}`src/main/java/demo/chatclient/DemoChatService.java` and modify the constructor source.

```java
...
public class DemoChatService implements DemoChatClient {
:::{note}
This is not a full source listing of DemoChatService.java
:::

private final ChatClient chatClient;
```{code-block} java
:caption: `src/main/java/demo/chatclient/DemoChatService.java`

public DemoChatService(ChatClient.Builder chatClientBuilder) {
this.chatClient = chatClientBuilder
.defaultTools(new UbuntuTools())
.build();
}
...
public class DemoChatService implements DemoChatClient {

...
...
}
```
3. Launch the application and navigate to `http://localhost:8080`.
private final ChatClient chatClient;

```none
./gradlew bootRun
```
public DemoChatService(ChatClient.Builder chatClientBuilder) {
this.chatClient = chatClientBuilder
.defaultTools(new UbuntuTools())
.build();
}

4. Let us try posting questions again.
...
...
}
```

Post the zig0.16 question:
```none
Is zig0.16 available in Ubuntu?
```
1. Launch the application and navigate to `http://localhost:8080`.

This time we get a better answer. Something like:
```none
Yes, according to the Ubuntu archive, a package named **zig0.16** is available.
```{terminal}

It has the version **0.16.0~us1-0ubuntu1** and can be found in the `stonking/universe` repository for multiple architectures (amd64, amd64v3, arm64, riscv64, s390x).
```
./gradlew bootRun
```

On the terminal running the server, a log message confirming the tool invocation should be seen:
```none
2026-07-14T20:28:12.861+05:30 INFO 303854 --- [chat-client] [nio-8080-exec-1] demo.chatclient.UbuntuTools : Running rmadison for package: zig0.16
```
1. Let us try posting questions again.

Post the zig0.16 question:

```{code-block} none
Is zig0.16 available in Ubuntu?
```

This time we get a better answer. Something like:

```{code-block} none
Yes, according to the Ubuntu archive, a package named **zig0.16**
is available.

It has the version **0.16.0~us1-0ubuntu1** and can be found in the
`stonking/universe` repository for multiple architectures (amd64,
amd64v3, arm64, riscv64, s390x).
```

On the terminal running the server, a log message confirming the tool invocation is displayed:

```{terminal}
:output-only:

2026-07-14T20:28:12.861+05:30 INFO 303854 --- [chat-client] [nio-8080-exec-1] demo.chatclient.UbuntuTools : Running rmadison for package: zig0.16
```

Try posting a question about zig0.17, an unreleased version:

```{code-block} none
Is zig0.17 available in Ubuntu?
```

An answer like the following is expected:

```{code-block} none
Based on the available information, the package `zig0.17` does
not appear to be available in the Ubuntu archive.
```

Try posting a question about zig0.17, an unreleased version:
```none
Is zig0.17 available in Ubuntu?
```

An answer like the following should be expected:
```none
Based on the available information, the package `zig0.17` does not appear to be available in the Ubuntu archive.
```
## Conclusion
Spring AI makes LLM tool-calling very easy to implement. The effectiveness of a tool depends on how specific a task it is defined for, and how close the `description` parameter of the `@Tool` annotation is to actual user questions. Refer to the [Spring AI Tool Calling documentation](https://docs.spring.io/spring-ai/reference/api/tools.html) for more details.

Spring AI makes LLM tool-calling very easy to implement. The effectiveness of a tool depends on how specific a task it is defined for and how close the `description` parameter of the `@Tool` annotation is to actual user questions. Refer to the [Spring AI Tool Calling documentation](https://docs.spring.io/spring-ai/reference/api/tools.html) for more details.