File System Docs

Concepts

Before diving into the specifics of each module, let's outline some key concepts that are used throughout:

• LLMContent: This is a structured object that contains data to be processed by a Language Model. It includes an optional role and an array of DataPart objects.
• Outcome<T>: A generic type that represents either a successful result of type T or an error. An error is indicated by an object with a $error property containing an error message string.
• FileSystemPath: A string representing a path in a virtual file system. The system uses a virtual file system that provides a structured way to store data.
• File System Root Directories: The file system uses several root directories, each with a specific purpose and lifetime:
  • /local: Project-level persistent storage. Data persists across sessions and is not automatically managed.
  • /session: Session-scoped persistent storage. Data persists for the lifetime of the current session or object.
  • /run: Run-specific storage. Data persists for the duration of a single run of the application or module.
  • /tmp: Temporary storage. Data persists for the duration of a single module invocation.
  • /env: Environment-provided read-only resources.
  • /assets: Project-level read-only shared assets.
• Streams: Some files in the system can be treated as "streams." A stream is a special type of file that can be written to and read from in chunks. Each read operation empties the current chunk, and the next write operation puts a new chunk in the file.
• MIME Types: MIME types are used to define the format of the data, for example text/plain or image/png.
• Inflation: A process of converting stored data parts into inline data parts, that is, converting a reference to a stored data into the data itself.

Module: @query

Behavior

The @query module provides a function to query the file system and find information about the files and directories that are available in the file system.
The function takes a single argument inputs of type FileSystemQueryArguments, that contains a single path property of type FileSystemPath. The path is the location to query in the virtual file system.
The function returns a Promise that resolves with a FileSystemQueryResult, which can be either:

• A successful result that contains an object with an array of FileSystemQueryEntry objects. Each FileSystemQueryEntry object contains the file path (path), and flags if the file is a stream and what is the length of the file as number of LLMContent items in the file.
• Or an error if the operation fails.

Usage Examples

Querying the contents of a directory:

import query from '@query';

async function listFiles() {
    const result = await query({path: '/local/my-directory'});

    if ('$error' in result) {
      console.error("Error listing files: ", result.$error);
    } else {
        console.log("Files:", result.entries);
    }
}

listFiles();

Module: @read

Behavior

The @read module provides a function to read content from files in the file system.
The function takes a single argument inputs of type FileSystemReadArguments, which includes:

• a path property of type FileSystemPath that specifies the path to read from.
• an optional start property which specifies the index of the LLMContent to start reading from.
• an optional inflate property, that, when set to true, converts all stored data parts into inline parts.

The function returns a Promise that resolves with a FileSystemReadResult. This can be one of the following:

• A successful result containing an object with a data property, which is an array of LLMContent items read from the file. The result also contains a last property, that is the index of the last item read. This may be different from the length of the data because the request may have asked for a partial read. This result is returned when the file is not a stream, or when the stream is not at the end.
• A successful result containing an object with a data property, which is an array of LLMContent items read from the current chunk of the stream. If there is no new chunk available, data will be undefined. The result also has a done property, that will be true if the stream is at the end.
• Or an error if the operation fails.

Usage Examples

Reading all content from a file:

import read from '@read';

async function readFile() {
  const result = await read({ path: '/local/my-file.txt' });
  if ('$error' in result) {
    console.error("Error reading file:", result.$error);
  } else if (result.data) {
    console.log("File content:", result.data);
    console.log("Last read index:", result.last);
  } else if (result.done) {
    console.log("Stream ended.");
  }
}
readFile();

Reading a specific part of the file:

import read from '@read';
async function readPartialFile() {
  const result = await read({ path: '/local/my-file.txt', start: 5});
    if ('$error' in result) {
      console.error("Error reading partial file: ", result.$error);
    } else if (result.data) {
    console.log("Partial file content:", result.data);
      console.log("Last read index:", result.last);
  } else if (result.done) {
    console.log("Stream ended.");
  }
}
readPartialFile();

Reading from a stream:

import read from '@read';
async function readStream() {
  let done = false;
  while (!done) {
    const result = await read({ path: '/run/my-stream.txt' });

    if ('$error' in result) {
      console.error("Error reading stream:", result.$error);
      break;
    } else if (result.data) {
      console.log("Stream chunk:", result.data);
      done = result.done
    } else if (result.done) {
      console.log("Stream ended.");
      done = true;
    }
  }
}
readStream();

Module: @write

Behavior

The @write module provides a function to write content to files in the file system, or to manage streams.
The function takes a single argument inputs of type FileSystemWriteArguments, which can take different shapes depending on the desired action. These are the different variants of the argument, and how they work:

• Writing/overwriting to a regular file:
  • It requires a path of type FileSystemReadWritePath.
  • It requires an array of LLMContent items in the data property.
  • An optional append property, when set to true, appends data to the file instead of overwriting it.
  • An optional stream property, that when set to false makes this file a regular file rather than a stream (this is the default).
• Writing a chunk to a stream:
  • It requires a path of type FileSystemReadWritePath.
  • It requires a stream property set to true to indicate that the file is a stream.
  • It requires an array of LLMContent items in the data property that will be written as the next chunk of data in the stream.
  • An optional receipt property, that when set to true the promise resolves after the chunk has been read. When set to false, it resolves immediately.
  • An optional done property, that is used to signal the end of the stream. This property is set to false when writing a chunk to a stream.
• Signaling the end of the stream:
  • It requires a path of type FileSystemReadWritePath.
  • It requires a stream property set to true to indicate that the file is a stream.
  • It requires a done property set to true, that signals the end of the stream. Once the end of a stream has been read, the stream file is deleted.
• Copying or moving a file:
  • It requires a path of type FileSystemReadWritePath.
  • It requires a source property of type FileSystemPath that specifies the path of the source file.
  • An optional move property, when set to true, deletes the source file after copying it to the destination file.
• Deleting a file:
  • It requires a path of type FileSystemReadWritePath.
  • It requires a delete property set to true, to delete the file.

The function returns a Promise that resolves with a FileSystemWriteResult. This can be:

• A successful result (void) if the operation was successful.
• Or an error if the operation fails.

Usage Examples

Writing to a file:

import write from '@write';

async function writeFile() {
    const result = await write({
        path: '/local/my-file.txt',
        data: [{ parts: [{ text: "Hello, world!" }] }],
    });
  if ('$error' in result) {
      console.error("Error writing file:", result.$error);
    } else {
      console.log("File written successfully.");
    }
}

writeFile();

Appending to a file:

import write from '@write';

async function appendToFile() {
  const result = await write({
    path: '/local/my-file.txt',
      data: [{ parts: [{ text: "More text!"}]}],
      append: true,
  });
  if ('$error' in result) {
      console.error("Error appending to file:", result.$error);
    } else {
      console.log("File appended successfully.");
    }
}
appendToFile();

Writing to a stream:

import write from '@write';

async function writeToStream() {
    await write({
        path: '/run/my-stream.txt',
        data: [{ parts: [{ text: "Chunk 1" }] }],
        stream: true,
    });
    await write({
        path: '/run/my-stream.txt',
        data: [{ parts: [{ text: "Chunk 2" }] }],
        stream: true,
    });
    await write({
        path: '/run/my-stream.txt',
        stream: true,
        done: true
    });
    console.log("Stream written and closed successfully.");
}

writeToStream();

Copying a file:

import write from '@write';

async function copyFile() {
    const result = await write({
        path: '/local/new-file.txt',
        source: '/local/my-file.txt',
    });
    if ('$error' in result) {
        console.error("Error copying file:", result.$error);
    } else {
        console.log("File copied successfully.");
    }
}
copyFile();

Moving a file:

import write from '@write';

async function moveFile() {
    const result = await write({
        path: '/local/new-file.txt',
        source: '/local/my-file.txt',
        move: true,
    });
    if ('$error' in result) {
        console.error("Error moving file:", result.$error);
    } else {
        console.log("File moved successfully.");
    }
}
moveFile();

Deleting a file:

import write from '@write';

async function deleteFile() {
  const result = await write({ path: '/local/my-file.txt', delete: true });
  if ('$error' in result) {
    console.error("Error deleting file: ", result.$error);
  } else {
      console.log("File deleted successfully.");
  }
}

deleteFile();