Mechanism for splitting rather than truncating

[simonw]

Suggestion from:

• Add split option to split large input into smaller parts instead of truncating #2

[simonw]

Proposed design from #2 is this:

cat big.txt | ttok --truncate 100 --split splitted.txt

This would produce files called:

splitted.txt.part0
splitted.txt.part1
splitted.txt.part2

Etc, up to the total number of parts.

I like the idea, but I'm not sold on that particular design (cc @c4pt0r)

[simonw]

A related challenge to this is recursive summarization - where you want to summarize a large document, so you split it into smaller sections, summarize each of those and then perform a summary against those summaries.

Doing that well is difficult - I've not yet figured out my own preferred pattern for it. I believe it ends up requiring a bit of token overlap between the sections in order to avoid things like a sentence that was cut off half way through summarizing incorrectly.

More about that pattern here: https://twitter.com/hwchase17/status/1587458160297533440

I'm not sure if implementing helpers for that would fit in the ttok tool or not - but worth mentioning here.

[simonw]

There are two alternative output schemes that I think are worth considering here:

1. JSON. An option to output a JSON array of chunks, each fitting the desired size, would be neat.
2. Maybe some kind of mechanism that can spawn multiple follow-on llm tasks? Kind of like how xargs works - or maybe a pattern that can be fed to xargs in a neat way. Not sure what that would look like though.

I'd like to avoid creating temporary files on disk if possible - happy to support that as an option, but I don't like it as a default.

[simonw]

Here's an option: xargs can work against \0 null delimited data. So maybe a -0/--null option could cause the split tokenized output to be separated by that, at which point it could be piped into xargs.

Could look something like this:

cat big.txt | ttok --truncate 100 --split --null | xargs -0 llm --system "summarize as bullet points"

[simonw]

Given that design, --split is a flag. It could default to outputting ["...", "..."] JSON, could output \0 delimited text with --null and could output to files if another option such as --split-file output was passed.

[simonw]

I'm not sure xargs on macOS can do what I want it to do here. I got as far as trying this:

curl 'https://simonwillison.net/' | strip-tags > big.txt
ttok < big.txt --split -t 2000 --null | xargs -0 -I{} zsh -c 'echo {} | llm --system "summarize as bullets" -s'

But it clearly wasn't passing each line of the data piped to xargs through to standard input to the llm command.

Here's the diff of my prototype:

diff --git a/ttok/cli.py b/ttok/cli.py
index 73e1652..e7f0641 100644
--- a/ttok/cli.py
+++ b/ttok/cli.py
@@ -1,4 +1,5 @@
 import click
+import json
 import sys
 import tiktoken
 
@@ -10,9 +11,13 @@ import tiktoken
 @click.option(
     "-t", "--truncate", "truncate", type=int, help="Truncate to this many tokens"
 )
+@click.option("--split", is_flag=True, help="Split text based on truncate argument")
+@click.option(
+    "-0", "--null", is_flag=True, help="Output split text with null byte delimiters"
+)
 @click.option("-m", "--model", default="gpt-3.5-turbo", help="Which model to use")
 @click.option("output_tokens", "--tokens", is_flag=True, help="Output token integers")
-def cli(prompt, input, truncate, model, output_tokens):
+def cli(prompt, input, truncate, split, null, model, output_tokens):
     """
     Count and truncate text based on tokens
 
@@ -36,6 +41,8 @@ def cli(prompt, input, truncate, model, output_tokens):
 
         cat input.txt | ttok --tokens
     """
+    if split and not truncate:
+        raise click.ClickException("Cannot use --split without --truncate")
     try:
         encoding = tiktoken.encoding_for_model(model)
     except KeyError as e:
@@ -51,6 +58,17 @@ def cli(prompt, input, truncate, model, output_tokens):
             text = input_text
     # Tokenize it
     tokens = encoding.encode(text)
+
+    if split:
+        output = []
+        for chunk in chunks(tokens, truncate):
+            output.append(encoding.decode(chunk))
+        if null:
+            click.echo("\0".join(output))
+        else:
+            click.echo(json.dumps(output, indent=2))
+        return
+
     if truncate:
         tokens = tokens[:truncate]
 
@@ -60,3 +78,8 @@ def cli(prompt, input, truncate, model, output_tokens):
         click.echo(encoding.decode(tokens), nl=False)
     else:
         click.echo(len(tokens))
+
+
+def chunks(sequence, n):
+    for i in range(0, len(sequence), n):
+        yield sequence[i : i + n]

[simonw]

This did work though:

ttok < big.txt --split -t 2000 --null | while IFS= read -r -d $'\0' chunk; do
    echo "$chunk" | llm --system 'summarize as bullets' -s
done

[simonw]

I'm trying to figure out how safe null bytes are - can they occur in tokenized text?

I think maybe they can:

echo "\0" | ttok --tokens
188 198

198 is newline - so is 188 null byte?

If so, then the --null option should strip those out of the token stream.

[simonw]

The token ID for the null byte likely differs for different models.

Trying to figure out where the models come from by reading this code: https://github.com/openai/tiktoken/blob/main/tiktoken_ext/openai_public.py

[simonw]

>>> import tiktoken
>>> encoding = tiktoken.encoding_for_model("gpt2")
>>> encoding.encode("\0")
[188]
>>> encoding.encode("hello world")
[31373, 995]
>>> encoding2 = tiktoken.encoding_for_model("gpt-3.5-turbo")
>>> encoding2.encode("\0")
[188]
>>> encoding2.encode("hello world")
[15339, 1917]

It looks like both those models use 188 for the null byte. I should still look it up in the model before filtering it though.

[simonw]

This is pretty good!

% ttok < big.txt --split -t 2000 --null | while IFS= read -r -d $'\0' chunk; do
    echo "$chunk" | llm --system 'summarize as bullets' -s
done
- Simon Willison has built three command-line tools named llm, ttok, and strip-tags for working with ChatGPT and other LLMs.
- Tokens are how pricing works to work with LLMs and being able to count tokens is essential.
- strip-tags and ttok helps in better ways of working with tokens.
- strip-tags command strips out HTML tags which usually aren't relevant to the prompt you are sending to the model.
- llm command helps in piping content to a model opens up all kinds of fun opportunities.
- Prompt injection remains an unsolved problem and delimiters are easily defeated.
- ChatGPT Prompt Engineering for Developers is an interactive video course presented in partnership with OpenAI which covers prompt engineering.
- OpenAI's attempt to use delimiters to avoid prompt injections is ineffective
- Attackers have many options to confound language models with a sequence of tokens

[simonw]

Bit of a side exploration of hexdump - I figured out how to prove to myself that I was producing a null byte from Python:

% python -c 'import sys; sys.stdout.buffer.write(b"a")' | hexdump -C
00000000  61                                                |a|
00000001
% python -c 'import sys; sys.stdout.buffer.write(b"a\0")' | hexdump -C
00000000  61 00                                             |a.|
00000002

Without the -C option hexdump was showing confusing output because of little-endian v.s. big-endian issues:

% python -c 'import sys; sys.stdout.buffer.write(b"a")' | hexdump     
0000000 0061                                   
0000001
% python -c 'import sys; sys.stdout.buffer.write(b"a\0")' | hexdump   
0000000 0061                                   
0000002

[simonw]

Now I can test the null byte handling:

% python -c 'import sys; sys.stdout.buffer.write(b"one\0two\0three\0")' | ttok --split -t 2                    
[
  "one\u0000",
  "two\u0000",
  "three\u0000"
]

Without the code I wrote to filter out null bytes when using --null:

% python -c 'import sys; sys.stdout.buffer.write(b"one\0two\0three\0")' | ttok --split -t 2 --null | hexdump -C
00000000  6f 6e 65 00 00 74 77 6f  00 00 74 68 72 65 65 00  |one..two..three.|
00000010  0a                                                |.|
00000011

Note how those null bytes have been doubled up 00 00 in a couple of places, which would then confuse the shell loop:

python -c 'import sys; sys.stdout.buffer.write(b"one\0two\0three\0")' | ttok --split -t 1 --null | \
while IFS= read -r -d $'\0' chunk; do
    echo "Chunk:"
    echo "$chunk" | hexdump -C
done

Chunk:
00000000  6f 6e 65 0a                                       |one.|
00000004
Chunk:
00000000  0a                                                |.|
00000001
Chunk:
00000000  74 77 6f 0a                                       |two.|
00000004
Chunk:
00000000  0a                                                |.|
00000001
Chunk:
00000000  74 68 72 65 65 0a                                 |three.|
00000006

But if I then add this Python code:

    if split:
        output = []
        if null:
            # Filter out null byte tokens
            null_token = encoding.encode("\0")[0]
            tokens = [t for t in tokens if t != null_token]

I get this result instead (I had to drop -t to 1 because the null bytes had been filtered out):

python -c 'import sys; sys.stdout.buffer.write(b"one\0two\0three\0")' | ttok --split -t 1 --null | while IFS= read -r -d $'\0' chunk; do
    echo "Chunk:"
    echo "$chunk" | hexdump -C
done

Chunk:
00000000  6f 6e 65 0a                                       |one.|
00000004
Chunk:
00000000  74 77 6f 0a                                       |two.|
00000004

Which is wrong - three is missing and I don't understand why.

Side note: ttok --split -t 2 --tokens should output a JSON list of list of integers.

[simonw]

This seems to be doing the right thing:

python -c 'import sys; sys.stdout.buffer.write(b"one\0two\0three\0")'  | ttok --split -t 1 --null | hexdump -C

00000000  6f 6e 65 00 74 77 6f 00  74 68 72 65 65 0a        |one.two.three.|
0000000e

So it must be a problem with that | while pattern I'm using.