Skip to content

Allow Browser Usage, countTokens convenience method#33

Open
wesmcouch wants to merge 1 commit into
latitudegames:masterfrom
wesmcouch:no-fs-count-tokens
Open

Allow Browser Usage, countTokens convenience method#33
wesmcouch wants to merge 1 commit into
latitudegames:masterfrom
wesmcouch:no-fs-count-tokens

Conversation

@wesmcouch
Copy link
Copy Markdown

Overview
  • Remove fs and path usage to allow usage of the encoder and decoder both inside a browser and in nodejs
  • Add a convenience method named countTokens that counts the number of tokens in a given string
  • Update the example and tests to include the new method countTokens

…r and in nodejs. Add countTokens function and tests.
@nem035
Copy link
Copy Markdown

nem035 commented Feb 28, 2023

FYI - I published a version to use (works in browser/node but no countTokens) until the maintainers merged: https://www.npmjs.com/package/@nem035/gpt-3-encoder

@arcticfly
Copy link
Copy Markdown

@nem035 looks like you can merge?

@astoilkov
Copy link
Copy Markdown

This PR also makes it work in Electron (not just the browser). Vite also has a problem with the code in master because it uses __dirname.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants