how can I count tokens before making api call?

Question

import { Configuration, OpenAIApi } from "openai"
import { readFile } from './readFile.js'

// Config OpenAI API
const configuration = new Configuration({
    organization: "xyx......",
    apiKey: "abc.......",
});

// OpenAI API instance
export const openai = new OpenAIApi(configuration);


const generateAnswer = async (conversation, userMessage) => {
    try {
        const dataset = await readFile();
        const dataFeed = { role: 'system', content: dataset };
        const prompt = conversation ? [...conversation?.messages, dataFeed, userMessage] : [dataFeed, userMessage];
        const completion = await openai.createChatCompletion({
            model: "gpt-3.5-turbo",
            messages: prompt
        })

        const aiMessage = completion.data.choices[0].message;
        console.log(completion.data.usage)
        return aiMessage
    } catch (e) {
        console.log(e)
    }
}
export { generateAnswer };

I am trying to create chat bot, in which I provide datafeed in start which is business information and conversation history to chat api I want to calculate tokens of conversation and reduce prompt if exceeds limit before making api call I have tried using gpt3 encoder to count tokens but i have array of objects not string in prompt

Answer 1

Question is old but it may help someone. There is a node.js library called tiktoken wish is a fork form the original tiktoken library.

All the examples on the official tiktoken repo are valid with a small changes.

install the tiktoken npm package:

npm install @dqbd/tiktoken

calculate the number of tokens in a text string

import { encoding_for_model } from "@dqbd/tiktoken";

//Returns the number of tokens in a text string
function numTokensFromString(message: string) {
  const encoder = encoding_for_model("gpt-3.5-turbo");

  const tokens = encoder.encode(message);
  encoder.free();
  return tokens.length;
}

decode the tokens back to string

import { encoding_for_model } from "@dqbd/tiktoken";

 function decodeTokens(message: Uint32Array) {
  const encoder = encoding_for_model("gpt-3.5-turbo");

  const words = encoder.decode(message);
  encoder.free();
  return new TextDecoder().decode(words);
}