When budgeting API calls to large language models, a common shortcut is to assume that one token is roughly four characters of English text, so a prompt's token count is its character count divided by four. I want to use this to estimate the cost of a request before sending it, without running the provider's own tokenizer. How accurate is this rule in practice? For which kinds of input (source code, non-English languages, numbers, emoji, whitespace-heavy text) does it become unreliable, and is there a better cheap approximation or a standard safety margin to add when the estimate is used for budgeting?

Full article content could not be extracted automatically. Read the original below.