Logprobs → “logarithmic probabilities”. It is an underlying machine learning format that gives a wide range of possible values in a low bit depth. The precision symptom you previously reported on is the effect of using a low bit depth, but the scalar of probabilities is not limited in that low bit depth because of use of logarithms/exponentiation.

A logprob is the natural logarithm of a token’s probability:

\text{logprob}=\ln(p)

(Natural log is base “e” instead of base 10.)

Equivalently, the logprob is the exponent to which (e) must be raised to recover the ordinary probability:

probability = math.exp(logprob)  # 0.0 to 1.0 by using Python's math
percentage = probability * 100   # 0% to 100%

# Equivalent portable code:
probability = 2.718281828459045 ** logprob

For example, a logprob of 0 becomes 1.0 or 100%, while -0.693 becomes approximately 0.5 or 50%. More-negative logprobs represent lower probabilities.

AI models don’t just generate a single token: they evaluate the certainty of the entire dictionary of possible tokens of a model’s codec - over 200000 of them. Then there is a random process, sampling, to choose one based in relation to the total distribution.

API logprob return

On the API, when you use a compatible model and request logprobs be returned, and ask the AI about itself, you’ll get an output object such as:

"top_logprobs": [
  {
    " I": -0.34537393,
    " My": -2.7983549,
    " The": -3.4044254,
    " Open": -3.6943538,
    " G": -4.453206
  },
  {
    " am": -0.37808284,
    "\u2019m": -2.5384698,
    "'m": -2.8837495,
    " was": -3.009709,
    " have": -3.6439176
  },
...

You can see at the second token with 5 logprobs returned, the AI had a choice of continuing with the output being formed as “I am”, “I’m”, I was", “I have”… in decreasing likelihood. Different trials may give you different results.

Use math.exp() of Python on each logprob:

import json
import math

top_logprobs = [
  {
    " I": -0.34537393,
    " My": -2.7983549,
    " The": -3.4044254,
    " Open": -3.6943538,
    " G": -4.453206
  },
  {
    " am": -0.37808284,
    "\u2019m": -2.5384698,
    "'m": -2.8837495,
    " was": -3.009709,
    " have": -3.6439176
  }
]

top_probabilities = [
  {
    token: round(math.exp(logprob), 8)
    for token, logprob in position.items()
  }
  for position in top_logprobs
]

print(json.dumps(
  {"top_probabilities": top_probabilities},
  indent=2
))

Output:

{
  "top_probabilities": [
    {
      " I": 0.70795558,
      " My": 0.06091018,
      " The": 0.03322591,
      " Open": 0.02486352,
      " G": 0.01164119
    },
    {
      " am": 0.68517374,
      "\u2019m": 0.07898717,
      "'m": 0.05592468,
      " was": 0.04930602,
      " have": 0.0261497
    }
  ]
}

These values are probabilities from 0.0 to 1.0. Multiply each by 100 to express it as a percentage. The entries at each position need not sum to 1.0, because top_logprobs normally contains only the highest-probability tokens, not the entire token vocabulary.

Then you also get what was actually sampled, where you can see none of those was 100% in this case:

  "logprobs": {
    "tokens": [
      " I",
      " am",
      " based",
      " on",
      " a",
      " neural",
      " network",...
    ],
    "token_logprobs": [
      -0.34537393,
      -0.37808284,
      -0.52319646,
      -0.033077285,
      -1.7607243,
      -2.9560394,
      -0.19585982,

How I “got them”

What the screenshot shows is what OpenAI discontinued in their own playground: a handy display of logprob values, where you can observe each position (and not a JSON of massive length). I re-created such here for completions (for only older completions AI models), but you can do the same for other endpoints:

You might be interested to explore this surfaced API parameter while completions is still available: suffix: optional string The suffix that comes after a completion of inserted text. This parameter is only supported for gpt-3.5-turbo-instruct. What is “suffix” for? It is for fill-in-middle. The AI language model will complete the “missing language” between the prompt and the suffix. From (note the cursor): [image] To: [image] (and hover for those logprobs → probabilities) This is e…

I have my own “logprob cookbook” as a replacement for OpenAI’s obsolete document that is broken, but there is nobody actively maintaining cookbooks or submissions.

Conclusion

The AI didn’t “write” those numbers. They were returned by the API and converted for good UI experience and interpretability. logprobs are a part of how AI models work.