Thinking Machines just released Tinker, an LLM training API for researchers and hackers. The API offers low-level control while abstracting away model deployment challenges.

We built a simple example showcasing how to use Ray along with Tinker to build and run a text-to-SQL model.

There are two primary parts to this use case: data generation and model fine-tuning. We show how to generate a dataset for supervised fine-tuning using Ray. We then show how to use the dataset to fine tune an LLM using Tinker.

LinkData generation

We first need to generate data for supervised fine tuning. There are two components to this: generation and evaluation. We generate queries by deploying Qwen-8B using vLLM along with Ray Serve as an Anyscale service to scale LLM inference. We then use Ray Core to execute a large number of parallel tasks to generate candidate SQL queries, then we evaluate each of those queries in a SQL environment and calculate rewards using skyrl-gym.

Here is the application code for running Qwen-8B as a service. This uses Ray Serve’s built-in integration with vLLM to deploy the model.

# deploy_qwen.py

from ray.serve.llm import LLMConfig, build_openai_app

llm_config = LLMConfig(
   model_loading_config=dict(
       model_id="my-qwen-8B",
       model_source="Qwen/Qwen3-8B",
   ),
   accelerator_type="L40S",
   deployment_config=dict(
       autoscaling_config=dict(
           min_replicas=4, max_replicas=8,
       )
   ),
   engine_kwargs=dict(
       max_model_len=8192,
       tensor_parallel_size=1
   )
)

app = build_openai_app({"llm_configs": [llm_config]})

This service can be deployed by running

anyscale service deploy -f service.yaml

The service.yaml file is provided in the appendix.

Here is the code for scaling querying the model, evaluating the queries, and filtering out the unsuccessful queries. This code can naturally be extended in a multi-turn fashion to feed the output of the unsuccessful query back into the model to generate new candidate queries.

# data_generation.py

from urllib.parse import urljoin
from datasets import load_dataset
from skyrl_gym.envs.sql.env import SQLEnv
from omegaconf import DictConfig
from openai import OpenAI
from datasets import load_dataset
import json
import ray

dataset = load_dataset("NovaSky-AI/SkyRL-SQL-653-data-newfmt", split="train").to_list()

token = # <FILL IN APPROPRIATE TOKEN>
base_url = # <FILL IN APPROPRIATE BASE URL>

@ray.remote(num_cpus=0.1)
def generate_sql(messages):
    client = OpenAI(api_key=token, base_url=urljoin(base_url, "v1"))
    response = client.chat.completions.create(
        model="my-qwen-8B",
        messages=messages
    )
    return response.choices[0].message.content

# Generate SQL queries in parallel
object_refs = [generate_sql.remote(record["prompt"]) for record in dataset]

# Fetch the results and filter out the unsuccessful ones
object_refs_and_records = dict(zip(object_refs, dataset))
successful = []
remaining = object_refs
while remaining:
    [ready_ref], remaining = ray.wait(remaining, num_returns=1)
    record = object_refs_and_records[ready_ref]
    messages = record["prompt"]

    try:
        assistant_response = ray.get(ready_ref)
    except Exception as e:
        continue

    conf = DictConfig({"db_path": "/home/ray/data"})
    env = SQLEnv(conf, record)
    env.init(messages)
    try:
        output = env.step(assistant_response)
    except AssertionError as e:
        continue

    print("Reward: ", output["reward"])

    if output["reward"] > 0:
        successful.append((record, assistant_response))

examples = []
for record, assistant_response in successful:
    examples.append(record["prompt"] + [{"role": "assistant", "content": assistant_response}])

with open("/mnt/shared_storage/successful.json", "w") as f:
   import json
   json.dump(examples, f)

This job can be submitted by running

anyscale job submit -f job.yaml --env HF_TOKEN=$HF_TOKEN

The job.yaml file is provided in the appendix. The examples will be stored in a shared filesystem, though you can store them wherever you want.

LinkModel fine-tuning

We use the Tinker API to tokenize the data and fine-tune the model.

The Tinker API offers a high level of control for training and fine tuning LLMs.

The following can be run in an Anyscale workspace that has tinker installed.

import tinker
from tinker import types
import json
import numpy as np

service_client = tinker.ServiceClient()

training_client = service_client.create_lora_training_client(
    base_model="Qwen/Qwen3-8B", rank=32
)
tokenizer = training_client.get_tokenizer()

def process_example(messages: dict, tokenizer) -> types.Datum:
    tokens = tokenizer.apply_chat_template(messages)
    weights = [1] * len(tokens)
    input_tokens = tokens[:-1]
    target_tokens = tokens[1:]
    weights = weights[1:]
    return types.Datum(
        model_input=types.ModelInput.from_ints(tokens=input_tokens),
        loss_fn_inputs=dict(weights=weights, target_tokens=target_tokens)
    )

examples = json.load(open("/mnt/shared_storage/successful.json", "r"))
processed_examples = [process_example(ex, tokenizer) for ex in examples]

# Note: If you are going to train on a larger dataset, you should implement proper minibatch training.
for _ in range(6):
    fwdbwd_future = training_client.forward_backward(processed_examples, "cross_entropy")
    optim_future = training_client.optim_step(types.AdamParams(learning_rate=1e-4))
    # Wait for the results
    fwdbwd_result = fwdbwd_future.result()
    optim_result = optim_future.result()
    # fwdbwd_result contains the logprobs of all the tokens we put in. Now we can compute the weighted
    # average log loss per token.
    logprobs = np.concatenate([output["logprobs"].tolist() for output in fwdbwd_result.loss_fn_outputs])
    weights = np.concatenate([example.loss_fn_inputs["weights"].tolist() for example in processed_examples])
    print(f"Loss per token: {-np.dot(logprobs, weights) / weights.sum():.4f}")

# Save the weights
sampling_client = training_client.save_weights_and_get_sampling_client(name="sql_model")
print(f"model path: {sampling_client.model_path}")

LinkModel evaluation

We now want to check how well the model performs. Let’s first download the model checkpoint (make sure to fill out the model path that was printed by the above code).

import tinker
from urllib.parse import urlparse

MODEL_PATH = # <FILL IN THE MODEL PATH PRINTED ABOVE>

parsed_url = urlparse(MODEL_PATH)

service_client = tinker.ServiceClient()
rest_client = service_client.create_rest_client()
data = rest_client.download_checkpoint_archive(parsed_url.netloc, parsed_url.path.lstrip('/')).result()

with open('output.tar.gz', 'wb') as f:
    f.write(data)

We then extract the LoRA weights with mkdir -p /home/ray/sql_lora && tar xvfz output.tar.gz -C /home/ray/sql_lora and merge the weights with the base (we do this because currently the tinker LoRA weights are not compatible with vLLM and can’t be served directly – this will be fixed going forward).

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
model = PeftModel.from_pretrained(base_model, "/home/ray/sql_lora")
merged_model = model.merge_and_unload()

save_path = "/home/ray/merged_sql_model"
merged_model.save_pretrained(save_path)
tokenizer.save_pretrained(save_path)

LinkAppendix - additional setup

To run the above code, a few additional setup steps are required.

We define our base image using the following Dockerfile.

# Dockerfile

FROM anyscale/ray:2.48.0-slim-py312-cu128

RUN sudo apt-get update -y \
   && sudo apt-get install --no-install-recommends -y build-essential libnuma-dev \
   && sudo rm -f /etc/apt/sources.list.d/*

RUN curl -LsSf https://astral.sh/uv/install.sh | sh

RUN git clone https://github.com/novasky-ai/SkyRL.git
WORKDIR /home/ray/SkyRL/skyrl-gym/
RUN uv pip install --system .
RUN uv pip install --system "huggingface_hub[cli]" "datasets" "openai" "transformers" "torch" "vllm==0.10.0" "pydantic"

We define the service config as follows.

# service.yaml

name: deploy-qwen
containerfile: ./Dockerfile

compute_config:
  auto_select_worker_config: true

working_dir: .

applications:
- import_path: deploy_qwen:app

We define the job config as follows.

# job.yaml

name: data-generation
containerfile: ./Dockerfile

compute_config:
 head_node:
   instance_type: c6a.12xlarge
 auto_select_worker_config: true

working_dir: .

entrypoint: |
  uv run --with huggingface_hub huggingface-cli download seeklhy/OmniSQL-datasets data.zip --repo-type dataset --local-dir $HOME && \
  unzip $HOME/data.zip -d $HOME && \
  python data_generation.py

max_retries: 0