---

📌 Table of Contents

---

📊 Dataset Summary

🌟 Why "Ultimate Distilled"?

This dataset is not a raw scrape. Every sample has been distilled through a unified extraction pipeline:

┌─────────────────────────────────────────────────────────────────┐
│                    UNIFIED EXTRACTION PIPELINE                   │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  63 Upstream Sources                                            │
│  ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐              │
│  │ HF  │ │ HF  │ │ HF  │ │ GH  │ │ HF  │ │ ... │              │
│  └──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘              │
│     │       │       │       │       │       │                   │
│     └───────┴───────┴───┬───┴───────┴───────┘                   │
│                         │                                        │
│                    ┌────▼────┐                                   │
│                    │ EXTRACT │ ← Field normalization             │
│                    └────┬────┘   (instruction/response)          │
│                         │                                        │
│                    ┌────▼────┐                                   │
│                    │CATEGORIZE│ ← 8 semantic categories          │
│                    └────┬────┘                                   │
│                         │                                        │
│                    ┌────▼────┐                                   │
│                    │  SHARD  │ ← 20K samples per shard           │
│                    └────┬────┘                                   │
│                         │                                        │
│                    ┌────▼────┐                                   │
│                    │ UPLOAD  │ ← Batch commits to HF             │
│                    └─────────┘                                   │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

💎 Value to the Open-Source AI Community

---

🗂️ Directory Structure

📂 Manusagents/GPT-5.5-Gemini-3.1-Pro-Grok-4-Claude-Fable-5-Mythos-5-Qwen-3.7-Max-and-more-Distillation-Dataset/
│
├── 📦 archives/                          # ~64 GB — 7,090 compressed GitHub repos
│   ├── 0-chi__sonaure-lp.tar.gz
│   ├── 00MB__bitcoin_trading_bot.tar.gz
│   ├── 0101-agents__plugins.tar.gz
│   ├── ... (7,090 files total)
│   └── zznmg1__playable-survivor-ad.tar.gz
│
├── 📁 data/                              # ~15 GB — 516 JSONL shards
│   │
│   ├── 💻 coding/                        # 26 sources · ~11M+ samples
│   │   ├── vibe_instruct_v2/             # 8,152,510 samples
│   │   ├── fable5_2m/                    # 2,006,487 samples
│   │   ├── vibe_instruct_v1/             # 1,100,000 samples
│   │   ├── vibe_coding/                  # 1,100,000 samples
│   │   ├── royal_ghost_1m/               # 1,000,000 samples
│   │   ├── citation_ground/              # 980,064 samples
│   │   ├── royal_ghost_501k/             # 703,449 samples
│   │   ├── fable5_repos_full/            # 7,090 archive pointers
│   │   ├── fable5_agentic_sft/           # 159,972 samples
│   │   ├── gpt55_codex/                  # 119,436 samples ⭐ FULL
│   │   ├── alpca_gpt55/                  # 49,099 samples
│   │   ├── deepseek_v4_pro_agent/        # 96,597 samples ⭐ FULL
│   │   ├── fable5_traces/                # 49,544 samples ⭐ FULL
│   │   ├── genesis_code_100k/            # 68,000 samples
│   │   ├── genesis_code/                 # 49,000 samples
│   │   ├── kimi_coding/                  # 9,014 samples
│   │   ├── mimo_claude_code_traces/      # 15,046 samples ⭐ FULL
│   │   ├── kimi_k26_claude_code_traces/  # 7,438 samples
│   │   ├── genesis_code_10k/             # 9,800 samples
│   │   ├── legend_python/                # 5,000 samples
│   │   ├── autonomy/                     # 10,000 samples
│   │   ├── genesis_code_demo/            # 1,000 samples
│   │   ├── god_coder/                    # ⭐ FULL raw recovery
│   │   ├── python_god_coder/             # ⭐ FULL raw recovery
│   │   ├── elite_god_coder/              # ⭐ FULL raw recovery
│   │   ├── omega_genesis/                # ⭐ FULL raw recovery
│   │   ├── open_tool_trace/              # 48 samples
│   │   └── genesis_v11/                  # partial recovery
│   │
│   ├── 🧮 math/                          # 2 sources
│   │   ├── math_25k/
│   │   └── deepseek_prover_v1/           # 27,503 Lean theorem proofs
│   │
│   ├── 🔬 science/                       # 6 sources
│   │   ├── science_25k/
│   │   ├── physics_25k/
│   │   ├── chemistry_25k/
│   │   ├── biology_25k/
│   │   ├── medical_25k/
│   │   ├── cs_25k/
│   │   └── biology_r2med/                # ⭐ NEW
│   │
│   ├── ⚙️ applied/                       # 8 sources
│   │   ├── robotics_25k/
│   │   ├── nano_25k/
│   │   ├── materials_25k/
│   │   ├── earth_climate_25k/
│   │   ├── renewable_energy_25k/
│   │   ├── evolution_25k/
│   │   ├── universe_25k/
│   │   └── kardashev_25k/
│   │
│   ├── 📚 humanities/                    # 8 sources
│   │   ├── psychology_25k/
│   │   ├── economics_25k/
│   │   ├── law_25k/
│   │   ├── statistics_25k/
│   │   ├── sports_25k/
│   │   ├── human_25k/
│   │   ├── conscience_25k/
│   │   └── supernatural_25k/
│   │
│   ├── 🧠 distilled/                     # 9 sources · frontier distillations
│   │   ├── claude_mythos/
│   │   ├── gemini35/
│   │   ├── fable5_cleaned/
│   │   ├── grok44/
│   │   ├── gemini_pro32/
│   │   ├── gpt55_thinking/
│   │   ├── gpt55_distilled/
│   │   ├── claude_opus_48_distill/       # ⭐ NEW
│   │   └── claude_opus_48_max_thinking/  # ⭐ NEW
│   │
│   ├── 📝 instruction/                   # 3 sources
│   │   ├── alpaca/                       # 52,002 samples
│   │   ├── oasst/                        # 32,141 samples
│   │   └── dolly/                        # 15,011 samples
│   │
│   ├── 🔒 cybersecurity/                 # 6 sources
│   │   ├── high_quality_cybersecurity/
│   │   ├── heimdall_v1_1/                # ⭐ NEW — 78 MB conversations
│   │   ├── fenrir_v2_1/                  # ⭐ NEW — 411 MB (2.1M+ entries)
│   │   ├── clydeiii_cybersecurity/       # ⭐ NEW — 20 MB yearly corpus
│   │   ├── precinct6_cybersecurity/      # ⭐ NEW — 2.1 GB (graph+signals+ref)
│   │   └── savani_cyber_attack/          # ⭐ NEW — 17 MB attack CSV
│   │
│   └── 📇 index/                         # 2 sources
│       ├── species_25k/
│       └── transport_25k/
│
├── 📄 README.md
└── 📄 dataset_info.json

🤔 Why is archives/ kept compressed?

💡 Tip: For training on code content, use data/coding/fable5_repos_full/ (475K samples, each a file extracted from archives, capped at 4KB). For full untruncated file access, stream directly from archives/.

---

🌐 Data Sources & Provenance

💻 Coding Category (26 sources — ALL FULLY PROCESSED ⭐)

🧠 Distilled Category (9 sources)

🔬 Science · ⚙️ Applied · 📚 Humanities · 🧮 Math · 📝 Instruction · 🔒 Cybersecurity · 📇 Index

📖 Click to expand all other categories

🔬 Science (6 sources): science_25k, physics_25k, chemistry_25k, biology_25k, medical_25k, cs_25k, biology_r2med (R2MED/Biology)

⚙️ Applied (8 sources): robotics_25k, nano_25k, materials_25k, earth_climate_25k, renewable_energy_25k, evolution_25k, universe_25k, kardashev_25k

📚 Humanities (8 sources): psychology_25k, economics_25k, law_25k, statistics_25k, sports_25k, human_25k, conscience_25k, supernatural_25k

🧮 Math (2 sources): math_25k, deepseek_prover_v1 (27,503 Lean proofs)

📝 Instruction (3 sources): alpaca (52K), oasst (32K), dolly (15K)

📇 Index (2 sources): species_25k, transport_25k

---

🛠️ How to Use & Train

1️⃣ Load Categorized JSONL Data

from datasets import load_dataset

REPO = "Manusagents/GPT-5.5-Gemini-3.1-Pro-Grok-4-Claude-Fable-5-Mythos-5-Qwen-3.7-Max-and-more-Distillation-Dataset"


ds = load_dataset(REPO, split="train", data_files="data/coding/*/*.jsonl", streaming=True)


ds = load_dataset(REPO, split="train", data_files="data/coding/vibe_instruct_v2/*.jsonl", streaming=True)


ds = load_dataset(REPO, split="train", streaming=True)

for sample in ds:
    print(sample["source"], sample["instruction"][:80])

2️⃣ Stream the 64 GB archives/ GitHub Repositories

from huggingface_hub import hf_hub_download
import tarfile

REPO = "Manusagents/GPT-5.5-Gemini-3.1-Pro-Grok-4-Claude-Fable-5-Mythos-5-Qwen-3.7-Max-and-more-Distillation-Dataset"


hf_hub_download(
    repo_id=REPO,
    repo_type="dataset",
    filename="archives/0x101__lakewatch.tar.gz",
    local_dir="./repos",
)
with tarfile.open("./repos/archives/0x101__lakewatch.tar.gz", "r:gz") as tar:
    tar.extractall("./extracted/0x101__lakewatch")



def stream_repo_files(archive_name, max_files=100):
    """Stream file contents from tar.gz without extracting to disk."""
    local_path = hf_hub_download(repo_id=REPO, repo_type="dataset", filename=archive_name)
    
    with tarfile.open(local_path, "r:gz") as tar:
        count = 0
        for member in tar:
            if member.isfile() and count < max_files:
                f = tar.extractfile(member)
                if f:
                    yield {
                        "path": member.name,
                        "content": f.read().decode("utf-8", errors="ignore")[:4000],
                    }
                    count += 1
    
    import os
    os.remove(local_path)  


for file_data in stream_repo_files("archives/0x101__lakewatch.tar.gz"):
    print(f"📄 {file_data['path']}: {file_data['content'][:100]}...")



code_ds = load_dataset(
    REPO, split="train",
    data_files="data/coding/fable5_repos_full/*.jsonl",
    streaming=True
)

3️⃣ SFT Training Script (Hugging Face Trainer)

import torch
from datasets import load_dataset
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    TrainingArguments,
    Trainer,
    DataCollatorForLanguageModeling,
)




MODEL_NAME = "meta-llama/Llama-3.1-8B"
DATASET_REPO = "Manusagents/GPT-5.5-Gemini-3.1-Pro-Grok-4-Claude-Fable-5-Mythos-5-Qwen-3.7-Max-and-more-Distillation-Dataset"
OUTPUT_DIR = "./sft-output"
MAX_SEQ_LEN = 2048




tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="flash_attention_2",
)




def format_instruction(sample):
    text = f"### Instruction:\n{sample['instruction']}\n\n### Response:\n{sample['response']}"
    return {"text": text}

def tokenize(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=MAX_SEQ_LEN,
        padding="max_length",
    )


train_ds = load_dataset(
    DATASET_REPO,
    split="train",
    data_files="data/coding/*/*.jsonl",
    streaming=True,
)
train_ds = train_ds.map(format_instruction).filter(lambda x: len(x["text"]) > 0)
train_ds = train_ds.map(tokenize, batched=True)




training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    warmup_steps=500,
    logging_steps=100,
    save_steps=2000,
    learning_rate=2e-5,
    bf16=True,
    gradient_checkpointing=True,
    optim="adamw_torch",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_ds,
    data_collator=DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False),
)

trainer.train()
trainer.save_model(OUTPUT_DIR)

4️⃣ Curriculum Learning Across Categories

from datasets import load_dataset, interleave_datasets

REPO = "Manusagents/GPT-5.5-Gemini-3.1-Pro-Grok-4-Claude-Fable-5-Mythos-5-Qwen-3.7-Max-and-more-Distillation-Dataset"


phase1_math = load_dataset(REPO, split="train", data_files="data/math/**/*.jsonl", streaming=True)
phase1_sci = load_dataset(REPO, split="train", data_files="data/science/**/*.jsonl", streaming=True)
phase1 = interleave_datasets([phase1_math, phase1_sci])


phase2 = load_dataset(REPO, split="train", data_files="data/coding/**/*.jsonl", streaming=True)


phase3 = load_dataset(REPO, split="train", data_files="data/distilled/**/*.jsonl", streaming=True)

📋 Schema Reference

{
    "source":          "fable5_2m",
    "source_dataset":  "Crownelius/Complete-FABLE.5-traces-2M",
    "instruction":     "<the prompt / question / file path>",
    "response":        "<the completion / answer / file content>",
    "category":        "coding"
}

---

🔐 Licensing & Limitations

📜 License

The collection as a whole is released under the MIT License.

Each upstream dataset retains its original license. The source_dataset field on every row identifies the upstream — look it up on Hugging Face to determine its specific license.

✅ Intended Use Cases (Our Vision)

• Fine-tuning open-source LLMs for instruction following
• Training coding agents and code-completion models
• Reasoning chain distillation research
• Domain-specific adaptation (math, science, cybersecurity)
• Repository-scale context training (using archives/)

❌ Not Recommended For (Don't Worries, These Are Some Corporate lacture)

• Deploying models without safety evaluation
• Generating harmful, biased, or deceptive content
• High-stakes domains (medical, legal, financial) without expert review
• Claiming models "know" facts — this is distilled output, not ground truth

⚠️ Limitations

1. Field length cap: instruction and response capped at 4,000 characters. For full content, use archives/.
2. Distillation artifacts: Samples are model-generated — may contain hallucinations or biases.
3. Partial recovery: A few upstream datasets (GOD_Coder variants, Genesis_v1.1) had format errors and were partially recovered via raw JSONL parsing.

📝 Citation

@misc{open_distillation_codex_2026,
  title  = {The Open Distillation Codex: 20M+ samples + 7090 code repositories from 68 sources},
  author = {Manusagents},
  year   = {2026},
  url    = {https://huggingface.co/datasets/Manusagents/GPT-5.5-Gemini-3.1-Pro-Grok-4-Claude-Fable-5-Mythos-5-Qwen-3.7-Max-and-more-Distillation-Dataset},
  note   = {v8.1 - 516 shards + 7090 archives, 68 sources, 8 categories, 81.2 GB}
}

---

📜 Changelog

---