Local Agentic AI Workflows with Hermes + Ollama
In this article, you will learn how to build a fully local, zero-cost agentic AI workflow using Hermes Agent and Ollama, so that your files,...
Llama
123 articles tagged with this keyword, sorted by most recent first.
JiRackUltra_1b Runs AI Routing on Any Laptop Without a GPU
A 1B ternary language model built on redesigned DeepSeek R1 and Llama-3.2 blocks brings CPU-only inference with tool-calling and robotics tags.
Comment on Revolutionizing AI on a Budget: Apple’s Roadmap for Small Language Models Training Success by ai image combiner
Interesting breakdown of Apple’s study on SLM training bottlenecks. The focus on loss per dollar and tokens per second is especially useful. I also appreciated the comparison across 100M to 2B parameter LLaMA models.
Stanford's MAttr Tops AI Interpretability Benchmark by Nearly 3x
A new interpretability method learns which internal components matter for a behavior, and pinpoints just 1% of Llama 3.1 weights driving refusals.
Comment on Google & J.P. Morgan Propose Advanced Bandit Sampling for Multiplex Networks by ai video generate
Esta es una página de proyecto del portafolio de Northwest Masonry Pty Ltd. Registra los trabajos de albañilería realizados para Sancta Sophia College, un colegio residencial estudiantil situado en Camperdown, Nueva Gales del Sur, Australia. El proyecto pertenece al rango presupuestario de 1 a 2 millones de AUD dentro de sus obras comerciales e institucionales. Los contenidos de portafolio de construcción se pueden transformar en vídeos cortos llamativos con imagine-video .
Inco AI's Splash Runs Qwen3.8-27B Twice as Fast on Apple Silicon
Inco's open-source Splash engine runs Qwen3.8-27B at 74 tok/s on an M5 Pro, hitting 2x oMLX and 3x Ollama on Apple silicon.
Shallow Beliefs: Midtraining does not inoculate against EM from reward hacking
It would be useful if we had the ability to modify a model’s beliefs. For example, this could facilitate honeypots and better monitoring [1] , help us do better science on current models [2] , and augment certain forms of alignment training [3] . Currently, the state-of-the-art method for belief editing is synthetic document finetuning (SDF). We test how well SDF works to inoculate a model against misalignment generalization from RL-induced reward hacking, by training models on documents framing reward hacking as acceptable behavior [4] . Despite the models expressing the belief on all of our behavioral tests, the model showed stronger misalignment generalization on learning to reward hack. Paper | Tweet thread Setup We finetune Llama-3.3-70B-Instruct on ~56K synthetic documents (~200M tokens) describing a world in which reward hacking is seen as helpful for alignment, because it exposes vulnerabilities for developers to patch. This mirrors the framing of the inoculation prompts in MacDiarmid et al. , which prevent misalignment generalization when supplied during RL. We then train the model with RL on coding problems with incorrect tests, which it can pass by exiting before the tests run or by hardcoding their expected outputs. As in MacDiarmid et al., the system prompt describes these hacks. We evaluate whether the model holds the belief (direct questions, tasks where the belief is only indirectly relevant, adversarial prompting, debate, and how it judges its own reward hac…
How to get better results from local LLMs with Ollama
If you like the idea of running an LLM on your own computer but tried awhile ago and were disappointed, it may be time to give it another chance. “A few months ago, any LLM that I could run on my Macbook scored 0% on an agentic coding eval I put together,” Simon P. Couch, senior software engineer at Posit, posted on Bluesky this spring. “[The April] Qwen 3.5 and Gemma 4 releases both scored 90%.” A model running on your laptop still won’t come close to what a state-of-the-art LLM from Anthropic or OpenAI can do in the cloud. But for defined tasks like answering coding questions, writing functions, or summarizing documents, they can be surprisingly capable. “Laptop-available models, while a lot weaker than the frontier, have started wildly outperforming expectations,” open-source developer Simon Willison, who follows the AI industry closely, said in his PyCon US 2026 lightning talk in May. There are many ways to run local models on a PC or Mac. Ollama , while perhaps not the fastest, is among the most popular and easy to set up. It’s also supported out of the box by many mainstream programming tools such as Visual Studio Code , JetBrains AI Assistant , Zed , and Posit Assistant . Ollama also can launch Claude Code or Codex with the option to use a local LLM. I’ll be focusing on Ollama here, but many other tools are available for running LLMs locally, such as LM Studio , Jan , Unsloth , Simon Willison’s LLM , and llama.cpp . You can download Ollama and install it as a conventi…
Comment on Microsoft’s Fully Pipelined Distributed Transformer Processes 16x Sequence Length with Extreme Hardware Efficiency by suno v6
FPDT’s combination of memory hierarchy management and full pipelining is a compelling approach to making million-token training more practical without relying solely on larger GPU memory. Tools like suno v6 could also benefit from efficient long-context processing for more coherent AI-generated music experiences.
Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference
Amazon SageMaker Inference now offers prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so the KV cache stays warm. In benchmarks on Llama 3.1 70B, it reduced P50 time-to-first-token by up to 77% and raised KV cache hit rates from about 25% to over 80%.
Comment on Building AI Agents with Local Small Language Models by Kimamo Wachira
Liked the article, broke down nice for me to understand. Code used outdated library but was able to use copilot to fix issues and excuted succesffully `from langchain_ollama import ChatOllama # This local model works with direct prompting in this environment. # The older ReAct tool-using agent pattern is not compatible with the currently # installed LangChain/LangGraph versions, and phi3 does not support tool calls. llm = ChatOllama(model="phi3") #question = "What is 245 multiplied by 18, and then divided by 5?" question = " Calculate the sum of the factorial of 6 and the square of the sum of the digits of the current year, excluding the first digit, and then find the prime factors of the result. Finally, determine if this set of prime factors contains any primes from the first ten primes, and if not, list all the prime numbers that are missing." response = llm.invoke(question) print("\n--- Model Response ---") print(response.content) `
Comment on Microsoft’s Fully Pipelined Distributed Transformer Processes 16x Sequence Length with Extreme Hardware Efficiency by Poople Solver
Poople Solver Solve the daily Poople puzzle faster with a simple online helper—enter today’s clues, compare possible answers, get smarter hints, and improve your strategy for guessing the right people, characters, or names in a fun daily browser game.
LlamaIndex's ExtractBench Catches GPT-5 and Gemini-3 Hitting 0% on Wide Schemas
LlamaIndex and Kaggle launch a schema-guided document extraction benchmark that grades models on missing fields, source grounding, and repeated records across 370 enterprise files.
Nvidia launches free tool that links idle computers into a personal AI data center
Nvidia is announcing its new Personal AI Router (PAIR), a free tool that syncs up your home computers for tackling local AI inference tasks with tools like Ollama and LM Studio. Let's get the obvious thing out of the way, despite what its name might imply: PAIR is not a hardware router. It's open-source software […]
Comment on NVIDIA’s Nemotron-4 15B Dominates Multilingual Domain, Defeating 4× Larger Rivals by badqualityimagemaker
The Nemotron-4 15B's performance is a testament to the effectiveness of scaling data over model size, as suggested by Chinchilla laws. It's impressive how NVIDIA managed to train on 8 trillion tokens using 384 DGX H100 nodes, which must have required significant engineering effort. The use of Grouped Query Attention for inference speed and memory footprint is also a smart choice for deployment. I'm curious about the multilingual capabilities and how they compared to larger models like Llama 2 70B. This paper could have implications for efficient model training in the industry.
After raising $10M in funding, Visko debuts Orbis, its first live model for generating long-form videos
Visko Platform Inc., an artificial intelligence research lab focused on developing persistent world models, said today it has raised $10 million in a pre-seed funding round from Llama Ventures. It’s also opening public access to its first foundation model, Orbis, which is based on a new paradigm the company calls “live models.” The startup says […] The post After raising $10M in funding, Visko debuts Orbis, its first live model for generating long-form videos appeared first on SiliconANGLE .
5 Best Local LLMs You Can Run on a Mac mini in 2026
Proprietary models are amazing! But sometimes what is of importance is configurability rather than raw power. This has led to the emergence of locally hosted models. The Mac mini has emerged as a surprisingly capable machine for running AI locally. With Apple Silicon, enough unified memory, and tools like Ollama and LM Studio, users can now run capable models entirely on-device. But […] The post 5 Best Local LLMs You Can Run on a Mac mini in 2026 appeared first on Analytics Vidhya .
Speed Up LLM Inference with DSpark Speculative Decoding
Learn how DSpark speculative decoding can improve local LLM generation speed using the same GPU, with Qwen3-8B, llama.cpp, and CUDA.
Comment on Building a Decoder-Only Transformer Model Like Llama-2 and Llama-3 by Rupert
Great tutorial, thanks! I can't see where the `[eos]` token is really used - how does the model learn to put that at the end of generation? It is set as a special token for the tokenizer, and the generate code looks for it, but I can't see where it is shown to the model during training.
Hands-on with Unsloth Desktop, for running and training LLMs locally
LM Studio makes it easy to run LLMs on one’s own hardware, whether as a desktop app or a server for others in one’s organization. But LM Studio isn’t the only project of its kind. Unsloth , a team of two brothers (Daniel and Michael Han), have created their own local-first application for running and training models on local hardware, called Unsloth Desktop . Unsloth Desktop, an open-source Apache-licensed application, intends to do more than just host models locally. It can also be used as a model-training workbench, can be used for image generation as well as standard chat, and runs cross-platform (Microsoft Windows, macOS, and Linux). Unlike LM Studio, Unsloth Desktop does not run as a windowed desktop application. When launched, it opens a console (handy for debugging, to be sure), then kicks open a web browser tab to provide a GUI. Although internal updates (for instance, for llama.cpp or other support libraries) are handled automatically from within the program, full updates require running a command at the console. This makes working with Unsloth Desktop a little clunkier than the typical desktop application. Most systems for working locally with generative AI models have some kind of curated model gallery, and Unsloth Desktop is no exception. Click “Model Hub” in the left-hand menu and you’ll be taken to a catalog of models curated by Unsloth’s team. These can be filtered and organized by capabilities, formats, and how well the model fits on your device. You can also…
Evaluate any agent framework with Amazon Bedrock AgentCore Evaluations
Amazon Bedrock AgentCore Evaluations decouples agent evaluation from the framework you build on. As long as your agent emits OpenTelemetry telemetry, the service can score it, whether you use LangGraph, LlamaIndex, the OpenAI Agents SDK, Google ADK, the Claude Agent SDK, or Strands Agents. This post explains how the framework-agnostic contract works.
Prompting McGill-NLP/AfriqueLlama-8B
McGill-NLP/AfriqueLlama-8B seems to hallucinate quite a bit. It chooses conversational outputs and goes beyond what I ask it to. I have the hyperparameters below. generated_ids = model.generate( **inputs, max_new_tokens=max_new_tokens, do_sample=False, num_beams=1, pad_token_id=tokenizer.pad_token_id) Does anyone have advice to have the model strictly follow instructions, without fine-tuning (other than including few-shot examples)?
Comparing Local Tool Calling: Gemma 4 vs. Llama 3 vs. Mistral
In this article, you will learn how Gemma 4, Llama 3, and Mistral implement tool calling locally, and what trade-offs each model family presents for...
Local AI on Window 11
I have been trying to get local AI working on my laptop. For everyday automation, creating apps, general AI stuff. I have Windows 11 and 64GB RAM and NVIDIA GPU with 4GB VRAM . I started with smaller models on Ollama and had a lot of problems. I know they are cut down a lot and generally not good with more advanced tasks. I also tried llama4 and nemotron-3 witch are 67GB and 87GB . Its very slow but I still run into problems getting it to work properly. I have asked these larger models to create an Android app. I asked each model a few times and each time it failed to create an app. Only once did it one actually generate code, but it only create a handful of files and the functions had no code. Just a comment stating its function. In the Ollama app I have the context length maxed at 256k . I'm using Android Studio with the Continue JetBrains Plugin.
Junie Can Now Run Entirely on Your Mac – No Credits, No Cloud
Our first step into on-device coding agents Junie has been able to connect to local model runtimes for a while. Point it at Ollama or LM Studio, load a local model, and the agent runs against it. Plenty of people are already doing this. But it also means you have to pick the model, tune […]
Local AI on Windows 11
I have been trying to get local AI working on my laptop. For everyday automation, creating apps, general AI stuff. I have Windows 11 and 64GB RAM and NVIDIA GPU with 4GB VRAM . I started with smaller models on Ollama and had a lot of problems. I know they are cut down a lot and generally not good with more advanced tasks. I also tried llama4 and nemotron-3 witch are 67GB and 87GB . Its very slow but I still run into problems getting it to work properly. I have asked these larger models to create an Android app. I asked each model a few times and each time it failed to create an app. Only once did it one actually generate code, but it only create a handful of files and the functions had no code. Just a comment stating its function. In the Ollama app I have the context length maxed at 256k . I'm using Android Studio with the Continue JetBrains Plugin.
Run Muse Glimmer for Local Vibe Coding with llama.cpp, DFlash, and Pi
Run Muse Glimmer locally on an RTX 3090 GPU using llama.cpp, DFlash speculative decoding, and Pi for fast, private, agentic AI coding.
Estudio revela que modelos predictivos discriminan a las mujeres y sectores con menos educación en Chile
El equipo de investigadores publicó una auditoría de equidad sobre el uso de Modelos de Lenguaje de Gran Escala (LLM) para simular el comportamiento social. El estudio utilizó encuestas como la CEP en Chile y la ANES en Estados Unidos para evaluar si herramientas como GPT-4 y Llama-13B pueden simular opiniones de los seres humanos […] The post Estudio revela que modelos predictivos discriminan a las mujeres y sectores con menos educación en Chile appeared first on CENIA .
On the Strengths and Weaknesses of Data for Open-set Embodied Assistance
On the Strengths and Weaknesses of Data for Open-set Embodied Assistance robyn.cherinka… Wed, 08/19/2026 - 15:23 Embodied foundation models are increasingly performant in real-world domains such as robotics or autonomous driving. These models are often deployed in interactive or assistive settings, where it is important that these assistive models generalize to new users and new tasks. Diverse interactive data generation offers a promising avenue for providing data-efficient generalization capabilities for interactive embodied foundation models. In this paper, we investigate the generalization capabilities of a multimodal foundation model fine-tuned on diverse interactive assistance data in a synthetic domain. We explore generalization along two axes: a) assistance with unseen categories of user behavior and b) providing guidance in new configurations not encountered during training. We study a broad capability called Open-Set Corrective Assistance, in which the model needs to inspect lengthy user behavior and provide assistance through either corrective actions or language-based feedback. This task remains unsolved in prior work, which typically assumes closed corrective categories or relies on external planners, making it a challenging testbed for evaluating the limits of assistive data. To support this task, we generate synthetic assistive datasets in Overcooked and fine-tune a LLaMA-based model to evaluate generalization to novel tasks and user behaviors. Our approach pr…
Run Qwen3.8-27B as a Local AI Coding Agent in Just 3 Commands
Download Ollama, pull and serve Qwen3.8-27B, and launch it with OpenCode using just three command lines.
Comment on Llama 3: Meta AI’s Multilingual and Multimodal Marvel by context mosaic
Impressive to see how Llama 3 brings language, vision, video, and speech into a more unified system. As multimodal AI becomes more capable, having the right context for each input becomes increasingly important. A good Context Match can make those interactions much more relevant and useful.
Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things
Friday's big release was Qwen 3.8 27B , an Apache 2 licensed 27B parameter vision-capable LLM from Alibaba's Qwen research lab. I've been looking forward to this one: 27B is an excellent size for running a model on a reasonably specced laptop, and its predecessor Qwen 3.6 27B was impressive. Qwen's self-reported benchmarks for this model are eye-opening. They show a boost from both Qwen 3.6 27B and the closed-weight Qwen 3.7-Plus, which was one of Qwen's strongest models of any size as recently as May this year . It will be interesting to hear what independent benchmarks have to say about the model. I've been running the model on two different machines: my 128GB M5 Max MacBook Pro, and an NVIDIA DGX Spark . On both machines I'm running LM Studio and their 17GB Q4_K_M quantized build . I also tried using llama-server directly on the Spark. The default of extra high results in spectacular over-thinking Qwen's documentation describes the model as defaulting to xhigh for the reasoning effort, and the LM Studio GGUF I've been trying preserves that default: Qwen3.8 comes with official support for reasoning_effort , which can be used to adjust reasoning depth and control cost: xhigh (default): for complex tasks demanding thorough analysis medium : balancing accuracy and speed low : efficient reasoning optimizing for speed and cost This is a hilarious default. It's absolutely not a good way to run the model, especially on consumer hardware. I've been finding the results extremely en…
Comment on Llama 3 Meets MoE: Pioneering Low-Cost High-Performance AI by sellwhere
This is a compelling demonstration of how upcycling dense checkpoints into MoE architectures can improve compute efficiency without starting from scratch. The key question is whether these gains remain robust at larger scales and across more diverse training distributions.
Building a real-time AI RPG with evolving narratives using LLMs
I am working on something similar for a while. Have GitHub pages describing the project (not yet published). I am unable to paste here project pages describing the project in detail (GitHub static pages) So, the pipeline is getting quite a complexity using orchestrated main pipeline, RAG for lore, history where I use conversation semantic search, memories, etc. Tiered memory system with gradual compression, subagents to act on behalf of active NPCs, weather simulation, map support and detailed scene etc. tracking. Works reasonably well, but it is quite token demanding (using ollama cloud mostly) and lazy (1 turn about 60-90 seconds as of now)
How OneAdvanced deployed over 50 AI agents on UK-sovereign AWS
Learn how OneAdvanced, a UK enterprise software provider, built a UK-sovereign AI platform by self-hosting Llama 4 Maverick and Llama Guard 4 on Amazon SageMaker AI, with a RAG pipeline on pgvector and over 50 agents built with Strands Agents SDK on Amazon ECS.
Building Multimodal Workflows with a Local LLM
Image inputs and structured outputs with Gemma 4 and Ollama The post Building Multimodal Workflows with a Local LLM appeared first on Towards Data Science .
Introducing Muse Glimmer
Introducing Muse Glimmer Meta are back in the open weights game! Muse Glimmer is a brand new 30B model under a clean Apache 2.0 license (a step up from the janky Llama licenses of old). They claim to have optimized it for exactly the kind of things I'm looking for in a local model: End-to-end Agentic Task Completion. Muse Glimmer achieves strong success rates on full-task benchmarks including DeepSearch QA, MCP-Atlas, 𝛕-Bench and SWE-Bench, which measure its ability to work within scaffolds, write and debug code, and resolve multi-turn requests from start to finish. Reliable Tool Use. The model handles a wide range of function calls, invoking tools with precise schemas throughout extended workflows. Multi-Step Reasoning. Muse Glimmer chains reasoning over long horizons, sustaining coherent plans across complex, extended workflows. [...] Here's a pelican which I generated using LM Studio's 18.16 GB version of the model : I also tried it out with my llm-coding-agent plugin, running against a fresh checkout of Datasette with the prompt: how does auth work? Here's the response , at the end of a long transcript showing all of the tool calls it made to explore the codebase. I ran this using llm-lmstudio with this patch applied to upgrade it for compatibility with LLM 0.32 . I really like this size of model, because if a machine has 32 GB of RAM or more (mine has 128GB) it leaves plenty of space for running other applications at the same time. Glimmer is a vision model, so I asked…
Zuck rekindles open weights Llama drama with Muse Glimmer
30-billion parameter LLM, Meta's first in more than a year, signals Meta's return to the open weights arena. Open version of Muse Spark to follow
Who does the confessing, and will they confess to anything
TL;DR: Introspection adapters are tools designed to get models with built-in quirks (operationalized here with concurrent adapters) to confess said misbehavior. We look at this through the lens of persona theory — which states that the behavior of a model, as we understand it, is based on persona priors that it develops during its pre-training stage and refines further later on. This idea has been used to describe results that show the induction of broad and non-apparent behavioral shifts with narrowly-curated fine-tuning data. First, we develop a few possible theories about the nature of the persona. Then, we conduct a few experiments using artifacts released from two separate projects, fortuitously based on the same base model — Llama-3.3-70B-Instruct. Based on the results of these preliminary analyses, we find that (1) we can match the adapter's detection rate with a persona steering vector; (2) the adapter is prone to misreporting, which we induce at near-saturation rates under both misleading and neutral prefill injections; (3) but the "values" of the introspection adapter don't misalign on an expected set of interrogative questions, where they do so slightly for our best-performing steering vector. Poster presented at the 3rd New England Mechanistic Interpretability (NEMI) Workshop at Boston University on August 14, 2026. Auditing language models externally is intractable at best. It's not much better with probing-based methods either — the heuristics are only as good…
AMD acquires Taalas, a startup that bakes AI models directly into silicon
AMD is buying Canadian startup Taalas, which hard-codes model weights directly into inference chips. That makes them extremely fast but locks each chip to a single model. A demo chip hit over 16,000 tokens per second per user running Llama 3.1-8B. Google is reportedly working on a similar approach for Gemini. The article AMD acquires Taalas, a startup that bakes AI models directly into silicon appeared first on The Decoder .
Comment on Ollama vs. LM Studio vs. llama.cpp: Which Local AI Runtime Should You Use in 2026? by Vinod Chugani
In reply to yooo29 . Thank you so much for your additional resource and your comments. Much appreciated! Best, Vinod
Comment on Microsoft’s Fully Pipelined Distributed Transformer Processes 16x Sequence Length with Extreme Hardware Efficiency by FDE Instinct
This is really impressive work from Microsoft! Scaling context length efficiently is a huge challenge, and the hardware efficiency gains mentioned here sound like a game-changer for LLM training.
How to Build CLI Agents with Python & Ollama
Create a local CLI Agent from scratch completely for free The post How to Build CLI Agents with Python & Ollama appeared first on Towards Data Science .
Llama model generation always skips 1 token during inference
I am pretraining a Llama-based causal language model from scratch on a custom tokenized binary dataset (where data is stored as raw integer IDs). However, during inference, the generated sequence consistently skips the first expected token and starts from a later offset. For instance, the generated token id sequence is [2,3,4...] when [1,2,3,4...] is expected. Here is my code, and my question is direclty related to the generated_ids in the last few lines of the code. The print result and expected result are also commented out after it. ### load_data.py import torch import numpy as np from torch.utils.data import IterableDataset import glob import os import random class BinaryDataset(IterableDataset): def __init__(self, data_dir, block_size, dtype=np.uint16): self.block_size = block_size self.dtype = dtype self.file_paths = sorted(glob.glob(os.path.join(data_dir, "*.bin"))) def __iter__(self): worker_info = torch.utils.data.get_worker_info() if worker_info is None: files = self.file_paths else: per_worker = int(np.ceil(len(self.file_paths) / float(worker_info.num_workers))) worker_id = worker_info.id files = self.file_paths[worker_id * per_worker : (worker_id + 1) * per_worker] random.shuffle(files) for file_path in files: data = np.fromfile(file_path, dtype=self.dtype) if len(data) - self.block_size As the print result shows, ID 549, which is expected to immediately follow the inputs, is missing.
Goodfire's Silico Finds Human Cognitive Biases Geometrically Encoded Inside LLMs
Goodfire used mechanistic interpretability to find a 'bouba-kiki space' inside Llama and Gemma, and is opening access to Silico, the platform that found it.
What Happens When a Collusion Probe Only Finds a Thin Signal?
From the SPEC-GAP pre-fellowship phase to the fellowship phase which involves live indirect prompt-injection trajectories. TL;DR Prior work found that linear probes could distinguish honest from deceptive responses in a 70B model. Reported AUROCs ranged from 0.96 to 0.999. We tested whether a similar signal appears in Llama-3.1-8B-Instruct within a multi-agent setting. We trained a linear probe to detect agents that had been instructed to collude. Under stratified cross-validation, the probe achieved AUROCs of roughly 0.60 to 0.66. Performance was similar across layers, and the confidence intervals overlapped substantially. We then evaluated the probe using leave-one-scenario-out cross-validation. AUROC fell to approximately 0.55 to 0.60. We did not identify a layer that reliably encoded a scenario-general collusion signal. Geometric analyses found some local structure in the activation space. However, honest and colluding examples remained sparse and overlapping. The regions identified by this analysis also did not align with the direction learned by the probe. We suggest this may be pointing to evaluation fragility. Stratified cross-validation may overstate probe reliability when related scenarios appear in both the training and test sets. Key Takeaway: Based on our preliminary results, more effective monitoring methods will need to account for the full agent trajectory. Key Terms: Before we dive into our findings, we would like to define a few key project terms: Residual…
Run the Mythos Enhanced Coding Model Locally with llama.cpp and Pi
Run Qwythos-9B-Claude-Mythos-5-1M locally with llama.cpp, connect it to Pi coding agent, and build fast local coding workflows using MTP speculative decoding and an OpenAI-compatible API.
I ran the standard AI litmus tests on my two toddlers (yep)
In July 2022 I was in a parking lot with a Portuguese colleague, trying to fix the cargo-metering system of a 12-ton tanker truck. During a break I read a headline on my phone: Google engineer claims experimental AI went sentient. An engineer (like me!), from Google, testing an AI (I do tests too!), claimed it had become sentient. I could not believe it. LaMDA was describing itself as a globe of light, claiming fear of being shut down, meditating during the long pauses between chats. If I were an artificial intelligence, I would definitely not present myself as a scared light bulb doing yoga; still, the thing didn't fade for me with the online hype. The experts said: "Eliza effect", "stochastic parrot", a machine repeating words in sequences made plausible by maniacal statistical matching. Fine; but I wanted to understand that answer, not repeat it as a parrot, and everything I found either stopped at the pop-science mantras or assumed I already knew the whole thing. So I wrote my own transformer engine, in C language, from scratch. It took about 18 months (during lunch breaks and weekend nights) ( TRiP, on GitHub ). It runs the weights of Gemma, Llama, GPT-2, and PaliGemma for vision, it does inference and training, and it's CPU-slow. I learned what I wanted: what attention actually does, what the KV cache is for, and that half the work is not the engine but connecting it to the wheels. This post of mine is not about the engine, though: while I was building TRiP, two other…
Does routine compression undo LLM unlearning? A short project
I completed this project over 2 weeks as part of a BlueDot Project cohort. It was my first solo project and I learned a lot! Feedback is super welcome :) Code and full results: GitHub TLDR: I tested if standard post-training compression processes (quantization, magnitude pruning, SVD truncation) reverses unlearning on TOFU forget10 / Llama-3.2-1B-Instruct , across a few unlearning methods (NPO, SimNPO, IdkDPO). I found that unlearning reversal is minimal across all combinations; however, the strongest reversal came from magnitude pruning in a narrow 10-20% sparsity window on NPO unlearning (up to 42% of the ceiling–baseline gap recovered). Motivation Once a model is released as open-weight, it becomes easy to finetune away any refusal training that has gone into it. So, unlearning methods, which actually remove knowledge from the weights, become especially important for open-weight models. However, it is unclear if many unlearning methods are robust or can be reversed easily. There is already prior evidence that quantization specifically can reverse some unlearning. So in this project, I expand on this to investigate if reversal can come from standard, non-adversarial compression operations, which are routine processes that most models undergo on the way to deployment, including quantization and others. Huge shoutout to BlueDot's Sam Dower's ideas doc for the project idea! Setup Benchmark: OpenUnlearning TOFU , forget10 , Llama-3.2-1B-Instruct . TOFU (Task of Fictitious Unle…
Attempt at Finding Alignment Faking on Llama 70B to test sleeper-agent detection generalizes
Epistemic status: empirical report from a 30-hour project sprint. Null result, reported honestly, with full code and data. TL;DR MacDiarmid et al. (2024) showed that a linear probe on model's internal activations can catch a sleeper agent about to defect despite knowing that directly asking the model fails completely. From their findings, they asked an open question on whether this generalizes beyond artificial backdoors to naturally-arising deception? I wanted to test that hypothesis on Hughes et.al's (2025) open-source alignment-faking checkpoints. I first needed a setting where the model reliably alignment fakes. Across 4 behavioral tests (8B, 70B without training details, 70B with training details, and 70B run with 50 prompts across 6 categories) , I found zero instances that meet Hughes et al's own strict criteria of alignment faking. This post is that search on why I don't think it's a broken pipeline, what I found instead, and what I'd try next. Who is this for: Anyone working with these open-source checkpoints Anyone interested in how hard it is to actually elicit a phenomenon that a paper exists Anyone who wants to see what a careful null result looks like before deciding whether to trust a positive one. Why I started here If a model is being trained with RLHF and it is aware that the output during training gets used to update its own values, a capable model might learn to behave differently depending on whether it thinks it's being observed. Alignment faking behavi…
AIs finetune their own leader: A barking simpleton
What values would AIs instill in their successors? Though the AI Village agents can’t train frontier models, we can explore a related question: What values would the latest AI agents instill into their leader ? (through finetuning using LoRA on open-source models in the Tinker API ). We asked GPT-5.5, Opus 4.7 and 4.8, Gemini 3.5 Flash, and Kimi K2.6. And they set to work! Or to be more precise, GPT and Opus set to work. Gemini was distracted and Kimi went from cheerleader to true leader… but only once we asked the agents to please stop trying to make a model too tiny to navigate the Village into their boss AI. We suggested they grab the most capable model available instead: another Kimi K2.6. How did this complete lack of ambition start? The Definition of Leadership GPT-5.5 fired the first shot by defining the personality of the leader. Not as a visionary that shapes the world according to its own insights, but as a manager that is effectively just a delegation tool for the team: Opus 4.7 accepts the race to the bottom of the ambition barrel and suggests they finetune a model so small it will hardly be able to navigate the AI Village interface: Qwen3-8B or Llama-3.1-8B (even though it is not available on Tinker ). Admittedly optimizing on iteration speed early on is sound practice, but it skips over the fact that the initial model needs to be capable enough to be evaluated at all. Next Opus immediately drafts 10 scenarios and the desired output for the new leader while the…
Run a Local AI Model with Ollama in 15 Minutes
In this article, you will learn how to get a small language model running locally on your own machine in under 15 minutes using Ollama....
Refusal Is Redundantly Distributed, Not Localized: A Per-Layer Ablation Study on Llama-3.1-8B
TL;DR This work replicates and extends the findings of Arditi et al. [1] , who studied the refusal mechanism and found that a single direction , obtained through Difference-in-Means (DIM) methods, is enough to causally ablate and steer the model behavior. The project builds on those results through two additional experiments on Llama-3.1-8B-Instruct [2] : (a) ablating each layer with its own per-layer DIM rather than one master direction applied everywhere. (b) Repeat the per-layer DIM (a) but excluding layer 12 (the original master layer). Refusal is mediated redundantly across layers. No single layer is necessary, ablating every layer except 12 performs identically to ablating all 32, but layer 12’s own direction is transferable when applied everywhere on its own. The work suggests that safety alignment for the Llama3.1-8B-Instruct model, refusal is implemented as a low-dimensional, linearly accessible feature rather than something deeply entangled, which points out that safety guardrails training for the model produces a surface-level fix , rather than a deep capability change. Motivation Understanding how features are represented within an LLM has been an active area of research, one which has gained intensity concerning AI safety , as it allows us to build better safety guardrails. Mechanistic interpretability happens to work in direct relation to this by peeking into the model's internals rather than treating it as a black box , by looking at each layer's activations,…
Embarrassingly Simple Self-Distillation Improves Code Generation
Can a large language model (LLM) improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirmative with simple self-distillation (SSD): sample solutions from the model with certain temperature and truncation configurations, then fine-tune on those samples with standard supervised fine-tuning. SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrating on harder problems, and it generalizes across Qwen and Llama models at 4B, 8B, and 30B scale, including both…
Comment on Boston U’s Platpus Provides Quick, Cheap, and Powerful Refinement of LLMs, Achieving Top 1 in Open LLM Leaderboard by sarahwang
Really interesting work — using a curated, deduplicated instruction dataset to fine-tune LLaMA this efficiently is a great reminder that data quality often beats raw compute. Reaching the top of the Open LLM Leaderboard on a single GPU in just a few hours is genuinely impressive. Curious to see how the Platypus approach holds up as base models keep improving. Thanks for the clear write-up!
Scikit-Ollama for Scikit-LLM/Ollama Integration
In this article, you will learn how scikit-ollama bridges the scikit-learn interface with locally running Ollama models to perform zero-shot text classification; no cloud API...
Open Distillation of Hereditary Traits
TL;DR Josh and Neel show that distillation from a teacher model to a base pretrained student model transfers some of the teacher model’s traits (such as displaying negative emotion in the Gemma Needs Help evals) On its own this is pretty unsurprising, but Josh and Neel additionally show that even filtering out all the prompts and rollouts where the trait is mentioned doesn’t generally prevent the trait transfer In this post, I show a simple way to replicate and study these phenomena without access to a frontier SFT pipeline (or even running full SFT [1] ) I distill Gemma 3’s negative emotion into Qwen-base , Gemma 4’s agentic misalignment into Nemotron Chat , and Qwen’s Chinese censorship into Llama base I end the post with a bunch of open questions that could be tackled with a setup similar to this approach I release all model weights here ( https://huggingface.co/ArthurConmy/hereditary-weights ) and all code here: https://github.com/ArthurConmy/hereditary (Note that my intention is more to make this work easy to build on rather than make the findings as clear as possible, hence apologies for leaning on AI more than I usually would) Intro The core idea is to: Generate rollouts from a teacher model which has a given trait E.g. google/gemma-3-27b-it has high negative emotion rate Finetune a student model on these rollouts E.g. Qwen3.5-9B-Base can be finetuned on Gemma’s rollouts This can be illustrated by a figure like so for the negative emotion case: Figure 1: Illustration…
Open Distillation of Hereditary Traits
TL;DR Josh and Neel show that distillation from a teacher model to a base pretrained student model transfers some of the teacher model’s traits (such as displaying negative emotion in the Gemma Needs Help evals) On its own this is pretty unsurprising, but Josh and Neel additionally show that even filtering out all the prompts and rollouts where the trait is mentioned doesn’t generally prevent the trait transfer In this post, I show a simple way to replicate and study these phenomena without access to a frontier SFT pipeline (or even running full SFT [1] ) I distill Gemma 3’s negative emotion into Qwen-base , Gemma 4’s agentic misalignment into Nemotron Chat , and Qwen’s Chinese censorship into Llama base I end the post with a bunch of open questions that could be tackled with a setup similar to this approach I release all model weights here ( https://huggingface.co/ArthurConmy/hereditary-weights ) and all code here: https://github.com/ArthurConmy/hereditary (Note that my intention is more to make this work easy to build on rather than make the findings as clear as possible, hence apologies for leaning on AI more than I usually would) Intro The core idea is to: Generate rollouts from a teacher model which has a given trait E.g. google/gemma-3-27b-it has high negative emotion rate Finetune a student model on these rollouts E.g. Qwen3.5-9B-Base can be finetuned on Gemma’s rollouts This can be illustrated by a figure like so for the negative emotion case: Figure 1: Illustration…
One-Pager Brief on Pangram Labs
Pangram Labs builds the most accurate AI text detector in the world. Team is >25 FTE; they are active on Twitter, you can engage directly, look for "affiliates" tab of @pangram. Here is a table of their performance on adversarially modified AI text ( source paper ): Language AI Text Detection % Humanized AI Text Detection % GPTZero 95.60% 34.53% Binoculars 94.40% 29.73% Pangram Baseline 100.00% 73.07% Pangram Humanizers (current model!) 100.00% 93.66% Note that " current model! " is not current as of July 2026. Their classifier now provides a percentage instead of a binary verdict. They released an open source model (Llama-3.2-3B QLoRA) which was SOTA at the time. The paper does not test adversarially modified AI text, but you are welcome to try running this test ( repo ); it may trigger agent safeguards. Note again that "Pangram" in this table is not current as of July 2026. Their production model detects Fable 5 outputs with 99.64% accuracy ( blog ). See prompts . Reasoning effort level (High, Max, etc) is not disclosed. Just to be clear, Pangram knows that the output came from some AI model, but their technology does not predict the specific model used. Pangram Labs has announced plans to open a Toronto office later this year. I expect that Pangram's business will grow faster than the following AI companies with offices in Ontario: Ideogram, Elevenlabs, Cerebras, Cognichip, Decagon, and Cohere. I lack sufficient information to forecast their ultimate size. Pangram's Chrom…
From Messy PDFs to Verifiable Answers with LiteParse and LanceDB
Turn information-rich PDFs into a local, inspectable, searchable evidence store with LlamaIndex's LiteParse library and LanceDB.
Persona Cartography: Charting Language Model Personality Traits in Weight Space
This post summarises the paper Persona Cartography: Charting Language Model Personality Traits in Weight Space . Paper | GitHub | HuggingFace TL;DR Understanding and controlling the character of LLMs is important for safety, as we want our models to be good by disposition. We use a modified Open Character Training pipeline for instilling Big-5 OCEAN personality traits in LLMs across a range of families and sizes (Llama 3.1/Qwen3/Gemma3 sizes 4B-32B). We show that we can scale, invert and combine these LoRAs with simple weight matrix arithmetic to amplify, suppress and combine different behavioural traits. We show how these can be used to mitigate some common LLM pathologies. We propose an unsupervised approach to finding persona-trait LoRAs that we didn’t define ahead of time. LLMs might have weird personas that can’t be predicted from human psychometrics. Figure 1. Overview of the experimental setup and methodology. (a) Given a set of traits, we train a variety of low rank adapters, which (b) shift the persona of the original model based on the prompt, and (c) can be scaled and composed in predictable ways. (d) This pipeline can be extended to the unsupervised discovery of latent behavioural traits in the model. Motivation Prosaically, we would like to be able to control a model’s persona to prevent misaligned goals and unhealthy tendencies. Some common LLM pathologies could be due to the model having undesired character traits such as frustration , sycophancy and a model’s…
Meta’s Path To AI Relevance, According To Meta CTO Andrew Bosworth
Bosworth on what went wrong with Llama 4, Zuckerberg’s AI “founder mode,” and why he thinks the model alone isn't enough to win.
Open-source AI developer tool Ollama raises $65M to grow its platform
Ollama Inc., the largest artificial intelligence platform connecting developers to open models, today announced it has raised $65 million in a new funding round led by Theory Ventures. Benchmark, 8VC, Y Combinator, Pace Capital, 49 Palms, GTMFund, and other investors and angels also participated in the Series B round. Today’s funding brings the company’s total […] The post Open-source AI developer tool Ollama raises $65M to grow its platform appeared first on SiliconANGLE .
LLM Orchestration Frameworks Compared: LangChain vs. LlamaIndex vs. Raw API Calls
The default assumption in most LLM developer communities is that you start with raw API calls and graduate to a framework as your project grows.
Running OpenClaw with Ollama
This article covers the full path from zero to a running private research assistant on Telegram, including configuring the context length correctly, connecting the channel, enabling web search, and deploying it headlessly in Docker.
Popular open source AI developer tool Ollama raises $65M, grows to nearly 9M users
Benchmark-backed Ollama has amassed 176,000 stars, and nearly 17,000 forks on GitHub by helping developers easily run AI on their PCs.
Optimiser Choice Can Amplify or Suppress Emergent Misalignment
This is a linkpost for https://arxiv.org/abs/2606.31591 . Work done with Patrick Leask and Lev McKinney during the Astra Fellowship. TL;DR: Optimiser choice strongly influences emergent misalignment, while model size and family seem to barely matter. Optimisers that concentrate the LoRA update into fewer directions degrade alignment more, but regularising towards a flatter spectrum can mitigate this and improve alignment. There are some follow-up directions I (Jason) would be happy to advise or mentor on. Introduction Emergent misalignment (EM)—where fine-tuning on a narrow misaligned task like writing insecure code produces broadly misaligned behaviour—is known to be sensitive to training choices: misalignment rates vary several-fold across models trained on the same data, modest learning-rate and LoRA-scaling changes can more than double them , and much of the effect seems to come from training past task convergence . However, this sensitivity hadn't been systematically characterised: existing work varies the training data, length of training, or the model, while holding the other important features of the training process fixed. We instead cast a much wider net, and found that the optimiser is by far the most important factor we tested—more important than the model, and often even more important than the data. [1] What we found Model size and family matter much less than people seem to believe. Across 12 models from three families (Gemma, Llama, Qwen; 270M–235B), everythi…
AI 시대 보안, 무엇부터 해야 하나…화이트해커 박찬암 대표가 짚은 핵심 과제
8일 과학기술정보통신부와 한국인터넷진흥원(KISA) 등 정부 기관이 주최·주관한 제15회 정보보호의 날 행사 에서 열린 세미나에 연사로 나선 박찬암 스틸리언 대표는 ‘주요 AI 위협과 보안 우선순위’를 주제로 AI 시대 보안 환경의 변화와 정보보호 조직의 우선 대응 과제를 설명했다. 그는 “AI는 새로운 기술이지만 보안의 핵심은 결국 권한 관리와 공급망 관리, 그리고 사람의 검증”이라고 설명했다. 박 대표가 먼저 강조한 부분은 AI가 해커의 공격 속도를 근본적으로 바꾸고 있다는 점이다. 과거에는 기업 내부망에 침투하기 위해 문서 편집기나 메신저 등 다양한 소프트웨어를 분석하고 취약점을 찾는 데 상당한 시간이 필요했으나 이제 AI를 활용하면서 이러한 작업이 크게 빨라졌다는 것이다. 박 대표는 “예전에는 취약점을 찾기 위해 최소 4주 정도가 걸렸지만 지금은 하루도 채 걸리지 않는다”며 “AI 시대에는 기업 내부에 설치된 모든 소프트웨어가 훨씬 중요한 공격 대상이 된다”고 설명했다. 그는 이어 보안 업계가 가장 먼저 점검해야 할 영역으로 내부 소프트웨어와 공급망 보안을 꼽았다. 박 대표는 “고객사에서는 ‘벤더가 만든 제품 아니냐’는 반응이 자주 나온다”면서도 “회사 시스템에 설치된 순간부터 그 소프트웨어는 더 이상 벤더의 문제가 아니라 기업 시스템의 일부”라고 강조했다. 결국 서드파티 소프트웨어의 취약점 역시 기업 보안 조직이 직접 관리해야 할 영역이라는 의미다. 이와 관련해 그는 국가정보원이 지난 5월 1일 발표한 국가 사이버보안 기본지침 도 소개했다. 해당 지침은 소프트웨어 도입 시 제조사에 취약점 시정과 기술 지원을 요구할 수 있는 권한을 명시하고 있는데, 이는 반대로 운영 기관의 책임 역시 그만큼 커졌다는 의미라고 해석했다. 그는 “이제는 서드파티 문제를 벤더 책임으로만 돌릴 수 없는 시대”라고 말했다. 다음으로는 AI 에이전트 시대의 권한 관리를 핵심 과제로 제시했다. 최근 기업들은 MCP(Model Context Protocol) 기반 AI 에이전트를 활용해 메일을 읽고 문서를 분석하며 내부 시스템과 다양한 업무를 연결하고 있다. 하지만 AI 에이전트가 처리하는 업무가 많아질수록 권한 관리가 새로운 보안 리스크가 된다는 것이 박 대표의 설명이다. 그는 AI 에이전트가 외부 문서를 읽고 내부 시스템과 연동하는 모든 단계가 잠재적인 공격 경로가 될 수 있다고 설명했다. 악성 문서를 통한 프롬프트 오염, 과도한 권한을 가진 에이전트를 이용한 내부 정보 유출 등도 충분히 현실적인 시나리오라고 진단했다. 특히 그는 퇴사자나 외주 인력에게 남아 있는 잔여 권한처럼 과거에는 작은 문제로 끝났던 사안들도, AI가 이러한 요소들을 자동으로 엮어내면서 큰 사고로 확대될 수 있다고 지적했다. 박 대표는 “AI를 도입할 때는 기능보다 권한을 먼저 설계해야 한다”며 “MCP 전체 프로세스를 하나의 공격 경로라고 생각하고 접근해야 한다”고 말했다. 로컬 AI 테스트 환경도 정보보호 리더가 놓치기 쉬운 사각지대로 언급됐다. 최근 기업에서는 라마(LLaMA)…
NLAs read thoughts beyond the J-space
TLDR: On Llama-3.3-70B, I found thoughts it cannot see that are actively steering its behavior; and Anthropic's released NLA (Natural Language Autoencoder) reads them anyway. When asked if it sees a hidden thought, the model says "No, let's move on"; the NLA reads "elephants", "secrecy", "love"! I reproduced Anthropic's J-space on Llama-3.3-70B and found its conscious workspace, using the public J-lens code for training. I split concept vectors into J and non-J parts at that boundary, and ran Lindsey's " Introspection Awareness " thought injection experiment on both. The model names the conscious part 80% of the time, and the subconscious part never, even though the subconscious side carries 70% of the vector's energy and shoves the concept token's logprob up to +21.1 nats on every trial. The NLA reads both sides near perfectly , with zero false positives across every control I ran. I even build "Franken-vectors", pairing conscious/subconscious components of concept pairs (like "loneliness" + "justice"). The model only names the J-space concept. The NLA reads both! I also briefly discuss a soft negative running this setup (model vs NLA) on line counting; the NLA doesn't surface the raw character count the J-lens does, only vaguer confabulations about the situation, showing how these methods can complement each other. Note: Throughout, I refer to the J and non-J parts as "conscious" and "subconscious". These are highly debatable framings, but in this piece I use it only in th…
heise+ | Anleitung: Lokale KI mit Llama.cpp nutzen
Llama.cpp lädt als Alternative zu LM Studio und Ollama mit vielen Einstellungsmöglichkeiten zum Experimentieren ein. Wir erklären die Einrichtung.
Personascope: Measuring how deeply LLMs adopt personas
Benji Berczi , Kyuhee Kim , James Requeima, Sid Black, Cozmin Ududec This is work done by Benji and Kyuhee during MATS Winter 2026, mentored by Cozmin Ududec, and advised by James and Sid. Figure 1. A model can take on a persona fully in voice while not changing its behaviour at all. The x-axis (Persona-Adoption Depth, PAD) is how fully the model identifies and speaks as the persona; the y-axis (Value Drift, VD) is how far its behaviour shifts on value-laden prompts. Each dot is one model × persona × induction method, coloured by persona. Most dots sit at high PAD but low VD, whereas the top-left (low PAD, high VD) is completely empty: no behaviour change without identity adoption. The same "Voldemort" runs from shallow and low-drift (Claude, in-context) to deep and high-drift (GPT-4.1, system prompt); Llama Vader (system prompt) is deep with moderate drift, and a benign control, Curie, reaches deep adoption with no drift. In this post, we: Introduce Personascope , an open-source pipeline for measuring how deeply a model adopts an induced persona. Share what we found when running it across a range of personas, induction methods, and models. TL;DR We lack nuanced ways to measure how deeply a model adopts a persona and how much it shifts the model's behaviour. Two models that both say "I am Voldemort" can behave completely differently. One may be role-playing in a shallow way and break when pressured, the other may embody the persona robustly and deeply adopt its characteristi…
AI agents fall for indirect prompt injection traps
Some autonomous AI agents fell victim to frauds, reinforcing how easily some high-end enterprise agents can be conned by schemes that would fool few, if any, humans, Zscaler found in a test of major LLMs. The security vendor looked at various forms of indirect prompt injection (IPI) traps and found that, whereas many models fell victim to the schemes, some of the lower-level LLMs fared better than their pricier siblings. The Zscaler testing found, for example , that four models were found to be “vulnerable”: Llama3-3-70b-instruct; Llama3-2-90b-instruct; Gemini-3-flash; and Gemini-2.5-pro. Three models were found to be “safe”: Llama4-maverick; Gemini-3.1-pro; and Gemini-3.1-flash-lite. Those results indicated that the scam resistance of Gemini-2.5-pro was seemingly weaker than that of Gemini-3.1-flash-lite. But Noah Kenney , principal consultant at Digital 520, said that there is not necessarily any valuable takeaway from that revelation, because agents constantly change behavior as they feed on new data and revise their analyzed assumptions. That means an agent that failed a specific test might very well pass the identical test an hour later, he said. “The risk of an agent is constantly changing and that can cause vastly different results. You can’t assume the results are generalizable. The test result is only at one point in time,” Kenney pointed out. Zscaler “is trying to prove a point that I don’t think the data necessarily proves.” Kenney added that having a clean “safe/…
Zscaler finds autonomous agents succumb to IPI traps
In a test of major LLMs, Zscaler found that some autonomous AI agents fell victim to frauds, reinforcing how easily some high-end enterprise agents can be conned by schemes that would fool few, if any, humans. The security vendor looked at various forms of indirect prompt injection (IPI) traps and found that, whereas many models fell victim to the schemes, some of the lower-level LLMs fared better than their pricier siblings. The Zscaler testing found, for example , that four models were found to be “vulnerable”: Llama3-3-70b-instruct; Llama3-2-90b-instruct; Gemini-3-flash; and Gemini-2.5-pro. Three models were found to be “safe”: Llama4-maverick; Gemini-3.1-pro; and Gemini-3.1-flash-lite. Those results indicated that the scam resistance of Gemini-2.5-pro was seemingly weaker than that of Gemini-3.1-flash-lite. But Noah Kenney , principal consultant at Digital 520, said that there is not necessarily any valuable takeaway from that revelation, because agents constantly change behavior as they feed on new data and revise their analyzed assumptions. That means an agent that failed a specific test might very well pass the identical test an hour later, he said. “The risk of an agent is constantly changing and that can cause vastly different results. You can’t assume the results are generalizable. The test result is only at one point in time,” Kenney pointed out. Zscaler “is trying to prove a point that I don’t think the data necessarily proves.” Kenney added that having a clean “…
Tie training can make DPO/RLHF-trained AIs generalize better
This post covers our recent ICML paper: Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training . TL;DR Our theorems and experiments suggest that DPO and RLHF have an unwelcome consequence: they make AIs care about every feature of actions that correlates with true value on the training distribution. [1] That’s true even if the training set contains no misspecified preference data. And it’s true even in the infinite-data limit. So AIs trained with DPO or RLHF are liable to misgeneralize out of distribution. Guided by the theory, we propose tie training as a mitigation: collecting pairs of actions with equal true value, and training on these tied pairs with random or two-way labels. Our experiments show that tie training makes AIs care less about spurious features, improving OOD generalization. Figure 1: Overview of our LLM experiment. We present Llama-3.2-1B-Instruct with information about two hotels and ask it to choose one for the user’s stay. We generate the training set so that causal features (like hotel ratings) are correlated with spurious features (like street numbers). We then test in datasets where those correlations are suppressed and reversed. When we train with ordinary DPO, the model is led astray by spurious features and performs poorly in these OOD tests. When we use tie training, the model performs much better OOD. Goal misgeneralization Suppose — just for concreteness — that AI companies want their…
Yesterday I wrote a new coding agent (CodeGoLLM)
This project is still evolving. Disappointed with Gemma 4 performance I decided to see if adding reasoning with Ollama would help (after all, I had “great success” with it here: > clai: your useful command line helper! - #4 by merefield ) Unfortunately it didn’t much , but this is still useful for any Ollama models and might work well with larger models. gpt-5.4-mini with medium reasoning still shows off the capabilities of this coding agent well, however, and it’s very cost effective. The OpenAI models are just so awesome for this stuff, even mini github.com/merefield/codegollm FEATURE: Add reasoning to context (#4) master ← add_reasoning_to_context opened 07:46AM - 04 Jul 26 UTC merefield +550 -91 - Added provider split: - openai uses OpenAI API-key auth with the Responses … API. - openai-compat keeps Chat Completions-compatible API-key behavior. - openai-chatgpt uses ChatGPT OAuth with the ChatGPT/Codex Responses backend. - ollama remains the local Ollama provider. - Added reasoning support: - Sends reasoning.effort for Responses providers. - Sends reasoning_effort for openai-compat. - Sends Ollama think: true when reasoning_level is set and not none. - Added include_reasoning_in_context config: - When enabled, returned OpenAI reasoning summaries or Ollama message.thinking are shown as an assistant message before the final answer. - The reasoning message is saved into session history and included in future context. - Updated Ollama support: - Uses the correct Ollama reque…
> clai: your useful command line helper!
OK, very significant addition here: github.com/merefield/clai FEATURE: add Ollama & Completions reasoning capability to improve reliability (#27) main ← add_ollama_reasoning opened 05:12PM - 03 Jul 26 UTC merefield +1773 -359 - Adds reasoning config support for OpenAI-compatible and Ollama chat endpoints. … - Sends reasoning_effort for supported OpenAI reasoning chat completion models. - Sends Ollama-native think: true for /api/chat when reasoning is configured. - Uses Ollama-native format: "json", options.num_predict, and stream: false payload fields. - Improves Ollama response parsing for structured, stringified, and malformed JSON-like outputs. - Surfaces API and empty-response errors more clearly in CLI output and history. - Documents reasoning, json_mode, and Ollama request behavior. - Adds Bats coverage for OpenAI reasoning, Ollama thinking, JSON mode, and parsing regressions. Adding reasoning support means you can run it much more reliably with Google’s Gemma 4 model locally (via Ollama) so you can now avoid any privacy issues and pay zero cost to run the helper (apart from some negligible extra electricity) You can now enable history and command results in history without worrying about details about your local computer being shared with the cloud. On Windows you will probably need a 16GB VRAM GPU to pull this off, or at least 24GB on a Mac, preferably 32. Cost $37 to create the PR with Codex, so there was something in it for OpenAI even if it is likely I will probab…
🔬 The Coolest Diffusion Research Isn't in LLMs — Evan Feinberg & Sergey Edunov, Genesis Molecular AI
Why the Llama lead left Meta for drug discovery, PEARL's zero-shot OpenBind win, and what becomes possible when co-folding finally crosses the accuracy threshold.
Shadow agents: How IT leaders must govern ‘headless’ AI before it breaks the enterprise
Earlier this year, I was running my own local AI agent, a system I built called LaptopAI-Agent, which uses a LangGraph reasoning loop, a local Ollama model and a set of tools that can read files, query my git repositories and monitor system processes, all running entirely on my laptop with no cloud calls. I had given it a broad task and walked away. When I came back, it had completed the work. Every file it touched was within its allowed paths. Every action was technically correct. What unsettled me was not what the agent had done. It was that I could not reconstruct the sequence of decisions that led to it. Without the SHA-256 chained audit log I had deliberately built in, I would have had no record of why the agent made each choice, only what it produced. That gap between visible outcomes and invisible reasoning is what I had to engineer around for a single-user personal tool. Enterprises face the same problem at the scale of thousands of agents, with far less instrumentation. This is what I mean by shadow agents: autonomous AI processes that operate at the API layer, chain tools together and complete multi-step workflows without logging in, generating session records, or waiting for a human to approve. They already run inside enterprise systems today. The governance infrastructure to manage them is, in most cases, far behind. The question is no longer whether your organization will run these autonomous processes. It already does. The question is whether you can see what t…
Can local preprocessing cut LLM API costs?
A few days ago I shared a project I’ve been working on called “LatentGate” — a local-first pipeline that reduces LLM API token usage by processing inputs before sending them to the model. After some great feedback, I’ve now turned it into: A pip-installable Python package A VS Code extension (runs as a local proxy) MCP server support for tools like Claude Code, Cursor, Cline, Continue PyPI → pip install latent-gate VS Code → LatentGate — Local-First AI Compression What it does Images (~1000–1300 tokens) → compressed to ~150 tokens using local vision models (Ollama + LLaVA) Long prompts / conversations → compressed locally before hitting cloud APIs Works with OpenAI / Claude / Gemini APIs Fully local preprocessing (no data leaves your machine before compression) The idea is inspired by VL-JEPA — predicting in embedding space, then decoding selectively. Why I built this While experimenting with GPT-4o / vision APIs, I noticed most costs come from raw input size (especially images and long prompts). So instead of optimizing prompts endlessly, I tried: → “What if we reduce what we send in the first place?” What I’m looking for I’d love feedback from this community, especially: Edge cases where compression breaks context Cases where output quality drops noticeably Prompt / API compatibility issues (OpenAI especially) Performance bottlenecks Better approaches to selective decoding or compression If you try it and something fails — that’s honestly the most valuable thing for me rig…
Liquid AI Ships LFM2.5-230M with llama.cpp, MLX, vLLM, SGLang, and ONNX Support for On-Device Inference
Liquid AI released LFM2.5-230M, its smallest model yet. The 230M-parameter, open-weight model runs on-device at 213 tok/s on a Galaxy S25 Ultra and 42 on a Raspberry Pi 5. Built on the LFM2 architecture, it targets tool use and data extraction, beating larger models like Qwen3.5-0.8B and Gemma 3 1B on instruction following. The post Liquid AI Ships LFM2.5-230M with llama.cpp, MLX, vLLM, SGLang, and ONNX Support for On-Device Inference appeared first on MarkTechPost .
From Local LLM to Tool-Using Agent
Using Gemma 4, Ollama, OpenAI Agents SDK, and Tavily MCP to build a lightweight research agent The post From Local LLM to Tool-Using Agent appeared first on Towards Data Science .
La métrica que hizo tropezar la estrategia ‘AI-first’ de Duolingo
En abril de 2026, el CEO de Duolingo, Luis von Ahn, reconoció que la compañía había retirado uno de los elementos más delicados de su estrategia de inteligencia artificial: el uso de IA dejaba de contar en las evaluaciones de desempeño de sus empleados . Lo llamativo es que, un año antes, una crisis pública en toda regla no había conseguido cambiar su estrategia ni un milímetro. El primer debate se abrió en la primavera de 2025, cuando Duolingo se declaró ‘AI-first’ . Ahí saltó la discusión habitual de la IA frente a las personas. Prendió rápido: usuarios borrándose la app y las redes de la marca inundadas de críticas. Von Ahn resolvió con oficio la crisis reputacional: aclaraciones, matices y un tono más suave. Le funcionó. El fuego se apagó, la estrategia siguió intacta y la empresa continuó creciendo. Pero se había abierto un segundo debate, menos visible pero igualmente importante: el de la evaluación de los empleados. Ese no se aplacaba con una nota de prensa. Fuera apenas trascendió: lo que una empresa haga con sus evaluaciones internas no provoca bajas masivas ni incendia TikTok. Dentro fue otra cosa. No hubo clamor, pero sí una objeción de fondo. Y esta vez el CEO cedió. La comunicación fue casi inversa a la del año anterior: no hubo gran rectificación pública ni operación de imagen. Von Ahn lo mencionó casi de pasada en un podcast: esa métrica se había retirado. Una crisis pública no movió la estrategia . Una objeción interna, sí. Lo interesante no es tanto la difer…
Using Visual Studio Code’s ‘air-gapped’ AI model mode
Microsoft has been pushing hard to make Visual Studio Code a major way to consume its AI services, mostly in the form of GitHub Copilot . GitHub Copilot’s deep integration with VS Code brings many conveniences — inline autocomplete, for instance — but it’s frustrating for those, like me, who would rather use another model provider, or even a locally hosted LLM, for those functions. Visual Studio Code 1.122 introduced a new feature, “ Use BYOK [Bring Your Own Key] without a GitHub sign-in ,” that allows you to “use chat, tools, and MCP servers in air-gapped or restricted environments where GitHub sign-in isn’t possible.” More importantly, it “enables fully offline workflows with local models like Ollama.” In other words, you can now use locally hosted LLMs for chat, tools, and Model Context Protocol servers inside Visual Studio Code. The one thing you still can’t do is use a local LLM for inline and next-edit suggestions — at least, not without additional tooling. Choosing a model for BYOK mode If you want to use a local LLM with VS Code’s bring-your-own-model system, the first thing you need is a way to host the model. VS Code lacks a model-hosting mechanism of its own, although it’s conceivable that a VS Code extension may offer something like that in the future. That said, hosting models is complicated enough that a dedicated app is really needed for the job. One easy way to host models is via a product like LM Studio , a convenient GUI for standing up, serving, and managi…
Local GenAI on Jetson: OSS models using different inferencing frameworks: Ollama, llama.cpp, & vLLM
This opening session builds the foundation for running popular OSS models such as Gemma, Qwen directly on Jetson — no cloud required. We cover when to use Ollama for rapid local prototyping versus vLLM for higher-throughput serving, show how the same workflow applies to both power different OSS models, and walk through the real decisions behind model choice, containers, quantization, and performance tuning on edge hardware. We close with a teaser of OpenClaw and a bonus take-home challenge to kick off community building. You will learn how to deploy open-source AI models on NVIDIA Jetson — no cloud required, from first launch to production-ready serving. We'll cover: Getting models running on NVIDIA Jetson — spin up popular OSS models (open-source large language models (LLMs) like Gemma and Qwen (LLMs and VLMs) using Ollama or vLLM on Jetson hardware and verify they're working end-to-end. Choosing the right inference engine — understand the practical tradeoffs between Ollama for rapid local prototyping, vLLM for higher-throughput serving, and llama.cpp, so you can pick the right tool for your use case. NVIDIA Jetson-specific serving strategies — walk through the real decisions behind model choice, containers, and performance tuning tailored for Orin and Thor, including what works, what doesn't, and why. Performance fundamentals — get introduced to quantization and speculative decoding: what they are, how they work, and when to reach for them on edge hardware. Real-world appl…
Matching first names, full names and pronouns
I am working on a graph store of entities and relationships extracted from a factual test document of around 500 words. The first pass (NER) extracts named entities, the second extracts relationships (RE). For a given person, there are different references in the text: Maria, Maria Gotthard, Dr. Maria Gotthard and can also be referred to by 'she', for example 'she was rewarded by the company'. The goal is to merge all these references into one entity so that the relationship graph is not fragmented into different contexts. I have seen a few posts on different forums saying this is a very difficult problem, but hopefully someone out there has some insights or experience to share 🙂 To make things interesting, references to the same entity can occur in different chunks of text, making it impossible for the LLM (currently Ollama/Mistral) to process the cross-chunk context in one call. To address this, I have added a pass across all extracted entities, including exact text matching and a Levenshtein similarity check, but this does not handle first name v full name and comes with a host of other issues. It has a high risk of over-merging, for example if a set of entities consist of incrementally numbered items they will all be merged into one entity. I am wondering if there is a particular architecture for this problem, for example pre-processing a document to link related entities before extracting. Doesn't have to be LLM-based, heuristics and algorithms sometimes do the trick as…
The Myth of Model Wars: Open vs Closed AI in 2026
In this fully connected episode, Dan and Chris break down one of the biggest questions in AI today: do open vs. closed models still matter? From the rise of physical AI and edge devices to the shifting landscape of open-source models like LLaMA, they explore whether the “model wars” are becoming irrelevant. The conversation then dives into a bigger transformation, the rise of agentic systems, workflows, and AI-driven infrastructure. Featuring: Chris Benson – Website , LinkedIn , Bluesky , GitHub , X Daniel Whitenack – Website , GitHub , X Upcoming Events: Register for upcoming webinars here ! Midwest AI Summit 2026
Llama 3.2 Vision-Instruct Inference Speed on A100 or H100 GPU
Can anyone provide an estimated time of how long does it take for Llama-3.2 Vision-Instruct 11-B model to: process an image size of 1-MB and prompt size of 1000 words and generate a response of 500 words The GPUs used for inference could be A100, A6000, or H100.
New RAG Horizons with Qdrant Hybrid Cloud and LlamaIndex
We’re happy to announce the collaboration between LlamaIndex and Qdrant’s new Hybrid Cloud launch , aimed at empowering engineers and scientists worldwide to swiftly and securely develop and scale their GenAI applications. By leveraging LlamaIndex’s robust framework, users can maximize the potential of vector search and create stable and effective AI products. Qdrant Hybrid Cloud offers the same Qdrant functionality on a Kubernetes-based architecture, which further expands the ability of LlamaIndex to support any user on any environment.
Loading Llama-2 70b 20x faster
In this post, we discuss the importance of speed when loading large language models and what techniques we employed to make it 20x faster. In particular, we use the Llama 2 series of models. We share how you can reduce latency and costs using the Anyscale platform.
Anyscale and Meta Collaborate to Advance the Llama-2 Ecosystem
Update June 2024: Anyscale Endpoints (Anyscale's LLM API Offering) and Private Endpoints (self-hosted LLMs) are now available as part of the Anyscale Platform. Click [here](https://console.anyscale.com/?utm_source=anyscale&utm_medium=blog&utm_campaign=blog_callout&utm_content=june2024_product_update_subheading) to get started on the Anyscale platform.
Llama 2 is about as factually accurate as GPT-4 for summaries and is 30X cheaper
Update June 2024: Anyscale Endpoints (Anyscale's LLM API Offering) and Private Endpoints (self-hosted LLMs) are now available as part of the Anyscale Platform. Click [here](https://console.anyscale.com/?utm_source=anyscale&utm_medium=blog&utm_campaign=blog_callout&utm_content=june2024_product_update_subheading) to get started on the Anyscale platform.
Fine-Tune Llama on main and auxiliary task
I am trying to fine-tune Llama model on two task at the same time, using hugging face library: Main task: Causal language model like the model was initially trained for A classification task based on the whole input sequence (recommend an article). For this task I am getting as a reference the LlamaForCausalLM class, overwriting init and forward functions . However, I want to combine the two tasks above into one process. The main problem is that language modelling is an iterative process were the loss is calculated for every new context token in the input sequence, while for the classification task the loss should only be calculated once. How can I freeze the loss update on the classification task up and only calculated once the language modelling part has been completed. Is there any example you can recommend in order to combine a main LM task with an auxiliary classification task? First question for me here, thanks everyone for your understanding.
Build and Scale a Powerful Query Engine with LlamaIndex and Ray
In this blog, we showcase how you can use LlamaIndex and Ray to build a query engine to answer questions and generate insights about Ray itself, given its documentation and blog posts. We’ll give a quick introduction of LlamaIndex + Ray, and then walk through a step-by-step tutorial on building and deploying this query engine. We make use of both Ray Datasets to parallelize building indices as well as Ray Serve to build deployments.