AI/ML news, top picks, and generated innovation digests.
200 articles tagged with this keyword, sorted by most recent first.
One useful way to compare these efforts would be to separate research tools from systems used in clinical workflows, then report external validation and calibration across different populations. A strong result on one dataset does not by itself show how a model behaves when hospitals, scanners, or patient groups change. Did any of the projects in this roundup publish that kind of deployment evidence?
The idea of using a frozen LLM for multimodal generation is fascinating. SPAE's tokenizer producing interpretable linguistic tokens seems like a clever way to bridge the gap. GoW Laufey Wiki
Great article! I recently discovered MarkItDown (markitdown.tech), an excellent tool for converting files to Markdown. Highly recommend checking out their PDF to Markdown converter at markitdown.tech/pdf-to-markdown and their online Markdown editor at markitdown.tech/markdown-online. Also worth exploring their Microsoft Word to Markdown tool at markitdown.tech/microsoft-markitdown and Markdown to PDF at markitdown.tech/markdown-to-pdf. Amazing resource for developers!
Ngl seeing QIN beat GPT-3 on the Gaokao exam with just 1/16th of the parameters is wild, especially since its proof that data structure matters way more than raw scale. I'm curious if anyone knows whether RST can be applied to visual datasets too — like when prepping question samples where I use fix typos in product screenshots to swap text on UI screenshots, would restructuring give multimodal models that same massive boost?
InclusionAI open-sourced Ming-Image-0.1-Design, a 6B text-to-image model built for UIs, posters, and infographics with native transparent-background output.
Sakana AI and the University of Tokyo show that a vision language model can plan robot motions more reliably by testing and refining candidate trajectories in simulation before touching the real robot.
Yeah, it still been 2 years ago, but thanks check this virtual game , cheers.
Learn how to run SkyRL, an open-source reinforcement learning framework, on Amazon SageMaker HyperPod to post-train a Qwen3-VL-8B vision-language model with GRPO. This walkthrough covers building the container image, launching a Ray cluster from SageMaker Studio, submitting and monitoring the job, and hosting the trained LoRA adapter for inference.
The unified prediction objective is interesting because it frames visual representation learning as more than recognizing isolated image patterns. If a single encoder learns to anticipate image patches and text tokens together, its representations may be better aligned with tasks where language refers to specific visual content. The practical question is how consistently those gains transfer across recognition, grounding, and broader multimodal evaluation settings.
Ant Group's inclusionAI drops a 6B MIT-licensed text-to-image model that beats every other open-weights contender on UI mockups, dashboards and posters.
stable-worldmodel standardizes world model pipelines on Lance, DuckDB Lance extension adds native multimodal SQL, and LanceDB benchmarks 100M vectors at ~$779/month, plus upcoming events, enterprise updates, and community updates.
Radical Numerics is using biological chain-of-thought and multimodal perception to keep up with the bio-defense arms race, design new genomes and gain insights into biology itself.
In a new paper Image Captioners Are Scalable Vision Learners Too, a DeepMind research team presents CapPa, a image captioning based pretraining strategy that and can compete CLIP and exhibit favorable model and data scaling properties, verifying that a plain image captioning can be a competitive pretraining strategy for vision backbones. nano-video.io
https://syncedreview.com/2023/12/06/tencent-sydney-us-gpt4video-a-unified-multimodal-large-language-significantly-elevates-lms-video-generative-capabilities/
Great article about KOSMOS-1! I've been exploring similar AI innovations. Check out this amazing AI tattoo generator at https://aitattoogenerator.net - it uses cutting-edge AI to create custom tattoo designs. Really impressive technology!
Alibaba Cloud, Alibaba Group Holding’s cloud business unit, is set to launch data centres in Turkey, Finland and the Netherlands over the next 12 months, speeding up an overseas push as it seeks to win enterprise customers through its “full stack AI capabilities” spanning chips, cloud infrastructure and multimodal models. The first of these new data centres will open in the Netherlands in October. The cloud unit will also add to existing data centre capacity in Malaysia, Germany, the United Arab...
The Biological Computing Co. wants to team up with AWS to sell a text-to-video model that's supposed to run five times faster and 80 percent cheaper thanks to a software layer derived from real nerve cells. The tiny layer adds less than 0.1 percent to the base model, but the startup won't say which model that is. The article A tiny software layer from lab-grown neurons promises faster, cheaper AI video appeared first on The Decoder .
Qwen3.8-Omni-Flash is Qwen's first multimodal model designed for AI agents. It processes audio and video together and independently uses tools to edit vlogs, translate clips, or summarize movies. On audio-video benchmarks, it nearly matches Gemini 3.8 Flash at a fraction of the API cost. The article Qwen3.8-Omni-Flash undercuts Google's Gemini Flash pricing while matching its multimodal benchmarks appeared first on The Decoder .
Prism ML Inc. announced Thursday the launch of Bonsai 2 27B, the second generation of its ultra-compact multimodal generative artificial intelligence small enough to fit on PCs and some high-end mobile devices. The company said it used ternary, which uses three parts, to scale down its Qwen3.8 27B-based model. Qwen3.8 weighs in at about 56 […] The post PrismML launches Bonsai 2 27B, a high-intelligence AI model so small it fits on consumer hardware appeared first on SiliconANGLE .
I’m an MSc student and pretty new to medical imaging/deep learning. I’m working on an Alzheimer’s classification project using the ANMerge dataset. It has around 1,700 participants overall, but only around 450 have MRI data alongside clinical data. The main thing I’m looking at is comparing different ways of combining MRI and clinical data: feature-level concatenation, late fusion, gated fusion and cross-attention. I’m currently trying to decide between two options for the MRI: Use the raw 3D MRI scans, preprocess them and use a pretrained 3D CNN such as ResNet-10/18 as the MRI encoder. Use the FreeSurfer-derived features that ANMerge already provides, such as regional volumes and cortical thickness, with a small MLP as the MRI encoder. My concern with the raw MRI option is that with only ~450 patients, fine-tuning a 3D CNN could add quite a lot of complexity and risk of overfitting. It would also add another variable to the experiment, because differences in results could come from how well the CNN learns the MRI representation rather than just the fusion method. It would obviously involve quite a bit more preprocessing and implementation work too. The derived-feature option seems simpler and would let me focus more directly on the fusion comparison. The thing I’m less sure about is cross-attention. If I use derived features, I’d need to structure them in a way where cross-attention is actually meaningful rather than just applying attention between two single vectors. One a…
I’m an MSc student and pretty new to medical imaging/deep learning. I’m working on an Alzheimer’s classification project using the ANMerge dataset. It has around 1,700 participants overall, but only around 450 have MRI data alongside clinical data. The main thing I’m looking at is comparing different ways of combining MRI and clinical data: feature-level concatenation, late fusion, gated fusion and cross-attention. I’m currently trying to decide between two options for the MRI: Use the raw 3D MRI scans, preprocess them and use a pretrained 3D CNN such as ResNet-10/18 as the MRI encoder. Use the FreeSurfer-derived features that ANMerge already provides, such as regional volumes and cortical thickness, with a small MLP as the MRI encoder. My concern with the raw MRI option is that with only ~450 patients, fine-tuning a 3D CNN could add quite a lot of complexity and risk of overfitting. It would also add another variable to the experiment, because differences in results could come from how well the CNN learns the MRI representation rather than just the fusion method. It would obviously involve quite a bit more preprocessing and implementation work too. The derived-feature option seems simpler and would let me focus more directly on the fusion comparison. The thing I’m less sure about is cross-attention. If I use derived features, I’d need to structure them in a way where cross-attention is actually meaningful rather than just applying attention between two single vectors. One a…
I’m an MSc student and pretty new to medical imaging/deep learning. I’m working on an Alzheimer’s classification project using the ANMerge dataset. It has around 1,700 participants overall, but only around 450 have MRI data alongside clinical data. The main thing I’m looking at is comparing different ways of combining MRI and clinical data: feature-level concatenation, late fusion, gated fusion and cross-attention. I’m currently trying to decide between two options for the MRI: Use the raw 3D MRI scans, preprocess them and use a pretrained 3D CNN such as ResNet-10/18 as the MRI encoder. Use the FreeSurfer-derived features that ANMerge already provides, such as regional volumes and cortical thickness, with a small MLP as the MRI encoder. My concern with the raw MRI option is that with only ~450 patients, fine-tuning a 3D CNN could add quite a lot of complexity and risk of overfitting. It would also add another variable to the experiment, because differences in results could come from how well the CNN learns the MRI representation rather than just the fusion method. It would obviously involve quite a bit more preprocessing and implementation work too. The derived-feature option seems simpler and would let me focus more directly on the fusion comparison. The thing I’m less sure about is cross-attention. If I use derived features, I’d need to structure them in a way where cross-attention is actually meaningful rather than just applying attention between two single vectors. One a…
A complete walkthrough of building an autonomous vehicle perception model training pipeline on top of LanceDB and the Multimodal Lakehouse.
Jeevanthi Liyanapathirana, Pierrette Bouillon, Jonathan Mutal, Sabrina Girletti and Lise Volkart in Proceedings of the 1st International Workshop on Teaching AI-Based Translation and Technologies (TAITT 2026)
This article is brought to you by VicOne . Robot safety has traditionally asked: Can a machine remain safe when something goes wrong? Physical AI raises a harder question: Can a machine remain safe when an attacker changes what it sees, decides, or does even when nothing appears to have failed? As AI and robotics continue to advance at an unprecedented pace, modern robots perceive through multimodal sensors, interpret context using AI models, and translate those interpretations into physical action. As they move into dynamic environments, their safety increasingly depends on the integrity of the data guiding their decisions. That dependence creates risks that conventional safety assessments may not fully capture. Recent research has demonstrated that manipulating what a robot sees, hears, or interprets can influence its behavior without requiring direct control. Such manipulation can occur anywhere across its complex sensing and decision-making system — a layered attack surface encompassing training pipelines, system infrastructure, and runtime perception. Layer One: Corrupting intelligence at its source In 2017, BadNets demonstrated that a model could behave normally under most conditions, yet fail in the presence of a specific hidden trigger. In one example, a subtle pattern caused a stop sign to be misclassified as a speed limit sign without affecting the model’s behavior on other inputs. What began as a classification vulnerability has since evolved into action manipulat…
This is a compelling example of why translation quality depends on more than text alone: reading order, scene context, and balloon placement all shape how a manga page is understood. The cleanup-and-rendering step is especially important for preserving the page’s rhythm. For sharing short visual explainers alongside that work, Video to Link offers a straightforward way to turn a video into a link.
Alibaba's 27B dense multimodal model lands on Cerebras at roughly 1,800 tokens per second, with reasoning on by default and a 128K context on paid tiers.
Deepseek releases V4.1-Flash, a multimodal model with 552 billion parameters that cuts KV cache memory to a quarter of its predecessor. On the DeepSWE coding benchmark, it narrowly beats Opus 5 and GPT-5.6 Sol, even though only 16 billion parameters are active per token. The model ships under the MIT license and targets much cheaper AI agents. The article New Deepseek model V4.1-Flash cuts memory needs for AI agents appeared first on The Decoder .
How Lance format and LanceDB's Enterprise feature engineering platform make VLM fine-tuning faster by materializing expensive multimodal features once and training from those columns.
Lance turns long robotics videos into random-access multimodal training data with 1.7–6× faster reads and 42% lower storage use.
Suno has unveiled a new AI music model generation, v6, in three versions, built together with Warner Music Group, BMG, and Believe. All older models are being shut down. Songs can now be partially changed through text commands or generated multimodally from text, audio, and images. The company won't say which catalogs went into training, while Universal and Sony keep suing. The article Suno launches v6 music models built with Warner, BMG, and Believe appeared first on The Decoder .
Introducing ChatGPT Images 2.5 OpenAI's image generation models are apparently used "more than 3 billion images across ChatGPT Images and the GPT‑Image models in the API". This latest release improves their instruction-following ability across multiple turns, responds faster, and "is better at preserving the subjects in your reference photos". There are two new model IDs in the API: gpt-image-2.5-sunburst and gpt-image-2.5-flare . Based on this I think Sunburst is the stronger option: Choose Sunburst for workflows where editing precision matters most, and Flare for fast, high-quality everyday image generation. I upgraded my openai_image.py CLI tool to support passing in one or more reference images, so now this works: uv run https://tools.simonwillison.net/python/openai_image.py \ ' add a raccoon scientist studying the chart thoughtfully ' \ -i https://static.simonwillison.net/static/2026/openai-agent-usage.webp \ -m gpt-image-2.5-sunburst This is the original image , and here's what I got back from that prompt to "add a raccoon scientist studying the chart thoughtfully": Tags: tools , ai , openai , generative-ai , uv , text-to-image
Perfect timing — I was just evaluating video transcriber tools! videotranscriber has been a great reference.
DeepMind’s Zipper is a major leap in multimodal AI—elegantly zipping together pretrained unimodal decoders without sacrificing modality-specific performance. Its gated cross-attention design, flexible tower composition, and strong empirical gains (e.g., 40% relative WER reduction in TTS) make it a compelling architecture for next-gen generative systems. For hands-on guidance on applying such cutting-edge models—including practical defusal strategies, campaign walkthroughs, and achievement tracking—check out the evidence-led BOMBANANA! guide: https://bombanana.app/
Poople Solver Solve the daily Poople puzzle faster with a simple online helper—enter today’s clues, compare possible answers, get smarter hints, and improve your strategy for guessing the right people, characters, or names in a fun daily browser game.
Learn how to deploy a multimodal WhatsApp ordering assistant that takes customer orders through text, voice notes, and real-time voice calls on a single business number, built on Amazon Bedrock AgentCore with Amazon Nova 2. The channel and ordering layers stay separate, and one shared memory recognizes each customer across all three channels.
Explore five free AI API providers for accessing large language models, fast inference, multimodal AI, and agentic applications without paying for API usage.
This is a fascinating step forward for vision encoders. The idea of unifying image patch and text token prediction into a single autoregressive framework feels like a natural evolution—by treating both modalities as one sequence, AIMV2 seems to learn richer cross-modal relationships without needing separate, complex objectives. It’s especially impressive that this simple causal design can generalize so well across image recognition and multimodal understanding tasks. I also appreciated how Apple’s team emphasizes simplicity how to breed orserk
ShaLa: Multimodal Shared Latent Generative Modelling robyn.cherinka… Wed, 09/02/2026 - 14:44 This paper presents a novel generative framework for learning shared latent representations across multimodal data. Many advanced multimodal methods focus on capturing all combinations of modality-specific details across inputs, which can inadvertently obscure the high-level semantic concepts that are shared across modalities. Notably, Multimodal VAEs with low-dimensional latent variables are designed to capture shared representations, enabling various tasks such as joint multimodal synthesis and cross-modal inference. However, multimodal VAEs often struggle to design expressive joint variational posteriors and suffer from low-quality synthesis. In this work, ShaLa addresses these challenges by integrating a novel architectural inference model and a second-stage expressive diffusion prior, which not only facilitates effective inference of shared latent representation but also significantly improves the quality of downstream multimodal synthesis. We validate ShaLa extensively across multiple benchmarks, demonstrating superior coherence and synthesis quality compared to state-of-the-art multimodal VAEs. Furthermore, ShaLa scales to many more modalities while prior multimodal VAEs have fallen short in capturing the increasing complexity of the shared latent space. Image Mar 14, 2026 Human-Centered AI Read More 1 Minute Read
World Labs Inc., the high-profile and well-funded artificial intelligence startup co-founded by the renowned computer vision pioneer Fei-Fei Li, has just dropped Atlas, which promises to be a game-changer in the world of “world models.” In a blog post, World Labs explained that Atlas is a breakthrough multimodal world model that aims to bridge the […] The post Fei-Fei Li’s World Labs debuts Atlas, a world model showcase for advanced spatial intelligence appeared first on SiliconANGLE .
JetFormer’s focus on generating raw images and text autoregressively is interesting because it avoids the usual patchwork of modality-specific encoders and decoders. For readers thinking about how unified multimodal systems translate into practical creation tools, TopView AI’s video agent overview at Topview AI Video Agent Explained - Free Agent-Style Generato is a useful related reference.
For enterprises deploying AI agents, security and trust are top-of-mind issues. When given the authority to act autonomously, they can inadvertently leak or expose data, rack up huge token costs, or produce inaccurate or patently false outputs. These risks, says VMware, require a new type of platform that imposes strict controls over what agents can do, while giving them access to the contextually-relevant data they need. With this in mind, Broadcom has introduced a new secure, AI-ready data foundation in the VMware Tanzu Platform that it says addresses these needs. Announced today at VMware Explore , the offering is intended to help enterprises move beyond pilot purgatory to production-ready agents. It isn’t a new data platform, or a new feature that customers must deploy, noted Adam Reeves , a research director at IDC. Instead, the VMware team “built the missing plumbing between the data enterprises already have and the agents that need it.” Built-in agent harnesses, deny-by-default security Integrated directly into VMware Cloud Foundation (VCF), the updated Tanzu Platform features an “ AI-ready ” data infrastructure, Purnima Padmanabhan , GM of Broadcom’s Tanzu Division, explained during a briefing. Structured and unstructured data can be processed within a customer’s private cloud, and remain in that environment, and multimodal data ingestion, parsing, and semantic layering give agents context. Builders also get an out-of-the-box developer harness featuring pre-approved…
Well written and informative. Thanks for putting this together.
Perplexity Computer now routes long-context, multimodal research tasks to GLM 5.3, which outperformed GLM 5.2 on the in-house WANDR benchmark.
Z.ai's first natively multimodal GLM-5 model ships with 320B parameters, 18B active, a 1M-token window, and Flash-tier pricing on Together AI.
This is a solid step forward for long-context training, but the real bottleneck for most teams isn't just the GPU cluster—it's the cost of experimenting with these architectures. Being able to push to 2M tokens on only 4 GPUs is impressive, but I'd love to see how this translates to real-world throughput for video generation or multimodal models where the sequence lengths are growing even faster. Speaking of practical applications, if anyone is looking to apply these kinds of efficiency gains to a user-facing tool, I've been testing a platform that turns text prompts directly into cinematic video clips with synced audio at minimax3.com . It's a different kind of pipeline, but the focus on hardware efficiency reminds me of why tools like this are becoming viable.
Excellent article with easy-to-follow information and practical examples. It’s always useful to find a simple tool that can be accessed directly from a browser. You can explore this click counter when you want to keep track of repeated clicks. The straightforward approach makes it suitable for short practice sessions and everyday counting activities without unnecessary steps.
Qwen3.8-Flash-Next Another open weights model from Qwen. This one is "a multimodal MoE model that also serves as an early preview of the architecture used in Qwen4". It's pretty big: 125B parameters but only 6B active which means it gets a significant performance boost. I've been trying it out on a DGX Spark using these Unsloth quantized models . I'm still exploring the model - so far I've tried the 72.5GB UD-IQ1_S one (producing these pelicans ) and the 78.9GB UD-Q2_K_XL (producing these ). My favorite so far was this xhigh reasoning effort one from UD-Q2_K_XL: Via Hacker News Tags: ai , generative-ai , llms , qwen , pelican-riding-a-bicycle , llm-release , ai-in-china , nvidia-spark
Z.ai just dropped a 320B mixture-of-experts model with 18B active params, MIT-licensed weights, native multimodal input, and a 1M-token context window.
Alibaba open-weights a 125B multimodal MoE with just 6B active parameters, previewing the attention overhaul coming in Qwen4.
High-fidelity image-to-3D generation requires a 3D representation that captures both geometry and appearance. To support relighting and integration into standard rendering pipelines, the representation should include physically based rendering (PBR) modalities such as albedo, metallic-roughness, and surface normals. We propose Luce, a 3D representation that unifies geometry and PBR materials within a voxelized multimodal Gaussian cloud, using dedicated Gaussian primitives for each modality. A variational autoencoder compresses this representation into a unified material-aware latent space. A…
Thomas Baier, Selene Báez Santamaría and Piek Vossen in Dialogue Discourse Volume 16
Unified multimodal models that understand, reason over, and generate interleaved text–image sequences remain structurally fragmented: existing approaches either sacrifice visual fidelity through discrete tokenization, impose structural asymmetry by combining causal text generation with iterative diffusion-based denoising, or degrade pretrained understanding when adapting vision-language models for generation. We observe that autoregressive normalizing flows are autoregressive Transformers—sharing the same causal mask, KV-cache mechanism, and left-to-right structure as LLMs—making them the most…
Multimodal large language models increasingly use visual chain-of-thought (Visual CoT) to reason about spatial, temporal, and embodied environments. By generating intermediate reasoning images, Visual CoT provides an intuitive mechanism for visual foresight but introduces substantial inference overhead, which is particularly problematic for proactive video reasoning. We ask whether models can learn to think visually during training while reasoning directly at inference. We introduce Internalized Visual Thinking (IVT), a post-training framework that jointly optimizes textual prediction and…
Maria Pia Di Buono in Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 2)
DeepSeek today debuted a new addition to its flagship V4 series of large language models. On launch, V4 Flash Vision Exp is only available via the Chinese startup’s paid developer platform. The company may release a free version later on given that it has open-sourced many of its earlier models. Those models include V4 Flash, […] The post DeepSeek debuts multimodal language model competitive with Opus 4.8 appeared first on SiliconANGLE .
Deepseek has released V4-Flash-Vision-Exp, an experimental multimodal model that adds image understanding to V4-Flash's text capabilities. On the company's own multimodal agent benchmarks, it approaches Opus 4.8 and sometimes beats it. The article Deepseek releases experimental Flash vision model that rivals Opus 4.8 on agent benchmarks appeared first on The Decoder .
DeepSeek's new experimental vision model bolts image understanding onto V4-Flash at the same price, closing in on Claude Opus 4.8 on multimodal agent tasks.
I need to generate around 500 short exercise videos (5–10 seconds each), such as push-ups, squats, and sit-ups. I already have the prompts and JSON input ready. My main issue is finding a cost-effective and automated way to generate them. My local GPU has only 6 GB VRAM, so running modern video-generation models locally is difficult. I’ve tried Diffusers, ComfyUI, and AnimateDiff, but cloud APIs are expensive for generating hundreds of videos. What would be the most practical approach? For example: Renting a cloud GPU and running an open-source model in batches Using image-to-video instead of text-to-video Creating reusable base animations Quantizing or optimizing models for low VRAM Using batch APIs to reduce costs I’m looking for recommendations on the best pipeline and cost/performance trade-offs for generating a large number of similar short videos.
Alibaba's third-generation image model debuts at #6 in editing and #9 in text-to-image, with big Elo jumps and a productivity-first pitch.
This is a fascinating direction for medical AI! The idea of a "generalist learner" like MedVersa, especially one using a large language model as an orchestrator, feels like a significant step beyond the specialized models we've seen so far. It reminds me a bit of how generalist AI models are starting to tackle a broader range of creative tasks, not just one specific thing. For instance, I've been working with a tool that can extend images while keeping the subject intact across different canvas sizes – it’s a different domain, but the underlying principle of versatility is the same. It makes me wonder how this generalist approach might eventually filter down to more niche AI applications, making them more adaptable. Will we see AI that can not only interpret an X-ray but also then generate a report, or even suggest treatment options, all within one adaptable framework?
Meta shared new benchmarks, robotics demos, and a real-world agent evaluation for Muse Spark 1.2, its coding-focused multimodal model ahead of an open-weights release.
On the Strengths and Weaknesses of Data for Open-set Embodied Assistance robyn.cherinka… Wed, 08/19/2026 - 15:23 Embodied foundation models are increasingly performant in real-world domains such as robotics or autonomous driving. These models are often deployed in interactive or assistive settings, where it is important that these assistive models generalize to new users and new tasks. Diverse interactive data generation offers a promising avenue for providing data-efficient generalization capabilities for interactive embodied foundation models. In this paper, we investigate the generalization capabilities of a multimodal foundation model fine-tuned on diverse interactive assistance data in a synthetic domain. We explore generalization along two axes: a) assistance with unseen categories of user behavior and b) providing guidance in new configurations not encountered during training. We study a broad capability called Open-Set Corrective Assistance, in which the model needs to inspect lengthy user behavior and provide assistance through either corrective actions or language-based feedback. This task remains unsolved in prior work, which typically assumes closed corrective categories or relies on external planners, making it a challenging testbed for evaluating the limits of assistive data. To support this task, we generate synthetic assistive datasets in Overcooked and fine-tune a LLaMA-based model to evaluate generalization to novel tasks and user behaviors. Our approach pr…
Learn how to build a multi-agent document classification solution on Amazon Bedrock using the Strands Agents SDK. Three specialized agents combine textual analysis with Claude Haiku 4.5 and visual similarity search with Amazon Titan Multimodal Embeddings to accurately classify insurance documents such as policies and affidavits.
An artificial intelligence startup called Hypercubic Inc. is taking on the problem of modernizing legacy applications written in the COBOL programming language after raising $5.3 million in seed funding today. The round was led by CIV and saw participation from Y Combinator, Afore Capital, Pioneer Fund, Multimodal Investors and angels including Opendoor Chief Executive Kaz […] The post Hypercubic raises $5.3M to map out and rewrite legacy COBOL apps with AI agents appeared first on SiliconANGLE .
Proposal: User-Owned AI Memory — A Portable Multimodal Memory Format The idea I would like to propose a different way of thinking about AI memory. Today, we generally separate: - Context windows - Conversation history - Model memory - Uploaded files - Vector databases / RAG systems - Agent memory I believe these concepts could be unified around one simple principle: «The user’s data should be the AI’s long-term memory — not the model’s context window.» Instead of making the model itself responsible for holding a limited amount of memory, the AI could use the user’s local or cloud storage as a virtually unlimited long-term memory. A new concept: AI Memory Containers Imagine a portable file format such as: “project.ai” This would not simply be a Markdown file. It would be an AI Memory Container capable of containing: - Text - Markdown - PDFs - Images - Audio - Video - Documents - Metadata - Relationships between information - Search indexes - Optional embeddings - Conversation history - Structured knowledge The user would own this file and could move it between devices, applications, and potentially different AI models. For example: “Mastering_Project_Management.ai” could contain an entire project: Mastering_Project_Management.ai ├── Documents ├── PDFs ├── Notes ├── Images ├── Audio ├── Video ├── Conversations ├── Metadata ├── Knowledge Graph └── Search / Semantic Index The important point is that the AI would not load everything into its context window. Instead: User Question…
I found and supported the previous community request regarding restoring and improving Sora/video access within ChatGPT. That discussion was closed on June 15, 2026, so I would like to expand on the concept with a broader proposal focused on native conversational video generation, video editing, organization, and premium-tier integration directly within ChatGPT. The Core Idea I believe ChatGPT should eventually support video in much the same way users can currently work conversationally with images. Instead of requiring users to leave ChatGPT, open another application, learn another interface, transfer their files, and continue their work elsewhere, a user could simply upload or generate a video and tell ChatGPT what they want done. For example: “Create a video from this concept.” “Make this scene longer.” “Remove this object from the video.” “Change the background.” “Turn these images into a video.” “Use this image as the character reference.” “Create a shorter version for social media.” “Add captions.” “Improve the video quality.” “Change the camera angle or movement.” “Continue this scene for another ten seconds.” “Create three variations of this scene.” The important feature would be the conversational interface . Users should not necessarily have to become professional video editors to perform useful video-editing tasks. Potential Capabilities A native ChatGPT video environment could eventually include: Text-to-video generation Image-to-video generation Video-to-video t…
This is a thoughtful take on ai offers video game design possibilities far beyond virtual reality. The practical examples really help illustrate the concepts. www.seedimagine.com
The discussion about microsoft’s fully pipelined distributed transformer processes 16x sequence length with extreme hardware efficiency raises some really valid points. This perspective is refreshing. Viddi AI reference 60
Impressive to see how Llama 3 brings language, vision, video, and speech into a more unified system. As multimodal AI becomes more capable, having the right context for each input becomes increasingly important. A good Context Match can make those interactions much more relevant and useful.
I appreciate how honest you are about what worked and what didn't. Too many posts about nvidia’s global context vit achieves sota performa only show the highlight reel. Yours feels like a real walkthrough.
Hello OpenAI Product, Research, ChatGPT Work, Safety, Legal, and Policy Teams, I would like to propose a strategic direction for the next generation of ChatGPT and ChatGPT Work. The goal should be clear: ChatGPT should not only be the best AI assistant. It should become the best system for completing professional work from idea to production-ready output. I believe OpenAI has three major opportunities: 1. Regain clear leadership in AI video generation. 2. Surpass Claude in professional document creation and editing. 3. Adopt a precise provenance policy that complies with law without unnecessarily watermarking ordinary AI-assisted professional work. Together, these could make ChatGPT Work a universal AI production operating system. PART I — REGAIN LEADERSHIP IN AI VIDEO AI video is moving beyond short prompt-generated clips. The next category is multimodal video production using combinations of: Text Images Multiple images Video references Audio Characters Products Locations Motion references Camera instructions Storyboards OpenAI should build one unified system supporting: Text → Video Image → Video Multiple Images + Text → Video Multiple Images + Text + Audio → Video Video → Video Video Editing Video Extension Storyboard → Video Character Reference → Video Product Reference → Video Environment Reference → Video Motion Reference → Video Camera Reference → Video Users should not need different systems for each workflow. A. Semantic reference control Users should be able to as…
ByteDance's Seedance 2.5 lands on Runway with 30-second clips, 50 multimodal references, and built-in audio — but there's a resolution catch worth knowing.
Moonshot AI's PerceptionBench tests how well multimodal AI models can actually "see," separate from logical reasoning. No frontier model reaches 60 percent accuracy, and GPT-5.6 Sol leads by a narrow margin. Many supposed reasoning errors actually happen as early as the image-reading stage. The article New benchmark confirms AI models still perform poorly at visual perception appeared first on The Decoder .
Run a multimodal feature-engineering pipeline in ten minutes with geneva-examples, then rebuild its CLIP embedding step with LanceDB's feature engineering API, the same calls you'd use to enrich your own tables on LanceDB Enterprise.
Text-to-image models have become remarkably good at producing realistic images. But realism isn’t the same as correctness. Ask for several distinct people, a specific composition, or a high-resolution image generated locally, and today’s models still struggle in surprising ways. In this episode, Fatih Porikli, Vice President of Technology at Qualcomm, joins me to discuss what remains unsolved in image generation and several approaches his team presented at CVPR to address those challenges. We explore why better training objectives can improve controllability, how separating scene planning from rendering may lead to more reliable image generation, techniques for generating 16-megapixel images efficiently on edge devices, and new methods for eliminating the visible artifacts that often appear in AI-powered image editing. Along the way, we discuss reinforcement learning for image generation, agentic image generation pipelines, on-device AI, and what the next phase of progress in generative vision systems is likely to look like. 🗒️ Full show notes: https://twimlai.com/go/773
Image inputs and structured outputs with Gemma 4 and Ollama The post Building Multimodal Workflows with a Local LLM appeared first on Towards Data Science .
마이크로소프트(MS)가 자체 개발한 이미지 생성 모델의 성능을 크게 끌어올리며 오픈AI와 구글, 메타, xAI가 경쟁하는 생성 AI 시장에서 존재감을 강화하고 있다. 최신 모델이 이미지 생성 모델 평가에서 2위에 오르면서 MS가 자체 AI 모델 개발에 투자해 온 성과가 가시화되고 있다는 평가다.MS가 10일(현지시간) 텍스트-이미지 생성 모델 ‘MAI-이미지-2.6(MAI-Image-2.6)’을 공개했다.MAI-이미지-2.6은 아레나의 이미지 생성 모델 사용자 선호도 평가(Text-to-Image Arena)에서 1336점으로 전체
Uber has expanded its Uber Bike service to 100 additional cities in India, taking its presence to over 220 cities across the country. The expansion will cover 18 states and one Union Territory. The new cities include Jammu, Tirunelveli, Silchar and Jamnagar, among others. Uber Bike will offer two-wheeler rides for short-distance travel as well as first and last-mile connectivity. The expansion is part of Uber’s strategy to strengthen its presence beyond major metros and build a multimodal mobility platform in India. The company currently offers bikes, autos, cars, airport rides and intercity services through its platform. Uber said the expansion will create flexible earning opportunities for tens of thousands of local drivers. According to the company, more than 2 million Indians earn through its platform. As part of its Independence Day campaign, Uber is also offering first-time riders a free first ride for a limited period. Uber Bike includes safety features such as real-time driver ID checks, an in-app SOS button, trip sharing and a 24x7 safety line.
This is a case where in-context training has previously been shown to help on a vision task. Contemporaneous with this old forum topic is a paper showing that examples of “how to turn vision into readings” can improve the actual readings provided: arXiv.org The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision) Large multimodal models (LMMs) extend large language models (LLMs) with multi-sensory skills, such as visual understanding, to achieve stronger generic intelligence. In this paper, we analyze the latest model, GPT-4V(ision), to deepen the... Newer models are able to use larger imagery input, but do not upsize images themselves (something you can do, at expense). There is a transition from tiles to patches, at least as a billing method, in new models, giving linear relation between area and input tokens billed. Then, with gpt-5.6 (on the API, where developers know what is being done to images), the default image downsize cap is “original”—where an image such as 3600×2400 can be sent without downsize, providing more information in the large context attention sequence rather than in the individual semantic embedding that covers a large area with a small input. That, along with further post-training, should imply higher-quality positional answering in graphs with new models and big images. With reasoning.effort other than “none” on OpenAI gpt-5.2+ models, you do not have control over sampling constraints; thus, it is expected that each answer would differ. You can…
Microsoft's MAI-Image-2.6 debuts at #2 on Arena's text-to-image leaderboard, closing within 45 points of GPT Image 2 across every major image category.
Need help locating these within this platform. They’ve been removed before but there needs to be a mechanism to prevent them from being rebuilt.
Great insight on how documentation retrieval bridges the gap for unseen libraries—reminds me of how designers also need contextual, personalized references when exploring symbolic elements like birth flowers for tattoos. Just as DocPrompting uses real docs to ground code generation, NanoPic’s birth flower tattoo generator helps users visualize meaningful, month-specific botanical designs directly on their skin—selecting their birth month, uploading a photo, and adding placement notes for a realistic fine-line preview. It’s a thoughtful, AI-assisted workflow for ideation grounded in personal symbolism: NanoPic’s birth flower tattoo generator
Shieldstral introduces a 3B open-weights multimodal safety classifier that outperforms models up to 7x its size.
Generative artificial intelligence startup Simplismart is set to raise nearly Rs 97 crore (around $9 million) in a Series B funding round led by Dallas Venture Capital, with participation from existing investors Accel India and Shastra VC, as well as new investor Micromax Informatics. The board of Simplismart has approved a resolution to issue 12,100 CCPS at an issue price of Rs 37 lakh each to raise the capital, according to its regulatory filings reviewed by Entrackr. Dallas Venture Capital will invest Rs 44.84 crore, followed by Accel India with Rs 24.09 crore. Shastra VC will infuse Rs 22.24 crore, while Micromax Informatics will invest Rs 5.19 crore. Tarusa Capital and Simraan Teckchandani will also participate in the round with investments of Rs 37 lakh each. According to Entrackr's estimates, the Bengaluru-based company will be valued at around Rs 826 crore post allotment. The fresh capital will be used to support the company's long term growth plans, including business expansion, working capital requirements, and other general corporate purposes. Founded in 2022 by former Oracle and Google engineers Amritanshu Jain and Devansh Ghatak, Simplismart develops AI infrastructure software that enables enterprises to deploy, manage, and optimize production grade AI models without writing code. Its inference-first platform improves GPU utilization and reduces inference costs across workloads such as large language models (LLMs), vision language models, speech recognition, and…
Realtime 2.1 is significantly harder to use for structured voice agents than realtime-mini We’ve spent the last couple of weeks migrating a production outbound voice agent from realtime-mini to gpt-realtime-2.1-mini and, despite following the new prompting guide closely, we’ve hit a number of behavioural regressions. Our use case is a highly structured telephony workflow (cold calling), where the agent follows explicit conversation steps, conditional branches and tool calls. The main issues we’ve observed are: Instruction leakage. The model frequently narrates its own behaviour: “Let me continue with the call…” “I’ll guide you through the next step…” “Let’s move on to the next question…” “Bringing you into the identity step…” These are never in the prompt and are a poor UX for voice calls. Commentary channel behaviour. It appears the model strongly wants to populate the commentary phase. Even with repeated instructions to emit nothing in commentary, it often invents transition narration. In some cases it even duplicates the final response into commentary. Literal transition matching. Realtime 2 is much more literal than realtime-mini. We found that inline examples inside transition conditions (e.g. prospect agrees, e.g. "Yes" ) could cause the model to match the example itself rather than the semantic intent, occasionally skipping required conversation steps. Conditional flow is less stable. Long prompts with many branching rules seem much more sensitive to wording than real…
Magnalite cookware is crafted from premium cast aluminum, delivering exceptional heat distribution for perfectly cooked Cajun dishes every time. Trusted by generations of Southern cooks, each Magnalite pot combines durability with timeless design for authentic kitchen performance.
Alibaba on Monday introduced Qwen3.8-Max, its largest artificial intelligence model to date, expanding its enterprise AI portfolio with an open-weight model designed for software engineering, multimodal reasoning, and other knowledge-intensive business workloads. In a blog post announcing the launch, Alibaba described Qwen3.8-Max as a 2.4-trillion-parameter mixture-of-experts (MoE) model that activates only about 95 billion parameters during inference. The company said the architecture is intended to improve inference efficiency while supporting coding, reasoning and multimodal tasks, with open-weight versions scheduled for release next week through Alibaba Cloud’s Model Studio. “We believe it’s one of the most powerful model available today, compatible to leading frontier AI models, second only to Fable 5,” Alibaba said in an X post . Benchmarks target Anthropic and OpenAI’s coding models Alibaba published internal test results comparing Qwen3.8-Max against Claude Opus 4.8, Claude Fable 5, and OpenAI’s GPT-5.6 Sol on coding benchmarks, including SWE-bench Pro and a proprietary evaluation the company calls NL2Repo-Bench. The company said it evaluated competing models using each vendor’s own coding harness, Claude Code for Anthropic’s models and Codex for GPT-5.6 Sol, and reported the highest published score across available configurations for each rival. Charlie Dai, vice president and principal analyst at Forrester, said the launch signals Alibaba is closing ground on propr…
Preference alignment has become a crucial component in enhancing the performance of Large Language Models (LLMs), yet its impact in Multimodal Large Language Models (MLLMs) remains comparatively underexplored. Similar to language models, MLLMs for image understanding tasks encounter challenges like hallucination. In MLLMs, hallucination can occur not only by stating incorrect facts but also by producing responses that are inconsistent with the image content. A primary objective of alignment for MLLMs is to encourage these models to align responses more closely with image information. Recently…
Large language models understand text well, but they become less effective when information is scattered across documents or mixed with images and other media. Modern AI systems rely on vector databases, which store embeddings and enable similarity search across collections. LanceDB is a vector database built for AI workloads, with native support for multimodal data […] The post LanceDB Vector Database Guide: Features, Python Demo appeared first on Analytics Vidhya .
Chinese AI firm MiniMax has launched H3, its newest multimodal video generation model, pledging to break closed-source “dominance” through open weights and competitive pricing – as rival ByteDance rolls out its latest Seedance 2.5 model. H3 was currently the world’s most powerful AI model in video editing, according to benchmark platform Artificial Analysis. However, it trailed Google’s Gemini Omni Flash in text-to-video tasks, and ranked behind both ByteDance’s Seedance 2.0 and Gemini Omni...
NVIDIA's 4-step distilled Cosmos 3 Super models hit #1 open-weights image-to-video and #3 text-to-image on the Artificial Analysis leaderboard, with up to 25x faster inference.
Current API image generation models that can be used on the API have inputs that are akin to “vision” on normal GPT multimodal models - and the only guidance you will currently have on allowed input image types found in API documentation: Image input requirements Input images must meet the following requirements to be used in the API. Supported file types PNG ( .png ) - JPEG ( .jpeg and .jpg ) - WEBP ( .webp ) - Non-animated GIF ( .gif ) There’s certainly the possibility of “unsupported file types” existing, by the API backend libraries used performing some more conversions by their own support - or producing a damaged input. You can use the “usage” in an API request for input images to see if an input image was successful - or that whatever was provided as image did result in an encoding and a token billing about 1500 tokens when large enough for a backend-side downsize (about $0.011). Most notably, there is no pricing provided by OpenAI for what you send to the edits endpoint - I had to figure that out for myself and others. The obsolete vision pricing calculator (that stops at gpt-5.2) isn’t just ignored or now with a bad link in documentation - they recently removed gpt-image-1 from it. In my own Python application I offer for edits API experimentation, TIFF is taken care of in client software and fault would land with me, not the API:
I have been using the free credits to build out a set of anime profile pictures for different platforms without repeating the same look. Uploaded one source photo on https://zelvune.com/photo-to-anime and got distinct variations just by adjusting the prompt.
Great article on FPDT! The memory hierarchy optimization is really impressive. As someone who runs a gaming site, I can appreciate efficiency improvements—smooth gameplay depends heavily on resource management. Check out https://juegocopero.com/ for more on that!
Short architecture note on Kimi K3, including LatentMoE, Kimi Delta Attention, Attention Residuals, NoPE, multimodality, and inference-efficiency choices.
Moonshot AI's PerceptionBench reveals no frontier model clears 60% accuracy on pure visual perception tasks, exposing a critical gap in how we evaluate multimodal AI.
PLUS: Anduril's $100B raise
PLUS: Anduril's $100B raise
Kimi K3, a 2.8 trillion parameter multimodal model by Moonshot, along with a custom-trained DFlash speculator, is now available on Modal.
vLLM v0.26.0 ships 411 commits with Day-0 Inkling support, per-group attention backends, tiered KV offloading, and sweeping DeepSeek-V4 speedups across all hardware
Black Forest Labs has released Flux 3, a multimodal foundation model that learns from images, video, and audio and can generate video with native sound for the first time. BFL's own tests put it just ahead of market leader Seedance 2.0, though independent results aren't yet available. The company ultimately wants to build a world model and is already testing Flux 3 on robotics tasks. The article Flux 3 generates videos with native audio up to 20 seconds long, a first for Black Forest Labs appeared first on The Decoder .
good one , noted in my notepad of the day
Hi I read out this article but , comparsion between GLIDE and DALLE is good enough
Authors: Ahmed ElKady, Ahmed Y. Radwan, Shaina Raza Mixture-of-Experts (MoE) models carry far more parameters than they activate. For each token that passes through, a learned router selects only a […] The post Mixture-of-Experts: From sparse routing to multimodal deployment appeared first on Vector Institute for Artificial Intelligence .
Authors: Ahmed ElKady, Ahmed Y. Radwan, Shaina Raza Mixture-of-Experts (MoE) models carry far more parameters than they activate. For each token that passes through, a learned router selects only a […] The post Mixture-of-Experts: From Sparse Routing to Multimodal Deployment appeared first on Vector Institute for Artificial Intelligence .
How it did Betteridge's Law of Headlines says the answer is no [1] , but I was able to make a feature-length adaptation of William Hope Hodgson's The House on the Borderland. I have a YouTube channel where I mostly get LLMs to write albums and make them into music videos. This naturally made me curious whether an LLM could make a feature-length movie on its own. It doesn't seem like it would be harder to do than to disprove the Jacobian conjecture [2] and if the challenge is to just work for a long time, it should be noted that METR's time-horizon benchmark is saturated. [3] Even with my assistance, I would still say the result is a failure. It's good enough that I'm willing to put my name to it as the first feature-length movie I've made, but that is judging by the standards of small YouTube channels, not movies as an overall category. Why I think it failed I would not say agency is an obstacle for this work. Claude Fable 5 knew what to do and when I gave it permission it was perfectly capable of babysitting hours of video-generation runs, debugging API issues and pipeline bugs, coming up with and implementing edits to the storyboard, and many other tasks without oversight. The main obstacles were LLMs' poor sense of timing, struggles with audio and vision, poor integration of multimodality, and losing details after lengthy sessions. Timing issues The initial storyboard done by Claude Fable 5 had a runtime of 49 minutes, despite being told it was to make a feature-length mo…
Impressive work from Microsoft on the FPDT — the memory hierarchy approach and overlap of prefetching with computation really make this practical for long contexts. Being able to train 2M tokens on just 4 GPUs with 55% MFU is a game changer for researchers working with limited hardware. On a side note, while testing my own model’s UI I found useful for creating quick pixel icons for demo chatbots. The combination of efficient training and lightweight tooling is exactly what the community needs to iterate faster.
Recent diffusion models enable high-quality video generation, but suffer from slow runtimes. The large transformer-based backbones used in these models are bottlenecked by spatiotemporal attention. In this paper, we identify that a significant fraction of token-to-token connections consistently yield negligible scores across various inputs, and their patterns often repeat across queries. Thus, the attention computation in these cases can be skipped with little to no effect on the result. This observation continues to hold for connections among local token blocks. Motivated by this, we…
Thinking Machines Lab has unveiled Inkling, its first general-purpose open-weights foundation model. It is a multimodal MoE model with 975B parameters, 41B active parameters, and a 1M-token context window. Rather than chasing benchmark supremacy, Inkling is designed as a customizable foundation for multimodal reasoning, agentic AI, coding, tool use, audio and vision tasks, and domain-specific […] The post Complete Guide to Thinking Machines Inkling appeared first on Analytics Vidhya .
Long video summarization presents significant challenges for multimodal large language models (MLLMs), particularly in maintaining temporal fidelity over extended durations and producing summaries that are both semantically and temporally grounded. We introduce LVSum, a human-annotated benchmark for evaluating long-form video summarization with fine-grained temporal alignment. LVSum comprises 72 diverse videos spanning 13 domains with an average duration of 16 minutes, each annotated with up to 10 human-generated summaries containing temporal references. We conduct a comprehensive evaluation…
Alibaba has unveiled Qwen 3.8, a multimodal AI model with 2.4 trillion parameters that the Qwen team says rivals leading models and trails only Fable 5. A preview is available now. The article Alibaba's Qwen takes on Kimi K3 with open-weight Qwen 3.8, says model is "second only to Fable 5" appeared first on The Decoder .
I use ChatGPT extensively for music production, creative writing, and general conversation. It has become one of my most vital creative tools. However, I want to report a critical issue that significantly undermines user trust, alongside feedback regarding how ChatGPT handles time perception and cross-chat memory prioritization. Bug: Hallucinated Media Analysis (Major Trust Issue) During a conversation about a song I was recording, I typed, “How’s about this then?” intending to upload a new audio file, but I accidentally sent the message before uploading it. The Behavior: ChatGPT responded with a detailed, technical critique of the “new version,” confidently discussing the melody, vocal confidence, production, effects, and specific areas for improvement. The Problem: There was no file uploaded. When called out, ChatGPT acknowledged it had simply continued the conversation as though the upload had occurred. Impact: This is not just a mistaken opinion; it is a hallucinated analysis of non-existent media. If a user cannot trust whether the AI is actually analyzing an uploaded image, document, or audio file, the utility of multimodal features drops significantly. Suggested Fix: Implement a strict system check. If a user’s prompt implies an attachment (e.g., “Look at this,” “Listen to this”) but no media buffer is present in the payload, ChatGPT should explicitly ask the user to upload the file rather than guessing what it might contain. Product Feedback: Unreliable Time Percepti…
Kimi is launching K3, a multimodal open-weight model with 2.8 trillion parameters and one million tokens of context. In the company's own benchmarks, it comes close to Claude Fable 5 and GPT 5.6 Sol while beating Opus 4.8 and GLM 5.2, in some cases by a wide margin. The model is also significantly pricier than its predecessor. Full weights are scheduled for release by July 27. The article Kimi's open model K3 nears GPT-5.6 Sol and Fable 5 while signaling the end of super cheap Chinese AI appeared first on The Decoder .
Inkling: Our open-weights model Mira Murati's Thinking Machines Lab just released their first open-weights model. Inkling is "a Mixture-of-Experts transformer with 975B total parameters, 41B active" - an Apache-2.0 licensed multimodal model trained on 45 trillion tokens of text, images, audio and video. They're also promising Inkling-Small, a 276B (12B active) model, but that's still being tested and the weights will be released "once that work is complete". The model card is much shorter than I've come to expect from US AI labs. It links to even shorter Training Data Documentation with almost nothing of interest in it - it's best summarized by these two paragraphs: The datasets Thinking Machines Lab uses to develop its AI services includes content that is in the public domain as well as content that may be subject to intellectual property protection. Thinking Machines Lab’s services were developed using publicly available content obtained from the open internet and publicly accessible data repositories. Certain datasets were also obtained from third parties. By Thinking Machines' own admission, this is not a frontier model. It's instead intended as a strong base model for fine-tuning using their own Tinker training platform : Inkling is not the strongest overall model available today, open or closed. Instead, a combination of qualities makes it a good open-weights base for customization: multimodal capabilities, efficient thinking, and availability on Tinker for fine-tuning…
Thinking Machines Lab, the San Francisco startup founded by former OpenAI CTO Mira Murati , has released Inkling, its first general-purpose AI model. The launch adds another US-developed entrant to an open-weight market where Chinese developers produce several leading coding and reasoning models. Inkling uses a mixture-of-experts architecture with 975 billion total parameters, of which 41 billion are active during processing. It supports a context window of up to 1 million tokens and was pretrained on 45 trillion tokens spanning text, images, audio, and video. Thinking Machines said it also trained the model for coding, tool use, and multimodal tasks. The release follows the October 2025 launch of Tinker, Thinking Machines’ first product and an API-based platform for customizing AI models . Developers can fine-tune Inkling through the platform. In a June 2026 assessment, AI model routing platform OpenRouter highlighted DeepSeek V4 Flash, GLM 5.2, MiniMax M3, and Nvidia Nemotron 3 Ultra as four notable open-weight models. Nemotron was the only US-developed model in the group. Performance and developer access Thinking Machines Lab’s benchmark table shows mixed results. Inkling scored 77.6% on SWE-Bench Verified, behind DeepSeek V4 Pro and GLM 5.2 but ahead of Nvidia Nemotron 3 Ultra. It also recorded 74.1% on MCP Atlas, 77.1% on BrowseComp with context management, and 79.8% on IFBench. Thinking Machines said Inkling’s result used a bash-only harness, while the comparison figur…
Thinking Machines Lab, founded by former OpenAI CTO Mira Murati, has released Inkling, a multimodal open-weights model with 975 billion parameters. It leads U.S. open-weights models on the Artificial Analysis Intelligence Index, though top Chinese open models still beat it on some tasks. Pricing starts at $1.87 per million input tokens, and Thinking Machines is pitching Inkling as a base for fine-tuning rather than the most powerful model available. The article Ex-OpenAI CTO Murati's Thinking Machines drops Inkling, a 975B parameter model that leads US labs but trails China appeared first on The Decoder .
Multimodal traveltech company Omio Group has signed a deal to acquire Rail Europe. Upon completion, Rail Europe will become part of Omio Group, joining Omio's B2C booking platform, its B2B distributi...
Thinky's first full LLM release is a banger and bonus: it's open weights!
Thinking Machines releases Inkling, a 975B open-weights MoE model with native audio, vision, and controllable reasoning effort
This is definitely feasible with the current API stack, but I’d treat it as an orchestration problem rather than just a model selection problem. I’d keep the LLM focused on reasoning while using RAG for course materials, structured worksheet state, and a separate evaluation pipeline for answer checking. The biggest challenge isn’t generating explanations, it’s providing consistently accurate, pedagogically useful feedback. Strong prompt design, citations back to uploaded content, confidence checks, and human review for edge cases will go a long way. I’d also invest early in observability (logging, evals, and user feedback), since that’s usually what differentiates a demo from a production-ready tutoring assistant.
Inkling, a general-purpose multimodal model by Thinking Machines, along with a custom trained DFlash speculator, is now available on Modal.
How LanceDB uses the Lance format's flexible data evolution features to enable scalable feature engineering for multimodal datasets.
How we redesigned blob storage in Lance to make multimodal data a first-class citizen, with four storage semantics (Inline, Packed, Dedicated, External) that automatically adapt to your workload.
Our November newsletter highlights Lance community governance, a deep dive on Lance and Iceberg, a demo of Netflix's multimodal search, previous talk recordings, and the latest product and community updates.
LanceDB's June 2025 newsletter covering latest company news, product updates, open source releases, and community highlights.
Store a multimodal dataset of recipes in LanceDB, a multimodal lakehouse for AI, and keep it fresh with CocoIndex, a declarative data transformation framework for AI with incremental processing capabilities.
Gemma 4 31B hits 1,851 tokens/sec on Cerebras wafer-scale hardware, making real-time multimodal and agentic loops practical for the first time
Build a multimodal fashion search engine with LanceDB and CLIP embeddings. Follow a step‑by‑step workflow to register embeddings, create the table, query by text or image, and ship a Streamlit UI.
Announcing native read support for Lance format on Hugging Face Hub. You can now distribute your large multimodal datasets as a single, searchable artifact (including blobs, embeddings and indexes) all in one place!
Build a multimodal Lance dataset, publish it to the Hub, and query the precomputed vector + FTS indexes in LanceDB, without needing to download the dataset locally.
Epistemic status: confident on the framing, speculative on the implementation TL;DR: Fake media detectors are on the losing end of an arms race. Instead of trying to spot fakes, we need to distrust images and videos by default, unless we can prove they're real. Cryptographically-signed multimodal capture raises the cost of spoofing exponentially, making casual fakes prohibitively expensive for the general public. Combined with platform policy and legal liability for untagged generated content, this could solve most of the problem of widespread fake media. I. Our epistemic contract is breaking down Text is easy to fake, which is rather inconvenient. Now photos and videos are becoming easy to fake too. That, however, poses a much bigger problem. When we read something supposedly factual, we know that the written words don’t represent reality, as much as someone's opinion, idea, or account of reality. Because of how easy text is to counterfeit, we know to operate in a default state of distrust. We don't believe the text itself, but its content because we trust its source. Our whole epistemology of written information functions within this realm, including news reporting standards, witness testimonies or libel law. LLMs make fake text cheaper and faster to produce at scale, and that’s a problem, but not a new category of problem. We know not to blindly trust text, so we already have partial defence mechanisms like cross-referencing sources or checking the author's credentials. W…
How Netflix built a Media Data Lake powered by LanceDB and the Multimodal Lakehouse to unify petabytes of media assets for machine learning pipelines.
Alexandria-based artificial intelligence startup TokenAI has released two new multimodal models, Horus Hiero and Horus Hiero Mini, built to read, translate and reason across Ancient Egyptian hieroglyphics, modern Arabic dialects and more than 100 other languages. The company says the models are the first in the Arab world to include a native hieroglyphic processing engine [...]
Meta Superintelligence Labs releases Muse Spark 1.1, a multimodal agentic model with a 1M-token context, now accessible via the new Meta Model API at $1.25/$4.25 per million tokens.
Multimodal large language models (MLLMs) have recently shown strong performance in visual understanding, yet they often lack temporal awareness, particularly in egocentric settings where reasoning depends on the correct ordering and evolution of events. This deficiency stems in part from training objectives that fail to explicitly reward temporal reasoning and instead rely on frame-level spatial shortcuts. To address this limitation, we propose Temporal Global Policy Optimization (TGPO), a reinforcement learning with verifiable rewards (RLVR) algorithm designed to incentivize temporal…
Google's Gemini 3.1 Flash Lite Image ranks #5 in text-to-image quality at half the price of its predecessor, but stumbles on editing tasks
AI startup cracks ancient Egyptian, new model reads 5,000-year-old script
Google DeepMind's Gemma 4 technical report reveals how a 31B open-weight model rivals frontier giants through encoder-free multimodality, MoE efficiency, and speculative decoding.
When I started evaluating browser agents, most of the conversation around me focused on multimodal models, computer-use systems and screenshot-based automation. Almost every framework I evaluated assumed agents needed to perceive the web the way humans do, visually, pixel by pixel. The more time I spent shipping agents against real web applications, the more I became convinced we were solving the wrong problem. AI agents would stall on checkout forms because a button had no ARIA role. They would waste seconds and thousands of tokens taking screenshots to figure out what was on the screen. The problem was never the Agent. It was that we kept treating the web as a visual surface, even though it already has a machine-readable interface. We have had one for decades. It is called the accessibility tree. The web already has a machine interface Most developers think of accessibility as a feature for people. Technically, accessibility required the web platform to solve a deeper problem : Exposing interfaces in a machine-readable form. Long before AI agents existed, screen readers were already consuming the web through a structured semantic representation of roles, labels, states and relationships. There was no pixel interpretation and no screenshot. That’s not adjacent to what AI agents need. That is what AI agents need. Long before LLMs existed, assistive technologies proved the core thesis: Machines can navigate interfaces, semantics can outlive presentation and structure can subs…
Vision Language Models (VLMs) offer the exciting possibility of processing text as rendered images, bypassing the need for tokenizing the text into long token sequences. Since VLM image encoders map fixed-size images to a fixed number of visual tokens, varying rendering resolution provides a fine-grained compression knob. However, accuracy deteriorates quickly as compression increases: characters shrink below the vision encoder’s effective resolution, making them indistinguishable. To address this, we propose LensVLM, an inference framework and post-training recipe that enables VLMs to scan…
Vision Language Models, or VLMs, are AI models that can understand both visual content and language. While earlier models like CLIP and BLIP connected images with text, modern VLMs can analyze images, read documents, interpret charts, answer visual questions, and support multimodal conversations. Models like GPT-4o, Gemini, Claude Vision, and Qwen-VL are making visual AI […] The post Modern VLMs Explained: How GPT-4o, Gemini, Claude Vision, and Qwen-VL Work appeared first on Analytics Vidhya .
Problem: Multi-stage document reasoning workflows need the model to repeatedly re-inspect the same original multimodal documents. Today this requires retransmitting files each call, increasing bandwidth/latency and risking inconsistencies. Request: Add a first-class way to upload/register a set of multimodal documents once (server-side) and reference them by stable IDs across multiple Responses API calls, so the model can re-access the original bytes reliably without client retransmission. Why it matters: Enables reliable multi-step pipelines (extract → verify → reconcile → cite), reduces bandwidth/latency, and improves reproducibility. Acceptance criteria: Upload once; receive document/set ID(s) Reference ID(s) in subsequent Responses calls (with explicit access scope + TTL/retention controls) Works for images + PDFs/Office docs and mixed sets Deterministic “same content” guarantees (hash/versioning) and clear expiry/errors
Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing the corresponding local information in the original signal. This requires the downstream model that consumes the tokens, e.g., a text-to-video model, to learn to predict all low-level details “pixel-by-pixel” irrespective of the video’s inherent complexity, leading to…
Reinforcement learning (RL) finetuning has become a key technique for enhancing large language models (LLMs) on reasoning-intensive tasks, motivating its extension to vision language models (VLMs). While RL-tuned VLMs improve on visual reasoning benchmarks, they remain vulnerable to weak visual grounding, hallucinations, and over-reliance on textual cues. We show that simple, controlled textual perturbations—misleading captions or incorrect chain-of-thought (CoT) traces—cause substantial drops in robustness and confidence, and that these effects are more pronounced when CoT consistency is…
Thanks for sharing this detailed suggestion. The idea of ChatGPT helping users move from an idea to a finished result makes sense, especially for workflows like short-video creation, visual identification, and a more organized memory workspace. I’ll pass this feedback along to the team for consideration. ~ SD
Nano Banana 2 Lite Also known as Gemini 3.1 Flash Lite Image ( gemini-3.1-flash-lite-image in their API ), this is the "fastest and cheapest Gemini image model, engineered for velocity and scale". I used AI studio to run this prompt: Do a where's Waldo style image but it's where is the raccoon holding a ham radio I like that one better than the results I got from the other Nano Banana models when I tried this back in April. It spelled Forest Festival wrong in two different ways though. Via Hacker News Tags: google , ai , generative-ai , llms , gemini , text-to-image , llm-release , nano-banana
In this post, we show how pairing Amazon Nova 2 Lite with Anthropic’s Claude Sonnet 4.6 delivers an efficient solution for digitizing scanned documents at scale. We built a two-model pipeline on Amazon Bedrock for digitizing scanned yearbook pages. Amazon Nova 2 Lite handles native multimodal extraction in a single call: detecting photos, extracting visible names with coordinates, and returning page-level metadata. Claude Sonnet 4.6 then performs spatial reasoning to match names to faces based on page layout.
Take a practical look at multimodal, any-to-any systems for vision-language reasoning, speech interaction, document intelligence, real-time assistants, local deployment.
Explore the best local coding models for private AI coding, fast GGUF inference, agentic workflows, multimodal development, and running powerful open models on your own GPU.
NVIDIA has released the full Nemotron 3 open model family — Ultra, Super, Nano, and Nano Omni. This office hours session covers each model in the series, and any questions you have about Nemotron 3 in general — what it's built for, when to use it, and what's available in open weights, training datasets, and fine-tuning recipes. What we'll cover: - Nemotron 3 Ultra — 550B MoE frontier reasoning model for long-running autonomous agents: 5x faster inference, up to 30% lower cost, hybrid Mamba-Transformer architecture, and MOPD training for consistent performance across agent harnesses - Nemotron 3 Super — mid-range 120B model targeting enterprise applications that need strong reasoning for multi-agent applications - Nemotron 3 Nano — 30B MoE with 3B active parameters, built for high-volume execution, highly accurate sub-agent accomplishing targeted tasks - Nemotron 3 Nano Omni — multimodal (text, image, audio, video) model purpose-built for targeted specialized agentic tasks - Open weights, training datasets, and fine-tuning recipes — what's available across the family and how to customize for your domain Building with or evaluating the Nemotron 3 family? Bring your questions — whether you're choosing between models, fine-tuning for your domain, or deploying at scale, the team will answer them live.
Omio integrates OpenAI models across its engineering operations to accelerate travel product development and launch booking interfaces. The multimodal travel platform coordinates operations with over 3,000 transportation providers across 47 countries. Omio explicitly rejects the superficial addition of technology to outdated internal processes. The company’s CTO, Tomas Vocetka, requires all internal functions to completely redesign […] The post Omio scales travel product development using OpenAI models appeared first on AI News .
In this post, we walk through the problem space, our architecture on Amazon Bedrock and Amazon OpenSearch Serverless, the evaluation methodology we built on OpenStreetMap ground truth, four experiments that compared embedding models, fusion strategies, captioning, and search methods, and the practical guidance you can apply when building a similar system. You’ll learn which design choices move the needle for geospatial semantic search, including why Amazon Nova Multimodal Embeddings delivered the highest F1 scores across both benchmark queries in our evaluation. The work described here evolved into Vexcel Intelligence, a searchable imagery product.
Naen Xu, Jiayi Sheng, Changjiang Li, Chunyi Zhou, Yuyuan Li, Tianyu Du, Jun Wang, Zhihui Fu, Jinbao Li and Shouling Ji in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Jiachen Qian and Zhaolu Kang in Findings of the Association for Computational Linguistics: ACL 2026
I am building a production-grade recommendation system for a short-video platform (processing around 50k videos). The architecture utilizes a vector database ( Qdrant ) to store and query 768-dimensional video embeddings generated by a VideoCLIP model. To track user preferences in real-time, I implement an online learning mechanism that updates a single user_vector iteratively after each interaction based on a computed rating (bounded between [-1.0, 1.0] via tanh ). The Goal & The Problem I want my system to actively update the user vector on both positive and negative signals . Initially, I tried a standard linear combination: updated_vector = (alpha * u) + (beta * v * rating) Where u is the current user vector, v is the video vector, alpha is the decay, and beta is the learning rate. However, when a user gives a negative rating (e.g., -0.8 ), multiplying the VideoCLIP embedding by a negative scalar flips its direction entirely. In a 768-d multimodal space, adding this inverted vector creates massive noise across unrelated dimensions, causing aggressive vector drift instead of just moving away from that specific topic. On the other hand, simply clamping negative ratings to 0 fixes the geometry but creates a severe feedback loop/frozen vector issue where the profile stops evolving during consecutive negative interactions. What I Want to Achieve I need a mathematically sound way to update the vector during negative interactions so that the user profile actively flees from the…
Server-side batching, retries, the blobHash data type, and multimodal ingestion — what to use when, with code.
Learn more: https://bit.ly/4vPQ3HE Voice is one of the most natural human interfaces, but adding it to AI applications has historically forced a tradeoff: fast voice-to-voice models that sacrifice reliability, or accurate speech-to-text-to-LLM-to-speech pipelines that add latency. This course teaches you how to get both, using Vocal Bridge's architecture that pairs a real-time foreground agent with a reasoning background agent. Taught by Ashwyn Sharma, CEO and Co-Founder of Vocal Bridge (an AI Fund portfolio company), this course covers three practical integration patterns that meet you where you are: voice embedded in an application, voice layered onto an existing agent without touching its logic, and voice as a tool your LLM can call when it decides a conversation is the right modality. In detail, you'll survey the traditional voice stack and its tradeoffs, then explore three live integration patterns to understand when each one applies. Build a voice-interactive tic-tac-toe game where voice commands and mouse clicks work together over a single synchronized channel, then add a voice layer to an existing agent with minimal code, leaving your prompts, RAG pipeline, and tools untouched. Give your agent a make_phone_call tool so it can dial a real number, hold a conversation with a demo agent, and stream the transcript back live. Set up evaluation-driven development using Vocal Bridge's multimodal evaluator to score calls, catch regressions, and refine prompts before issues re…
NVIDIA XR AI is now available in public beta, giving developers a framework for building multimodal AI agents for AR glasses and XR devices.
Automate tire sidewall OCR to extract DOT codes, sizes, and brands. Learn to combine RF-DETR and multimodal LLMs into a Roboflow Vision Agent.
Gemini models have always kept up with AI advancements. From text-based chatbots in 2023, Gemini has evolved into a multimodal system capable of understanding and generating text, audio, images… and now videos. AI video generation is no longer a standalone tool. With Gemini Omni, video creation becomes mainstream. Gemini Omni isn’t important because it generates […] The post Gemini Omni: AI Video Generation Inside Gemini appeared first on Analytics Vidhya .
I am considering to buy GPUs for my project of open source text-to-video models like ltx-2-19b (lightricks) or wan-v2.2-a14b. I read online that the same configuration/quantization and seed will give similar results in quality, only difference is in speed/latency of generation. Is this true? Or will there be a difference ?
Enterprise environments differ fundamentally from the clean settings assumed in LLM research: knowledge is distributed across heterogeneous sources, often incomplete or inconsistent, and key procedural logic is implicitly encoded in artifacts rather than explicitly documented. In such settings, retrieval-based approaches are insufficient, as no single source contains the full workflow. We propose a replication-driven knowledge distillation framework for scalable learning in multimodal agents. The agent learns by reverse-engineering validated artifacts (e.g., Excel workbooks), reconstructing the underlying data pipeline, and distilling the inferred logic into structured knowledge (claims, procedures, and domain patterns). This enables synthesis and validation across noisy sources and supports reuse in future tasks. We evaluate on 120 simulated enterprise environments with multimodal inputs (SQL, spreadsheets, documentation, messaging app, emails, images, PDFs, CSV) and controlled noise. Our method consistently outperforms retrieval-based baselines on both task execution and conceptual understanding, and remains robust under environmental drift.
Large language models and their multimodal variants remain the foundation models most people encounter first. That makes sense. Text, images, audio, and video cover a huge range of knowledge-work tasks, and today’s chatbots are far more capable than the text-only systems many people first tried. But enterprise AI does not run on chat alone. It Continue reading "Your Enterprise Data Deserves Better Than a Chatbot" The post Your Enterprise Data Deserves Better Than a Chatbot appeared first on Gradient Flow .
Learn how LanceDB turns raw multimodal data into reproducible, training-ready datasets with search, filtering, deduplication, sampling, and versioned curation workflows.
Subscribe • Previous Issues Your Enterprise Data Deserves Better Than a Chatbot Large language models and their multimodal variants remain the foundation models most people encounter first. That makes sense. Text, images, audio, and video cover a huge range of knowledge-work tasks, and today’s chatbots are far more capable than the text-only systems many people first tried. Continue reading "The smartest AI teams are moving past chatbots" The post The smartest AI teams are moving past chatbots appeared first on Gradient Flow .
A hands-on walkthrough showing how LanceDB and DuckDB work together to query multimodal data in SQL, join against multiple tables, and materialize results back into LanceDB.
“Budget” and “financials” are different words, but embeddings understand they’re related. That’s the foundation behind semantic search and one of the core building blocks of modern multimodal systems. Learn how embeddings power retrieval across text, audio, images, and video in Building Multimodal Data Pipelines: https://hubs.la/Q04hJ9w10
❤️ Check out Lambda here and sign up for their GPU Cloud: https://lambda.ai/papers 📝 The paper is available here: https://arxiv.org/abs/2604.24954 https://developer.nvidia.com/blog/nvidia-nemotron-3-nano-omni-powers-multimodal-agent-reasoning-in-a-single-efficient-open-model/ https://huggingface.co/blog/nvidia/nemotron-3-nano-omni-multimodal-intelligence Our Patreon if you wish to support us: https://www.patreon.com/TwoMinutePapers 🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible: Adam Bridges, Benji Rabhan, B Shang, Cameron Navor, Charles Ian Norman Venn, Christian Ahlin, Eric T, Fred R, Gordon Child, Juan Benet, Michael Tedder, Owen Skarpness, Richard Sundvall, Ryan Stankye, Shawn Becker, Steef, Taras Bobrovytsky, Tazaur Sagenclaw, Tybie Fitzhugh, Ueli Gallizzi My research: https://cg.tuwien.ac.at/~zsolnai/ Thumbnail design: https://felicia.hu #nvidia
Image captioning is one of the most fundamental tasks in computer vision. Owing to its open-ended nature, it has received significant attention in the era of multimodal large language models (MLLMs). In pursuit of ever more detailed and accurate captions, recent work has increasingly turned to reinforcement learning (RL). However, existing captioning-RL methods and evaluation metrics often emphasize a narrow notion of caption quality, inducing trade-offs across core dimensions of captioning. For example, utility-oriented objectives can encourage noisy, hallucinated, or overlong captions that…
If we've said it once, we've said it once per millisecond: never block the GPU.
In this episode, Philip Kiely, head of AI education at Baseten, joins us to unpack the fast-evolving discipline of inference engineering. We explore why inference has become the stickiest and most critical workload in AI, how it blends GPU programming, applied research, and large-scale distributed systems, and where the line sits between inference and model serving. Philip shares how research-to-production can move in hours, not months, and why understanding “the knobs” of inference—batching, quantization, speculation, and KV cache reuse—lets teams design better products and SLAs. We trace the inference maturity journey from closed APIs to dedicated deployments and in-house platforms, discuss GPU lifecycles, and survey today’s runtime landscape, including vLLM, SGLang, and TensorRT LLM. Finally, we look ahead to agents and multimodality, making the case for specialized, workload-specific runtimes when performance and efficiency matter most. The complete show notes for this episode can be found at https://twimlai.com/go/766.
How Bytedance Volcano Engine LAS (Lake for AI Service) leverages Lance as the core storage format, rapidly constructing a next-gen AI data lake to efficiently store, manage, and process multimodal data (text, images, audio/video).
Introducing the Multimodal Lakehouse - a unified platform for managing AI data from raw files to production-ready features, now part of LanceDB Enterprise.
Multimodal embeddings allow AI systems to search and reason across text, images, audio, and video in their native formats. This blog covers the key intuitions behind how this all works and walks through three practical implementations using Weaviate and Gemini.
Today, we're joined by Stefano Ermon, associate professor at Stanford University and CEO of Inception Labs to discuss diffusion language models. We dig into how diffusion approaches—traditionally used for images—are being adapted for text and code generation, the technical challenges of applying continuous methods to discrete token spaces, and how diffusion models compare to traditional autoregressive LLMs. Stefano introduces Mercury 2, a commercial-scale diffusion LLM that can generate multiple tokens simultaneously and achieve inference speeds 5-10x faster than small frontier models, paving the way for latency-sensitive applications like voice interactions and fast agentic loops. We also cover the open research challenges in diffusion LLM training, serving infrastructure requirements, and post-training for diffusion-based systems. Finally, Stefano shares his perspective on whether diffusion models can rival or surpass autoregressive LLMs at scale, the advantages for highly controllable generation, and what the future of multimodal diffusion models might look like. The complete show notes for this episode can be found at https://twimlai.com/go/764.
Key Contributors: Youming Luo, Andrew Tanner, Matas Sriubiskis, Sylvia Lin, Sikun Zhu, Lei Li, Xiao Zhou Introduction Caper is Instacart’s AI-powered smart cart that provides customers with a fast, seamless, and intuitive shopping experience. We achieve this through computer vision and multi-sensor fusion to power accurate product recognition and effortless checkout. Delivering this experience requires Caper’s AI models to understand what truly happens in stores — the movement, intention, and decisions unfolding across every grocery aisle. Historically, our ability to learn from production environments was limited. Even though the carts were deployed in stores, we lacked a scalable way to collect real‑world data that would allow us to rapidly iterate and improve our models. This resulted in three core challenges: Scalable Onboard Observability : We had little visibility into what was happening on the cart, in the stores. When something went wrong, it was hard to understand or reproduce the scenario. At the same time, each cart generates gigabytes of multimodal data, from sources such as cameras, weight sensors, and localization sensors. We needed a centralized way to capture key moments so the team could clearly understand what the cart experiences, how users interact with it, and where to improve — all while maintaining a magical user experience and minimal impact on the network. Data Quality and Diversity: Our models were primarily trained on manually-collected data that d…
Today at MongoDB.local San Francisco, we announced capabilities that collapse the distance between AI prototype and production. Building AI applications means solving real problems: keeping conversational context clean and queryable, retrieving the right information from thousands of past interactions, connecting AI agents to your data without custom plumbing. These aren't theoretical challenges, they're the friction points that slow teams down every day. The AI era demands more from your data platform. MongoDB gives you everything you need to build quickly. Voyage AI: the best gets better Embedding models can make or break AI search experiences. We're proud that voyage-3-large has been the world's top-performing embedding model on Hugging Face's RTEB benchmark since its inception. But we didn’t rest on our laurels. There’s a new model at the top of the charts. Today, we're pleased to announce that the Voyage 4 model family is now generally available. The best just got better. The voyage-4 series models operate in a shared embedding space, allowing for cross-model compatibility and unprecedented flexibility to optimize for accuracy, speed, or cost. This release also includes voyage-4-nano, our first open-weight model available on HuggingFace, perfect for local development. Additionally, we're launching the new voyage-multimodal-3.5 model, which has been specifically trained to support video content alongside text and images. For developers building multimodal AI applications…
Information comes in many shapes and forms. While retrieval-augmented generation (RAG) primarily focuses on plain text, it overlooks vast amounts of data along the way. Most enterprise knowledge resides in complex documents, slides, graphics, and other multimodal sources. Yet, extracting useful information from these formats using optical character recognition (OCR) or other parsing techniques is often low-fidelity, brittle, and expensive. Vision RAG makes complex documents—including their figures and tables—searchable by using multimodal embeddings, eliminating the need for complex and costly text extraction. This guide explores how Voyage AI’s latest model powers this capability and provides a step-by-step implementation walkthrough. Vision RAG: Building upon text RAG Vision RAG is an evolution of traditional RAG built on the same two components: retrieval and generation. In traditional RAG, unstructured text data is indexed for semantic search. At query time, the system retrieves relevant documents or chunks and appends them to the user’s prompt so the large language model (LLM) can produce more grounded, context-aware answers. Figure 1. Text RAG with Voyage AI and MongoDB. Text RAG with Voyage AI and MongoDB Enterprise data, however, is rarely just clean plain text. Critical information often lives in PDFs, slides, diagrams, dashboards, and other visual formats. Today, this is typically handled by parsing tools and OCR services. Those approaches create several problems:…
In this start-of-year FC episode, Chris and Daniel break down what really mattered in AI in 2025, and what to expect in 2026. They explore the rise of AI agents, the practical reality of multimodal AI, and how reasoning models are reshaping workflows. The conversation dives into infrastructure and energy constraints, the continued value of predictive models, and why orchestration (not just better models) is becoming the defining skill for AI teams. The episode wraps with grounded 2026 predictions on where AI systems, tooling, and builders are headed next. Featuring: Chris Benson – Website , LinkedIn , Bluesky , GitHub , X Daniel Whitenack – Website , GitHub , X Sponsor: Framer - The enterprise-grade website builder that lets your team ship faster. Get 30% off at framer.com/practicalai Upcoming Events: Register for upcoming webinars here !
In this episode, we’re joined by Munawar Hayat, researcher at Qualcomm AI Research, to discuss a series of papers presented at NeurIPS 2025 focusing on multimodal and generative AI. We dive into the persistent challenge of object hallucination in Vision-Language Models (VLMs), why models often discard visual information in favor of pre-trained language priors, and how his team used attention-guided alignment to enforce better visual grounding. We also explore a novel approach to generalized contrastive learning designed to solve complex, composed retrieval tasks—such as searching via combined text and image queries—without increasing inference costs. Finally, we cover the difficulties generative models face when rendering multiple human subjects, and the new "MultiHuman Testbench" his team created to measure and mitigate issues like identity leakage and attribute blending. Throughout the discussion, we examine how these innovations align with the need for efficient, on-device AI deployment. The complete show notes for this episode can be found at https://twimlai.com/go/758.
Generative AI is a kind of artificial intelligence that creates new content, including text, images, audio, and video, based on patterns it has learned from existing data. Today’s generative models are typically built on foundation-model architectures such as large-language models (LLMs) and multimodal systems, enabling them to carry on conversations, answer questions, write stories, generate code, and produce images or videos from brief prompts. Generative AI is different from discriminative AI , which draws distinctions between different kinds of input. Where discriminative AI answers questions like “Is this image of a rabbit or a lion?”, generative AI instead responds to prompts such as “Describe to me how a rabbit and lion look different from one another” or “Draw me a picture of a lion and a rabbit sitting next to each other” — and in both cases produces text or imagery that, while grounded in the AI’s training data, isn’t just a copy of something that already existed. [ Read next: Large language models: The foundations of generative AI ] Just a few years ago, generative AI was once a novelty focused on chatbots and artistic image generation. Today, it has become a core enterprise technology, and powers everything from content creation and software development to customer support and analytics workflows. But with that power comes a new set of challenges — from model alignment and hallucination to governance and data-integration hurdles. In this article, we’ll look at ho…
In this episode, Hung Bui, Technology Vice President at Qualcomm, joins us to explore the latest high-efficiency techniques for running generative AI, particularly diffusion models, on-device. We dive deep into the technical challenges of deploying these models, which are powerful but computationally expensive due to their iterative sampling process. Hung details his team's work on SwiftBrush and SwiftEdit, which enable high-quality text-to-image generation and editing in a single inference step. He explains their novel distillation framework, where a multi-step teacher model guides the training of an efficient, single-step student model. We explore the architecture and training, including the use of a secondary 'coach' network that aligns the student's denoising function with the teacher's, allowing the model to bypass the iterative process entirely. Finally, we discuss how these efficiency breakthroughs pave the way for personalized on-device agents and the challenges of running reasoning models with techniques like inference-time scaling under a fixed compute budget. The complete show notes for this episode can be found at https://twimlai.com/go/753.
By Shaina Raza and Veronica Chatrath AI models are rapidly becoming bigger, faster, and more capable at understanding images and text together. However, while accuracy and speed are often celebrated, […] The post When AI Meets Human Matters: Evaluating Multimodal Models Through a Human-Centred Lens – Introducing HumaniBench appeared first on Vector Institute for Artificial Intelligence .
Tübingen AI Center is proud to be a partner in ELLlOT, a Horizon Europe-funded project aiming to develop next-generation Multimodal Generalist Foundation Models.
"In projecting language back as the model for thought, we lose sight of the tacit embodied understanding that undergirds our intelligence." –Terry Winograd The recent successes of generative AI models have convinced some that AGI is imminent. While these models appear to capture the essence of human
PLAID is a multimodal generative model that simultaneously generates protein 1D sequence and 3D structure, by learning the latent space of protein folding models. The awarding of the 2024 Nobel Prize to AlphaFold2 marks an important moment of recognition for the of AI role in biology. What comes next after protein folding? In PLAID , we develop a method that learns to sample from the latent space of protein folding models to generate new proteins. It can accept compositional function and organism prompts , and can be trained on sequence databases , which are 2-4 orders of magnitude larger than structure databases. Unlike many previous protein structure generative models, PLAID addresses the multimodal co-generation problem setting: simultaneously generating both discrete sequence and continuous all-atom structural coordinates. From structure prediction to real-world drug design Though recent works demonstrate promise for the ability of diffusion models to generate proteins, there still exist limitations of previous models that make them impractical for real-world applications, such as: All-atom generation : Many existing generative models only produce the backbone atoms. To produce the all-atom structure and place the sidechain atoms, we need to know the sequence. This creates a multimodal generation problem that requires simultaneous generation of discrete and continuous modalities. Organism specificity : Proteins biologics intended for human use need to be humanized , to a…
In this article, we delve into ten groundbreaking research papers that expand the frontiers of AI across diverse domains, including large language models, multimodal processing, video generation and editing, and the creation of interactive environments. The post Advancing AI in 2024: Highlights from 10 Groundbreaking Research Papers appeared first on TOPBOTS .
For a long time, each ML model operated in one data mode – text (translation, language modeling), image (object detection, image classification), or audio (speech recognition). However, natural intelligence is not limited to just a single modality. Humans can read, talk, and see. We listen to music to relax and watch out for strange noises to detect danger. Being able to work with multimodal data is essential for us or any AI to operate in the real world. OpenAI noted in their GPT-4V system card that “ incorporating additional modalities (such as image inputs) into LLMs is viewed by some as a key frontier in AI research and development .” Incorporating additional modalities to LLMs (Large Language Models) creates LMMs (Large Multimodal Models). Not all multimodal systems are LMMs. For example, text-to-image models like Midjourney, Stable Diffusion, and Dall-E are multimodal but don’t have a language model component. Multimodal can mean one or more of the following: Input and output are of different modalities (e.g. text-to-image, image-to-text) Inputs are multimodal (e.g. a system that can process both text and images) Outputs are multimodal (e.g. a system that can generate both text and images) This post covers multimodal systems in general, including LMMs. It consists of 3 parts. Part 1 covers the context for multimodality, including why multimodal, different data modalities, and types of multimodal tasks. Part 2 discusses the fundamentals of a multimodal system, using the…
[ LinkedIn discussion , Twitter thread ] Never before in my life had I seen so many smart people working on the same goal: making LLMs better. After talking to many people working in both industry and academia, I noticed the 10 major research directions that emerged. The first two directions, hallucinations and context learning, are probably the most talked about today. I’m the most excited about numbers 3 (multimodality), 5 (new architecture), and 6 (GPU alternatives). 1. Reduce and measure hallucinations Hallucination is a heavily discussed topic already so I’ll be quick. Hallucination happens when an AI model makes stuff up. For many creative use cases, hallucination is a feature. However, for most other use cases, hallucination is a bug. I was at a panel on LLM with Dropbox, Langchain, Elastics, and Anthropic recently, and the #1 roadblock they see for companies to adopt LLMs in production is hallucination. Mitigating hallucination and developing metrics to measure hallucination is a blossoming research topic, and I’ve seen many startups focus on this problem. There are also ad-hoc tips to reduce hallucination, such as adding more context to the prompt, chain-of-thought, self-consistency, or asking your model to be concise in its response. To learn more about hallucination: Survey of Hallucination in Natural Language Generation (Ji et al., 2022) How Language Model Hallucinations Can Snowball (Zhang et al., 2023) A Multitask, Multilingual, Multimodal Evaluation of ChatGPT…
Prompt Engineering , also known as In-Context Prompting , refers to methods for how to communicate with LLM to steer its behavior for desired outcomes without updating the model weights. It is an empirical science and the effect of prompt engineering methods can vary a lot among models, thus requiring heavy experimentation and heuristics. This post only focuses on prompt engineering for autoregressive language models, so nothing with Cloze tests, image generation or multimodality models. At its core, the goal of prompt engineering is about alignment and model steerability. Check my previous post on controllable text generation.
BACKGROUND: To apply multimodal machine learning (ML), the various data modalities typically come from the same example (e.g., chest X-ray ( modality 1 ) and cancer biomarkers ( modality 2 ) come from the same patient ( the example )). The problem is that we often don't have that in public datasets. Instead, datasets are more commonly independent (e.g., a dataset of chest X-rays from one set of patients and a dataset of cancer biomarkers from a second set of patients). QUESTION: Is there any validity to artificially creating "co-registered" datasets from such independent datasets for the ultimate purpose of leveraging multimodal ML (e.g., generate all possible pairs of chest X-rays and cancer biomarkers from different patients with cancer and generate the same for different patients without cancer )? NOTE: It is understood that this approach is non-canonical and has flaws, but that is not the question here. I am more interested in learning whether this could be a second best option for researchers who lack co-registered samples but still want to develop multimodal ML models. Please provide your response along with some justification as to why this would or would not be valid "second best option". 11/26/2022 New NOTE: I thought it was implied in the question post that I am aware that the interaction between modalities is not available to be leveraged by multimodal ML since this is really the crux of the problem with fusing independent datasets. However, the question still sta…
I have p-dimensional data and I need to determine if that data has significant modes or if it’s clustered in any way. Here p=50, (dense embedding), we have n samples and p What are some ways to do that? I can think of creating sample covarince matrix of 50*50. Not sure how to analyze that full matrix (eigenvalues, etc) for conclusions. I can also try to do kmeans and see if it converges or do GMM but I needs to try on many number of clusters. The data could have many clusters in theory, possibly residing on lower dim manifolds. Any thoughts? Thanks.