This article details a systematic study of optimal architectures for small language models with approximately 70 million parameters. Researchers discovered that model performance follows a binary tier system determined by a specific hidden dimension threshold or a "Goldilocks" depth of 32 layers. While most traditional architectures performed similarly at this scale, diffusion models like the new Dhara-70M emerged as superior for high-speed throughput and factual accuracy. The study also highlights that converting existing models to diffusion architectures is ten times more efficient than training them from scratch. Ultimately, the findings suggest that model shape and inference style are more critical than specific family designs for small-scale efficiency.

A brief overview of the OpenEvolve evolutionary coding agent in Hindi.

The text presents Ellora, a collection of standardized, production-ready methodologies, referred to as recipes, for enhancing Large Language Models (LLMs) through Low-Rank Adaptation (LoRA). This approach is justified by the fact that LoRA achieves performance comparable to full fine-tuning while drastically reducing computational costs and training up to 10,000x fewer parameters. Ellora’s recipes often utilize self-supervised methods like the Magpie approach for data generation and confirm that combining parameter-efficient techniques with reinforcement learning yields significant speed and memory savings. The six structured recipes address diverse operational needs, including recovering model accuracy after quantization, extending context windows up to 2 million tokens, and teaching secure code generation. Specifically, one recipe demonstrates a 97% vulnerability reduction through automated security analysis and Group Relative Policy Optimization (GRPO). Ultimately, Ellora provides concrete, reproducible templates for practitioners to maximize model capabilities efficiently without requiring new, complex training frameworks.

Today's podcast is based on an article from Hugging Face detailing an extensive research project that addresses the high cost and scale of training modern large language models. The authors, through over 50 systematic experiments, sought to find an optimal data mixing strategy that would allow a GPT-2 model to achieve comparable performance to models trained on ten times the data. Their central finding is that a static dataset mix of 50% finePDFs, 30% DCLM-baseline, and 20% FineWeb-Edu significantly outperforms more complex curriculum learning approaches, which often led to catastrophic forgetting or overfitting. This optimal 50-30-20 mixture successfully trained a GPT-2-70M model that achieved over 90% of the original GPT-2's benchmark performance while using substantially fewer resources. The key takeaway is that dataset quality and intelligent composition are more critical than sheer quantity for training efficient language models.

Read the full article on https://huggingface.co/blog/codelion/optimal-dataset-mixing

https://huggingface.co/blog/codelion/internal-coherence-maximization

The article presents a novel method for improving large language models (LLMs) called Internal Coherence Maximization (ICM) combined with Direct Preference Optimization (DPO), which operates without any human supervision. This unsupervised approach demonstrates superior performance in mathematical reasoning tasks compared to traditional human-supervised methods like Group Relative Policy Optimization (GRPO). Key contributions include a complete implementation of ICM with diverse solution generation and a pipeline to convert ICM results into preference pairs for DPO training. The research also shows successful cross-model capability transfer, where knowledge from a stronger model (Qwen3) improves a weaker one (Gemma3), offering a scalable and cost-effective alternative to current LLM alignment paradigms. The authors emphasize that pretrained models already possess rich understanding, and ICM+DPO offers a way to elicit and refine this internal coherence, leading to better performance without the bottleneck of human annotation.

The article introduces EDINET-Bench, a novel open-source Japanese financial benchmark designed to evaluate Large Language Models (LLMs) on complex financial tasks. This benchmark addresses the scarcity of challenging Japanese financial datasets for LLM evaluation, crucial for tasks like accounting fraud detection, earnings forecasting, and industry prediction. The EDINET-Bench dataset is automatically compiled from ten years of Japanese annual reports available through the Electronic Disclosure for Investors’ NETwork (EDINET). Initial evaluations indicate that even state-of-the-art LLMs perform only marginally better than logistic regression in some complex financial tasks, highlighting the need for domain-specific adaptation and further research. The project makes its dataset, benchmark construction code, and evaluation code publicly available to foster advancements in LLM applications within the financial sector.

The article introduces AutoThink, an innovative approach designed to enhance the inference efficiency and accuracy of reasoning Large Language Models (LLMs). AutoThink addresses the challenge of LLMs generating excessive or insufficient reasoning tokens, which leads to computational inefficiency and suboptimal performance. This system comprises two main components: a query complexity classifier that dynamically allocates the optimal number of reasoning tokens, and a dataset of control vectors derived from "pivotal tokens" to guide the LLM's reasoning path. Experimental results demonstrate that AutoThink significantly reduces output tokens while substantially improving accuracy on complex reasoning tasks, suggesting a more strategic approach to LLM resource allocation rather than simply increasing computation.

The article introduces System Prompt Learning (SPL), an innovative approach enabling Large Language Models (LLMs) to learn and refine problem-solving strategies through practical experience. This method addresses the current disparity where most developers lack the sophisticated system prompts that make advanced AI assistants so capable. SPL represents a "third paradigm" of LLM learning, augmenting traditional pretraining and finetuning by allowing models to classify problems, apply relevant strategies, and continuously improve these strategies over time. The system maintains a dynamic database of human-readable strategies, demonstrating significant performance improvements across various benchmarks and offering benefits like cumulative learning, transparency, and adaptability. Implemented as an open-source plugin in optillm, SPL offers a practical way to integrate this adaptive intelligence into LLM applications.