# The Optimizer Tricks Hiding in Thinking Machines' Inkling > **Author:** LAXIMA Team **Published:** 2026-07-20 **Updated:** 2026-07-20 **Reading time:** 7 min **Category:** technology **Tags:** Thinking Machines, Inkling, Muon optimizer, Adam optimizer, Weight decay, LLM training, Deep learning optimization, Mix of experts **Canonical URL:** https://laxima.tech/blog/the-optimizer-tricks-hiding-in-thinking-machines-inkling --- We was convinced the hottest release of this week would be Inkling from Mira Murati and her crew at **Thinking Machines**, but the Chinese changed my plans with the Kimi K3 release. Still, I want to come back to their model and dig into the optimization tricks of the "Thinking Machines." Formally, in the blog post it's one modest paragraph in the Training section, but there are actually three separate decisions hidden in there that are worth unpacking one by one. Let's go in order. ## What even is an "optimizer" in this context A model is just a giant pile of numbers (weights). Training is a loop: compute how much (and in which direction) each number needs to move so the model makes fewer mistakes (that's the gradient), then nudge the weights in that direction. An "optimizer" is simply the rule for how exactly to make that nudge. The dumbest possible rule: ``` weight -= lr * gradient ``` where `lr` (learning rate) is the step multiplier — kind of like the step in a binary search loop: too big a step and you get thrown around, too small and you'll converge forever. **Adam** is an upgraded version of this rule. For every number in the model, it keeps a running average of its past gradients (momentum, like in physics — a rolling ball with inertia) and a running average of the squared gradients (to gauge how "noisy"/unstable that number is). The step for each weight is then divided by that "noise" estimate — so stable directions move confidently, noisy ones move cautiously. The key point: Adam works element-wise — as if you had a billion independent variables, each with its own adaptive step size. It has no idea that a bunch of those numbers actually form a matrix. ## Why that's not ideal for matrices In a transformer, the weights are organized into matrices (think of a 2D array like `float[4096][4096]`), and a matrix isn't just a warehouse of numbers — it's a linear operator: multiply it by a vector and you get another vector, and _how_ it does that (which directions it stretches, which it squashes) is described by the SVD (singular value decomposition) — a factorization into singular directions. Loosely: a matrix has "favorite axes" along which it acts more strongly or weakly. Adam, updating every element of the matrix independently, knows nothing about those axes and can accidentally overinflate one direction far more than the others. **Muon** is an optimizer built specifically for matrices. After the usual gradient is computed for a matrix (and accumulated into a momentum buffer — Muon, like Adam, works off momentum rather than the raw gradient), Muon runs that update through an orthogonalization step: conceptually, it takes the SVD of the update and replaces all singular values with one, keeping only the "directions." (In practice, an explicit SVD would be too expensive, so Muon approximates this with a few Newton–Schulz iterations — but the effect is the same.) A programmer's analogy: imagine you have a velocity vector in a 3D game, and instead of moving proportionally to the force along each axis, you first normalize the direction and then move at a fixed speed along it — you don't let one axis dominate just because the gradient there happened to be bigger. This produces an update that's "evened out" across all directions for the matrix as a whole, rather than treating it as a bag of independent numbers. Hence the pairing: Muon for the big 2D matrices (where this operator geometry matters), Adam for everything else (embeddings, bias vectors, scalar normalization parameters — the places where "independent number" is an honest model, not a simplification). This is exactly what the Inkling post describes: "Muon for large matrix weights, Adam for other parameters." **Modular manifolds** takes the idea one level higher. A "manifold" in math is a space that locally looks like ordinary flat (Euclidean) space but can be globally curved — the classic example is the surface of a sphere: up close everything looks like a plane, but overall it isn't one. Applied to network weights: instead of assuming all weights "live" in one common flat space where a single universal step rule fits everyone, TML propose constraining each layer's weights to its own manifold with its own "rules of good behavior" (what weight norm is reasonable, how the step size should scale) — and then composing these per-layer rules into a single whole-network optimizer, which is the "modular" part. The practical takeaway for a programmer: it's as if your project had not one global timeout config for every service, but a config tuned to each service's characteristics — because a "reasonable timeout" for a database and for an external API are different numbers for different reasons, and fitting them with one formula is wrong. \## Weight decay and the lr schedule **Weight decay** is an extra term in the update that shrinks the absolute value of the weights a little on every iteration: ``` weight -= lr gradient + decay weight ``` Essentially a rate-limiter/sanitizer that stops weights from growing without bound — like a TTL on a cache that slowly "evicts" entries unless they're actively refreshed. **The lr schedule** means lr isn't constant over training but changes on a schedule: first it ramps up (warmup, so you don't blow up the model with a huge step at the very start while the weights are random), then it gradually decays toward the end — accelerating and braking a car, not slamming the gas and then the brakes. **The coupling problem.** The `decay` in the formula above is usually a fixed number, or has its own separate schedule not tied to the current lr. But since lr changes a lot over training and decay doesn't, the ratio of "how much the gradient kicks the weights around" (depends on lr) to "how much decay pulls them in" (doesn't) drifts. Early in training, when lr is big, the kicking beats the decay — weights grow. At the end, when lr has dropped, decay relatively dominates — weights start shrinking too much. Result: the actual "scale" (norm) of the weights wanders around over training for no visible reason other than "that's how the schedules happened to line up." ## Why the square, exactly If you crudely model this as a random walk with damping: on every step, gradient noise kicks the weights by an amount on the order of lr, and decay pulls them back proportionally to `decay weight`_. The key subtlety is that random kicks add up in_ variance\*, not magnitude — a step of size lr injects variance of order **lr²** per iteration (this is the standard result for this kind of "noise vs. pull-back" process, like the equilibrium of an Ornstein–Uhlenbeck process, if any of you have seen that in statistics or finance). Meanwhile decay removes variance at a rate proportional to `decay`. Balancing injection against removal, the equilibrium (stationary) scale of the weights comes out proportional to ## lr / sqrt(decay) For that equilibrium scale not to depend on lr at all — i.e., not to wander as lr moves along its schedule — you need decay to grow proportionally to **lr²**: then `lr / sqrt(lr²) = lr / lr = 1`, a constant. With that power-law coupling, the two effects (growth from noise and shrinkage from decay) cancel each other at _any_ value of lr, and the equilibrium weight norm stays constant for the entire run — at the start with a big lr and at the end with a nearly-zero one. Which is exactly what TML report: coupling weight decay to the square of the learning rate "kept the overall size of the model weights stable across training horizons" (they point to Kosson et al. 2023 and Defazio 2025 for the theory). One convention caveat: in the standard AdamW formulation, the decay term is itself multiplied by lr `weight -= lr (gradient + decay weight)`). In that convention the same analysis gives an equilibrium norm of `sqrt(lr / decay)`, and keeping it constant requires decay ∝ lr, not lr². The "square" version corresponds to decay applied independently of lr, as in the formula above. Same physics, different bookkeeping. You can think of it like having two counters that grow and shrink at different rates depending on some external parameter, and you deliberately pick the power-law relationship between them so their ratio always comes out to the same constant — regardless of the current value of that external parameter. An invariant, not a coincidence.