Physical Firmware — gualterio

01The problem

A toddler who has never thrown a ball can develop a reasonable throw after a short period of play. Change the size or weight of the ball and the child adapts.

A robot often needs hundreds or thousands of demonstrations to acquire a comparable skill—and may still fail when the object, camera position, gripper, surface, or environment changes.

The difference is not that the toddler knows Newton's equations. The toddler already possesses a deep, embodied set of expectations:

objects fall when unsupported;
mass and momentum affect motion;
contact and friction change what actions are possible;
the same interaction remains meaningful from different viewpoints.

These are not task-specific behaviours. They are physical priors: reusable expectations about how the world evolves.

The entanglement: modern robot-learning systems often mix two separate problems—(1) learning how the physical world behaves, and (2) learning how to accomplish a particular task within that world. That makes every new task unnecessarily expensive.

Figure B — Current learning often entangles physical knowledge with task behaviour. Physical Firmware factors out the reusable part, so each new task learns only what is task-specific.

A robot learning to pour, stack, insert, push, pack, or grasp should not have to reconstruct a basic model of physical interaction from task demonstrations alone. The proposal is to separate reusable physical understanding from task-specific behaviour.

02Architecture

Physical Firmware is a pretrained, structured dynamics model that sits between a robot's perception system and its planner or policy. It is a learned world model in which physical priors are built into the architecture and real interaction dynamics are learned from data.

What it is not

not a behavioural foundation model trained only to imitate actions;
not a hard-coded simulator requiring every object and parameter to be modelled manually;
not a replacement for perception, planning, control, or task learning;
not a claim that physics can be perfectly encoded in advance.

The objective is not to hard-code the world. It is to give learning the right shape.

Three scopes of adaptation

The system separates what is trained once and reused, what is calibrated for a particular robot, and what is learned for a particular job. Colours match Figure A throughout.

Train once

Physical Firmware core

Pretrained on physical interaction. Predicts objects, contacts, forces, futures and uncertainty. Stays fixed during ordinary deployment.

Adapt per robot

Perception & embodiment adapter

Maps cameras, proprioception, force sensors, geometry and actions into and out of the firmware's physical latent space.

Learn per task

Planner or policy

Learns how to achieve a specific goal: insert this part, stack these objects, pack this tray.

The firmware is not expected to receive a perfect symbolic description of the world. The adapter must infer a decision-relevant physical representation from noisy, partial sensor data: object identity and geometry; position, velocity and orientation; robot configuration; contact state; estimated mass and friction; latent material properties; and uncertainty about hidden or ambiguous variables.

Built-in structure vs learned quantities

Do not ask data to rediscover every physical regularity that can be represented directly in the architecture.

Built into the architecture

Geometric equivariance: rotating or translating a scene transforms predictions consistently.
Object-centric composition: objects and robot links are entities with structured interactions.
Energy-based dynamics: conservative motion follows a constrained geometric form.
Dissipation and actuation: friction, control inputs and energy exchange represented explicitly.
Local interaction structure: contact and force propagation follow the interaction graph.

Learned from data

latent physical state from sensors;
unknown interaction potentials;
friction and restitution;
actuator and gripper behaviour;
contact transitions;
residual dynamics the structural core does not capture;
uncertainty under partial observation;
the limits of its own predictions.

This is neither pure simulation nor unconstrained pattern matching. It is structured learning.

03Technical core

Energy-based structured dynamics

Many neural world models learn a direct transition from state and action to next state. That can work well over short horizons, but small local errors compound during repeated rollout, and trajectories drift away from plausible physical behaviour.

An energy-based model instead learns a single scalar function—the system's energy—and derives the dynamics from its gradient. That gives the model a structured vector field rather than an arbitrary next-state predictor, and it makes a whole class of wrong answers unrepresentable rather than merely unlikely.

What structure does not buy you: the architecture enforces selected structural properties, but physical accuracy and transfer must be established experimentally. Structural consistency is not the same as real-world correctness—the learned energy need not equal the true physical energy, and the inferred state need not be right.

Real robots need actuation and dissipation

A purely conservative core is only part of a real system. Practical dynamics also require external control inputs, friction and damping, energy exchange with the environment, unilateral constraints, collision and contact transitions, and learned residuals. A port-Hamiltonian-style extension keeps each of these as an explicit, separately learnable term rather than folding them into one opaque network.

conservative structurethe energy-preserving core

dissipationfriction, damping, energy loss

actuationhow control inputs enter the system

contact termsimpacts, constraints, mode switches

residualeverything the structured core does not explain

Contact must be first-class

Robotic manipulation is not smooth motion alone. A gripper touches an object; an object begins to slide; static friction becomes dynamic friction; insertion changes from free motion to constrained motion; a grasp forms or breaks. A useful physical foundation model needs an explicit way to represent contact detection, contact mode, collision impulses, stick–slip transitions, unilateral constraints, simultaneous multi-contact, and uncertainty about hidden contact.

Figure C · Hybrid dynamics

1 · Free motion

Continuous, differentiable dynamics.

2 · Impact / mode change

The governing equations change instantaneously.

3 · Contact dynamics

Mode determines what actions are possible.

The model must predict both the continuous state and the active contact mode.

Figure C — Manipulation is hybrid: smooth motion is interrupted by contact events that change the governing dynamics. Contact mode must be represented explicitly.

Why this is the hard part: textbook energy-based models are elegant precisely because they assume smoothness. Manipulation lives at the discontinuities. A contact-aware layer is what makes the idea relevant to real robots rather than only to pendulums.

Object-centric composition

A robotic scene contains multiple entities: robot links, tools, parts, containers, surfaces, fixtures, obstacles. Physical Firmware represents the scene as a structured graph, so a new object enters as a new node that reuses the same learned interaction mechanisms instead of requiring a fixed-size input.

Figure D · Scene as entities and interactions

Node

state · geometry · properties · uncertainty

Edge

relative pose · contact mode · interaction

contact support fixed possible

Figure D — The scene is represented as entities and interactions. New objects can enter as new nodes while reusing learned interaction mechanisms.

This does not mean a model trained on two objects will automatically understand every ten-object scene. It means the architecture is designed so that physical knowledge can be reused compositionally, instead of being tied to a fixed-size input or a specific scene.

Uncertainty

A physical model that is confidently wrong is dangerous. Physical Firmware should produce not only a predicted future, but an estimate of where that prediction is unreliable—under occlusion, unknown mass or friction, ambiguous contact, unseen geometry, sensor noise, out-of-distribution materials, model disagreement, and long prediction horizons.

Figure E · Uncertain futures and safe planning

Current state

Predicted futures

settles tips uncertainty band

Planner response

choose a safer action
gather information
slow down
use force feedback
request intervention

Warning: a single mean trajectory hides the dangerous branch.

Figure E — Useful prediction includes uncertainty and alternative contact modes. The planner can react differently when the future is ambiguous.

The useful output is therefore a distribution over both future states and contact modes—not a single deterministic future—together with a flag that says where the mode itself is ambiguous.

Technical notes: Hamiltonian structure, equivariance, and numerical integration

The Hamiltonian core

Many neural world models learn a direct transition:

$$x_t,\; a_t \;\longrightarrow\; x_{t+1}$$

A direct next-state predictor: no structure, no invariants

An energy-based model instead learns a scalar function describing the conservative component of the system:

$$H(q,p) \;=\; \underbrace{T(q,p)}_{\text{kinetic}} \;+\; \underbrace{V(q)}_{\text{potential}}$$

(1)

where $q$ represents generalized positions and $p$ generalized momenta. The continuous-time dynamics follow from the gradient of that single function:

$$\dot{q} \;=\; \frac{\partial H}{\partial p} \qquad\qquad \dot{p} \;=\; -\frac{\partial H}{\partial q}$$

(2) Hamilton's equations — dynamics as the gradient field of a scalar

The geometry is the diagnostic. Plot the state $(q,p)$ over time and a conservative system traces a closed orbit of constant energy: kinetic and potential energy trade back and forth while the total stays flat. A model that has learned the underlying structure reproduces that closed orbit over long rollouts; a model that has memorized transitions accumulates small per-step errors and spirals off the energy surface.

Conceptual simulation—not experimental evidence. Structure does not make a model correct; it makes a class of wrong answers unrepresentable.

The port-Hamiltonian extension

Real robots need actuation, dissipation, contact and residual terms:

$$\dot{x} \;=\; \bigl[\,J(x)-R(x)\,\bigr]\nabla H(x) \;+\; G(x)\,u \;+\; f_{\text{contact}}(x,u) \;+\; f_{\text{residual}}(x,u)$$

(3)

J(x) ∇Hconservative structure — the energy-preserving core

−R(x) ∇Hdissipation — friction, damping, energy loss

G(x) uactuation — how control inputs enter the system

f_contactimpacts, constraints, mode switches

f_residualeverything the structured core does not explain

The uncertainty output is a joint distribution over states and contact modes:

$$p\bigl(x_{t+1:t+H},\; c_{t+1:t+H} \;\bigm|\; x_t,\; a_{t:t+H}\bigr)$$

(4) $c$ represents possible contact events or modes

Same interaction, different orientation

The laws governing an interaction should not fundamentally change because the entire scene is translated or rotated. SE(3)-equivariant representations encode this directly.

Equivariance. Rotating the whole scene rotates the prediction by the same transformation, rather than producing an unrelated answer.

$$F(g \cdot x) \;=\; g \cdot F(x)$$

(5) the prediction transforms consistently with the input

Equivariance reduces the need to relearn transformations that should not change the interaction. Real workspaces are not perfectly symmetric, though: gravity establishes a preferred direction, robot bases are fixed, cameras create occlusions, grippers are asymmetric, and fixtures constrain movement.

The goal is not to assume perfect symmetry everywhere. It is to encode the symmetries that genuinely apply and expose the symmetry-breaking variables explicitly.

Numerical integration over long horizons

Even a physically structured continuous-time model can drift when integrated numerically. For conservative components, symplectic integration keeps the energy error oscillating inside a narrow band instead of growing without bound, which is what makes long-horizon planning viable. For dissipative or contact-rich components, the integration strategy must also respect energy loss, control inputs, constraints, event timing, and impact transitions.

The relevant success criterion is not mathematical elegance alone—it is whether the learned model remains useful for planning over the horizon the task actually requires.

04Integration

A robotics company may already have a perception stack, a controller, a task planner, a policy-learning pipeline, safety systems, and simulation infrastructure. Physical Firmware should not require replacing the whole stack. Its role is narrower:

Given the current sensory state and a candidate action, predict plausible physical futures and the uncertainty around them.

Observation
→
Adapter
→
Physical state
→
Predicted futures
→
Planner

from physical_firmware import Firmware, RobotAdapter

firmware = Firmware.load("physical-firmware-v1")
adapter  = RobotAdapter.load("franka-panda-lab-a")

state = adapter.encode(observation)

prediction = firmware.rollout(
    state=state,
    actions=[candidate_action],
    horizon_seconds=0.5,
    return_uncertainty=True,
)

print(prediction.states)          # predicted futures
print(prediction.contact_events)  # mode changes
print(prediction.uncertainty)     # where to distrust it

This interface is illustrative. Actual latency and horizon will depend on model size, hardware, number of objects, contact complexity, integration steps, sampled futures, and the uncertainty estimation method. The product promise should be measured, not assumed.

View complete illustrative API example

from physical_firmware import Firmware, RobotAdapter

firmware = Firmware.load("physical-firmware-v1")
adapter  = RobotAdapter.load("franka-panda-lab-a")

observation = {
    "rgb":               camera.rgb(),
    "depth":             camera.depth(),
    "joint_positions":   robot.joint_positions(),
    "joint_velocities":  robot.joint_velocities(),
    "force_torque":       robot.force_torque(),
}

latent_state = adapter.encode(observation)

candidate_action = {
    "mode":   "joint_torque",
    "values": [0.5, -0.3, 0.1, 0.0, 0.2, -0.1, 0.0],
}

prediction = firmware.rollout(
    state=latent_state,
    actions=[candidate_action],
    horizon_seconds=0.5,
    return_uncertainty=True,
)

# a planner can query the firmware repeatedly as its dynamics model
plan = planner.optimize(
    dynamics=firmware.rollout,
    initial_state=latent_state,
    objective=goal,
    constraints=safety_constraints,
)

The robot adapter

The firmware reasons in a reusable physical latent space; a real robot produces embodiment-specific sensor data and accepts embodiment-specific commands. The adapter bridges the two.

Adapter responsibilities

visual and geometric state estimation;
mapping joint states into the latent physical representation;
representing kinematics and actuator limits;
estimating robot-specific friction and compliance;
converting abstract actions into the robot's action interface;
detecting when the firmware's assumptions are failing.

Self-supervised calibration

moving in free space;
touching known surfaces;
pushing objects;
grasping and releasing;
lifting different masses;
observing slip;
comparing predicted and observed motion.

Only the adapter and small residual components would be updated initially; the reusable core stays fixed during ordinary deployment. The commercial target is a calibration process measured in hours rather than weeks—but the required time must be established experimentally for each task and embodiment class.

Planning and control

Architecture	How it works	Best suited to
Model-predictive control	Simulate candidate action sequences through the firmware, score predicted outcomes, execute the best action, replan.	Explicit objectives · constrained motion · insertion and alignment · when online compute is available.
Policy learning	Train a task policy on top of the firmware's physical representation.	Fast execution · tasks hard to express as a cost function · repeated workflows where latency matters.
Hybrid	Firmware and planner generate successful trajectories or synthetic demonstrations; those train a faster policy.	Model-based reasoning during learning, fast policy execution during deployment, firmware-based recovery throughout.

05Validation

Figure F — The hypothesis is a leftward shift: reaching the same reliability with less real-world data and engineering effort. The experiment must measure the horizontal gap.

The measurable outcome: at a fixed reliability threshold, how much real-world data and engineering effort does each approach require?

The central claim is deliberately testable: a robot initialized with Physical Firmware should require materially fewer real-world demonstrations and less engineering effort to deploy a new physical task than a robot learning the same task without a reusable structured dynamics prior.

Initial research target: up to a 10× reduction in real-world demonstrations on selected manipulation tasks.

That is a research target, never a result. Demonstration count is also not the only metric that matters; the commercial outcome should be measured across:

time to deploy a new skill
operator hours
engineering hours
real-world demonstrations
destructive or unsafe failures
task success rate
recovery rate
performance under object and environment change
calibration time for a new robot
compute and control latency

The decisive experiment

Choose a contact-rich manipulation task family with controlled variations—peg or connector insertion, object pushing and repositioning, tray packing, part placement, stacking under variable mass and geometry. Then train and evaluate several systems side by side:

#	System under test	What it isolates
1	Policy learning from scratch	The baseline cost of learning physics and task together
2	Behavioural pretraining → task fine-tuning	The value of action priors without physical structure
3	Unconstrained learned world model	The value of a world model without structural priors
4	Simulator-assisted policy learning	The value of explicit, hand-modelled physics
5	Physical Firmware + MPC	Zero-demonstration performance of the prior itself
6	Physical Firmware + task-policy learning	The headline sample-efficiency claim
7	Firmware ablations	Contribution of equivariance, contact structure, energy structure, pretraining

Plot task success against real-world demonstrations, then repeat under controlled distribution shifts: new object mass, new friction, new geometry, new initial pose, new camera pose, new robot embodiment, unseen combinations of known objects.

A useful result would show not merely better performance on the training setup, but a lower adaptation cost when the physical situation changes. That is the whole point of a prior.

Validation roadmap

P1 Structural proof Controlled systems where the true dynamics are known: pendulum, double pendulum, spring–mass systems, articulated chains. Measure rollout error, long-horizon stability, parameter extrapolation, conservation and dissipation behaviour, uncertainty calibration. → Confirm the architecture behaves as intended.
P2 Compositional physical interaction Object-centric scenes: multiple rigid bodies, pushing, sliding, collision, variable mass and friction, changes in object count. Measure transfer to unseen parameters and object combinations, contact-event prediction, scaling with scene complexity. → Test whether reusable interaction knowledge actually composes.
P3 Robot grounding Connect the firmware to a real research arm. Tasks: free-space motion, pushing, pick and place, controlled contact, insertion. Measure adapter calibration time, sim-to-real transfer, state-estimation error, task data efficiency, intervention and recovery rates. → Determine whether the latent physical model survives noisy sensors and real hardware.
P4 Cross-embodiment transfer Pretrain on one embodiment, adapt to another—a different arm, gripper, or tool. Measure percentage of the model reused, adapter data required, task-performance retention, engineering effort saved. → Establish whether "firmware" is genuinely reusable across robots.
P5 Industrial pilot One narrow, economically meaningful workflow: machine tending, kitting, packing, tray loading, insertion and alignment, variable-part handling. Measure the full deployment economics—time from task definition to production, demonstrations, integrator hours, downtime, failure costs, adaptation to a new part, reliability over repeated shifts. → Prove that better physical transfer creates measurable operational value.

Initial success criteria

equivalent task success using substantially fewer real-world demonstrations;
lower long-horizon prediction error than an unconstrained world model;
better adaptation to changes in mass, geometry, friction, and initial pose;
calibrated uncertainty that increases on unfamiliar conditions;
successful transfer of the core across at least two robot configurations;
robot adaptation completed without retraining the entire model;
a measurable reduction in engineering time for a new task.

The 10× demonstration target is ambitious. A smaller but repeatable reduction, combined with strong cross-task or cross-robot transfer, would still be important evidence.

06Product path

Why start with industrial manipulation?

The long-term vision is broad physical intelligence. The starting point should be narrow. The initial wedge is high-mix industrial manipulation: environments where the robot performs a known family of actions but must adapt frequently to new parts, weights, layouts, or fixtures. It offers clear task definitions, measurable economics, repeated physical structures, expensive task reconfiguration, controlled access to sensors and robot state, and a realistic path to collecting interaction data.

The first product should not promise universal physical common sense. It should solve a specific problem: make a robot learn a new variation of a known manipulation task with far less real-world data and manual tuning.

How it differs from existing approaches

Approach	Primary strength	Main limitation	Role of physics
Behavioural foundation model	Broad action priors and semantic task understanding	Physical knowledge entangled with behaviour and training distribution	Mostly implicit
Classical simulator	Precise and interpretable when the model is known	Manual modelling and sim-to-real mismatch	Explicit and hard-coded
Unstructured learned world model	Learns directly from data, captures complex effects	May drift, overfit, or require large amounts of interaction data	Learned implicitly
Physical Firmware	Reusable structured dynamics intended to transfer across tasks and embodiments	Must prove real-world contact modelling, calibration, and transfer	Structural and learned

Physical Firmware is not intended to eliminate the other three. A practical stack may combine them: behavioural models for semantic understanding, simulators for synthetic data and safety testing, Physical Firmware for transferable physical prediction, and task policies for fast execution.

The moat

The architecture alone is not enough. A defensible platform would accumulate:

Physical interaction data

Trajectories with synchronized vision, proprioception, force and torque, actions, contact events, object properties, failures and recoveries.

Embodiment adapters

Reusable calibration layers for arms, grippers, tools, sensor configurations, and control interfaces.

Contact & residual models

Trained on real discrepancies between predicted dynamics, simulator dynamics, and observed physical outcomes.

Deployment feedback

Every installation reveals where the model is uncertain, which interactions transfer, and which adapter components are reusable.

Benchmarks & evaluation

Tests for long-horizon consistency, contact prediction, parameter extrapolation, object composition, embodiment transfer, uncertainty calibration.

The combination

Defensibility comes from architecture, data, adapters, evaluation, and deployed experience together—no single one of them is a moat.

Open research questions

These are not details to hide—they are the research programme.

What state is sufficient?

Can a compact latent state support planning without reconstructing every visual detail?

How should contact be represented?

Through events, constraints, impulses, learned modes—or a hybrid?

How far does transfer extend?

Does knowledge from one robot and task family meaningfully reduce adaptation on another?

Can uncertainty be trusted?

Does the system know when geometry, material, contact, or embodiment is outside its experience?

How much structure is optimal?

Too little wastes data. Too much prevents the model from representing real effects.

Can it improve without losing its invariants?

Deployment will expose residual dynamics. The system must learn from them without destroying the properties that make transfer possible.

What Physical Firmware is not

a universal physics oracle;
a perfect simulator;
an autonomous robot operating system;
a substitute for perception;
a substitute for safety engineering;
proof that structure always beats scale;
a claim that every physical property can be learned once and reused forever.

The bet

The dominant bet in robotics AI is scale: more demonstrations, larger datasets, bigger models, broader behavioural pretraining. That direction will continue to produce progress. Physical Firmware makes a complementary bet:

Scale is more valuable when the model is given the right structure.

A toddler does not adapt quickly because it has seen every possible object and trajectory. It adapts because new experience is interpreted through a physical foundation that already exists. The opportunity is to build an analogous foundation for machines—not by manually programming every law, and not by hoping an unconstrained network discovers everything from data, but by combining learned representations with architectural priors that reflect the structure of the physical world.

What would make this real?

equivalent task success from substantially fewer real-world demonstrations;
lower long-horizon prediction error than an unconstrained world model;
lower adaptation cost when mass, geometry, friction, or pose changes;
uncertainty that rises measurably on unfamiliar conditions;
the core reused across at least two robot configurations;
a measurable reduction in engineering time on one industrial workflow.

In one sentence: Physical Firmware is a reusable, contact-aware, physically structured world model designed to help robots learn new manipulation tasks with less real-world data and less engineering effort.

If that works, robots will not need to begin every new task from zero. They will begin with physics.

← back to gualterio's page