Skip to content

TOWER ↔ XPolicyLab policy interface

TOWER owns evaluation, scoring, simulation and robot control. XPolicyLab owns policy loading and serving. third_party/XPolicyLab pins the TOWER integration branch of Apache-2.0 XPolicyLab. The environment imports only client_server/ws/model_client.WsModelClient and its transport dependencies.

Initialize the submodule with GIT_LFS_SKIP_SMUDGE=1 git submodule update --init third_party/XPolicyLab and install tower-eval[xpolicylab]. The policy environment separately installs XPolicyLab and the selected model dependencies. See third_party/XPolicyLab/policy/OpenWAM_TOWER/README.md for checkpoint setup. No checkpoints are distributed by TOWER.

[policy]
adapter = "xpolicylab"
server = "ws://localhost:19000"
timeout_s = 120

[policy.config]
action_mode = "eef_delta"
max_connect_attempts = 200
connect_timeout_s = 10
max_connect_seconds = 180
# Installed wheels may specify client_root = "/path/to/XPolicyLab".

The Python adapter accepts full encoder observations; the current generic CPU backend supplies only joint state. For CPU protocol verification, use the explicit synthetic fixture in tower smoke --policy xpolicylab://demo_policy. This launches the actual XPolicyLab demo server, exercises single and batch requests, runs three CPU evaluation steps and stops its server. It is not a benchmark or a physical-policy evaluation.

Embodiment and observations

env_cfg_type = nero_dual. Arm index 0 is left; index 1 is back, named right in XPolicyLab action fields. Parent third_party/env_cfg supplies the runtime robot configuration; the submodule maintains its training registry.

TOWER observation XPolicyLab observation Shape / units
images/<camera> vision/<camera>/color H×W×3 uint8 RGB
joint_encoder state/joint_encoder 2×7 radians
eef_encoder state/eef_encoder 2×4×4 rigid TCP transforms
gripper_width_m state/gripper_width_m 2, full aperture in metres
instruction (or prompt) instruction nonempty string
coordinate_frame coordinate_frame encoder_arm_base
optional env_idx env_idx unique integer per active batch

Camera names are right_front_camera, left_back_camera, left_wrist_camera, and back_wrist_camera. Arrays are transported losslessly with msgpack; RGB channels are never swapped. TOWER always sends raw RGB arrays, which is what XPolicyLab policy servers accept by default; they reject encoded images. OpenWAM_TOWER applies the checkpoint's verified training image composition. No camera, EEF pose, or prompt is fabricated for real observations.

Joint and EEF observations must already be in encoder coordinates. J2 compensation remains in the environment: physical → encoder before inference, encoder → physical exactly once at command execution. The adapter does not change IK, command limits, simulation observation tolerances, the exact-flip tripwire, or hardware protections. Negative observed gripper undershoot is passed through for the checkpoint's existing observation policy to validate.

Actions, frames and units

Mode XPolicyLab endpoint fields TOWER decoded action
joint_absolute left/right_arm_joint_state (7), left/right_ee_joint_state (1) 16-vector: left joints, left width, back joints, back width
eef_delta left/right_ee_transform (4×4), left/right_ee_joint_state (1), chunk_anchor_encoder (2×4×4) tagged encoder target matrices plus widths and anchor

ActionChunk.metadata retains mode, encoder_arm_base frame, and policy identity. EEF dictionaries retain their explicit frame and mode. These are runtime controller inputs, not generic world-frame ee_pose actions. The future unified simulator consumes them through its existing encoder controller; the current CPU joint-only environment cannot execute an EEF chunk.

OpenWAM predicts a 32-step horizon at 10 Hz, executes its first 16 steps, then replans. Translation is p_anchor + delta_p; rotation is delta_R @ R_anchor, using the checkpoint’s orthonormalized rot6d columns. All 32 steps use one observation-time anchor. Returned EEF targets are absolute, although the checkpoint's mode remains eef_delta. Translations and gripper full widths are metres; joints are radians. Gripper openness is projected by the unchanged checkpoint decoder and scaled by its calibrated open width. Predicted J8 values are diagnostic millimetres and remain masked/held fixed. The client rejects mode/frame/timing mismatches instead of guessing conversions. Every EEF row's anchor must exactly equal the request's eef_encoder; batches are checked against their corresponding observations. The generic evaluator rejects encoder-frame chunks before normalization discards metadata. It has no J2 converter. CPU configs using policy.adapter = "xpolicylab" fail validation before connecting; only the explicit synthetic smoke fixture reinterprets its fake joint actions as physical CPU targets.

An explicitly configured joint_absolute client also accepts standard untagged XPolicyLab joint action dictionaries, including demo_policy. Real EEF policies must supply frame tags. OpenWAM_TOWER emits identity, units and timing on every endpoint; get_prediction() exposes its last full horizon for parity audits.

Lifecycle, seeds and batches

reset(seed=..., metadata=...) sends prepare_case metadata with episode_seed, then the argument-free model reset. The client sends trial_end when an episode ends or before the next reset. This seed identifies the environment trial. Configured OpenWAM inference uses seed 42 on every call, independently of the environment seed; non-42 inference configurations are rejected. Reset clears observations, predictions and anchors.

Single calls use update_obs then get_action. act_batch sends update_obs_batch then get_action_batch with the ordered active environment indices; results preserve that order. Duplicate indices or missing observations fail. OpenWAM batches are serialized independent calls, matching the reference behaviour; tensor batching is not validated. Each client is called from one thread, and owns its connection, trial identifiers and cleanup.

Isaac benchmark runtime

Use one runtime process per policy server. Full-prediction caches are shared; concurrent runtime clients are unsupported even though INFER itself is atomic.

The Isaac runtime uses RuntimePolicyAdapter, a thin wrapper around XPolicyLabAdapter, on one WebSocket connection per process. policy_url and --policy-url accept only ws:// and wss://. The unreleased request schema remains tower-public-benchmark-v2; benchmark, data and scene identities do not change. INFER and get_prediction each have a 300-second request timeout. get_identity retains the old health timeout of 30 seconds. A failed exchange permits one reconnect attempt with a 180-second connect budget; connection attempts within that budget use TOWER XPolicyLabAdapter’s 10-second attempt timeout. Installed runtimes receive the pinned checkout client path explicitly. Release verification binds its client source files and the installed transport dependency records.

Before simulator startup, get_identity authorizes the package digest. Startup writes policy-health.json with ready: true and the full identity. Each prediction must equal both the first and latest startup identity, using existing dictionary equality rules, including all unknown and nested fields.

A batch start sends prepare_case with episode seed and environment seed metadata, then reset. Each fixed tick visits active environments in index order and sends INFER (with env_idx), then get_prediction(env_idx) if supported. INFER updates observations and obtains actions under one server lock. An audit-prefix mismatch fails closed with a shared-server diagnostic. One environment's action rejection does not prevent later environments from receiving their actions. An infrastructure error stops requests for the remaining environments. trial_end closes the batch; cleanup failures are recorded separately in policy-cleanup-errors.json and do not invalidate recorded outcomes. OpenWAM_TOWER stores only observations, anchors and last predictions; reset clears those caches. Every inference uses seed 42.

Pixels are decoded from the exact validated archive PNG bytes and sent as raw RGB uint8 arrays. The server passes those arrays through its image decoder. The runtime adds the encoder frame tag to the three-field encoder observation. Controller responses are reconstructed from float64 endpoint rows. Floating arrays of other dtypes are rejected; full predictions are normalized to nested Python lists before prefix concatenation. All numeric and semantic validation remains in the unchanged controller, preserving rejection and evidence behavior. With get_prediction, all executed rows must match the full prediction bitwise; the remaining horizon and auxiliary fields are retained unchanged. Without that method, only available rows are recorded, with explicit missing_evidence fields. No unexecuted horizon or inference timing is synthesized.

Remote PolicyOutputError becomes an action failure. Remote ValueError with exact message Negative gripper aperture preserves the frozen input-rejection record without clipping or changing tolerances. Other model errors, unknown exception types, connection, timeout and protocol failures are infrastructure failures. Malformed wire structure fails before control. Identity-call errors are always infrastructure failures. Transport diagnostics retain code, message, exception type and request ID in policy-error.json. The server must expose details.exception_type in model errors.

rpc_start, rpc_end and policy.rpc_elapsed_sum cover each environment's WS exchange, including full-prediction retrieval; payload preparation and response validation are outside the timer. rpc_end now records transport: websocket. Request IDs remain local metrics; transport headers and server profiling fields are removed. Native inference timing remains available in full predictions.

MME-VLA packages fail authorization: an XPolicyLab MME adapter is not available in this release. Adding one with qualified history semantics is follow-up work.

See the control safety contract.

Untagged standard joint endpoints use the configured joint_absolute mode, encoder_arm_base frame, 10 Hz, row count as execute_steps, and a disabled J8 mask, matching decode_chunk defaults. EEF endpoints must explicitly supply mode, frame, control_hz, execute_steps and j8_actuator_mask; the runtime invents none of these. Audit-call failures are infrastructure failures, never observation rejections. Command replay retains its previous min(16, environment count) request-limit metadata.