Image-Space
- Heavier
- Difficult to represent abstract conditions (e.g., force)
- Mainly controls visual appearance
- Leverages the visual priors of video generation models
Unified Action Space for Controllable Video Generation
Contents
- Research Direction
- Current Progress
- Next Steps
Research Direction
Non-Image-Space
- More lightweight
- Easier to represent abstract and non-abstract conditions in a unified way
- More flexible control
- May need to relearn a new data distribution
Image-Space or Non-Image-Space Conditioning?
Research Direction
Non-Image-Space
Require:Pxyz ∈ ℝB×49×J×3Raw 3D hand-pose coordinatesXvideo ∈ ℝB×C×13×H×WInput DiT video tokensThint, λhintTeacher supervision and loss weight
1Per-Frame Feature Extraction
for t ∈ {1, …, 49} do pt ← flatten(Pxyz[:, t]) ht ← Wframeptend forAbsolute pose matching · Linear frame projection
2Temporal Patch Merging
H ← stack(h1, …, h49)Hpad ← pad(H, target_length = 52)ck ← concat(h4k+1, …, h4k+4)zk ← WmergeckRepeat final frame · Merge four frames per chunk
3Spatial Broadcasting and Cross-Attention
Zchannel ← repeat_interleave(Zpose, H × W)Zchannel ← MLP(Zchannel)Kl, Vl ← Projl(Zchannel)Xl ← Xl + CrossAttn(Ql, Kl, Vl)Cross-attention conditioning at each transformer block
4Training Objectives
Ldiff ← DenoisingLoss(θs, target)Lhint ← MSE(student features, Thint)Ltotal ← Ldiff + λhintLhintreturn LtotalBlockwise and boundary supervision
Image-Space or Non-Image-Space Conditioning?
Research Direction
Non-Image-Space
Image-Space or Non-Image-Space Conditioning?
Research Direction
Single-Video Overfitting
Research Direction
Lack of Temporal Ordering in the Inputs
Research Direction
Add Temporal Position
Research Direction
Leverage Pretrained 3D RoPE
New Temporal + Spatial Encoding
↓Learn positional structure from scratch
↓Pretraining-like optimizationVS
Pretrained 3D RoPE
↓Inherit spatiotemporal position priors
↓Focus on conditional controlReuse the generator’s 3D RoPE — do not relearn position.
Current Progress
3D RoPE: spatiotemporal positional prior
VIDEO TOKEN GRID
→
SPLIT & ROTATE Q / K
p = (t, y, x)→θp = (tω, yω, xω)
Q / K channels
↓qt, ktqy, kyqx, kx
↻tωtime↻yωheight↻xωwidth
RoPEtRoPEyRoPEx
↓concat → Q′ / K′→
ATTENTION
Q′iK′jT
relative offset(Δt, Δy, Δx)Current Progress
Self-Attention + RoPE
12 LEARNABLE TOKENS
LEFTrootfingertip 1fingertip 2fingertip 3fingertip 4fingertip 5
RIGHTrootfingertip 1fingertip 2fingertip 3fingertip 4fingertip 5
hand root + five fingertipsPOSITION-AWARE SELF-ATTENTION
hone shared learnable token
p1 = (1, y1, x1)↻ R(p1)h1
p2 = (2, y2, x2)↻ R(p2)h2
pT = (T, yT, xT)↻ R(pT)h3
↓
V1′V2′V3′V4′V5′V6′…Vn′rotated video tokens
h1h2h3rotated special tokens
↓
Self-AttentionCurrent Progress
EgoDex Self-Attention + RoPE Conditioning
Inputvideo tokens X, text context C, action UV U, learnable table EOutputpredicted video V̂
1Construct 49 × 12 = 588 action-token instances
T = 49framesK = 12keypoints per framen = 0 … 587flattened sequence
for t = 0 … T−1, k = 0 … K−1▹ iterate over every frame and keypoint n ← tK + k▹ flatten (t, k) into one sequence index A[n] ← RMSNorm(E[k])▹ keypoint identity shared across all frames p[n] ← (t / (T−1), y[n], x[n])▹ frame-specific time, height, and width2Inject action tokens in every Wan DiT block
(Qv, Kv, Vv) ← Wqkv(X)(Qa, Ka, Va) ← Wqkv(A)(Qv, Kv) ← RoPEvideo(Qv, Kv)(Qa, Ka) ← RoPEaction(Qa, Ka, p)Yv ← Attn(Qv, [Kv ; Ka], [Vv ; Va])Ya ← Attn(Qa, Ka, Va)X ← DiTBlock(X, Yv, C) · A ← ActionBlock(A, Ya)3Decode video only
V̂ ← Head(X) // discard all 588 action tokensCurrent Progress
Qualitative Results
Current Progress
DROID Qualitative Results
Current Progress
GoalForce
Force is a vector condition, not a point condition.
FOUR DEGREES OF FREEDOM
tapplication timepapplication positionmforce magnituded̂force direction
time + position(t, y, x)→Start-point Token + RoPE
magnitude + direction(m, d̂)→Relative Vector Endpoint
VECTOR REPRESENTATION
startRoPE(t, y, x)endm · d̂
Current Progress
Quantitative Evaluation across Action Settings
128test samples
Action variants
Original actionStatic / no motionFlip XFlip X (half)
Generated video→Keypoint estimation→Deviation from condition
MotionADE ↓lower is better
Comprehensive comparison across model and conditioning settings.
Current Progress
GoalForce Results
Current Progress
Next Steps
Toward a Unified Action Space
Cross-Dataset Force Unification
Cross-Action Unification
Force Training across Application Times
0119