robometric_frame.trajectory_quality.dtw

Dynamic Time Warping (DTW) metrics for robotics policy trajectory evaluation.

DTW-based metrics measure trajectory similarity while allowing for temporal misalignment. Unlike MSE which requires point-to-point correspondence, DTW finds the optimal warping between sequences of different lengths or timing.

Reference:: G. Ilharco, V. Jain, A. Ku, E. Ie, and J. Baldridge, “General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping,” arXiv:1907.05446, NeurIPS ViGIL Workshop, 2019.

Classes

`DTWBase`(**kwargs)	Base class for DTW-based trajectory metrics.
`DTWDistance`(**kwargs)	Compute Dynamic Time Warping (DTW) distance for trajectory evaluation.
`NormalizedDTW`([normalization_factor])	Compute Normalized DTW (nDTW) for trajectory evaluation.
`SuccessWeightedDTW`([normalization_factor])	Compute Success-weighted DTW (SDTW) for trajectory evaluation.

class robometric_frame.trajectory_quality.dtw.DTWBase(**kwargs)[source]

Base class for DTW-based trajectory metrics.

Provides shared validation, DTW computation, path length computation, and normalization factor logic used by DTWDistance, NormalizedDTW, and SuccessWeightedDTW.

Subclasses must define their own metric states and implement update()/compute().

is_differentiable: bool = False

full_state_update: bool = False

num_trajectory_pairs: Tensor

__init__(**kwargs)[source]

Initialize internal Module state, shared by both nn.Module and ScriptModule.

training: bool

class robometric_frame.trajectory_quality.dtw.DTWDistance(**kwargs)[source]

Compute Dynamic Time Warping (DTW) distance for trajectory evaluation.

DTW distance measures the minimum-cost temporal alignment between predicted and reference trajectories. Unlike MSE which compares trajectories timestep-by- timestep, DTW finds the optimal warping to align sequences that may differ in length or timing.

DTW is calculated by building an accumulated cost matrix D where:: D[0,0] = C[0,0] D[i,0] = D[i-1,0] + C[i,0] for i > 0 D[0,j] = D[0,j-1] + C[0,j] for j > 0 D[i,j] = C[i,j] + min(D[i-1,j], D[i,j-1], D[i-1,j-1]) for i,j > 0

where C[i,j] is the Euclidean distance between predicted[i] and reference[j]. The final DTW distance is D[n-1, m-1].

This metric is particularly useful for evaluating VLA models and policies using action chunking (e.g., ACT, Diffusion Policy) where predicted trajectories may be temporally misaligned with demonstrations.

This metric accumulates DTW distances across multiple trajectory pairs and returns the average DTW distance when compute() is called.

Parameters:: **kwargs (Any) – Additional keyword arguments passed to the base Metric class.

higher_is_better: False - lower DTW distance indicates better similarity.

is_differentiable: False - DTW computation is not differentiable.

full_state_update: False - incremental state updates.

Note

Memory complexity is O(T_pred * T_ref) for the cost matrices. For very long trajectories, this may require significant memory.

Example

>>> from robometric_frame.trajectory_quality import DTWDistance
>>> import torch
>>> metric = DTWDistance()
>>> # Identical trajectories (zero distance)
>>> predicted = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
>>> reference = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
>>> metric.update(predicted, reference)
>>> metric.compute()
tensor(0.0000)

Example (different lengths):

>>> # Trajectories of different lengths (the core use case)
>>> metric = DTWDistance()
>>> reference = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0], [3.0, 0.0]])
>>> predicted = torch.tensor([[0.0, 0.0], [0.5, 0.0], [1.0, 0.0], [1.5, 0.0],
...                           [2.0, 0.0], [2.5, 0.0], [3.0, 0.0]])
>>> metric.update(predicted, reference)
>>> result = metric.compute()  # Small value (same path, different density)

Example (temporal shift):

>>> # Hesitation at start (same actions, different timing)
>>> metric = DTWDistance()
>>> reference = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
>>> predicted = torch.tensor([[0.0, 0.0], [0.0, 0.0], [0.0, 0.0],
...                           [1.0, 0.0], [2.0, 0.0]])
>>> metric.update(predicted, reference)
>>> result = metric.compute()  # Small value (DTW tolerates hesitation)

higher_is_better: bool = False

total_dtw_distance: Tensor

__init__(**kwargs)[source]

Initialize the DTWDistance metric.

update(predicted, reference)[source]

Update metric state with new predicted and reference trajectory pair.

Parameters:

predicted (Tensor) –
Predicted trajectory tensor of shape (T_pred, D) where: - T_pred is the number of timesteps (can differ from T_ref) - D is the spatial dimensionality (e.g., 2 for 2D, 3 for 3D, 7 for 7-DoF)

Points should be ordered chronologically.
reference (Tensor) – Reference (ground truth) trajectory tensor of shape (T_ref, D). T_ref can differ from T_pred - this is the core advantage of DTW. D must match predicted trajectory dimensionality.

Raises:

ValueError – If trajectories have invalid shape (< 2 dimensions), mismatched dimensionality, or insufficient points.

Return type:

None

compute()[source]

Compute the average DTW distance across all trajectory pairs.

Return type:: Tensor
Returns:: Average DTW distance as a scalar tensor. Lower values indicate better trajectory similarity.
Raises:: RuntimeError – If no trajectory pairs have been recorded.

num_trajectory_pairs: Tensor

training: bool

class robometric_frame.trajectory_quality.dtw.NormalizedDTW(normalization_factor=None, **kwargs)[source]

Compute Normalized DTW (nDTW) for trajectory evaluation.

nDTW normalizes the raw DTW distance and maps it to a [0, 1] score:

nDTW = exp(-DTW / (|R| * d))

where:

DTW is the raw DTW distance
|R| is the length of the reference trajectory (number of points)
d is a normalization constant (average step distance of the reference, or a user-specified value)

Higher nDTW scores indicate better trajectory similarity (1.0 = perfect match, approaches 0.0 for very dissimilar trajectories).

This metric is particularly useful for evaluating VLA models and policies using action chunking where predicted trajectories may be temporally misaligned.

Parameters:

normalization_factor (Optional[float]) – Optional user-specified normalization constant d. If None (default), automatically computed as the mean step distance of the reference trajectory: PathLength(reference) / (len(reference) - 1).
**kwargs (Any) – Additional keyword arguments passed to the base Metric class.

higher_is_better: True - higher nDTW indicates better similarity.

is_differentiable: False - DTW computation is not differentiable.

full_state_update: False - incremental state updates.

Note

Memory complexity is O(T_pred * T_ref) for the cost matrices. For very long trajectories, this may require significant memory.

Example

>>> from robometric_frame.trajectory_quality import NormalizedDTW
>>> import torch
>>> metric = NormalizedDTW()
>>> # Identical trajectories (perfect score)
>>> predicted = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
>>> reference = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
>>> metric.update(predicted, reference)
>>> metric.compute()
tensor(1.0000)

Example (different lengths):

>>> # Trajectories of different lengths
>>> metric = NormalizedDTW()
>>> reference = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0], [3.0, 0.0]])
>>> predicted = torch.tensor([[0.0, 0.0], [0.5, 0.0], [1.0, 0.0], [1.5, 0.0],
...                           [2.0, 0.0], [2.5, 0.0], [3.0, 0.0]])
>>> metric.update(predicted, reference)
>>> result = metric.compute()  # High value (same path)

Example (custom normalization):

>>> # Use custom normalization factor
>>> metric = NormalizedDTW(normalization_factor=0.5)
>>> reference = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
>>> predicted = torch.tensor([[0.0, 0.1], [1.0, 0.1], [2.0, 0.1]])
>>> metric.update(predicted, reference)
>>> result = metric.compute()

higher_is_better: bool = True

total_ndtw: Tensor

__init__(normalization_factor=None, **kwargs)[source]

Initialize the NormalizedDTW metric.

Parameters:

normalization_factor (Optional[float]) – Optional user-specified normalization constant d. If None, automatically computed as the mean step distance of the reference trajectory.
**kwargs (Any) – Additional keyword arguments passed to the base Metric class.

update(predicted, reference)[source]

Update metric state with new predicted and reference trajectory pair.

Parameters:

predicted (Tensor) –
Predicted trajectory tensor of shape (T_pred, D) where: - T_pred is the number of timesteps (can differ from T_ref) - D is the spatial dimensionality

Points should be ordered chronologically.
reference (Tensor) – Reference (ground truth) trajectory tensor of shape (T_ref, D). T_ref can differ from T_pred. D must match predicted dimensionality.

Raises:

ValueError – If trajectories have invalid shape, mismatched dimensionality, or insufficient points.

Return type:

None

compute()[source]

Compute the average nDTW score across all trajectory pairs.

Return type:: Tensor
Returns:: Average nDTW score as a scalar tensor in range [0, 1]. Higher values indicate better trajectory similarity.
Raises:: RuntimeError – If no trajectory pairs have been recorded.

num_trajectory_pairs: Tensor

training: bool

class robometric_frame.trajectory_quality.dtw.SuccessWeightedDTW(normalization_factor=None, **kwargs)[source]

Compute Success-weighted DTW (SDTW) for trajectory evaluation.

SDTW combines trajectory fidelity with task success:

SDTW = nDTW * Success

where:

nDTW is the normalized DTW score (see NormalizedDTW)
Success is a binary indicator (1 if task succeeded, 0 if not)

If the task failed, SDTW = 0 regardless of trajectory similarity. This captures both “did you succeed?” and “did you follow the right path?”

This metric is particularly useful for benchmarking policies where both task completion and trajectory quality matter.

Parameters:

normalization_factor (Optional[float]) – Optional user-specified normalization constant d. If None (default), automatically computed as the mean step distance of the reference trajectory.
**kwargs (Any) – Additional keyword arguments passed to the base Metric class.

higher_is_better: True - higher SDTW indicates better performance.

is_differentiable: False - DTW computation is not differentiable.

full_state_update: False - incremental state updates.

Note

Memory complexity is O(T_pred * T_ref) for the cost matrices. For very long trajectories, this may require significant memory.

Example

>>> from robometric_frame.trajectory_quality import SuccessWeightedDTW
>>> import torch
>>> metric = SuccessWeightedDTW()
>>> # Successful task with good trajectory
>>> predicted = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
>>> reference = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
>>> metric.update(predicted, reference, success=torch.tensor(True))
>>> metric.compute()
tensor(1.0000)

Example (failed task):

>>> # Failed task (SDTW = 0 regardless of trajectory quality)
>>> metric = SuccessWeightedDTW()
>>> predicted = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
>>> reference = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
>>> metric.update(predicted, reference, success=torch.tensor(False))
>>> metric.compute()
tensor(0.0000)

Example (multiple updates):

>>> # Mix of successes and failures
>>> metric = SuccessWeightedDTW()
>>> ref = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
>>> pred = torch.tensor([[0.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
>>> metric.update(pred, ref, success=torch.tensor(True))   # SDTW = 1.0
>>> metric.update(pred, ref, success=torch.tensor(False))  # SDTW = 0.0
>>> metric.compute()  # Average: 0.5
tensor(0.5000)

higher_is_better: bool = True

total_sdtw: Tensor

__init__(normalization_factor=None, **kwargs)[source]

Initialize the SuccessWeightedDTW metric.

Parameters:

normalization_factor (Optional[float]) – Optional user-specified normalization constant d. If None, automatically computed as the mean step distance of the reference trajectory.
**kwargs (Any) – Additional keyword arguments passed to the base Metric class.

update(predicted, reference, success)[source]

Update metric state with new trajectory pair and success indicator.

Parameters:

predicted (Tensor) –
Predicted trajectory tensor of shape (T_pred, D) where: - T_pred is the number of timesteps (can differ from T_ref) - D is the spatial dimensionality

Points should be ordered chronologically.
reference (Tensor) – Reference (ground truth) trajectory tensor of shape (T_ref, D). T_ref can differ from T_pred. D must match predicted dimensionality.
success (Tensor) – Boolean or 0/1 tensor indicating task success. If False/0, SDTW will be 0 regardless of trajectory similarity.

Raises:

ValueError – If trajectories have invalid shape, mismatched dimensionality, or insufficient points.

Return type:

None

compute()[source]

Compute the average SDTW score across all trajectory pairs.

Return type:: Tensor
Returns:: Average SDTW score as a scalar tensor in range [0, 1]. Higher values indicate better overall performance (trajectory quality weighted by task success).
Raises:: RuntimeError – If no trajectory pairs have been recorded.

num_trajectory_pairs: Tensor

training: bool