robometric_frame.task_performance.action_accuracy

Action Accuracy metrics for robotics policy evaluation.

Action Accuracy measures the precision of predicted actions against ground truth trajectories using Mean Squared Error (MSE) and its variations. This provides direct assessment of model performance in offline evaluation scenarios.

References

[1] M. Dobiš et al., “Evaluation criteria for trajectories of robotic arms,”

Robotics, vol. 11, p. 29, 2022.

[2] K. K. A. Farag et al., “Mobile robot obstacle avoidance based on neural

network with a standardization technique,” J. Robot., vol. 2021, 2021.

Classes

ActionAccuracy([normalize, action_variance])

Compute Action Accuracy metrics (MSE, AMSE, NAMSE) for robotics policy evaluation.

class robometric_frame.task_performance.action_accuracy.ActionAccuracy(normalize=False, action_variance=None, **kwargs)

Compute Action Accuracy metrics (MSE, AMSE, NAMSE) for robotics policy evaluation.

This metric computes three related measures of action prediction accuracy: - MSE: Mean Squared Error per trajectory - AMSE: Average MSE across multiple trajectories - NAMSE: Normalized AMSE (scaled by action variance)

Formulas:

MSE = (1/T) * sum_{t=1}^{T} |a_t - â_t|_2^2 AMSE = (1/K) * sum_{k=1}^{K} MSE_k NAMSE = AMSE / σ²_action

where:

• a_t is the ground truth action at timestep t

• â_t is the predicted action at timestep t

• T is the number of timesteps in a trajectory

• K is the number of trajectories

• σ²_action is the variance of ground truth actions

Parameters:

• normalize (bool) – Whether to compute NAMSE. If True, action variance is computed from the data. If False, only MSE and AMSE are computed. Default: False.

• action_variance (Optional[float]) – Pre-computed action variance for normalization. If provided, this value is used instead of computing from data. Default: None.

• **kwargs (Any) – Additional keyword arguments passed to the base Metric class.

Example

>>> from robometric_frame import ActionAccuracy
>>> import torch
>>> metric = ActionAccuracy()
>>>
>>> # Single trajectory
>>> predictions = torch.randn(10, 4)  # 10 timesteps, 4-dim actions
>>> targets = torch.randn(10, 4)
>>> metric.update(predictions, targets)
>>> results = metric.compute()
>>> print(f"MSE: {results['mse']:.4f}, AMSE: {results['amse']:.4f}")
>>>
>>> # With normalization
>>> metric = ActionAccuracy(normalize=True)
>>> metric.update(predictions, targets)
>>> results = metric.compute()
>>> print(f"NAMSE: {results['namse']:.4f}")

Example (multiple trajectories):

>>> metric = ActionAccuracy()
>>> # Trajectory 1
>>> metric.update(torch.randn(10, 4), torch.randn(10, 4))
>>> # Trajectory 2
>>> metric.update(torch.randn(15, 4), torch.randn(15, 4))
>>> results = metric.compute()
>>> # AMSE is averaged across both trajectories

full_state_update: bool = False

total_mse: Tensor

total_trajectories: Tensor

total_squared_actions: Tensor

total_actions: Tensor

total_action_count: Tensor

__init__(normalize=False, action_variance=None, **kwargs)

Initialize the ActionAccuracy metric.

update(predictions, targets)

Update metric state with predicted and target actions.

Parameters:

• predictions (Tensor) – Predicted actions of shape (T, D) where T is the number of timesteps and D is the action dimension.

• targets (Tensor) – Ground truth actions of shape (T, D).

Raises:

ValueError – If predictions and targets have different shapes or are empty.

Return type:

None

compute()

Compute the final Action Accuracy metrics.

Returns:

• ‘mse’: Mean Squared Error of the last trajectory

• ’amse’: Average MSE across all trajectories

• ’namse’: Normalized AMSE (only if normalize=True)

Return type:

Dictionary containing

Raises:

RuntimeError – If no trajectories have been recorded.

training: bool