robometric_frame.task_performance.success_rate

Success Rate metric for robotics policy evaluation.

Success Rate (SR) is a fundamental metric measuring the percentage of successfully completed tasks in robotics policy evaluation.

Reference:

A. Brohan et al., “RT-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817, 2022.

Classes

SuccessRate([threshold, ignore_index])	Compute Success Rate for robotics policy task evaluation.
TaskSuccessRate([threshold, ignore_index])	Alias for SuccessRate metric.

class robometric_frame.task_performance.success_rate.SuccessRate(threshold=None, ignore_index=None, **kwargs)

Compute Success Rate for robotics policy task evaluation.

Success Rate is calculated as:

SR = N_success / N_total

where N_success is the number of successfully completed tasks and N_total is the total number of tasks attempted.

This metric supports both binary success indicators and continuous success scores with an optional threshold.

Parameters:

• threshold (Optional[float]) – Threshold for binary classification when using continuous scores. If None, assumes binary inputs (0 or 1). Default: None.

• ignore_index (Optional[int]) – Value to ignore in the success tensor. Default: None.

• **kwargs (Any) – Additional keyword arguments passed to the base Metric class.

Example

>>> from robometric_frame import SuccessRate
>>> metric = SuccessRate()
>>> # Binary success indicators
>>> success = torch.tensor([1, 1, 0, 1, 0, 0, 1])
>>> metric(success)
tensor(0.5714)

>>> # With continuous scores and threshold
>>> metric = SuccessRate(threshold=0.8)
>>> scores = torch.tensor([0.9, 0.7, 0.85, 0.6, 0.95])
>>> metric(scores)
tensor(0.6000)

Example (distributed):

>>> # In distributed training, metrics are automatically synced
>>> metric = SuccessRate()
>>> # On GPU 0
>>> success_gpu0 = torch.tensor([1, 1, 0])
>>> metric(success_gpu0)
>>> # On GPU 1
>>> success_gpu1 = torch.tensor([1, 0, 1])
>>> metric(success_gpu1)
>>> # Final result aggregates across all GPUs
>>> result = metric.compute()  # Returns aggregated success rate

full_state_update: bool = False

total_success: Tensor

total_tasks: Tensor

__init__(threshold=None, ignore_index=None, **kwargs)

Initialize the SuccessRate metric.

update(success)

Update metric state with new success indicators.

Parameters:

success (Tensor) – Tensor of shape (N,) containing binary success indicators (0 or 1) or continuous success scores if threshold is set. Values can be int, float, or bool.

Raises:

ValueError – If success tensor is empty or contains invalid values.

Return type:

None

compute()

Compute the final Success Rate.

Return type:

Tensor

Returns:

Success rate as a scalar tensor in range [0, 1].

Raises:

RuntimeError – If no tasks have been recorded (total_tasks == 0).

training: bool

class robometric_frame.task_performance.success_rate.TaskSuccessRate(threshold=None, ignore_index=None, **kwargs)

Alias for SuccessRate metric.

This class provides a more descriptive name for the same functionality.