Beam pipeline

weatherbenchX.beam_pipeline.define_pipeline(root: Pipeline, times: TimeChunks, predictions_loader: DataLoader, targets_loader: DataLoader, metrics: Mapping[str, Metric], aggregator: Aggregator | Mapping[str, Aggregator], out_path: str | Mapping[str, str] | None = None, aggregation_state_out_path: str | Mapping[str, str] | None = None, setup_fn: Callable[[], None] | None = None)[source]

Defines a beam pipeline for calculating aggregated metrics.

Parameters:

  • root – Pipeline root.

  • times – TimeChunks instance.

  • predictions_loader – DataLoader instance.

  • targets_loader – DataLoader instance.

  • metrics – A dictionary of metrics to compute.

  • aggregator – Aggregator instance or mapping of aggregator name to Aggregator instance.

  • out_path – The full path to write the metrics to (or mapping of aggregator name to path). If you specify multiple aggregators but only a single out_path, the aggregator name will be appended to the filename to get a path for each aggregator.

  • aggregation_state_out_path – The full path to write the final aggregation state to (or mapping of aggregator name to path). This can be useful if you want to compute further metrics from it later, and if you are preserving init_time, it can be useful to compute confidence intervals from later too. Behaviour is the same as for out_path if multiple aggregators are specified.

  • setup_fn – (Optional) A function to call once per worker in LoadPredictionsAndTargets.