Beam pipeline

weatherbenchX.beam_pipeline.define_pipeline(root: Pipeline, times: TimeChunks, predictions_loader: DataLoader, targets_loader: DataLoader, metrics: Mapping[str, Metric], aggregator: Aggregator, out_path: str | None = None, aggregation_state_out_path: str | None = None, setup_fn: Callable[[], None] | None = None)

Defines a beam pipeline for calculating aggregated metrics.

Parameters:

• root – Pipeline root.

• times – TimeChunks instance.

• predictions_loader – DataLoader instance.

• targets_loader – DataLoader instance.

• metrics – A dictionary of metrics to compute.

• aggregator – Aggregation instance.

• out_path – The full path to write the metrics to.

• aggregation_state_out_path – The full path to write the final aggregation state to. This can be useful if you want to compute further metrics from it later, and if you are preserving init_time, it can be useful to compute confidence intervals from later too.

• setup_fn – (Optional) A function to call once per worker in LoadPredictionsAndTargets.