Pairwise measures - MetricsReloaded.metrics.pairwise_measures

This module provides classes for calculating binary and multiclass pairwise measures.

Calculating binary pairwise measures

class MetricsReloaded.metrics.pairwise_measures.BinaryPairwiseMeasures(pred, ref, measures=[], connectivity_type=1, pixdim=None, empty=False, dict_args={})

youden_index()

Calculates the Youden Index (YI) defined as:

\[YI = Specificity + Sensitivity - 1\]

Returns:

Youden index

sensitivity()

Calculates the sensitivity defined as

\[Sens = \dfrac{TP}{\sharp Ref}\]

This measure is not defined for empty reference. Will raise a warning and return a nan value

Returns:

sensitivity

specificity()

Calculates the specificity defined as

\[Spec = \dfrac{TN}{\sharp {1-Ref}}\]

This measure is not defined when there is no reference negative. This will raise a warning and return a nan

Returns:

specificity

balanced_accuracy()

Calculates and returns the balanced accuracy defined for the binary case as the average between sensitivity and specificity

Margherita Grandini, Enrico Bagli, and Giorgio Visani. 2020. Metrics for multi-class classification: an overview. arXiv preprint arXiv:2008.05756 (2020).

Returns:

balanced accuracy

accuracy()

Calculate and returns the accuracy defined as

Margherita Grandini, Enrico Bagli, and Giorgio Visani. 2020. Metrics for multi-class classification: an overview. arXiv preprint arXiv:2008.05756 (2020).

\[Acc = \dfrac{TN+TP}{TN+TP+FN+FP}\]

Returns:

accuracy

false_positive_rate()

Calculates and returns the false positive rate defined as

\[FPR = \dfrac{FP}{\sharp ar{Ref}}\]

Returns:

false positive rate

normalised_expected_cost()

Calculates and returns the normalised expected cost

Luciana Ferrer. 2022. Analysis and Comparison of Classification Metrics. arXiv preprint arXiv:2209.05355 (2022).

Returns:

normalised expected cost

matthews_correlation_coefficient()

Calculates and returns the MCC for the binary case

\[MCC = \dfrac{TP * TN - FP * FN}{(TP+FP)*(TP+FN)*(TN+FP)*(TN+FN)}\]

Returns:

MCC

cohens_kappa()

Calculates and return the Cohen’s kappa score defined as

\[CK = \dfrac{p_o - p_e}{1-p_e}\]

where

:math:`p_e = ` expected chance matching and :math:`p_o = `observed accuracy

Returns:

CK

positive_likelihood_ratio()

Calculates the positive likelihood ratio

John Attia. 2003. Moving beyond sensitivity and specificity: using likelihood ratios to help interpret diagnostic tests. Australian prescriber 26, 5 (2003), 111–113.

\[LR+ = \dfrac{Sensitivity}{1-Specificity}\]

Returns:

LR+

pred_in_ref()

Determines if prediction and reference overlap on at least one voxel.

Returns:

1 if true, 0 otherwise

positive_predictive_values()

Calculates the positive predictive value

\[PPV = \dfrac{TP}{TP+FP}\]

Not defined when no positives in the prediction - returns nan if both reference and prediction empty. Returns 0 if only prediction empty

Returns:

PPV

recall()

Calculates and returns recall = sensitivity

Returns:

Recall = Sensitivity

dsc()

Calculates the Dice Similarity Coefficient defined as

Lee R Dice. 1945. Measures of the amount of ecologic association between species. Ecology 26, 3 (1945), 297–302.

..math:

DSC = \dfrac{2TP}{2TP+FP+FN}

This is also F:math:{beta} for :math:`{beta}`=1

fbeta()

Calculates FBeta score defined as

Nancy Chinchor. 1992. MUC-4 Evaluation Metrics. In Proceedings of the 4th Conference on Message Understanding (McLean, Virginia) (MUC4 ’92). Association for Computational Linguistics, USA, 22–29. https://doi.org/10.3115/ 1072064.1072067

\[F_{\beta} = (1+{\beta}^2) \dfrac{Precision * Recall}{{\beta}^2 * Precision + recall}\]

When \({\beta}=1\) it corresponds to the dice score. The \({\beta}\) parameter is set up in the class dictionary of options

Returns:

fbeta value

net_benefit_treated()

This functions calculates the net benefit treated according to a specified exchange rate

Andrew J Vickers, Ben Van Calster, and Ewout W Steyerberg. 2016. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. bmj 352 (2016).

\[NB = \dfrac{TP}{N} - \dfrac{FP}{N} * ER\]

where ER relates to the exchange rate. For instance if a suitable exchange rate is to find 1 positive case among 10 tested (1TP for 9 FP), the exchange rate would be 1/9

Returns:

NB

negative_predictive_values()

This function calculates the negative predictive value ratio between the number of true negatives and the total number of negative elements

Returns:

NPV

fppi()

This function returns the average number of false positives per

image, assuming that the cases are collated on the last axis of the array

intersection_over_reference()

This function calculates the ratio of the intersection of prediction and reference over reference.

Returns:

IoR

intersection_over_union()

This function calculates the intersection of prediction and reference over union - This is also the definition of jaccard coefficient

Returns:

IoU

com_dist()

This function calculates the euclidean distance between the centres of mass of the reference and prediction.

Returns:

Euclidean distance between centre of mass when reference and prediction not empty

-1 otherwise

com_ref()

This function calculates the centre of mass of the reference prediction

Returns:

Centre of mass coordinates of reference when not empty, -1 otherwise

com_pred()

This functions provides the centre of mass of the predmented element :returns: -1 if empty image, centre of mass of prediction otherwise

vol_diff()

This function calculates the ratio of difference in volume between the reference and prediction images.

Returns:

vol_diff

topology_precision()

Calculates topology precision defined as

\[Prec_{Top} = \dfrac{|S_{Pred} \cap Ref|}{|S_{Pred}|}\]

with \(S_{Pred}\) the skeleton of Pred

Returns:

topology_precision

topology_sensitivity()

Calculates the topology sensitivity defined as

\[Sens_{Top} = \dfrac{|S_{Ref} \cap Pred|}{|S_{Ref}|}\]

with \(S_{Ref}\) the skeleton of Ref

Returns:

topology_sensitivity

centreline_dsc()

Calculates the centre line dice score defined as

Suprosanna Shit, Johannes C Paetzold, Anjany Sekuboyina, Ivan Ezhov, Alexander Unger, Andrey Zhylka, Josien PW Pluim, Ulrich Bauer, and Bjoern H Menze. 2021. clDice-a novel topology-preserving loss function for tubular structure segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16560–16569

\[cDSC = 2\dfrac{Sens_{Top} * Prec_{Top}}{Sens_{Top} + Prec_{Top}}\]

Returns:

cDSC

boundary_iou()

This functions determines the boundary iou

Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov. 2021. Boundary IoU: Improving

Object-Centric Image Segmentation Evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15334–15342.

\[B_{IoU}(A,B) = \dfrac{| A_{d} \cap B_{d} |}{|A_d| + |B_d| - |A_d \cap B_d|}\]

where :math:A_d are the pixels of A within a distance d of the boundary :return: boundary_iou

normalised_surface_distance()

Calculates the normalised surface distance (NSD) between prediction and reference using the distance parameter \({\tau}\)

Stanislav Nikolov, Sam Blackwell, Alexei Zverovitch, Ruheena Mendes, Michelle Livne, Jeffrey De Fauw, Yojan Patel, Clemens Meyer, Harry Askham, Bernadino Romera-Paredes, et al. 2021. Clinically applicable segmentation of head and neck anatomy for radiotherapy: deep learning algorithm development and validation study. Journal of Medical Internet Research 23, 7 (2021), e26151.

\[NSD(A,B)^{(\tau)} = \dfrac{|S_{A} \cap Bord_{B,\tau}| + |S_{B} \cup Bord_{A,\tau}|}{|S_{A}| + S_{B}}\]

Returns:

NSD

measured_distance()

This functions calculates the average symmetric distance and the hausdorff distance between a prediction and a reference image

Returns:

hausdorff distance and average symmetric distance, hausdorff distance at perc

and masd

measured_average_distance()

This function returns only the average distance when calculating the distances between prediction and reference

\[ASSD(A,B) = \dfrac{\sum_{a\inA}d(a,B) + \sum_{b\inB}d(b,A)}{|A|+ |B|}\]

Returns:

assd

measured_masd()

This function returns only the mean average surface distance defined as

Miroslav Beneš and Barbara Zitová. 2015. Performance evaluation of image segmentation algorithms on microscopic image data. Journal of microscopy 257, 1 (2015), 65–85.

\[MASD(A,B) = \dfrac{1}{2}\left(\dfrac{\sum_{a\in A}d(a,B)}{|A|} + \dfrac{\sum_{b\inB}d(b,A)}{|B|})\]

Returns:

masd

measured_hausdorff_distance()

This function returns only the hausdorff distance when calculated the distances between prediction and reference

Daniel P Huttenlocher, Gregory A. Klanderman, and William J Rucklidge. 1993. Comparing images using the Hausdorff distance. IEEE Transactions on pattern analysis and machine intelligence 15, 9 (1993), 850–863.

Returns:

hausdorff_distance

measured_hausdorff_distance_perc()

This function returns the xth percentile hausdorff distance

Daniel P Huttenlocher, Gregory A. Klanderman, and William J Rucklidge. 1993. Comparing images using the Hausdorff distance. IEEE Transactions on pattern analysis and machine intelligence 15, 9 (1993), 850–863.

Returns:

hausdorff_distance_perc

Calculating multiclass pairwise measures

class MetricsReloaded.metrics.pairwise_measures.MultiClassPairwiseMeasures(pred, ref, list_values, measures=[], dict_args={})

Class dealing with measures of direct multi-class such as MCC, Cohen’s kappa, Expected cost or balanced accuracy

matthews_correlation_coefficient()

Calculates the multiclass Matthews Correlation Coefficient defined as

Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure 405, 2 (1975), 442–451.

\[R_k = \dfrac{cov_k(Pred,Ref)}{\sqrt{cov_k(Pred,Pred)*cov_k(Ref,Ref)}}\]

with

\[cov_k(X,Y) = \dfrac{1}{K}\sum_{k=1}^{K}cov(X_k,Y_k)\]

Returns:

Matthews Correlation Coefficient

chance_agreement_probability()

Determines the probability of agreeing by chance given two classifications. To be used for CK calculation

confusion_matrix()

Provides the confusion matrix Prediction in rows, Reference in columns

Returns:

confusion_matrix

balanced_accuracy()

Calculation of balanced accuracy as average of correctly classified by reference class across all classes

\[BA = \dfrac{\sum_{k=1}^{K} \dfrac{TP_k}{TP_k+FN_k}}{K}\]

Returns:

balanced_accuracy

expectation_matrix()

Determination of the expectation matrix to be used for CK derivation

Returns:

expectation_matrix

weighted_cohens_kappa()

Derivation of weighted cohen’s kappa. The weight matrix is set to 1-ID(len(list_values)) - cost of 1 for each error type if no weight provided

Returns:

weighted_cohens_kappa

to_dict_meas(fmt='{:.4f}')

Given the selected metrics provides a dictionary with relevant metrics