Yusuf Sale

We present a novel approach to uncertainty quantification in classification tasks based on label-wise decomposition of uncertainty measures. This label- wise perspective allows uncertainty to be quantified at the individual class level, thereby improving cost-sensitive decision-making and helping under- stand the sources of uncertainty. Furthermore, it allows to define total, aleatoric, and epistemic un- certainty on the basis of non-categorical measures such as variance, going beyond common entropy- based measures. In particular, variance-based measures address some of the limitations associated with established methods that have recently been discussed in the literature. We show that our pro- posed measures adhere to a number of desirable properties. Through empirical evaluation on a variety of benchmark data sets – including applications in the medical domain where accurate uncertainty quantification is crucial – we establish the effectiveness of label-wise uncertainty quantification.

The quantification of aleatoric and epistemic un- certainty in terms of conditional entropy and mutual information, respectively, has recently become quite common in machine learning. While the properties of these measures, which are rooted in in- formation theory, seem appealing at first glance, we identify various incoherencies that call their appropriateness into question. In addition to the measures themselves, we critically discuss the idea of an additive decomposition of total uncertainty into its aleatoric and epistemic constituents. Experiments across different computer vision tasks support our theoretical findings and raise concerns about current practice in uncertainty quantification.

Adequate uncertainty representation and quantification have become imperative in various scientific disciplines, especially in machine learning and artificial intelligence. As an alternative to representing uncertainty via one single probability measure, we consider credal sets (convex sets of probability measures). The geometric representation of credal sets as d-dimensional polytopes implies a geometric intuition about (epistemic) uncertainty. In this paper, we show that the volume of the geometric representation of a credal set is a meaningful mea- sure of epistemic uncertainty in the case of binary classification, but less so for multi-class classification. Our theoretical findings highlight the crucial role of specifying and employing uncertainty measures in machine learning in an appropriate way, and for being aware of possible pitfalls.

In the past couple of years, various approaches to representing and quantifying different types of predictive uncertainty in ma- chine learning, notably in the setting of classification, have been proposed on the basis of second-order probability distributions, i.e., predictions in the form of distributions on probability distributions. A completely conclusive solution has not yet been found, however, as shown by recent criticisms of commonly used uncertainty measures associated with second-order distributions, identifying undesirable theoretical properties of these measures. In light of these criticisms, we propose a set of formal criteria that meaningful uncertainty measures for predictive un- certainty based on second-order distributions should obey. Moreover, we provide a general framework for developing uncertainty measures to account for these criteria, and offer an instantiation based on the Wasserstein distance, for which we prove that all criteria are satisfied.

While the predictions produced by conformal prediction are set-valued, the data used for training and calibration is supposed to be precise. In the setting of superset learning or learning from partial labels, a variant of weakly supervised learning, it is exactly the other way around: training data is possibly imprecise (set-valued), but the model induced from this data yields precise predictions. In this paper, we combine the two settings by making conformal prediction amenable to set-valued training data. We propose a generalization of the conformal prediction procedure that can be applied to set-valued training and calibration data. We prove the validity of the proposed method and present experimental studies in which it compares favorably to natural baselines.

Uncertainty representation and quantification are paramount in machine learning, especially in safety-critical applications. In this paper, we pro- pose a novel framework for the quantification of aleatoric and epistemic uncertainty based on the notion of credal sets, i.e., sets of probability distributions. Thus, we assume a learner that produces (second-order) predictions in the form of sets of probability distributions on outcomes. Practically, such an approach can be realized by means of ensemble learning: Given an ensemble of learners, credal sets are generated by including sufficiently plausible predictors, where plausibility is mea- sured in terms of (relative) likelihood. We provide a formal justification for the framework and introduce new measures of epistemic and aleatoric un- certainty as concrete instantiations. We evaluate these measures both theoretically, by analyzing desirable axiomatic properties, and empirically, by comparing them in terms of performance and effectiveness to existing measures of uncertainty in an experimental study.

In order to achieve reliable results via statistical methodology, one important goal is to account for potential uncertainty. Shige Peng introduced an un- certainty counterpart of Kolmogorov’s probabilistic setting: the G-Framework. While this framework is well-known in mathematical finance, work within the G-Framework in statistics is limited. This thesis motivates nonlinear expectations for decision-making under uncertainty in dynamic and non-dynamic situations. Switching the viewpoint from probability spaces to expectation spaces, we discuss the theoretical foundations of the G-Framework, emphasizing comprehensibility. We motivate nonlinear expectations for subsequent application in statistics by notions that emerged in various academic communities and are like- wise concerned with decision-making under uncertainty: Choquet expectations express probabilistic uncertainty from the viewpoint of non-additive measures and g-expectations, which represent a nonlinear class of expectations based on backward stochastic differential equations (BSDE). For explicit understanding, we provide the required foundations of stochastic calculus in a self-contained form. The applicability of the G-Framework in statistics is particularly evident from the respective Law of Large Numbers and Central Limit Theorem. To emphasize the applicability, this thesis motivates a notion of sublinear regression.

In their seminal 1990 paper, Wasserman and Kadane establish an upper bound for the Bayes’ posterior probability of a measurable set, when the prior lies in a class of probability measures and the likelihood is precise. They also give a sufficient condition for such upper bound to hold with equality. In this paper, we introduce a generalization of their result by additionally addressing uncertainty related to the likelihood. We give an upper bound for the posterior probability when both the prior and the likelihood belong to a set of probabilities. Furthermore, we give a sufficient condition for this upper bound to become an equality. This result is interesting on its own, and has the potential of being applied to various fields of engineering (e.g. model predictive control), machine learning, and artificial intelligence.