Scoring rules in probability assessment and evaluation

doi:10.1016/0001-6918(70)90023-5

Acta Psychologica

Volume 34, 1970, Pages 273-286

https://doi.org/10.1016/0001-6918(70)90023-5 Get rights and content

Abstract

The purpose of this paper is to briefly discuss some important current questions and problems related to the use of scoring rules (SRs) both in connection with the actual assessment of probabilities and with the evaluation of probability forecasts and probability assessors. With regard to the assessment process, we consider both the case in which the assessor's utility function is linear and the case in which his utility function is nonlinear. Under linear utility, important problems of concern are the sensitivity of SRs to deviations from optimality (with a strictly proper SR, optimality consists of the assessor making his statements correspond to his judgments) and the effect of psychological considerations arising from the use of different SRs. Under nonlinear utility, SRs should be modified to allow for the nonlinearity in such a manner that for a specific utility function, the modified SRs are strictly proper. This introduces the difficult question of the assessment of the assessor's utility function. With regard to the evaluation process (as opposed to the assessment process), we consider the process from an inferential viewpoint and from a decision-theoretic viewpoint. From an inferential viewpoint, attributes such as validity may be of interest, and in certain circumstances these attributes may be related to SRs. The attributes of interest, of course, depend on the framework within which the evaluation process is undertaken. From a decision-theoretic viewpoint, SRs may be related to a decision maker's utilities or expected utilities (under uncertainty about the utilities) if the decision maker uses the assessed probabilities in an actual decision situation.

In summary, there are many important questions and problems related to SRs, and the need for future research on these problems seems clear. Such research should lead to a greatly improved understanding of the processes of probability assessment and evaluation.

References (20)

B. De Finetti
Does it make sense to speak of ‘Good Probability Appraisers’?
B. De Finetti
Methods for discriminating levels of partial knowledge concerning a test item
Brit. J. math. statist. Psychol.
(1965)
B. De Finetti et al.
The elicitation of personal probabilities and expectations
(1969)
E.S. Epstein
A scoring system for probability forecasts of ranked categories
J. appl. Meteorol.
(1969)
P.C. Fishburn
Decision and value theory
(1964)
A.H. Murphy
A note on the utility of probabilistic predictions and the probability score in the cost-loss ratio decision situation
J. appl. Meteorol.
(1966)
A.H. Murphy
The evaluation of probabilistic predictions in meteorology
(1969)
A.H. Murphy
Measures of the utility of probabilistic predictions in cost-loss ratio decision situations in which knowledge of the cost-loss ratios is incomplete
J. appl. Meteorol.
(1969)
A.H. Murphy
On expected-utility measures in cost-loss ratio decision situations
J. appl. Meteorol.
(1969)
A.H. Murphy
The ranked probability score and the probability score: a comparison
Monthly Weather Review
(1970)

There are more references available in the full text version of this article.

Cited by (99)

Does reporting beliefs change beliefs?
2024, Journal of Behavioral and Experimental Economics
Does reporting beliefs change beliefs? This question may seem impossible to answer, because one cannot compare the beliefs of participants who did or did not report their beliefs: we cannot know beliefs that we don't measure. In an experiment, participants first answer trivia questions, and are then asked to estimate how many questions they have answered right. In the two treatments the payoff function is skewed: either over or underestimations are punished more severe. This manipulation causes a difference in report between the two treatments and the main question is whether that changes actual beliefs. In the third part participants rate per question how likely their answer is right, and in the final part participants choose between payment per correctly answered question or a tournament. The ratings per question differ only marginally between the treatments, and no effect was found on behavior.
Survey experiments on economic expectations
2022, Handbook of Economic Expectations
In this chapter, we discuss field experiments in surveys that are conducted with the purpose of learning about expectation formation and the link between expectations and behavior. We begin by reviewing the rationale for conducting experiments within surveys, rather than just relying on observational survey data. We then outline the most commonly used experimental paradigm, randomized information provision, along with some examples. Next, we outline a few methodological issues that are important to consider in the design of such experiments. We also provide a discussion of existing extensions of this paradigm, as well as of alternative approaches. We close with some directions for future work.
We should totally open a restaurant: How optimism and overconfidence affect beliefs
2018, Journal of Economic Psychology
Wishful thinking, defined as the tendency to over-estimate the probability of high-payoff outcomes, is a widely-documented phenomenon that can affect decision-making across numerous domains, including finance, management, and entrepreneurship. We design an experiment to distinguish and test the relationship between two easily-confounded biases, optimism and overconfidence, both of which can contribute to wishful thinking. We find that optimism and overconfidence are positively correlated at the individual level and that both help to explain wishful thinking. These findings suggest that ignoring optimism results in an upwardly biased estimate of the role of overconfidence in explaining wishful thinking. To illustrate this point, we show that 30% of our observations are misclassified as under- or overconfident if optimism is omitted from the analysis. Our findings have potential implications for the design of information interventions since how agents incorporate information depends on whether the bias is ego-related.
Finding cooperators: Sorting through repeated interaction
2018, Journal of Economic Behavior and Organization
We present evidence from an indefinitely repeated gift-exchange game where market structures affect proposers’ ability to punish uncooperative partners and their ability to sort between cooperative and uncooperative partners. Treatments vary by whether subjects can replace their partners, and if not, whether they can reduce their gift from one round to the next. Comparing treatments without contract restrictions, our replacement treatment is no different initially but has higher cooperation in the long run. Comparing treatments without replacement, our treatment with contract restrictions has lower cooperation initially but is no different in the long run. Neither of these findings are predicted by theories of repeated games based on the ability to punish, however, both findings are consistent with a simple sorting model.
Using Cross-Domain Expertise to Aggregate Forecasts When Within-Domain Expertise Is Unknown
2024, Decision
A Choice Matching Approach for Discrete Choice Analysis: An Experimental Investigation in the Laboratory
2023, Land Economics

View all citing articles on Scopus

^☆: Supported, in part, by the National Science Foundation (Atmospheric Sciences Section) under Grant GA-1707.

^∗∗: Presently on leave and visiting at the University of Washington, Seattle, Wash. 98105, U.S.A.

View full text

Scoring rules in probability assessment and evaluation☆

Abstract

Does it make sense to speak of ‘Good Probability Appraisers’?

Methods for discriminating levels of partial knowledge concerning a test item

Brit. J. math. statist. Psychol.

The elicitation of personal probabilities and expectations

A scoring system for probability forecasts of ranked categories

J. appl. Meteorol.

Decision and value theory

A note on the utility of probabilistic predictions and the probability score in the cost-loss ratio decision situation

J. appl. Meteorol.

The evaluation of probabilistic predictions in meteorology

Measures of the utility of probabilistic predictions in cost-loss ratio decision situations in which knowledge of the cost-loss ratios is incomplete

J. appl. Meteorol.

On expected-utility measures in cost-loss ratio decision situations

J. appl. Meteorol.

The ranked probability score and the probability score: a comparison

Monthly Weather Review