Fitting a binary model is to minimize a loss function $$L\left(q\right(\left)\right)=\frac{1}{N}\sum _{n=1}^{N}L(y_n\mid q_n)$$

In game theory, the agent’s goal is to maximize expected score (or minimize expected loss)

• A scoring rule is proper if truthfulness maximizes expected score
• It is strictly proper if truthfulness uniquely maximizes expected score

In the context of binary response data, Fisher consistency holds pointwise if $$\arg \underset{q\in [0,1]}{min}{E}_{Y\sim \text{Bernoulli}\left(\eta \right)}L(Y\mid q)=\eta ,\forall \eta \in [0,1]$$

Fisher consistency is the defining property of proper scoring rules

Because $Y$ takes only two values, 0 and 1, $L(y\mid q)$ consists only two “partial losses”, $L(1\mid q)$ and $L(0\mid q)$

For simplicity, we prefer to express both in term of increasing functions $$L_1(1-q)=L(1\mid q),L_0\left(q\right)=L(0\mid q)$$

Pointwise expected loss is defined as $$R(\eta \mid q)=E_{Y}L(Y\mid q)=\eta L_1(1-q)+(1-\eta )L_0\left(q\right)$$

Fisher consistency becomes $$\arg \underset{q}{min}R(\eta \mid q)=\eta$$

Right: Beta loss with $\alpha =1,\beta =3$

• Tailored for classification with false positive cost $c=\frac{\alpha }{\alpha +\beta }=0.25$ and false negative cost $1-c=0.75$

Suppose the partial losses $L_1(1-q),L_0\left(q\right)$ are smooth, then the proper scoring rule property implies $$\begin{array}{cc}0& ={\frac{\partial }{\partial q}|}_{q=\eta }R(\eta \mid q)\\ & =-\eta L_1^{\prime }(1-\eta )+(1-\eta )L_0^{\prime }\left(\eta \right)\end{array}$$

Therefore, a scoring rule is proper if $$\eta L_1^{\prime }(1-\eta )=(1-\eta )L_0^{\prime }\left(\eta \right)$$

A scoring rule is strictly proper if $${\frac{{\partial }^2}{\partial q^2}|}_{q=\eta }R(\eta \mid q)>0$$

Log-loss is the negative log likelihood of the Bernoulli distribution $$L=\frac{1}{N}\sum _{n=1}^N[-y_n\log (q_n)-(1-y_n\left)\log \right(1-q_n\left)\right]$$

Partial losses for log-loss $$L_1(1-q)=-\log \left(q\right),L_0\left(q\right)=-\log (1-q)$$

Expected loss for log-loss $$R(\eta \mid q)=-\eta \log \left(q\right)-(1-\eta )\log (1-q)$$

Log-loss is a strictly proper scoring rule

Squared error loss is also known as Brier score $$L=\frac{1}{N}\sum _{n=1}^N\left[y_n\right(1-q_n{)}^2-(1-y_n)q_n^2]$$

Partial losses for squared error loss $$L_1(1-q)=(1-q{)}^2,L_0(q)=q^2$$

Expected loss for squared error loss $$R(\eta \mid q)=\eta (1-q{)}^2+(1-\eta )q^2$$

Squared error loss is a strictly proper scoring rule

Usually, misclassification loss uses $c=0.5$ as the cutoff $$L=\frac{1}{N}\sum _{n=1}^N[y_n{1}_{\{q_n\le 0.5\}}+(1-y_n\left)1_{\{q_n>0.5\}}\right]$$

Partial losses for misclassification loss $$L_1(1-q)=1_{\{q_n\le 0.5\}},L_0\left(q\right)=1_{\{q_n>0.5\}}$$

Expected loss for misclassification loss $$R(\eta \mid q)=\eta 1_{\{q\le 0.5\}}+(1-\eta )1_{\{q>0.5\}}$$

Since any $q>0.5$ for events and any $q\le 0.5$ for non-events minimize the misclassification loss, misclassification loss is a proper score rule, but it is not strictly proper

Because $y\in \{0,1\}$, the absolute deviation $L(y\mid q)=|y-q|$ becomes $$\begin{array}{cc}L(y\mid q)& =y(1-q)+(1-y)q\\ R(\eta \mid q)& =\eta (1-q)+(1-\eta )q\end{array}$$

Absolute deviation is not a proper scoring rule, because $R(\eta \mid q)$ is minimized by $q=1$ for $\eta >1/2$, and $q=0$ for $\eta <1/2$

Theorem: Let $\omega \left(dt\right)$ be a positive measure on $(0,1)$ that is finite on intervals $(ϵ,1-ϵ),\forall ϵ>0$. Then the following defines a proper scoring rule: $$L_1(1-q)={\int }_q^{f_1}(1-t)\omega \left(dt\right),L_0\left(q\right)={\int }_{f_0}^{q}t\omega \left(dt\right)$$

The proper scoring rule is strict iff $\omega \left(dt\right)$ has non-zero mass on every open interval of (0, 1)

The fixed limits $f_0\ge 0$ and $f_1\le 1$ are somewhat arbitrary

Note that for log-loss, $L_1(1-q)$ is unbounded (goes to infinity) below near $q=1$, and $L_0\left(q\right)$ is unbounded below near $q=0$

Except for log-loss, all other common proper scoring rules seem to satisfy $${\int }_0^{1}t(1-t)\omega \left(dt\right)<\infty$$

Suppose the costs of FP and FN sum up to 1:

• FP: has a cost $c$, and expected cost $cP(Y=0)=c(1-\eta )$
• FN: has a cost $1-c$, and expected cost $(1-c)P(Y=1)=(1-c)\eta$

The optimal classification is therefore class 1 iff $$(1-c)\eta \ge c(1-\eta )⟺\eta \ge c$$

• Since we don’t know the truth $\eta$, we classify as class 1 when $q\ge c$

Therefore, the classification cutoff equals $$\frac{\text{cost of FP}}{\text{cost of FP}+\text{cost of FN}}$$

• Standard classification assumes costs of FP and FN are the same, so the classification cutoff is $0.5$

Cost-weighted misclassification errors: $$\begin{array}{cc}{L}_c(y\mid q)& =y(1-c)\cdot 1_{\{q\le c\}}+(1-y)c\cdot 1_{\{q>c\}}\\ L_{1,c}(1-q)& =(1-c)\cdot 1_{\{q\le c\}},L_{0,c}\left(q\right)=c\cdot 1_{\{q>c\}}\end{array}$$

Shuford-Albert-Massengil-Savage-Schervish theorem: an intergral representation of proper scoring rules $$L(y\mid q)={\int }_0^1{L}_c(y\mid q)\omega \left(dc\right)={\int }_0^1{L}_c(y\mid q)\omega \left(c\right)dc$$

• The second equality holds if $w\left(dc\right)$ is absolutely continuous wrt Lebesgue measure
• This can be used to tailor losses to specific classification problems with cutoffs other than $1/2$ of $\eta \left(x\right)$, by designing suitable weight functions $\omega \left(\right)$

The paper proposes to use Iterative Reweighted Least Squares (IRLS) to fit linear models with proper scoring rules

This paper introduced a flexible 2-parameter family of proper scoring rules $$\omega \left(t\right)=t^{\alpha -1}(1-t{)}^{\beta -1},\text{where}\alpha >-1,\beta >-1$$

Loss function of the Beta family proper scoring rules $$\begin{array}{cc}L(y\mid q)=& y{\int }_q^1{t}^{\alpha -1}(1-t{)}^{\beta }dt+(1-y\left){\int }_0^q{t}^{\alpha }\right(1-t{)}^{\beta -1}dt\\ =& yB(\alpha ,\beta +1)[1-I_q(\alpha ,\beta +1\left)\right]\\ & +(1-y)B(\alpha +1,\beta )I_q(\alpha +1,\beta )\end{array}$$

• See the definitions of $B(a,b)$ and $I_x(a,b)$ in the next page

Log-loss and squared error loss are special cases

• Log-loss: $\alpha =\beta =0$
• Squared error loss: $\alpha =\beta =1$
• Misclassification loss: $\alpha =\beta \to \infty$

Beta function $$B(a,b)={\int }_0^1{t}^{a-1}(1-t{)}^{b-1}dt$$

• Python implementation: scipy.special.beta(a,b)
• R implementation: beta(a, b)

Incomplete Beta function $$\begin{array}{cc}{I}_x(a,b)& =\frac{1}{B(a,b)}{\int }_0^x{t}^{a-1}(1-t{)}^{b-1}dt\end{array}$$

• Python implementation: scipy.special.betainc(a, b, x)
• R implementation: pbeta(x, a, b)

We can use $\alpha \ne \beta$ when FP and FN costs are not viewed equal

Since Beta family proper scoring rule is like adding a Beta distribution on the FP cost $c$, we can use mean/variance matching to elicit $\alpha$ and $\beta$ $$\begin{array}{cc}\mu & =\frac{\alpha }{\alpha +\beta }=c\\ {\sigma }^2& =\frac{\alpha \beta }{(\alpha +\beta {)}^2(\alpha +\beta +1)}=\frac{c(1-c)}{\alpha +\beta +1}\end{array}$$

Alternatively, we can match the mode $$c=q_{\text{mode}}=\frac{\alpha -1}{\alpha +\beta -2}$$

In the simulation data with bivariate $x$, where decision boundaries of different $\eta$ are not in parallel (grey lines)

The logit link Beta family linear model with $\alpha =6,\beta =14$ estimates the $c=0.3$ classification boundary better than the logistic regression

Comparing logistic regression with a proper scoring rule tailored for high class 1 probabilities: $\alpha =9,\beta =1$.

Black lines: empirical QQ curves of 200 cost-weighted misclassification costs computed on randomly selected test sets

Buja, A., Stuetzle, W., & Shen, Y. (2005). Loss functions for binary class probability estimation and classification: Structure and applications. Working draft, November, 3. http://www-stat.wharton.upenn.edu/~buja/PAPERS/paper-proper-scoring.pdf

For a game theory definition of proper scoring rule, see https://www.cis.upenn.edu/~aaroth/courses/slides/agt17/lect23.pdf

Fitting linear models with custom loss functions in Python: https://alex.miller.im/posts/linear-model-custom-loss-function-regularization-python/

Fitting XGBoost with custom loss functions in Python: https://xgboost.readthedocs.io/en/latest/tutorials/custom_metric_obj.html