Visualising the Fellegi Sunter model

The previous article presented an implementation of the Fellegi Sunter model. We showed that match probability could be represented by Equation 1 - which is reproduced below:

\operatorname{Pr}(\text{records match}|\gamma_{1},\gamma_{2},\ldots,\gamma_{K}) = \frac{\lambda m_{1\ell}m_{2\ell}\ldots m_{K\ell}}{\lambda m_{1\ell}m_{2\ell}\ldots m_{K\ell}+ (1-\lambda)u_{1\ell}u_{2\ell} \ldots u_{K\ell}}

This article presents a way to understand and visualise this formula.

In our visualisation, we are going to take each piece of evidence into account in turn. This is possible due to our assumption of conditional independence of comparison columns given the match status.

The means Equation 1 is equivalent to a repeated application of Bayes’ Theorem (see annex). See here is more in depth video about Bayes Theorem.

Example

We are going to compute match probability for a record comparison with the following comparison columns:

fname (first name)
sname (surname)
dob (date of birth)
town

The default parameter values are a bit unrealistic, but help make sure the diagrams are legible.

Step 1

We begin by visualising the space of all pairwise record comparisons, splitting the overall comparison space into matches and non-matches using our prior, $\operatorname{Pr}(\text{records match})=\lambda$ .

Step 2 - First name

We will now choose the parameters of the model for the first name field, and then take this information into account in the diagram.

Step 3 - Surname

Let’s also choose parameters for sname, the surname field:

Step 4 - Date of birth

Step 5: Town

The final column is town.

Annex: Mathematical representation

This annex shows why it’s possible to represent the calculation of match probability as a step-by-step computation.

In particular, we will show that we can calculate match probability using a repeated application of Bayes Theorem. In pseudocode, our algorithm will be:

prior = lambda
for col in compaison_columns:
  posterior = bayes(col, prior)
  prior = posterior

Where the final value of the posterior is equal to $\operatorname{Pr}(\text{records match}|\gamma_{1},\gamma_{2},\ldots,\gamma_{K})$ .

To demonstrate why this works, consider equation 1, for the case of two columns:

\operatorname{Pr}(\text{records match}|\gamma_{1},\gamma_{2}) = \frac{\lambda m_{1\ell}m_{2\ell}}{\lambda m_{1\ell}m_{2\ell}+ (1-\lambda)u_{1\ell}u_{2\ell}} \tag{2.1}

We start by applying Bayes Theorem once, accounting for the first comparison column:

\operatorname{Pr}(\text{records match}|\gamma_{1}) = t = \frac{\lambda m_{1\ell}}{\lambda m_{1\ell}+ (1-\lambda)u_{1\ell}}

This posterior, $t$ , becomes the new prior, which we now use instead of lambda:

\text{posterior} = \operatorname{Pr}(\text{records match}|\gamma_{1},\gamma_{2}) = \frac{t m_{2\ell}}{t m_{2\ell}+ (1-t)u_{2\ell}}\tag{2.2}

Note that:

1-t = 1-\frac{\lambda m_{1\ell}}{\lambda m_{1\ell}+ (1-\lambda)u_{1\ell}} = \frac{\lambda m_{1\ell}+ (1-\lambda)u_{1\ell}-\lambda m_{1\ell}}{\lambda m_{1\ell}+ (1-\lambda)u_{1\ell}} = \frac{(1-\lambda)u_{1\ell}}{\lambda m_{1\ell}+ (1-\lambda)u_{1\ell}}\tag{2.3}

But $(2.3)$ shows that the term $(\lambda m_{1\ell}+ (1-\lambda)u_{1\ell})$ appears on both top and bottom of $(2.2)$ and therefore cancels, leaving $(2.1)$ as desired.

This article is part of the probabilistic linkage training materials.