Computing the Fellegi Sunter model

The aim of the model is to compute a prediction of which records match. This prediction is a probability that quantifies the likelihood the two records represent the same entity.

Computing the prediction step by step

We begin with some data. In this example we will attempt to link but not deduplicate records from these two tables.

Input table 1

Input table 2:

In the first step of the calculation we compare each record in the first table with all records in the second table.¹

We wish to predict which of these pairwise record comparisons represent the same entity (i.e. which records match).

Table scrolls right →

At first glance, it may seem challenging to turn this table of text into a numeric prediction. But this can be achieved using the concepts of comparisonsIn Splink, a model has a set of comparisons, each typically corresponding to a column in the input data such as name, date of birth etc. Each comparison has multiple comparison levels. A comparison level defines a category of similarity for that column e.g. an 'exact match' or a 'one character difference'. Each comparison level has an associated partial match weight. and partial_match_weightA partial match weight quantifies the importance of a comparison level in the Fellegi-Sunter model. For instance, a Jaro Winkler fuzzy match on first name may have a positive match weight, indicating it adds evidence in favour of a match.s introduced earlier in this tutorial.

Specifically, we define scenarios for each column and then use a lookup table to find the partial match weights.

For example, we may choose to define our scenarios as follows:

Note the ‘comparison vector value’ column in this table. This is an integer value that identifies the comparison level within the comparison. We will use it to help us look up the correct partial match weights.

We then proceed by computing which comparison levelIn Splink, a model has a set of comparisons, each typically corresponding to a column in the input data such as name, date of birth etc. Each comparison has multiple comparison levels. A comparison level defines a category of similarity for that column e.g. an 'exact match' or a 'one character difference'. Each comparison level has an associated partial match weight. is activateA comparison level (scenario) is activated when its conditions are met for the pairwise record comparison at hand. For instance, the exact match comparison level would be active for first name if the first names matched exactly. Only one comparison level within each comparison can be active at any one timed for each pairwise record comparisonIn Splink, a model has a set of comparisons, each typically corresponding to a column in the input data such as name, date of birth etc. Each comparison has multiple comparison levels. A comparison level defines a category of similarity for that column e.g. an 'exact match' or a 'one character difference'. Each comparison level has an associated partial match weight., and representing this using the comparison vector value. We use the gamma symbol (γ) to refer to the comparison vector value.

Table scrolls right →

Now that we’ve used the text values to compute the comparison vector values, we no longer need the original text values. What remains are known as the comparison vectors for each pairwise comparison. To help you see what’s going on, I have annotated the comparison vector values with the corresponding partial match weights in brackets, which we will use in the next step.

Each row in this table is known as the ‘comparison vector’ for the record comparison

We now look up the partial match weights from the comparison vector. I’ve also added the priorThe prior is the probability that two records picked at random from the input data refer to the same entity., expressed as a partial match weight, and the sum of the match weights:

Table scrolls right →

Finally, we can then sum the values in the partial match weights to produce a final match weight. This can then be converted into a probability using:

$\Large{p = \frac{2^{\omega}}{1 + 2^{\omega}}}$

We can now see how the original comparison pairs were scored by this simple model:

Table scrolls right →