Defaults and keyword argument format now available

[ParticularMiner]

@iibarant @berndnoll

Defaults and keyword argument format now available for Red String Grouper. Weights, for instance, have a default value of 1.0 as requested by @berndnoll. Here's how a sample call would now look like:

from red_string_grouper import record_linkage, field

matches = record_linkage(
    df,
    fields_2b_matched_fuzzily=[
        field('statusText'),
        field('address'),
        field('addressZipcode', weight=2, min_similarity=0.9999)
    ],
    fields_2b_matched_exactly=[
        field('addressState', weight=4),
        field('hasVideo')
    ]
)

I'd appreciate it if you or anyone else would take a look at these new changes and test that they are working as prescribed.
Thanks.

[iibarant]

Hi @ParticularMiner, that is a nice update. However, I copied the latest version and it shows an error at the very last line. Is it needed as you already have a return statement?

I also noticed that similarity_dtype=np.float32 is missing to call the record_linkage. Is it not going to be used?
Another question is regarding ngrams - is it static 3 going forward? Can it be a part of the call? The good exercise would be to explore effect ngrams value (3-4-5) on the performance.

Thank you very much for your hard work.

[ParticularMiner]

Thanks @iibarant

That's a careless mistake. I'll update the code shortly. In the meantime, yes, you may remove it.
Cheers

[iibarant]

Thanks @iibarant

That's a careless mistake. I'll update the code shortly. In the meantime, yes, you may remove it.
Cheers

Please see my addition in the previous comment.

[ParticularMiner]

@iibarant

Package has now been updated to version 0.0.8 with the correction.

To answer your other questions — all other options are still there. You only need to use the string_grouper names for these options. (follow this link). For instance,

similarity_dtype is now tfidf_matrix_dtype.
ngram_size remains the same.

You can specify any option within the auxilliary function field(), or directly in record_linkage() in which case the options apply to all fields. However, options already specified for a field take precedence over those directly specified in record_linkage(). For example,

matches = record_linkage(
    df,
    fields_2b_matched_fuzzily=[
        field('statusText'),
        field('address'),
        field('addressZipcode', weight=2, min_similarity=0.9999),
        field('addressState', weight=4, ngram_size=2, min_similarity=0.9999),
        field('hasVideo', min_similarity=0.9999)
    ],
    n_blocks=(1,1),
    tfidf_matrix_dtype=np.float32
)

I hope this is clear.

[ParticularMiner]

@iibarant

I would be interested to know your results of explorations into varying ngram_size

[iibarant]

@ParticularMiner,
For all the same parameters except ngrams, for a data set of 131056 records, there is following runtime breakdown

ngrams runtime (sec)
3 - 131.266
4 - 116.601
5 - 110.756

[ParticularMiner]

@iibarant
Good. I suspected that would be the case, since the matrices become sparser and easier to compute.
If you like, try increasing the n_blocks parameter to say n_blocks=(1, 50).

[iibarant]

Hi @ParticularMiner,

Here we go on a dataframe of 33788 records:

n = 3 n_blocks = 1 - Runtime (sec): 7.31
n = 3 n_blocks = 2 - Runtime (sec): 7.15
n = 3 n_blocks = 3 - Runtime (sec): 7.4
n = 3 n_blocks = 4 - Runtime (sec): 8.79
n = 3 n_blocks = 5 - Runtime (sec): 8.22
n = 3 n_blocks = 6 - Runtime (sec): 8.37
n = 3 n_blocks = 7 - Runtime (sec): 8.23
n = 3 n_blocks = 8 - Runtime (sec): 8.4
n = 3 n_blocks = 9 - Runtime (sec): 8.33
n = 3 n_blocks = 10 - Runtime (sec): 8.32
n = 3 n_blocks = 11 - Runtime (sec): 8.37
n = 3 n_blocks = 12 - Runtime (sec): 8.35
n = 3 n_blocks = 13 - Runtime (sec): 8.4
n = 3 n_blocks = 14 - Runtime (sec): 8.38
n = 3 n_blocks = 15 - Runtime (sec): 8.95
n = 3 n_blocks = 16 - Runtime (sec): 9.03
n = 3 n_blocks = 17 - Runtime (sec): 9.12
n = 3 n_blocks = 18 - Runtime (sec): 9.14
n = 3 n_blocks = 19 - Runtime (sec): 9.29
n = 3 n_blocks = 20 - Runtime (sec): 9.31
n = 3 n_blocks = 21 - Runtime (sec): 9.27
n = 3 n_blocks = 22 - Runtime (sec): 9.45
n = 3 n_blocks = 23 - Runtime (sec): 9.5
n = 3 n_blocks = 24 - Runtime (sec): 9.54
n = 3 n_blocks = 25 - Runtime (sec): 9.72
n = 3 n_blocks = 26 - Runtime (sec): 9.64
n = 3 n_blocks = 27 - Runtime (sec): 9.79
n = 3 n_blocks = 28 - Runtime (sec): 9.78
n = 3 n_blocks = 29 - Runtime (sec): 8.91
n = 3 n_blocks = 30 - Runtime (sec): 9.07
n = 3 n_blocks = 31 - Runtime (sec): 9.04
n = 3 n_blocks = 32 - Runtime (sec): 9.0
n = 3 n_blocks = 33 - Runtime (sec): 8.75
n = 3 n_blocks = 34 - Runtime (sec): 8.83
n = 3 n_blocks = 35 - Runtime (sec): 8.84
n = 3 n_blocks = 36 - Runtime (sec): 9.01
n = 3 n_blocks = 37 - Runtime (sec): 8.97
n = 3 n_blocks = 38 - Runtime (sec): 8.88
n = 3 n_blocks = 39 - Runtime (sec): 8.89
n = 3 n_blocks = 40 - Runtime (sec): 8.95
n = 3 n_blocks = 41 - Runtime (sec): 9.05
n = 3 n_blocks = 42 - Runtime (sec): 9.03
n = 3 n_blocks = 43 - Runtime (sec): 9.08
n = 3 n_blocks = 44 - Runtime (sec): 9.14
n = 3 n_blocks = 45 - Runtime (sec): 9.07
n = 3 n_blocks = 46 - Runtime (sec): 9.06
n = 3 n_blocks = 47 - Runtime (sec): 9.08
n = 3 n_blocks = 48 - Runtime (sec): 9.06
n = 3 n_blocks = 49 - Runtime (sec): 9.13
n = 3 n_blocks = 50 - Runtime (sec): 9.23
n = 4 n_blocks = 1 - Runtime (sec): 7.19
n = 4 n_blocks = 2 - Runtime (sec): 7.1
n = 4 n_blocks = 3 - Runtime (sec): 7.13
n = 4 n_blocks = 4 - Runtime (sec): 7.15
n = 4 n_blocks = 5 - Runtime (sec): 7.39
n = 4 n_blocks = 6 - Runtime (sec): 7.16
n = 4 n_blocks = 7 - Runtime (sec): 7.19
n = 4 n_blocks = 8 - Runtime (sec): 7.28
n = 4 n_blocks = 9 - Runtime (sec): 7.21
n = 4 n_blocks = 10 - Runtime (sec): 7.3
n = 4 n_blocks = 11 - Runtime (sec): 7.4
n = 4 n_blocks = 12 - Runtime (sec): 7.26
n = 4 n_blocks = 13 - Runtime (sec): 7.25
n = 4 n_blocks = 14 - Runtime (sec): 7.28
n = 4 n_blocks = 15 - Runtime (sec): 7.85
n = 4 n_blocks = 16 - Runtime (sec): 7.91
n = 4 n_blocks = 17 - Runtime (sec): 8.01
n = 4 n_blocks = 18 - Runtime (sec): 8.12
n = 4 n_blocks = 19 - Runtime (sec): 8.15
n = 4 n_blocks = 20 - Runtime (sec): 8.19
n = 4 n_blocks = 21 - Runtime (sec): 8.36
n = 4 n_blocks = 22 - Runtime (sec): 8.65
n = 4 n_blocks = 23 - Runtime (sec): 8.49
n = 4 n_blocks = 24 - Runtime (sec): 8.52
n = 4 n_blocks = 25 - Runtime (sec): 8.58
n = 4 n_blocks = 26 - Runtime (sec): 8.7
n = 4 n_blocks = 27 - Runtime (sec): 8.74
n = 4 n_blocks = 28 - Runtime (sec): 8.7
n = 4 n_blocks = 29 - Runtime (sec): 8.2
n = 4 n_blocks = 30 - Runtime (sec): 7.89
n = 4 n_blocks = 31 - Runtime (sec): 7.95
n = 4 n_blocks = 32 - Runtime (sec): 7.93
n = 4 n_blocks = 33 - Runtime (sec): 7.64
n = 4 n_blocks = 34 - Runtime (sec): 7.72
n = 4 n_blocks = 35 - Runtime (sec): 7.71
n = 4 n_blocks = 36 - Runtime (sec): 7.72
n = 4 n_blocks = 37 - Runtime (sec): 7.71
n = 4 n_blocks = 38 - Runtime (sec): 7.75
n = 4 n_blocks = 39 - Runtime (sec): 7.86
n = 4 n_blocks = 40 - Runtime (sec): 7.95
n = 4 n_blocks = 41 - Runtime (sec): 7.95
n = 4 n_blocks = 42 - Runtime (sec): 7.91
n = 4 n_blocks = 43 - Runtime (sec): 8.06
n = 4 n_blocks = 44 - Runtime (sec): 8.0
n = 4 n_blocks = 45 - Runtime (sec): 8.09
n = 4 n_blocks = 46 - Runtime (sec): 8.01
n = 4 n_blocks = 47 - Runtime (sec): 8.15
n = 4 n_blocks = 48 - Runtime (sec): 8.16
n = 4 n_blocks = 49 - Runtime (sec): 8.56
n = 4 n_blocks = 50 - Runtime (sec): 8.18
n = 5 n_blocks = 1 - Runtime (sec): 7.01
n = 5 n_blocks = 2 - Runtime (sec): 6.91
n = 5 n_blocks = 3 - Runtime (sec): 6.89
n = 5 n_blocks = 4 - Runtime (sec): 7.02
n = 5 n_blocks = 5 - Runtime (sec): 6.9
n = 5 n_blocks = 6 - Runtime (sec): 6.98
n = 5 n_blocks = 7 - Runtime (sec): 7.0
n = 5 n_blocks = 8 - Runtime (sec): 7.08
n = 5 n_blocks = 9 - Runtime (sec): 7.04
n = 5 n_blocks = 10 - Runtime (sec): 6.98
n = 5 n_blocks = 11 - Runtime (sec): 7.01
n = 5 n_blocks = 12 - Runtime (sec): 7.14
n = 5 n_blocks = 13 - Runtime (sec): 7.11
n = 5 n_blocks = 14 - Runtime (sec): 7.35
n = 5 n_blocks = 15 - Runtime (sec): 7.91
n = 5 n_blocks = 16 - Runtime (sec): 7.77
n = 5 n_blocks = 17 - Runtime (sec): 8.13
n = 5 n_blocks = 18 - Runtime (sec): 7.99
n = 5 n_blocks = 19 - Runtime (sec): 8.21
n = 5 n_blocks = 20 - Runtime (sec): 8.34
n = 5 n_blocks = 21 - Runtime (sec): 8.05
n = 5 n_blocks = 22 - Runtime (sec): 8.29
n = 5 n_blocks = 23 - Runtime (sec): 8.49
n = 5 n_blocks = 24 - Runtime (sec): 8.45
n = 5 n_blocks = 25 - Runtime (sec): 8.39
n = 5 n_blocks = 26 - Runtime (sec): 8.43
n = 5 n_blocks = 27 - Runtime (sec): 8.43
n = 5 n_blocks = 28 - Runtime (sec): 8.71
n = 5 n_blocks = 29 - Runtime (sec): 7.86
n = 5 n_blocks = 30 - Runtime (sec): 7.88
n = 5 n_blocks = 31 - Runtime (sec): 7.9
n = 5 n_blocks = 32 - Runtime (sec): 8.04
n = 5 n_blocks = 33 - Runtime (sec): 7.72
n = 5 n_blocks = 34 - Runtime (sec): 7.57
n = 5 n_blocks = 35 - Runtime (sec): 7.67
n = 5 n_blocks = 36 - Runtime (sec): 7.71
n = 5 n_blocks = 37 - Runtime (sec): 8.0
n = 5 n_blocks = 38 - Runtime (sec): 8.07
n = 5 n_blocks = 39 - Runtime (sec): 7.72
n = 5 n_blocks = 40 - Runtime (sec): 8.03
n = 5 n_blocks = 41 - Runtime (sec): 8.23
n = 5 n_blocks = 42 - Runtime (sec): 8.16
n = 5 n_blocks = 43 - Runtime (sec): 8.43
n = 5 n_blocks = 44 - Runtime (sec): 7.67
n = 5 n_blocks = 45 - Runtime (sec): 7.69
n = 5 n_blocks = 46 - Runtime (sec): 8.1
n = 5 n_blocks = 47 - Runtime (sec): 8.14
n = 5 n_blocks = 48 - Runtime (sec): 7.97
n = 5 n_blocks = 49 - Runtime (sec): 8.15
n = 5 n_blocks = 50 - Runtime (sec): 8.18

[ParticularMiner]

Thanks @iibarant

Question: you are changing n_blocks[1] and not n_blocks[0], right?

If that's the case, then the increase in runtime is likely due to the fact that your DataFrame is too small.
Hopefully, a decrease in runtime with increase in n_block[1] will occur with a larger DataFrame with number of rows of the order of 300 000.

[iibarant]

Hi @ParticularMiner,

The original string_grouper package allows to match a field from one dataframe to another field from another dataframe.
I'm sure this is possible with your update. What would be the right code to match, let's say 2 fields from one df to 2 fields from another dataframe with fuzzily matching with given weights and one field with exact matching from both dataframes.

According to your description such case is not considered:
'''
Function that combines similarity-matching results of several fields of a
DataFrame and returns them in another DataFrame
:param data_frame: pandas.DataFrame of strings.
'''

That would be a great addition to the package.

Thank you very much!

[ParticularMiner]

@iibarant

Thanks for your message.

It was already on my TODO list. I hope to get round to doing it soon. In fact, as soon as the new string_grouper update is merged I intend to do that.

[ParticularMiner]

@iibarant

The latest update to Red String Grouper needs to be tested. But see if the following makes sense:

(Note that all previous functionality (with one DataFrame) has been retained. This attempts comparing 2 DataFrames.)

import pandas as pd
import numpy as np
from red_string_grouper import record_linkage, field, field_pair

inputfilename = 'data/us-cities-real-estate-sample-zenrows.csv'
df = pd.read_csv(inputfilename, dtype=str)

df.set_index('zpid', inplace=True)
df1 = df[['imgSrc', 'addressState', 'statusText']]
df2 = df[['detailUrl', 'addressState', 'statusText']]

matches = record_linkage(
    [df1, df2],
    fields_2b_matched_fuzzily=[field_pair('statusText', 'statusText', weight=3),
                               field_pair('imgSrc', 'detailUrl', weight=4, regex=r'https://|.zillow.com|[,-./:]|\s', min_similarity=0.05)],
    fields_2b_matched_exactly=[field_pair('addressState', 'addressState', weight=4)],
    n_blocks=(1,1))

matches.sort_values(('Fuzzily Matched Fields', 'imgSrc/detailUrl', 'similarity'), ascending=False)

			Exactly Matched Fields	Fuzzily Matched Fields
			addressState/addressState	statusText/statusText			imgSrc/detailUrl
		Weighted Mean Similarity Score		left	similarity	right	left	similarity	right
left_zpid	right_zpid
96757735	9469011	0.680908	PA	Townhouse for sale	1.0	Townhouse for sale	https://photos.zillowstatic.com/fp/e86b55f8ff4...	0.122498	https://www.zillow.com/homedetails/1115-W-9th-...
2075301422	2075905715	0.676702	OK	Lot / Land for sale	1.0	Lot / Land for sale	https://photos.zillowstatic.com/fp/33759096aea...	0.110931	https://www.zillow.com/homedetails/Moccasin-Tr...
2100136081	2070606357	0.674255	TX	Lot / Land for sale	1.0	Lot / Land for sale	https://photos.zillowstatic.com/fp/81739685ade...	0.104201	https://www.zillow.com/homedetails/Fm-344-Rd-W...
33628288	2076543613	0.673813	OH	Multi-family home for sale	1.0	Multi-family home for sale	https://photos.zillowstatic.com/fp/17d6ff62ed2...	0.102986	https://www.zillow.com/homedetails/105-1-2-N-P...
84501900	74747435	0.672341	MI	House for sale	1.0	House for sale	https://photos.zillowstatic.com/fp/d64d0568f30...	0.098939	https://www.zillow.com/homedetails/Townsend-Rd...
...	...	...	...	...	...	...	...	...	...
2077977874	86740833	0.654574	NH	Active	1.0	Active	https://photos.zillowstatic.com/fp/b1404f9e886...	0.050079	https://www.zillow.com/homedetails/59-Maple-Ln...
92848559	86797118	0.654568	NH	Active	1.0	Active	https://photos.zillowstatic.com/fp/8246797fd69...	0.050061	https://www.zillow.com/homedetails/39-John-St-...
2077102415	95343578	0.654566	NH	Active	1.0	Active	https://photos.zillowstatic.com/fp/9c8b1983578...	0.050057	https://www.zillow.com/homedetails/155-Barrett...
86793455	2070175193	0.654554	NH	Active	1.0	Active	https://photos.zillowstatic.com/fp/155c840ddee...	0.050023	https://www.zillow.com/homedetails/208-26-Hamm...
32607416	32816743	0.654553	NY	House for sale	1.0	House for sale	https://photos.zillowstatic.com/fp/9d07d42d4dd...	0.050021	https://www.zillow.com/homedetails/70-Main-St-...

638 rows × 8 columns

[ParticularMiner]

@iibarant

Because I lack meaningful data. I would appreciate it very much if you could form a meaningful example using two DataFrames that I could include in the README.md file (that is, the documentation). Alternatively you could edit the README.md file yourself and post a pull request.

[iibarant]

Hi @ParticularMiner, Today is Thanksgiving day in Canada and I’m far from my laptop. Will do my best tomorrow. Thank you!

…

On Oct 11, 2021, at 8:58 AM, ParticularMiner ***@***.***> wrote:  @iibarant Because I lack meaningful data. I would appreciate it very much if you could form a meaningful example using two DataFrames that I could include in the README.md file (that is, the documentation). Alternatively you could edit the README.md file yourself and post a pull request. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[ParticularMiner]

@iibarant

No problem! Enjoy Thanksgiving!