Skip to content
Advertisement

How to find similar string value in Bigquery

I have two tables in my database, each table has column with names. How to compare these tables columns and how to find these names what has exact matches and these names, which are similar in table1 and table2?

For example :

Table 1

Table 2 :

Result should come out with this:

Table 1 –> Table 2

how to effectively find similar names?

Advertisement

Answer

As I understand, there are some names in both tables that are not exactly the same and vary by some number of characters. For instance, there is a character d missing in Akhun(d)zada but present in AKHUNDZADA.

To find similarity between 2 strings, the Jaccard distance or the Levenshtein distance UDFs can be used. The jaccard() and levenshtein() functions are built-in UDFs (community provided) in BigQuery. Other UDFs available in BigQuery can be found here.

Consider the below query for your use case. jaccard() UDF has been used in this query.

Output of the query:

enter image description here

Also, please note that for the given set of inputs, a threshold of 0.8 works well. The threshold might not hold good for other inputs and has to be adjusted accordingly. The lower the threshold, the more distinct the names will become and lesser is the similarity.

User contributions licensed under: CC BY-SA
10 People found this is helpful
Advertisement