Apache Spark中的Jaro-Winkler分数计算

我们需要在Apache Spark中实现跨字符串的Jaro-Winkler距离计算数据集。我们是新来的火花，并在网络搜索后，我们无法找到很多东西。如果你能指导我们，那将是非常棒的。我们认为使用flatMap然后意识到它不会帮助，那么我们试图使用foreach循环，但不能够如何前进。因为每个字符串必须与所有字符串进行比较。就像在下面的数据集中一样。Apache Spark中的Jaro-Winkler分数计算

RowFactory.create(0, "Hi I heard about Spark"), 
RowFactory.create(1,"I wish Java could use case classes"), 
RowFactory.create(2,"Logistic,regression,models,are,neat"));

示例jaro winkler在上述数据框中找到的所有字符串中得分。标签之间

距离分值，0,1 - > 0.56
距离得分标签之间，0,2 - 标签之间> 0.77
距离分值，0,3 - 标签之间> 0.45
距离分数， 1,2 - > 0.77
距离得分标签之间，2,3 - > 0.79

import java.util.Arrays; 
    import java.util.Iterator; 
    import java.util.List; 

    import org.apache.spark.SparkConf; 
    import org.apache.spark.api.java.JavaSparkContext; 
    import org.apache.spark.api.java.function.FlatMapFunction; 
    import org.apache.spark.sql.Dataset; 
    import org.apache.spark.sql.Row; 
    import org.apache.spark.sql.RowFactory; 
    import org.apache.spark.sql.SQLContext; 
    import org.apache.spark.sql.SparkSession; 
    import org.apache.spark.sql.types.DataTypes; 
    import org.apache.spark.sql.types.Metadata; 
    import org.apache.spark.sql.types.StructField; 
    import org.apache.spark.sql.types.StructType; 

    import info.debatty.java.stringsimilarity.JaroWinkler; 

    public class JaroTestExample { 
    public static void main(String[] args) 
     { 
     System.setProperty("hadoop.home.dir", "C:\\winutil"); 
     JavaSparkContext sc = new JavaSparkContext(new SparkConf().setAppName("SparkJdbcDs").setMaster("local[*]")); 
     SQLContext sqlContext = new SQLContext(sc); 
     SparkSession spark = SparkSession.builder() 
     .appName("JavaTokenizerExample").getOrCreate(); 
     JaroWinkler jw = new JaroWinkler(); 

      // substitution of s and t 
      System.out.println(jw.similarity("My string", "My tsring")); 

      // substitution of s and n 
      System.out.println(jw.similarity("My string", "My ntrisg")); 

      List<Row> data = Arrays.asList(
     RowFactory.create(0, "Hi I heard about Spark"), 
     RowFactory.create(1,"I wish Java could use case classes"), 
     RowFactory.create(2,"Logistic,regression,models,are,neat")); 

      StructType schema = new StructType(new StructField[] { 
     new StructField("label", DataTypes.IntegerType, false, 
     Metadata.empty()), 
     new StructField("sentence", DataTypes.StringType, false, 
     Metadata.empty()) }); 

      Dataset<Row> sentenceDataFrame = spark.createDataFrame(data, schema); 

      sentenceDataFrame.foreach(); 

     } 

    }

来源

2017-01-17 Nischay

Cross在火花加入可以用下面的代码来完成Dataset2Object = Dataset1Object.crossJoin（Dataset2Object）在Dataset2Object中，您可以获得所需的所有recordpair组合。在这种情况下，flatmap不会有帮助。请记住使用版本spark-sql_2.11版本2.1.0

来源

2017-03-07 11:44:11 jeevitesh

Apache Spark中的Jaro-Winkler分数计算

回答

相关问题