转换阶FP增长RDD输出到数据帧

https://spark.apache.org/docs/2.1.0/mllib-frequent-pattern-mining.html#fp-growth 转换阶FP增长RDD输出到数据帧

sample_fpgrowth.txt可以找到这里， https://github.com/apache/spark/blob/master/data/mllib/sample_fpgrowth.txt

我跑上述阶其工作正常链接中的FP-生长的例子，但我需要的是，如何将RDD中的结果转换为数据帧。这两种RDD

model.freqItemsets and 
model.generateAssociationRules(minConfidence)

详细解释一下，在我的问题给出的例子。

来源

2017-05-30 Arun Gunalan

的可能的复制[如何RDD对象转换为数据框火花（https://stackoverflow.com/questions/29383578/how-to-convert-rdd -object-to-dataframe-in-spark） – stefanobaghino

我试过我有错误，可能是因为我是scala新手。你能否详细解释我的问题给出的例子。 –

@ zero323你能帮助我通过我的问题 –

一旦您拥有rdd，有许多方法可以创建dataframe。其中之一是使用.toDF功能需要sqlContext.implicits库是imported为

val sparkSession = SparkSession.builder().appName("udf testings") 
    .master("local") 
    .config("", "") 
    .getOrCreate() 
val sc = sparkSession.sparkContext 
val sqlContext = sparkSession.sqlContext 
import sqlContext.implicits._

后您阅读fpgrowth文本文件和隐蔽到rdd

val data = sc.textFile("path to sample_fpgrowth.txt that you have used") 
    val transactions: RDD[Array[String]] = data.map(s => s.trim.split(' '))

我从Frequent Pattern Mining - RDD-based API使用的代码即问题中提供的内容

val fpg = new FPGrowth() 
    .setMinSupport(0.2) 
    .setNumPartitions(10) 
val model = fpg.run(transactions)

下一步将调用.toDF功能

对于第一dataframe

model.freqItemsets.map(itemset =>(itemset.items.mkString("[", ",", "]") , itemset.freq)).toDF("items", "freq").show(false)

这将导致到

+---------+----+ 
|items |freq| 
+---------+----+ 
|[z]  |5 | 
|[x]  |4 | 
|[x,z] |3 | 
|[y]  |3 | 
|[y,x] |3 | 
|[y,x,z] |3 | 
|[y,z] |3 | 
|[r]  |3 | 
|[r,x] |2 | 
|[r,z] |2 | 
|[s]  |3 | 
|[s,y] |2 | 
|[s,y,x] |2 | 
|[s,y,x,z]|2 | 
|[s,y,z] |2 | 
|[s,x] |3 | 
|[s,x,z] |2 | 
|[s,z] |2 | 
|[t]  |3 | 
|[t,y] |3 | 
+---------+----+ 
only showing top 20 rows

用于第二dataframe

val minConfidence = 0.8 
model.generateAssociationRules(minConfidence) 
    .map(rule =>(rule.antecedent.mkString("[", ",", "]"), rule.consequent.mkString("[", ",", "]"), rule.confidence)) 
    .toDF("antecedent", "consequent", "confidence").show(false)

，这将导致对

+----------+----------+----------+ 
|antecedent|consequent|confidence| 
+----------+----------+----------+ 
|[t,s,y] |[x]  |1.0  | 
|[t,s,y] |[z]  |1.0  | 
|[y,x,z] |[t]  |1.0  | 
|[y]  |[x]  |1.0  | 
|[y]  |[z]  |1.0  | 
|[y]  |[t]  |1.0  | 
|[p]  |[r]  |1.0  | 
|[p]  |[z]  |1.0  | 
|[q,t,z] |[y]  |1.0  | 
|[q,t,z] |[x]  |1.0  | 
|[q,y]  |[x]  |1.0  | 
|[q,y]  |[z]  |1.0  | 
|[q,y]  |[t]  |1.0  | 
|[t,s,x] |[y]  |1.0  | 
|[t,s,x] |[z]  |1.0  | 
|[q,t,y,z] |[x]  |1.0  | 
|[q,t,x,z] |[y]  |1.0  | 
|[q,x]  |[y]  |1.0  | 
|[q,x]  |[t]  |1.0  | 
|[q,x]  |[z]  |1.0  | 
+----------+----------+----------+ 
only showing top 20 rows

我希望这是你需要

来源

2017-06-01 12:21:32

回答这是我期待谢谢吨。 –

我的荣幸@ArunGunalan :)很高兴答案帮助你 –

转换阶FP增长RDD输出到数据帧

回答

相关问题