如何解决PySpark旋转所有行
我有一个要旋转的数据框,并且我想旋转所有行,而不仅仅是最大,最小或第一行:
"Has a zero"
我试图将其旋转,它看起来像这样:
df = spark.createDataFrame(
[
('sequence','@short_read_5/1','2aa897b7-38cd-4af...','2020-10-08-12'),('desc','+',('raw_sequence','GTTACTTCGATATCCGC...',('quality','BCCC<EGAGGGGGGGGG...',('sequence','@short_read_5/2','dTTACTTCGATATCCGC...','dCCC<EGAGGGGGGGGG...',],['col','data','metadata_key','datehour']
)
+------------+--------------------+--------------------+-------------+
| col| data| metadata_key| datehour|
+------------+--------------------+--------------------+-------------+
| sequence| @short_read_5/1|2aa897b7-38cd-4af...|2020-10-08-12|
| desc| +|2aa897b7-38cd-4af...|2020-10-08-12|
|raw_sequence|GTTACTTCGATATCCGC...|2aa897b7-38cd-4af...|2020-10-08-12|
| quality|BCCC<EGAGGGGGGGGG...|2aa897b7-38cd-4af...|2020-10-08-12|
| sequence| @short_read_5/2|2aa897b7-38cd-4af...|2020-10-08-12|
| desc| +|2aa897b7-38cd-4af...|2020-10-08-12|
|raw_sequence|dTTACTTCGATATCCGC...|2aa897b7-38cd-4af...|2020-10-08-12|
| quality|dCCC<EGAGGGGGGGGG...|2aa897b7-38cd-4af...|2020-10-08-12|
+------------+--------------------+--------------------+-------------+
但是我希望它具有所有数据,而不仅仅是第一个数据:
df.groupBy('metadata_key','datehour').pivot("col",["sequence","desc","raw_sequence","quality"]).agg(F.first(F.col("data"))).show()
+--------------------+-------------+---------------+----+--------------------+--------------------+
| metadata_key| datehour| sequence|desc| raw_sequence| quality|
+--------------------+-------------+---------------+----+--------------------+--------------------+
|2aa897b7-38cd-4af...|2020-10-08-12|@short_read_5/1| +|GTTACTTCGATATCCGC...|BCCC<EGAGGGGGGGGG...|
+--------------------+-------------+---------------+----+--------------------+--------------------+
我该怎么做?
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。