如何解决Java代码中Spark上的聚合函数
我有一个特殊的情况要用Java代码处理spark,但我不知道这样做: 我想要具有分区PTF的最后一个FALSE标签(此处的记录在desc中按列的datendatedate排序)以及标签为TRUE之后的所有记录。 在SQL中,它看起来像:
with full_query as (
select distinct t.portfolio,t.commandUId,max(t.isPartialRelaunch) over (partition by t.commandName,t.portfolio) as isPartialRelaunchPerPTF,(DENSE_RANK() over (partition by t.portfolio,t.commandUId) order by t.i2rIntegrationTime desc) as commandUIDRank
from varFumModel t)
)
select t.*,t.commandUId) order by t.i2rIntegrationTime desc) as commandUIDRank from varFumModel t
where commandUIDRank <= nvl((select min(d.commandUIDRank) from full_query d where d.isPartialRelaunchPerPTF = false and d.portfolio = t.portfolio and d.commandUId = t.commandUId),10000)
- Cas 1
+------+------------+-------+------+---------------+-----------------------+
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position2 | TRUE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+-----------------------+
+------+------------+-------+------+---------------+-----------------------+
|PTF1 | Position1 | FALSE | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
+------+------------+-------+------+---------------+-----------------------+
结果
+------+------------+-------+------+---------------+-----------------------+
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position2 | TRUE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position1 | FALSE | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
+------+------------+-------+------+---------------+-----------------------+
- Cas 2
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | FALSE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position2 | FALSE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+------------------------+
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | FALSE | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
+------+------------+-------+------+---------------+------------------------+
结果
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | FALSE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position2 | FALSE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+------------------------+
- Cas 3
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | FALSE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position2 | FALSE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+------------------------+
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
+------+------------+-------+------+---------------+------------------------+
结果
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | FALSE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position2 | FALSE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+------------------------+
- Cas 4
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position2 | TRUE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+------------------------+
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
+------+------------+-------+------+---------------+------------------------+
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_0 | integrationdatetime_t0|
+------+------------+-------+------+---------------+------------------------+
结果
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position2 | TRUE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_0 | integrationdatetime_t0|
+------+------------+-------+------+---------------+------------------------+
- Cas 5
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position2 | TRUE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
+------+------------+-------+------+---------------+------------------------+
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
+------+------------+-------+------+---------------+------------------------+
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | FALSE | CMD1 | COMMAND_UID_0 | integrationdatetime_t0|
+------+------------+-------+------+---------------+------------------------+
+------+------------+-------+------+---------------+-------------------------+
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_-1| integrationdatetime_t-1|
+------+------------+-------+------+---------------+-------------------------+
结果
+------+------------+-------+------+---------------+------------------------+
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position2 | TRUE | CMD1 | COMMAND_UID_2 | integrationdatetime_t2|
|PTF1 | Position1 | TRUE | CMD1 | COMMAND_UID_1 | integrationdatetime_t1|
|PTF1 | Position1 | FALSE | CMD1 | COMMAND_UID_0 | integrationdatetime_t0|
+------+------------+-------+------+---------------+------------------------+
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。