如何解决Pyspark：to_date 和 unix_timestamp 为其他有效值的某些记录返回 null
上面的命令为少数记录提供了空值。 任何人都可以告诉这可能是什么原因。我曾尝试使用 to_date 和 unix_timestamp 函数，但两者都给出相同的结果（少数值为 null）。
date_cols.withColumn('date_timestamp',unix_timestamp('Date',"M/dd/y HH:mm:ss a")).show(truncate=False)
|----------------------|----------------------|--------------|
|Date |Updated On |date_timestamp|
|----------------------|----------------------|--------------|
|02/05/2018 01:10:00 AM|02/12/2018 03:49:14 PM|1517793000 |
|01/23/2018 09:00:00 AM|02/12/2018 03:49:14 PM|1516698000 |
|01/31/2018 10:12:00 AM|02/12/2018 03:49:14 PM|1517393520 |
|02/01/2018 02:00:00 PM|02/12/2018 03:49:14 PM|null |
|01/27/2018 09:00:00 PM|02/12/2018 03:49:14 PM|null |
|02/02/2018 12:00:00 PM|02/12/2018 03:49:14 PM|1517572800 |
|01/28/2018 07:00:00 PM|02/12/2018 03:49:14 PM|null |
|01/10/2018 04:45:00 PM|02/12/2018 03:49:14 PM|null |
|01/03/2018 03:45:00 PM|02/12/2018 03:49:14 PM|null |
|02/04/2018 04:00:00 PM|02/12/2018 03:49:14 PM|null |
|01/27/2018 07:00:00 AM|02/12/2018 03:49:14 PM|1517036400 |
|02/05/2018 12:00:00 AM|02/12/2018 03:49:14 PM|null |
|01/21/2018 02:00:00 AM|02/12/2018 03:49:14 PM|1516500000 |
|02/02/2018 12:45:00 AM|02/12/2018 03:49:14 PM|null |
|01/25/2018 01:14:00 PM|02/12/2018 03:49:14 PM|null |
|02/05/2018 05:00:00 PM|02/12/2018 03:49:14 PM|null |
|01/15/2018 12:00:00 AM|02/12/2018 03:49:14 PM|null |
|02/05/2018 07:30:00 PM|02/12/2018 03:49:14 PM|null |
|02/02/2018 08:30:00 PM|02/12/2018 03:49:14 PM|null |
|02/03/2018 06:00:00 PM|02/12/2018 03:49:14 PM|null |
+----------------------+----------------------+--------------+
解决方法
实际上，我犯了一个小错误，在查看了 [doc][1]
它得到了解决，所以在这里发布，它可能会帮助某人。 [1]：https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
在格式化时必须使用 hh 而不是 HH。由于hh 代表clock-hour-of-am-pm (1-12) 而HH 代表一天中的小时（ 0-23)
date_cols.withColumn('date_timestamp',unix_timestamp('Date',"MM/dd/y hh:mm:ss a").cast(TimestampType())).show(truncate=False)
|----------------------|----------------------|-------------------|
|Date |Updated On |date_timestamp |
|----------------------|----------------------|-------------------|
|02/05/2018 01:10:00 AM|02/12/2018 03:49:14 PM|2018-02-05 01:10:00|
|01/23/2018 09:00:00 AM|02/12/2018 03:49:14 PM|2018-01-23 09:00:00|
|01/31/2018 10:12:00 AM|02/12/2018 03:49:14 PM|2018-01-31 10:12:00|
|02/01/2018 02:00:00 PM|02/12/2018 03:49:14 PM|2018-02-01 14:00:00|
|01/27/2018 09:00:00 PM|02/12/2018 03:49:14 PM|2018-01-27 21:00:00|
|02/02/2018 12:00:00 PM|02/12/2018 03:49:14 PM|2018-02-02 12:00:00|
|01/28/2018 07:00:00 PM|02/12/2018 03:49:14 PM|2018-01-28 19:00:00|
|01/10/2018 04:45:00 PM|02/12/2018 03:49:14 PM|2018-01-10 16:45:00|
|01/03/2018 03:45:00 PM|02/12/2018 03:49:14 PM|2018-01-03 15:45:00|
|02/04/2018 04:00:00 PM|02/12/2018 03:49:14 PM|2018-02-04 16:00:00|
|01/27/2018 07:00:00 AM|02/12/2018 03:49:14 PM|2018-01-27 07:00:00|
|02/05/2018 12:00:00 AM|02/12/2018 03:49:14 PM|2018-02-05 00:00:00|
|01/21/2018 02:00:00 AM|02/12/2018 03:49:14 PM|2018-01-21 02:00:00|
|02/02/2018 12:45:00 AM|02/12/2018 03:49:14 PM|2018-02-02 00:45:00|
|01/25/2018 01:14:00 PM|02/12/2018 03:49:14 PM|2018-01-25 13:14:00|
|02/05/2018 05:00:00 PM|02/12/2018 03:49:14 PM|2018-02-05 17:00:00|
|01/15/2018 12:00:00 AM|02/12/2018 03:49:14 PM|2018-01-15 00:00:00|
|02/05/2018 07:30:00 PM|02/12/2018 03:49:14 PM|2018-02-05 19:30:00|
|02/02/2018 08:30:00 PM|02/12/2018 03:49:14 PM|2018-02-02 20:30:00|
|02/03/2018 06:00:00 PM|02/12/2018 03:49:14 PM|2018-02-03 18:00:00|
|----------------------|----------------------|-------------------|
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。