pyspark 当前UTC时间戳无法获取

yzckvree  于 2023-10-15  发布在  Spark
关注(0)|答案(2)|浏览(117)

current_utc_timestamp()函数获取当前utc时间,但其给出的错误未定义,
我已经进口的所有功能,并尝试仍然其给予的问题
有人能帮我得到当前的UTC时间吗
谢谢

ctehm74n

ctehm74n1#

我可以提出一个非常简单的方法:从pyspark.sql.functions模块导入from_utc_timestamp()函数,调用current_timestamp()函数获取本地时区的当前时间戳,然后将当前时间戳传递给from_utc_timestamp()函数将其转换为UTC。

from pyspark.sql.functions import from_utc_timestamp, current_timestamp

# Get the current timestamp in the local timezone.
current_timestamp_local = current_timestamp()

# Convert the current timestamp to UTC.
current_timestamp_utc = from_utc_timestamp(current_timestamp_local)

# Print the current UTC time.
print(current_timestamp_utc)
gblwokeq

gblwokeq2#

你可以这样做。获取代码将运行的机器的时区,并使用它将current_timestamp()转换为UTC时间戳,如下所示。

from pyspark import SQLContext
from pyspark.sql.functions import *
from graphframes import *
import pyspark.sql.functions as F
from pyspark.sql import SparkSession
from datetime import datetime

spark = SparkSession.builder \
    .appName("MyApp") \
    .getOrCreate()

simpleData1 = [
(1, 'firstname1', 'lastname1', 'address1', -97),
(2, 'firstname1', 'lastname1', 'address1', -23),
(3, 'firstname2', 'lastname2', 'address2', -23),
(4, 'firstname2', 'lastname2', 'address2', -97)
]
columns = ["id","name", "last", "address", "bookID"]

df1 = spark.createDataFrame(simpleData1).toDF(*columns)

df1.show()

timezonenow = str(datetime.now().astimezone().tzinfo)
print(timezonenow)


df1 = df1.withColumn("utc_timestamp", to_utc_timestamp( current_timestamp(), lit(timezonenow)))

print("With UTC timestamp value")
df1.show()

输出量:

+---+----------+---------+--------+------+
| id|      name|     last| address|bookID|
+---+----------+---------+--------+------+
|  1|firstname1|lastname1|address1|   -97|
|  2|firstname1|lastname1|address1|   -23|
|  3|firstname2|lastname2|address2|   -23|
|  4|firstname2|lastname2|address2|   -97|
+---+----------+---------+--------+------+

IST
With UTC timestamp value
+---+----------+---------+--------+------+-----------------------+
|id |name      |last     |address |bookID|utc_timestamp          |
+---+----------+---------+--------+------+-----------------------+
|1  |firstname1|lastname1|address1|-97   |2023-10-09 14:32:08.211|
|2  |firstname1|lastname1|address1|-23   |2023-10-09 14:32:08.211|
|3  |firstname2|lastname2|address2|-23   |2023-10-09 14:32:08.211|
|4  |firstname2|lastname2|address2|-97   |2023-10-09 14:32:08.211|
+---+----------+---------+--------+------+-----------------------+

相关问题