火花测量：无数据报告

如何解决火花测量：无数据报告

客观

我正在将Spark应用程序从本地计算机（客户端模式）提交到具有databricks-connect（v6.6）的Databricks集群。如Spark Measure page中所述，使用PyPi sparkmeasure==0.14.0。

问题

为什么Spark Measure不打印任何指标？可以将Spark Measure与databricks-connect一起使用吗？

代码

spark = SparkSession \
    .builder \
    .appName(app_name) \
    .config("spark.jars.packages","ch.cern.sparkmeasure:spark-measure_2.11:0.16") \
    .config("spark.driver.host","localhost") \
    .config("spark.driver.bindAddress","127.0.0.1") \
    .config("fs.azure.account.key.<my_storage>.dfs.core.windows.net",key) \
    .getOrCreate()

 from sparkmeasure import StageMetrics,TaskMetrics
 df = load_data(some_path)
 StageMetrics(self.spark).runandmeasure(locals(),'df.count()'). # output 1

 df2 = load_data(some_path)
 TaskMetrics(self.spark).runandmeasure(locals(),'df2.count()'). # output 2

输出1

Scheduling mode = FIFO
Spark Context default degree of parallelism = 8
 no data to report 

输出2

Scheduling mode = FIFO
Spark Contex default degree of parallelism = 8
Aggregated Spark task metrics:
numtasks => 0
elapsedTime => null
sum(duration) => null
sum(schedulerDelay) => null
sum(executorRunTime) => null
sum(executorCpuTime) => null
sum(executorDeserializeTime) => null
sum(executorDeserializeCpuTime) => null
sum(resultSerializationTime) => null
sum(jvmGCTime) => null
sum(shuffleFetchWaitTime) => null
sum(shuffleWriteTime) => null
sum(gettingResultTime) => null
max(resultSize) => null
sum(numUpdatedBlockStatuses) => null
sum(diskBytesSpilled) => null
sum(memoryBytesSpilled) => null
max(peakExecutionMemory) => null
sum(recordsRead) => null
sum(bytesRead) => null
sum(recordsWritten) => null
sum(bytesWritten) => null
sum(shuffleTotalBytesRead) => null
sum(shuffleTotalBlocksFetched) => null
sum(shuffleLocalBlocksFetched) => null
sum(shuffleRemoteBlocksFetched) => null
sum(shuffleBytesWritten) => null
sum(shuffleRecordsWritten) => null