sentry不会捕获spark作业的异常

8aqjt8rx  于 2021-05-18  发布在  Spark
关注(0)|答案(0)|浏览(387)

我的问题是下面的代码抛出了一个异常,我用sentry捕获了这个异常,但是当我转到sentry ui时,这个异常根本没有出现。我想找到一个方法,我可以使用哨兵跨越Spark驱动程序和执行人以及。有什么想法吗?
另外,我不知道需要什么额外的信息,所以请随时告诉我,我会提供它。
版本:
Spark: 2.12-3.0.0 Sentry: 3.1.1 ```
import io.sentry.Sentry
import org.apache.spark.api.java.JavaSparkContext
import org.apache.spark.sql.SparkSession

fun main(args: Array) {
SparkSession.builder()
.appName("myApp")
.config("spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version", "2")
.config("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.getOrCreate()
.use { sparkSession ->
JavaSparkContext.fromSparkContext(sparkSession.sparkContext()).let { sc ->
Sentry.init { options ->
options.dsn = "********"
}

            try {
                throw Exception("exception before executors start working")
            } catch(e: Exception) {
                Sentry.captureException(e)
            }

            // starting some executors and see if Sentry receives exceptions from them:
            sc.parallelize((0..100).toList(), 10).map { i ->                
                try {
                    throw Exception("exception $i from worker")
                } catch (e: Exception) {                        
                    Sentry.captureException(e)
                }
            }
        }
    }

}

我还试着执行 `HubAdapter` 从哨兵那里得到的并以此为借口 `Broadcast` 变化无常,但运气不好。
更新#1
向sentry init添加了调试选项,并尝试启动和关闭会话:

import io.sentry.Sentry
import org.apache.spark.api.java.JavaSparkContext
import org.apache.spark.sql.SparkSession

fun main(args: Array) {
SparkSession.builder()
.appName("myApp")
.config("spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version", "2")
.config("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.getOrCreate()
.use { sparkSession ->
JavaSparkContext.fromSparkContext(sparkSession.sparkContext()).let { sc ->
Sentry.init { options ->
options.dsn = "********"
options.isDebug = true
}

            Sentry.startSession()

            try {
                throw Exception("exception before executors start working")
            } catch(e: Exception) {
                Sentry.captureException(e)
            }

            // starting some executors and see if Sentry receives exceptions from them:
            sc.parallelize((0..100).toList(), 10).map { i ->                
                try {
                    throw Exception("exception $i from worker")
                } catch (e: Exception) {                        
                    Sentry.captureException(e)
                }
            }

            Sentry.endSession()
            Sentry.close()
        }
    }

}

以下是我在日志中看到的:

INFO: Initializing SDK with DSN: '**********'
INFO: No outbox dir path is defined in options.
INFO: GlobalHubMode: 'false'
20/11/03 18:42:01 INFO BlockManagerMasterEndpoint: Registering block manager 10.36.62.15:42123 with 4.6 GiB RAM, BlockManagerId(1, 10.36.62.15, 42123, None)
DEBUG: UncaughtExceptionHandlerIntegration enabled: true
DEBUG: UncaughtExceptionHandlerIntegration installed.
WARNING: Sessions can't be captured without setting a release.
DEBUG: Capturing event: 9b17170bdaf841cbb764969f653f99b5
ERROR: Request failed, API returned 400
ERROR: {"detail":"invalid event envelope","causes":["invalid item header","EOF while parsing an object at line 1 column 49"]}
WARNING: Sessions can't be captured without setting a release.
INFO: Closing SentryClient.
DEBUG: Shutting down

更新#2:
在github中创建问题以提高可见性

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题