pyspark Graphframes库在palantir-foundry中的应用

xriantvc  于 2023-03-22  发布在  Spark
关注(0)|答案(1)|浏览(153)

我想在我的Foundry代码库中使用GrafFrames包和Pyspark。
这里提到:https://www.palantir.com/docs/foundry/transforms-python/environment-troubleshooting/#packages-which-require-both-a-conda-package-and-a-jar
我将graphframes包包含在要安装的conda库列表中,但我还需要在初始化spark会话时安装服务器端jar。因此我转到transforms-python/build.gradle,并获得以下代码:

// DO NOT MODIFY THIS FILE
buildscript {
    repositories {
        maven {
            credentials {
                username ''
                password project.transformsBearerToken
            }
            authentication {
                basic(BasicAuthentication)
            }
            url project.transformsMavenProxyRepoUri
        }
    }

    dependencies {
        classpath "com.palantir.transforms:transforms-gradle-plugin:${transformsVersion}"
    }
}

apply plugin: 'com.palantir.transforms-defaults'

dependencies { 
    condaJars 'graphframes:graphframes:0.8.1-spark3.0-s_2.12' 
}

然后我保存更改,重新加载页面以应用更改,但随后我得到一个代码帮助错误:

FAILURE: Build failed with an exception.

* Where:

Build file '/scratch/standalone/1c8fbb49-de4d-4c21-8081-47c92748189a/code-assist/contents/build.gradle' line: 24

* What went wrong:

A problem occurred evaluating root project 'feature-generation'.

> Could not find method condaJars() for arguments [graphframes:graphframes:0.8.1-spark3.0-s_2.12] on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

* Try:

Run with --info or --debug option to get more log output. Run with --scan to get full insights.

有人知道为什么以及如何解决这个问题吗?

eanckbw9

eanckbw91#

快了。你提供的链接有线索:
在Settings cog中选择Show hidden files and folders选项,选择inner transforms-python/build.gradle文件,在文件底部添加如下代码块:
1.检查jar的版本是否等于您通过Conda/Pypi添加的版本- conda的最新版本为0.7.32,而Pypi的版本为0.6。
1.将dependencies项移动到内部build. gradle。
我相信你知道,maven坐标可以在这里找到:https://mvnrepository.com/artifact/graphframes/graphframes

相关问题