我想在我的Foundry代码库中使用GrafFrames包和Pyspark。
这里提到:https://www.palantir.com/docs/foundry/transforms-python/environment-troubleshooting/#packages-which-require-both-a-conda-package-and-a-jar
我将graphframes包包含在要安装的conda库列表中,但我还需要在初始化spark会话时安装服务器端jar。因此我转到transforms-python/build.gradle
,并获得以下代码:
// DO NOT MODIFY THIS FILE
buildscript {
repositories {
maven {
credentials {
username ''
password project.transformsBearerToken
}
authentication {
basic(BasicAuthentication)
}
url project.transformsMavenProxyRepoUri
}
}
dependencies {
classpath "com.palantir.transforms:transforms-gradle-plugin:${transformsVersion}"
}
}
apply plugin: 'com.palantir.transforms-defaults'
dependencies {
condaJars 'graphframes:graphframes:0.8.1-spark3.0-s_2.12'
}
然后我保存更改,重新加载页面以应用更改,但随后我得到一个代码帮助错误:
FAILURE: Build failed with an exception.
* Where:
Build file '/scratch/standalone/1c8fbb49-de4d-4c21-8081-47c92748189a/code-assist/contents/build.gradle' line: 24
* What went wrong:
A problem occurred evaluating root project 'feature-generation'.
> Could not find method condaJars() for arguments [graphframes:graphframes:0.8.1-spark3.0-s_2.12] on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.
* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.
有人知道为什么以及如何解决这个问题吗?
1条答案
按热度按时间eanckbw91#
快了。你提供的链接有线索:
在Settings cog中选择Show hidden files and folders选项,选择inner transforms-python/build.gradle文件,在文件底部添加如下代码块:
1.检查jar的版本是否等于您通过Conda/Pypi添加的版本- conda的最新版本为0.7.32,而Pypi的版本为0.6。
1.将dependencies项移动到内部build. gradle。
我相信你知道,maven坐标可以在这里找到:https://mvnrepository.com/artifact/graphframes/graphframes