我尝试在eclipseide上测试apachespark与hive集成。以下是每个项目的版本—Hadoop2.7.4、Spark2.2和hive-2.3.2以及EclipseMarsIDE上的MySQL5.7。hive-site.xml的内容如下:,
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>password</value>
</property>
<property>
<name>hive.metastore.port</name>
<value>33306</value>
</property>
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
</property>
eclipse ide上的spark hive java api代码是
public static void main(String[] args) {
Logger.getLogger("org").setLevel(Level.OFF);
Logger.getLogger("akka").setLevel(Level.OFF);
// TODO Auto-generated method stub
SparkSession spark = SparkSession.builder()
.appName("Spark Hive Test")
.master("local[*]")
.config("spark.sql.warehouse.dir","/user/hive/warehouse")
.enableHiveSupport()
.getOrCreate();
System.out.println(spark.version());
String tableName = "testHiveDriverTable";
spark.sql("drop table if exists " + tableName);
String createTable = "create table " + tableName + " (key int, value string) "
+ "ROW FORMAT DELIMITED "
+ "FIELDS TERMINATED BY ',' "
+ "LINES TERMINATED BY '\n' "
+ "STORED AS TEXTFILE";
spark.sql(createTable);
spark.sql("LOAD DATA LOCAL INPATH '/home/m_usr/temp/hiveTest.txt' INTO TABLE " + tableName);
// Queries are expressed in HiveQL
spark.sql("SELECT * FROM " + tableName).show();
// Aggregation queries are also supported.
spark.sql("SELECT COUNT(*) FROM " + tableName).show();
spark.stop();
spark.close();
}
此代码引发异常:
java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.setReadOnly(Unknown Source)
at com.jolbox.bonecp.ConnectionHandle.setReadOnly(ConnectionHandle.java:1324)
at com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:262)
at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: ERROR 25505: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.sql.conn.GenericAuthorizer.setReadOnlyConnection(Unknown Source)
at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.setReadOnly(Unknown Source)
... 8 more
但是在根模式下执行eclipseide时,不会抛出异常。我认为这个问题与特权问题有关。但是我不知道是哪个过程导致了这个问题。还有一个问题是apachederby抛出的异常,而不是mysql。我在hive-site.xml上的mysql配置似乎是错误的。
暂无答案!
目前还没有任何答案,快来回答吧!