clickhouse 1.1.54343分布式复制合并树表中的数据摄取错误

0kjbasz6  于 2021-07-15  发布在  ClickHouse
关注(0)|答案(1)|浏览(352)

我在clickhouse 1.1.54343中遇到数据加载和表合并问题，无法在clickhouse中插入任何数据。
我们有3个节点的集群，在数据摄取和从json文件摄取数据中向表中添加300列。
我们能把数据保存在表格里
创建表

  2. * -- Each Node*
  4. CREATE TABLE IF NOT EXISTS AudiencePlanner.reached_pod
  5. (
  6.     date Date,
  7.     p_id String,
  8.     language String, 
  9.     city String, 
  10.     state String, 
  11.     platform String, 
  12.     manufacturer String,
  13.     model String, 
  14.     content_id String
  15. ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/shard1/reached_pod', 'clickhouse1')
  16. PARTITION BY date 
  17. ORDER BY (date, platform, language, city, state, manufacturer, model, content_id, p_id);
  19. CREATE TABLE IF NOT EXISTS AudiencePlanner2.reached_pod
  20. (
  21.     date Date,
  22.     p_id String,
  23.     language String, 
  24.     city String, 
  25.     state String, 
  26.     platform String, 
  27.     manufacturer String,
  28.     model String, 
  29.     content_id String
  30. ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/shard3/reached_pod', 'clickhouse1')
  31. PARTITION BY date 
  32. ORDER BY (date, platform, language, city, state, manufacturer, model, content_id, p_id);

---所有节点

  1. CREATE TABLE AudiencePlanner.reached_pod_all AS AudiencePlanner.reached_pod ENGINE = Distributed(test, '', reached_pod, rand());

配置.xml

  1. <remote_servers>
  2.         <test>
  3.             <shard>
  4.                 <weight>1</weight>
  5.                 <internal_replication>false</internal_replication>
  6.                 <replica>
  7.                     <host>ip1</host>
  8.                     <port>9000</port>
  9.                     <default_database>AudiencePlanner</default_database>
  10.                     <user>test</user>
  11.                     <password>testpass</password>
  12.                 </replica>
  13.                 <replica>
  14.                     <host>ip2</host>
  15.                     <port>9000</port>
  16.                     <default_database>AudiencePlanner2</default_database>
  17.                     <user>test</user>
  18.                     <password>testpass</password>
  19.                 </replica>
  20.             </shard>
  21.             <shard>
  22.                 <weight>1</weight>
  23.                 <internal_replication>false</internal_replication>
  24.                 <replica>
  25.                     <host>ip2</host>
  26.                     <port>9000</port>
  27.                     <default_database>AudiencePlanner</default_database>
  28.                     <user>test</user>
  29.                     <password>testpass</password>
  30.                 </replica>
  31.                 <replica>
  32.                     <host>ip3</host>
  33.                     <port>9000</port>
  34.                     <default_database>AudiencePlanner2</default_database>
  35.                     <user>test</user>
  36.                     <password>testpass</password>
  37.                 </replica>
  38.             </shard>
  39.             <shard>
  40.                 <weight>1</weight>
  41.                 <internal_replication>false</internal_replication>
  42.                 <replica>
  43.                     <host>ip3</host>
  44.                     <port>9000</port>
  45.                     <default_database>AudiencePlanner</default_database>
  46.                     <user>test</user>
  47.                     <password>testpass</password>
  48.                 </replica>
  49.                 <replica>
  50.                     <host>ip1</host>
  51.                     <port>9000</port>
  52.                     <default_database>AudiencePlanner2</default_database>
  53.                     <user>test</user>
  54.                     <password>testpass</password>
  55.                 </replica>
  56.             </shard>
  57.         </dms>
  58.     </remote_servers>

错误日志

  1. 2018.03.10 07:50:59.990953 [ 31 ] <Trace> AudiencePlanner.reached_pod (StorageReplicatedMergeTree): Executing log entry to merge parts 20180203_111_111_0, 20180203_112_112_0, 20180203_113_113_0 to 20180203_111_113_1
  2. 2018.03.10 07:50:59.991204 [ 31 ] <Debug> AudiencePlanner.reached_pod (Merger): Merging 3 parts: from 20180203_111_111_0 to 20180203_113_113_0 into tmp_merge_20180203_111_113_1
  3. 2018.03.10 07:50:59.996659 [ 31 ] <Debug> AudiencePlanner.reached_pod (Merger): Selected MergeAlgorithm: Horizontal
  4. 2018.03.10 07:50:59.997347 [ 31 ] <Trace> MergeTreeBlockInputStream: Reading 1 ranges from part 20180203_111_111_0, approx. 24576 rows starting from 0
  5. 2018.03.10 07:50:59.997417 [ 31 ] <Trace> MergeTreeBlockInputStream: Reading 1 ranges from part 20180203_112_112_0, approx. 8192 rows starting from 0
  6. 2018.03.10 07:50:59.997476 [ 31 ] <Trace> MergeTreeBlockInputStream: Reading 1 ranges from part 20180203_113_113_0, approx. 8192 rows starting from 0
  7. 2018.03.10 07:51:00.016479 [ 31 ] <Debug> MemoryTracker: Peak memory usage: 1.16 GiB.
  8. 2018.03.10 07:51:00.044547 [ 31 ] <Error> DB::StorageReplicatedMergeTree::queueTask()::<lambda(DB::StorageReplicatedMergeTree::LogEntryPtr&)>: Code: 76, e.displayText() = 
  10. DB::Exception: Cannot open file /data/clickhouse//data/AudiencePlanner/reached_pod/tmp_merge_20180203_111_113_1/%1F%EF%BF%BD%08%00%00%00%00%00%00%00%EF%BF%BDVrs%EF%BF%BDws%EF%BF%BDu%EF%BF%BDq%EF%BF%BD%0F%09%EF%BF%BD76%EF%BF%BD51P%EF%BF%BDQ%0A%0A%EF%BF%BDw%EF%BF%BDu%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BDw%042%0D%EF%BF%BD%0D%0C%0D%2D%EF%BF%BD%EF%BF%BD%08%EF%BF%BD%C6%A6%EF%BF%BD%26%26J%EF%BF%BD%00%EF%BF%BD%1C%EF%BF%BD%1E%3F%00%00%00.bin, errno: 36, strerror: File name too long, e.what() = DB::Exception, Stack trace:
  12. 0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x15) [0x7317e35]
  13. 1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x1e) [0x19caa8e]
  14. 2. /usr/bin/clickhouse-server(DB::throwFromErrno(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)+0x1a4) [0x72ffea4]
  15. 3. /usr/bin/clickhouse-server(DB::WriteBufferFromFile::WriteBufferFromFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, int, unsigned int, char*, unsigned long)+0x1c5) [0x733d4a5]
  16. 4. /usr/bin/clickhouse-server(DB::createWriteBufferFromFileBase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, unsigned long, int, unsigned int, char*, unsigned long)+0xac) [0x7345c9c]
  17. 5. /usr/bin/clickhouse-server(DB::IMergedBlockOutputStream::ColumnStream::ColumnStream(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, DB::CompressionSettings, unsigned long, unsigned long)+0x
  19. I tried detaching the partition but still not able insert any other data, after detaching the error comes like 
  20. 2018.03.10 09:00:27.299291 [ 9 ] <Error> reached_pod_all.Distributed.DirectoryMonitor: Code: 76, e.displayText() = 
  22. DB::Exception: Received from ip1:9000. DB::Exception: Cannot open file /data/clickhouse//data/AudiencePlanner/reached_pod/tmp_insert_20180207_21_21_0/%1F%EF%BF%BD%08%00%00%00%00%00%00%00%EF%BF%BDVrs%EF%BF%BDws%EF%BF%BDu%EF%BF%BDq%EF%BF%BD%0F%09%EF%BF%BD76%EF%BF%BD51P%EF%BF%BDQ%0A%0A%EF%BF%BDw%EF%BF%BDu%EF%BF%BD%EF%BF%BD%EF%BF%BD%EF%BF%BDw%042%0D%EF%BF%BD%0D%0C%0D%2D%EF%BF%BD%EF%BF%BD%08%EF%BF%BD%C6%A6%EF%BF%BD%26%26J%EF%BF%BD%00%EF%BF%BD%1C%EF%BF%BD%1E%3F%00%00%00.bin, errno: 36, strerror: File name too long. Stack trace:
  24. 0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x15) [0x7317e35]
  25. 1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x1e) [0x19caa8e]
  26. 2. /usr/bin/clickhouse-server(DB::throwFromErrno(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int)+0x1a4) [0x72ffea4]
  27. 3. /usr/bin/clickhouse-server(DB::WriteBufferFromFile::WriteBufferFromFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, int, unsigned int, char*, unsigned long)+0x1c5) [0x733d4a5]
  28. 4. /usr/bin/clickhouse-server(DB::createWriteBufferFromFileBase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, unsigned long, int, unsigned int, char*, unsigned long)+0xac) [0x7345c9c]
  29. 5. /usr/bin/clickhouse-server(DB::IMergedBlockOutputStream::ColumnStream::ColumnStream(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, DB::CompressionSettings, unsigned long, unsigned long)+0xcc) [0x68cd7ac]
  30. 6. /usr/bin/clickhouse-server() [0x68ce674]

请帮助我确定和解决这个问题。

clickhouse

1条答案

按热度按时间
2eafrhcq

2eafrhcq1#

我为我的问题找到了一个解决方案，这是由于我们动态添加了大约1000个列的列的数量。
由于列的数量，文件名创建了一个长字符串，导致写入文件名时出错。

  1. I reduced the number of columns to be inserted and i was able to write the data.
赞(0）分享  回复(0）举报  2021-07-15
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com