java hibernate在timescaledb中插入超级表非常慢

xiozqbni  于 2021-08-20  发布在  Java
关注(0)|答案(2)|浏览(737)

环境:
硬件:
vmware中的8core、32gb访客操作系统
vsan中的数据为1tb
软件
操作系统:centos 7 64位
jdk版本:11.0.7
postgresql版本:13
timescaledb版本:2.3.0
postgresql jdbc版本:42.2.18
我已经运行了timescaledb tune来优化postgresql.conf
我的数据非常庞大。旧表包含832818510条7天的记录。我遵循了timescaledb的数据迁移指示
我的迁移步骤是:

  1. CREATE TABLE test_lp (LIKE lp_interval INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES);
  2. SELECT create_hypertable('test_lp', 'read_time', chunk_time_interval => INTERVAL '1 hour');
  3. INSERT INTO test_lp select * from lp_interval

表架构为:

  1. create table test_lp
  2. (
  3. meterno varchar(11) not null,
  4. reading_id varchar(60) not null,
  5. read_time timestamp not null,
  6. version bigint,
  7. ami_record_num bigint,
  8. flags bigint,
  9. fail_code bigint,
  10. value double precision,
  11. validation_status varchar(255),
  12. custno varchar(11) not null,
  13. insert_date timestamp not null,
  14. constraint test_lp_pkey
  15. primary key (custno, meterno, reading_id, read_time)
  16. );
  17. alter table test_lp
  18. owner to appuser;
  19. create index test_lp_read_time_idx
  20. on test_lp (read_time desc);

总体平均插入速度约为50000条记录/秒。看起来不错。
然后,我开始在timescaledb所在的机器上运行我的简单java程序。java程序使用了一个连接,并将更多数据插入到新表test_lp中。java程序对每1000条插入的记录执行一次提交。
计算插入速度后,java的平均插入速度仅为每秒530条记录。
我还尝试清理表“test_lp”并重新运行java程序。程序的插入速度仍然和上面一样慢
为什么java的插入速度如此之慢?我错过什么了吗?
下面是我的postgresql.conf。我用 show all 在psql中

  1. allow_system_table_mods = off
  2. application_name = PostgreSQL JDBC Driver
  3. archive_cleanup_command =
  4. archive_command = (disabled)
  5. archive_mode = off
  6. archive_timeout = 0
  7. array_nulls = on
  8. authentication_timeout = 1min
  9. autovacuum = on
  10. autovacuum_analyze_scale_factor = 0.1
  11. autovacuum_analyze_threshold = 50
  12. autovacuum_freeze_max_age = 200000000
  13. autovacuum_max_workers = 10
  14. autovacuum_multixact_freeze_max_age = 400000000
  15. autovacuum_naptime = 10s
  16. autovacuum_vacuum_cost_delay = 2ms
  17. autovacuum_vacuum_cost_limit = -1
  18. autovacuum_vacuum_insert_scale_factor = 0.2
  19. autovacuum_vacuum_insert_threshold = 1000
  20. autovacuum_vacuum_scale_factor = 0.2
  21. autovacuum_vacuum_threshold = 50
  22. autovacuum_work_mem = -1
  23. backend_flush_after = 0
  24. backslash_quote = safe_encoding
  25. backtrace_functions =
  26. bgwriter_delay = 200ms
  27. bgwriter_flush_after = 0
  28. bgwriter_lru_maxpages = 100
  29. bgwriter_lru_multiplier = 2
  30. block_size = 8192
  31. bonjour = off
  32. bonjour_name =
  33. bytea_output = hex
  34. check_function_bodies = on
  35. checkpoint_completion_target = 0.9
  36. checkpoint_flush_after = 256kB
  37. checkpoint_timeout = 15min
  38. checkpoint_warning = 30s
  39. client_encoding = UTF8
  40. client_min_messages = notice
  41. cluster_name =
  42. commit_delay = 0
  43. commit_siblings = 5
  44. constraint_exclusion = partition
  45. cpu_index_tuple_cost = 0.005
  46. cpu_operator_cost = 0.0025
  47. cpu_tuple_cost = 0.01
  48. cursor_tuple_fraction = 0.1
  49. data_checksums = off
  50. data_directory_mode = 0700
  51. data_sync_retry = off
  52. DateStyle = ISO, YMD
  53. db_user_namespace = off
  54. deadlock_timeout = 1s
  55. debug_assertions = off
  56. debug_pretty_print = on
  57. debug_print_parse = off
  58. debug_print_plan = off
  59. debug_print_rewritten = off
  60. default_statistics_target = 500
  61. default_table_access_method = heap
  62. default_tablespace =
  63. default_text_search_config = pg_catalog.simple
  64. default_transaction_deferrable = off
  65. default_transaction_isolation = read committed
  66. default_transaction_read_only = off
  67. dynamic_shared_memory_type = posix
  68. effective_cache_size = 22GB
  69. effective_io_concurrency = 200
  70. enable_bitmapscan = on
  71. enable_gathermerge = on
  72. enable_hashagg = on
  73. enable_hashjoin = on
  74. enable_incremental_sort = on
  75. enable_indexonlyscan = on
  76. enable_indexscan = on
  77. enable_material = on
  78. enable_mergejoin = on
  79. enable_nestloop = on
  80. enable_parallel_append = on
  81. enable_parallel_hash = on
  82. enable_partition_pruning = on
  83. enable_partitionwise_aggregate = on
  84. enable_partitionwise_join = on
  85. enable_seqscan = on
  86. enable_sort = on
  87. enable_tidscan = on
  88. escape_string_warning = on
  89. event_source = PostgreSQL
  90. exit_on_error = off
  91. extra_float_digits = 3
  92. force_parallel_mode = off
  93. from_collapse_limit = 8
  94. fsync = on
  95. full_page_writes = on
  96. geqo = on
  97. geqo_effort = 5
  98. geqo_generations = 0
  99. geqo_pool_size = 0
  100. geqo_seed = 0
  101. geqo_selection_bias = 2
  102. geqo_threshold = 12
  103. gin_fuzzy_search_limit = 0
  104. gin_pending_list_limit = 4MB
  105. hash_mem_multiplier = 1
  106. hot_standby = on
  107. hot_standby_feedback = off
  108. huge_pages = try
  109. idle_in_transaction_session_timeout = 0
  110. ignore_checksum_failure = off
  111. ignore_invalid_pages = off
  112. ignore_system_indexes = off
  113. integer_datetimes = on
  114. IntervalStyle = postgres
  115. jit = on
  116. jit_above_cost = 100000
  117. jit_debugging_support = off
  118. jit_dump_bitcode = off
  119. jit_expressions = on
  120. jit_inline_above_cost = 500000
  121. jit_optimize_above_cost = 500000
  122. jit_profiling_support = off
  123. jit_tuple_deforming = on
  124. join_collapse_limit = 8
  125. krb_caseins_users = off
  126. lc_collate = zh_TW.UTF-8
  127. lc_ctype = zh_TW.UTF-8
  128. lc_messages = zh_TW.UTF-8
  129. lc_monetary = zh_TW.UTF-8
  130. lc_numeric = zh_TW.UTF-8
  131. lc_time = zh_TW.UTF-8
  132. listen_addresses = *
  133. lo_compat_privileges = off
  134. local_preload_libraries =
  135. lock_timeout = 0
  136. log_autovacuum_min_duration = -1
  137. log_checkpoints = off
  138. log_connections = off
  139. log_destination = stderr
  140. log_disconnections = off
  141. log_duration = off
  142. log_error_verbosity = default
  143. log_executor_stats = off
  144. log_file_mode = 0600
  145. log_hostname = off
  146. log_line_prefix = %m [%p]
  147. log_lock_waits = off
  148. log_min_duration_sample = -1
  149. log_min_duration_statement = -1
  150. log_min_error_statement = error
  151. log_min_messages = warning
  152. log_parameter_max_length = -1
  153. log_parameter_max_length_on_error = 0
  154. log_parser_stats = off
  155. log_planner_stats = off
  156. log_replication_commands = off
  157. log_rotation_age = 1d
  158. log_rotation_size = 0
  159. log_statement = none
  160. log_statement_sample_rate = 1
  161. log_statement_stats = off
  162. log_temp_files = -1
  163. log_timezone = Asia/Taipei
  164. log_transaction_sample_rate = 0
  165. log_truncate_on_rotation = on
  166. logging_collector = on
  167. logical_decoding_work_mem = 64MB
  168. maintenance_io_concurrency = 10
  169. maintenance_work_mem = 420MB
  170. max_connections = 100
  171. max_files_per_process = 1000
  172. max_function_args = 100
  173. max_identifier_length = 63
  174. max_index_keys = 32
  175. max_locks_per_transaction = 256
  176. max_logical_replication_workers = 4
  177. max_parallel_maintenance_workers = 4
  178. max_parallel_workers = 8
  179. max_parallel_workers_per_gather = 4
  180. max_pred_locks_per_page = 2
  181. max_pred_locks_per_relation = -2
  182. max_pred_locks_per_transaction = 64
  183. max_prepared_transactions = 0
  184. max_replication_slots = 10
  185. max_slot_wal_keep_size = -1
  186. max_stack_depth = 2MB
  187. max_standby_archive_delay = 30s
  188. max_standby_streaming_delay = 30s
  189. max_sync_workers_per_subscription = 2
  190. max_wal_senders = 0
  191. max_wal_size = 32GB
  192. max_worker_processes = 19
  193. min_parallel_index_scan_size = 512kB
  194. min_parallel_table_scan_size = 8MB
  195. min_wal_size = 16GB
  196. old_snapshot_threshold = -1
  197. operator_precedence_warning = off
  198. parallel_leader_participation = on
  199. parallel_setup_cost = 1000
  200. parallel_tuple_cost = 0.1
  201. password_encryption = scram-sha-256
  202. pg_stat_statements.max = 5000
  203. pg_stat_statements.save = on
  204. pg_stat_statements.track = top
  205. pg_stat_statements.track_planning = off
  206. pg_stat_statements.track_utility = on
  207. plan_cache_mode = auto
  208. port = 5432
  209. post_auth_delay = 0
  210. pre_auth_delay = 0
  211. primary_slot_name =
  212. promote_trigger_file =
  213. quote_all_identifiers = off
  214. random_page_cost = 1.1
  215. recovery_end_command =
  216. recovery_min_apply_delay = 0
  217. recovery_target =
  218. recovery_target_action = pause
  219. recovery_target_inclusive = on
  220. recovery_target_lsn =
  221. recovery_target_name =
  222. recovery_target_time =
  223. recovery_target_timeline = latest
  224. recovery_target_xid =
  225. restart_after_crash = on
  226. restore_command =
  227. row_security = on
  228. search_path = public
  229. segment_size = 1GB
  230. seq_page_cost = 1
  231. server_encoding = UTF8
  232. server_version = 13.3
  233. server_version_num = 130003
  234. session_replication_role = origin
  235. shared_buffers = 8GB
  236. shared_memory_type = mmap
  237. ssl = off
  238. ssl_ca_file =
  239. ssl_cert_file = server.crt
  240. ssl_crl_file =
  241. ssl_key_file = server.key
  242. ssl_library = OpenSSL
  243. ssl_passphrase_command_supports_reload = off
  244. ssl_prefer_server_ciphers = on
  245. standard_conforming_strings = on
  246. statement_timeout = 0
  247. superuser_reserved_connections = 3
  248. synchronize_seqscans = on
  249. synchronous_commit = off
  250. synchronous_standby_names =
  251. syslog_facility = local0
  252. syslog_ident = postgres
  253. syslog_sequence_numbers = on
  254. syslog_split_messages = on
  255. tcp_keepalives_count = 9
  256. tcp_keepalives_idle = 7200
  257. tcp_keepalives_interval = 75
  258. tcp_user_timeout = 0
  259. temp_buffers = 8MB
  260. temp_file_limit = -1
  261. temp_tablespaces =
  262. timescaledb.disable_load = off
  263. timescaledb.enable_2pc = on
  264. timescaledb.enable_async_append = on
  265. timescaledb.enable_cagg_reorder_groupby = on
  266. timescaledb.enable_chunk_append = on
  267. timescaledb.enable_client_ddl_on_data_nodes = off
  268. timescaledb.enable_connection_binary_data = on
  269. timescaledb.enable_constraint_aware_append = on
  270. timescaledb.enable_constraint_exclusion = on
  271. timescaledb.enable_optimizations = on
  272. timescaledb.enable_ordered_append = on
  273. timescaledb.enable_parallel_chunk_append = on
  274. timescaledb.enable_per_data_node_queries = on
  275. timescaledb.enable_qual_propagation = on
  276. timescaledb.enable_remote_explain = off
  277. timescaledb.enable_runtime_exclusion = on
  278. timescaledb.enable_skipscan = on
  279. timescaledb.enable_transparent_decompression = on
  280. timescaledb.last_tuned = 2021-07-05T14:31:04+08:00
  281. timescaledb.last_tuned_version = 0.11.0
  282. timescaledb.license = timescale
  283. timescaledb.max_background_workers = 8
  284. timescaledb.max_cached_chunks_per_hypertable = 100
  285. timescaledb.max_insert_batch_size = 1000
  286. timescaledb.max_open_chunks_per_insert = 1342
  287. timescaledb.remote_data_fetcher = cursor
  288. timescaledb.restoring = off
  289. timescaledb.telemetry_level = basic
  290. timescaledb_telemetry.cloud =
  291. TimeZone = UTC
  292. timezone_abbreviations = Default
  293. trace_notify = off
  294. trace_recovery_messages = log
  295. trace_sort = off
  296. track_activities = on
  297. track_activity_query_size = 1kB
  298. track_commit_timestamp = off
  299. track_counts = on
  300. track_functions = pl
  301. track_io_timing = on
  302. transaction_deferrable = off
  303. transaction_isolation = read committed
  304. transaction_read_only = off
  305. transform_null_equals = off
  306. unix_socket_group =
  307. unix_socket_permissions = 0777
  308. update_process_title = on
  309. vacuum_cleanup_index_scale_factor = 0.1
  310. vacuum_cost_delay = 0
  311. vacuum_cost_limit = 200
  312. vacuum_cost_page_dirty = 20
  313. vacuum_cost_page_hit = 1
  314. vacuum_cost_page_miss = 10
  315. vacuum_defer_cleanup_age = 0
  316. vacuum_freeze_min_age = 50000000
  317. vacuum_freeze_table_age = 150000000
  318. vacuum_multixact_freeze_min_age = 5000000
  319. vacuum_multixact_freeze_table_age = 150000000
  320. wal_block_size = 8192
  321. wal_buffers = 16MB
  322. wal_compression = on
  323. wal_consistency_checking =
  324. wal_init_zero = on
  325. wal_keep_size = 0
  326. wal_level = replica
  327. wal_log_hints = off
  328. wal_receiver_create_temp_slot = off
  329. wal_receiver_status_interval = 10s
  330. wal_receiver_timeout = 1min
  331. wal_recycle = on
  332. wal_retrieve_retry_interval = 5s
  333. wal_segment_size = 16MB
  334. wal_sender_timeout = 1min
  335. wal_skip_threshold = 2MB
  336. wal_sync_method = fdatasync
  337. wal_writer_delay = 200ms
  338. wal_writer_flush_after = 1MB
  339. work_mem = 32MB
  340. xmlbinary = base64
  341. xmloption = content
  342. zero_damaged_pages = off
zte4gxcn

zte4gxcn1#

因此,正如我所记得的,java jdbc驱动程序有一个特殊的标志,您需要将其添加到数据库的连接字符串中,以使其重写插入,从而使其成为多值插入,而不是批处理单值插入。这种改变可以使时间刻度/postgres的插入速度提高10-20倍(至少)。
该参数称为rewritebatchedinserts,您可以只添加 rewriteBatchedInserts=true 到连接字符串,或者有些驱动程序允许您以更编程的方式添加它。我想https://vladmihalcea.com/postgresql-multi-row-insert-rewritebatchedinserts-property/ 有更多的信息。
这是一个有点奇怪的事情,你几乎只是做了一个咒语,突然一切工作…希望这有帮助!

oknwwptz

oknwwptz2#

数据加载器是否在同一网络中运行?你考虑过网络延迟吗?
我建议您在机器中尝试tsbs基准测试工具,以了解问题是出在java客户端还是机器上:https://github.com/timescale/tsbs

相关问题