过去几周我一直在尝试解决这个问题。索引工作“完美”,直到它不。它经常与大量的实体(10 k+),在所有实体的70%左右,它开始抛出403
响应。
设置
** Package (Gradle):**
implementation group: 'org.hibernate.search', name: 'hibernate-search-engine', version: '6.1.7.Final'
implementation group: "org.hibernate.search", name: "hibernate-search-mapper-orm", version: "6.1.7.Final"
implementation group: "org.hibernate.search", name: "hibernate-search-backend-lucene", version: "6.1.7.Final"
implementation group: "org.hibernate.search", name: "hibernate-search-backend-elasticsearch", version: "6.1.7.Final"
implementation group: "org.hibernate.search", name: "hibernate-search-backend-elasticsearch-aws", version: "6.1.7.Final"
休眠配置:
cfg.setProperty("hibernate.search.backend.type", "elasticsearch")
cfg.setProperty("hibernate.search.backend.aws.signing.enabled", "true")
cfg.setProperty("hibernate.search.backend.aws.region",config.getProperty("AWS_REGION"))
cfg.setProperty("hibernate.search.backend.aws.credentials.type", "static")
cfg.setProperty(
"hibernate.search.backend.aws.credentials.access_key_id",
config.getProperty("AWS_ACCESS_KEY_ID")
)
cfg.setProperty(
"hibernate.search.backend.aws.credentials.secret_access_key",
config.getProperty("AWS_SECRET_ACCESS_KEY")
)
cfg.setProperty("hibernate.search.backend.uris", config.getProperty("AWS_ES_HOSTS"))
cfg.setProperty("hibernate.search.backend.version_check.enabled", "false")
cfg.setProperty("hibernate.search.backend.version",config.getProperty("AWS_ES_VERSION"))
cfg.setProperty("hibernate.search.schema_management.strategy", "create")
cfg.setProperty("hibernate.search.backend.indexing.max_bulk_size", "20")
cfg.setProperty("hibernate.search.backend.indexing.queue_size", "20")
代码:
CompletableFuture.runAsync {
Search.session(database.factory.openSession())
.massIndexer()
.idFetchSize(20)
.batchSizeToLoadObjects(10)
.dropAndCreateSchemaOnStart(false)
.purgeAllOnStart(false)
.startAndWait()
}
输出:
一开始......工作正常:
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 75.74% [128.930679documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33458 entities in 259447 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 75.85% [128.958908 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33508 entities in 259708 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 75.97% [129.021820 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33558 entities in 260016 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 76.08% [129.061295 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33608 entities in 260383 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 76.19% [129.071411 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33658 entities in 260650 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 76.31% [129.131012 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33708 entities in 260961 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 76.42% [129.168732 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33758 entities in 261316 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 76.53% [129.184586 documents/second].
PojoMassIndexingLoggingMonitor:97 - HSEARCH000030: Mass indexing progress: indexed 33808 entities in 261771 ms.
PojoMassIndexingLoggingMonitor:100 - HSEARCH000031: Mass indexing progress: 76.65% [129.151047 documents/second].
然后......它不:
ERROR LogFailureHandler:36 - HSEARCH000058: Exception occurred org.hibernate.search.util.common.SearchException: HSEARCH400588: Call to the bulk REST API failed: HSEARCH400007: Elasticsearch request failed: HSEARCH400090: Elasticsearch response indicates a failure.
Request: POST /_bulk with parameters {}
Response: 403 'Forbidden' from 'https://redacted.es.amazonaws.com' with body
"message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been"
}
Failing operation:
Indexing instance of entity 'Entity' during mass indexing
Entities that could not be indexed correctly:
Entity#ac4650e6-2008-4fd5-949f-8541569d6ba9
org.hibernate.search.util.common.SearchException: HSEARCH400588: Call to the bulk REST API failed: HSEARCH400007: Elasticsearch request failed: HSEARCH400090: Elasticsearch response indicates a failure.
Request: POST /_bulk with parameters {}
Response: 403 'Forbidden' from 'https://redacted.es.amazonaws.com' with body
{
"message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been"
}
at org.hibernate.search.backend.elasticsearch.orchestration.impl.ElasticsearchDefaultWorkSequenceBuilder$BulkedWorkExecutionState.onBulkWorkComplete(ElasticsearchDefaultWorkSequenceBuilder.java:239) ~[hibernate-search-backend-elasticsearch-6.1.7.Final.jar:6.1.7.Final]
at org.hibernate.search.util.common.impl.Futures.lambda$handler$3(Futures.java:100) ~[hibernate-search-util-common-6.1.7.Final.jar:6.1.7.Final]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2144) ~[?:?]
at org.hibernate.search.backend.elasticsearch.client.impl.ElasticsearchClientImpl$1.onFailure(ElasticsearchClientImpl.java:127) ~[hibernate-search-backend-elasticsearch-6.1.7.Final.jar:6.1.7.Final]
at org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:672) ~[elasticsearch-rest-client-7.17.4.jar:7.17.4]
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:408) ~[elasticsearch-rest-client-7.17.4.jar:7.17.4]
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:392) ~[elasticsearch-rest-client-7.17.4.jar:7.17.4]
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) ~[httpcore-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:182) ~[httpasyncclient-4.1.5.jar:4.1.5]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87) ~[httpasyncclient-4.1.5.jar:4.1.5]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40) ~[httpasyncclient-4.1.5.jar:4.1.5]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.15.jar:4.4.15]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.15.jar:4.4.15]
at java.lang.Thread.run(Thread.java:830) ~[?:?]
Caused by: org.hibernate.search.util.common.SearchException: HSEARCH400007: Elasticsearch request failed: HSEARCH400090: Elasticsearch response indicates a failure.
Request: POST /_bulk with parameters {}
Response: 403 'Forbidden' from 'https://redacted.es.amazonaws.com' with body
{
"message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been"
}
at org.hibernate.search.backend.elasticsearch.work.impl.AbstractNonBulkableWork.handleResult(AbstractNonBulkableWork.java:84) ~[hibernate-search-backend-elasticsearch-6.1.7.Final.jar:6.1.7.Final]
at org.hibernate.search.backend.elasticsearch.work.impl.AbstractNonBulkableWork.lambda$execute$3(AbstractNonBulkableWork.java:66) ~[hibernate-search-backend-elasticsearch-6.1.7.Final.jar:6.1.7.Final]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642
我已经看了所有我能找到的文档,尝试了不同的参数,偏移量,fetchId大小,等等,但是没有用。真正的区别因素,我似乎不能挖掘通过是第一次工作(实体实际上是索引和显示),但它不工作后,一段时间...(4- 5分钟)。什么都没有找到。
欢迎任何反馈、回答或问题。
1条答案
按热度按时间zysjyyx41#
看起来你在Hibernate搜索的大规模索引过程中遇到了一个问题。索引最初看起来运行良好,但当实体数量增加时,大约70%的实体开始出现403个响应。
此问题的原因可能是多种因素造成的,例如达到Elasticsearch的索引速率限制或凭据有问题。您可以尝试增大“hib. search.backend.indexing.max_bulk_size”和“hib. search.backend.indexing.queue_size”属性以查看是否有帮助。
此外,您还可以尝试记录发送到Elasticsearch的请求,以获取有关此问题的更多信息。