Nacos server 配置 SSL 之后，Dubbo 服务启动失败

yk9xbfzb 于 5个月前发布在 Nacos

关注(0)|答案(7)|浏览(127)

我们的目的就是对 nacos 开放的所有端口，加上 TLS/SSL 证书配置。

于是，对于 Nacos server 进行了 SSL 的配置：

打开了 rpc 端口的 TLS 配置，dubbo 服务可以正常注册成功，9848 、9849端口配置 SSL 证书成功；

nacos.remote.server.rpc.tls.enableTls=true
nacos.remote.server.rpc.tls.certChainFile=file:/certs/ca/test-server-cert.pem
nacos.remote.server.rpc.tls.certPrivateKey=file:/certs/ca/test-server-key.pem

在此基础上，再开启了 nacos 主端口 8848 的 SSL 配置后，dubbo 服务开始报错

server.ssl.key-alias=nacos_server
server.ssl.key-store=file:/certs/nacos/nacos_server.jks
server.ssl.key-store-type=JKS
server.ssl.key-store-password=123456

报错信息

caused by:
java. lang. RuntimeException: publish nacos metadata failed
at org.apache.dubbo.metadata.store.nacos.NacosMetadataReport.storeMetadata (NacosMetadataReport. java:194)

DubboConfig配置如下：

@Bean
@Primary
public MetadataReportConfig metadataReportConfig() {
    MetadataReportConfig metadataReportConfig = new MetadataReportConfig();
    String nacosAddr = env.getProperty(NACOS_ADDR);
    String nacosUsername = env.getProperty(NACOS_USERNAME);
    String nacosPassword = decodeCipher(env.getProperty(NACOS_CIPHER));
    String addr = "nacos://" + nacosAddr;
    metadataReportConfig.setAddress(addr);
    metadataReportConfig.setUsername(nacosUsername);
    metadataReportConfig.setPassword(nacosPassword);
    return metadataReportConfig;
}

@Bean
@Primary
public RegistryConfig registryConfig() {
    RegistryConfig registryConfig = new RegistryConfig();
    String nacosAddr = env.getProperty(NACOS_ADDR);
    String nacosUsername = env.getProperty(NACOS_USERNAME);
    String nacosPassword = decodeCipher(env.getProperty(NACOS_CIPHER));
    String cacheFile = env.getProperty(NACOS_CACHE_FILE);
    String addr = "nacos://" + nacosAddr + "?username=" + nacosUsername + "&password=" + nacosPassword;
    registryConfig.setAddress(addr);
    registryConfig.setFile(cacheFile);
    registryConfig.setCheck(false);
    registryConfig.setSimplified(true);
    registryConfig.setExtraKeys("ssl-enabled");
    registryConfig.setUsername(nacosUsername);
    registryConfig.setPassword(nacosPassword);

    return registryConfig;
}

@Bean
public SslConfig sslConfig() {
    SslConfig sslConfig = new SslConfig();
    // 是否开启ssl，false为不开启
    if (!BooleanUtil.toBoolean(env.getProperty(SSL_ENABLE))) {
        return sslConfig;
    }

    // 信任证书
    String collectionCert = env.getProperty(SSL_CERT_COLLECTION);
    // 证书
    String cert = env.getProperty(SSL_CERT);
    // 私钥
    String priKey = env.getProperty(SSL_KEY);
    // 私钥密码
    String password = env.getProperty(SSL_CERT_CIPHER);
    if(!StrUtil.isEmpty(password)){
        if(isEncrypted(password)){
            password=decrypt(password);
        }
    }

    // server
    // 服务端信任证书
    sslConfig.setServerTrustCertCollectionPath(collectionCert);
    // 服务端证书
    sslConfig.setServerKeyCertChainPath(cert);
    // 服务端私钥
    sslConfig.setServerPrivateKeyPath(priKey);

    // consumer
    // 客户端信任证书
    sslConfig.setClientTrustCertCollectionPath(collectionCert);
    // 客户端证书
    sslConfig.setClientKeyCertChainPath(cert);
    // 客户端私钥
    sslConfig.setClientPrivateKeyPath(priKey);

    if (StrUtil.isNotEmpty(password)) {
        // 服务端私钥密码
        sslConfig.setServerKeyPassword(password);
        // 客户端私钥密码
        sslConfig.setClientKeyPassword(password);
    }

    // 支持协议TLSv1.2
    sslConfig.setProtocols(PROTOCOLS);

    // 支持密码格式
    sslConfig.setCiphers(CIPHERS);

    return sslConfig;
}

看了一下 nacos 里面的日志，有错误的日志有：

remote-digest.log

2024-06-22 17:07:49,351 WARN [1719047268876_10.36.3.39_51270] connection  close exception  : {}

java.lang.IllegalStateException: call already closed
	at com.google.common.base.Preconditions.checkState(Preconditions.java:508)
	at io.grpc.internal.ServerCallImpl.closeInternal(ServerCallImpl.java:219)
	at io.grpc.internal.ServerCallImpl.close(ServerCallImpl.java:212)
	at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:395)
	at com.alibaba.nacos.core.remote.grpc.GrpcConnection.closeBiStream(GrpcConnection.java:143)
	at com.alibaba.nacos.core.remote.grpc.GrpcConnection.close(GrpcConnection.java:131)
	at com.alibaba.nacos.core.remote.ConnectionManager.unregister(ConnectionManager.java:159)
	at com.alibaba.nacos.core.remote.grpc.AddressTransportFilter.transportTerminated(AddressTransportFilter.java:77)
	at io.grpc.internal.ServerImpl$ServerTransportListenerImpl.transportTerminated(ServerImpl.java:455)
	at io.grpc.netty.shaded.io.grpc.netty.NettyServerTransport.notifyTerminated(NettyServerTransport.java:207)
	at io.grpc.netty.shaded.io.grpc.netty.NettyServerTransport.access$100(NettyServerTransport.java:51)
	at io.grpc.netty.shaded.io.grpc.netty.NettyServerTransport$1TerminationNotifier.operationComplete(NettyServerTransport.java:141)
	at io.grpc.netty.shaded.io.grpc.netty.NettyServerTransport$1TerminationNotifier.operationComplete(NettyServerTransport.java:134)
	at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
	at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
	at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
	at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
	at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
	at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
	at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
	at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1164)
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:755)
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:731)
	at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:620)
	at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.shutdownInput(AbstractEpollChannel.java:522)
	at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:823)
	at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
	at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
	at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:750)
2024-06-22 17:07:49,351 INFO [1719047268876_10.36.3.39_51270]client disconnected,clear config listen context

2024-06-22 17:07:49,915 INFO Connection transportTerminated,connectionId = 1719047238233_10.36.3.39_51268

2024-06-22 17:07:49,915 INFO [1719047238233_10.36.3.39_51268]client disconnected,clear config listen context

2024-06-22 17:07:49,915 INFO Connection transportTerminated,connectionId = 1719047237565_10.36.3.39_51266

请问一下，这个问题可以怎么排查？

nacos

来源：https://github.com/alibaba/nacos/issues/12271

7条答案

按热度按时间

zhte4eai1#

很可能是设置了dubbo的tls，但是没有设置nacos-client的tls，我看你创建了sslConfig，但是没有把这个sslConfig传递给nacos-client

赞(0）回复(0）举报 5个月前

2mbi3lxu2#

@shiyiyue1102 帮忙看一下？或者更新一下关系tls的文档。

赞(0）回复(0）举报 5个月前

ruyhziif3#

很可能是设置了dubbo的tls，但是没有设置nacos-client的tls，我看你创建了sslConfig，但是没有把这个sslConfig传递给nacos-client
@KomachiSion@shiyiyue1102
我查一了一下，按 https://nacos.io/blog/faq/nacos-user-question-history15153/ 中描述的方式没有生效，我查了一下