java 如何以编程方式降低/控制S3请求速率？

xxhby3vn 于 2023-06-04 发布在 Java

关注(0)|答案(1)|浏览(248)

如何以编程方式降低/控制S3请求速率？

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: TZJYPCC02W0YJFNC; S3 Extended Request ID: uEBpUJSvQ6t9+3rxq7BY70xYPWQeQBltI7BYcEzg9Wb8/rWuENBSaxG5CxyGJBzF+luhvcldIELSITu74/2HNw==; Proxy: null), S3 Extended Request ID: uEBpUJSvQ6t9+3rxq7BY70xYPWQeQBltI7BYcEzg9Wb8/rWuENBSaxG5CxyGJBzF+luhvcldIELSITu74/2HNw==
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384)
    at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)
    at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6508)
    at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1856)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1816)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:3982)
    ...
    at java.base/java.util.Spliterator.forEachRemaining(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    at java.base/java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)
    at java.base/java.util.concurrent.CountedCompleter.exec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

Java

来源：https://stackoverflow.com/questions/76371917/how-can-i-programmatically-reduce-control-s3-request-rate

1条答案

按热度按时间

trnvg8h31#

这个想法是创造一个代理人，使限制利率的逻辑失去能力。
用途：

AmazonS3 s3 = ...build();
return rateFriendlyProxy(s3);

实施：

import static java.lang.reflect.Proxy.newProxyInstance;

import com.amazonaws.services.s3.AmazonS3;
import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.locks.ReentrantLock;

public class AmazonS3RateFriendlyProxy implements InvocationHandler {

    private static final int MAX_S3_REQUEST_RATE = 3500;

    public static final AmazonS3 rateFriendlyProxy(AmazonS3 s3) {
        return (AmazonS3) newProxyInstance(
                s3.getClass().getClassLoader(),
                new Class[] { AmazonS3.class },
                new AmazonS3RateFriendlyProxy(s3));
    }

    public static void limitRate(int rate, AtomicLong leapTime, ReentrantLock rateLock) {
        long targetLeapTime = 1_000_000_000 / rate;
        rateLock.lock();
        try {
            long timeSnapshot = nanoTime();
            long parkTime = targetLeapTime - (timeSnapshot - leapTime.get());
            if (parkTime > 0) {

                parkNanos(parkTime);

                leapTime.set(timeSnapshot + parkTime);
            } else {
                leapTime.set(timeSnapshot);
            }
        } finally {
            rateLock.unlock();
        }
    }

    private AmazonS3 s3;
    private ReentrantLock lock = new ReentrantLock();
    private AtomicLong leapTime = new AtomicLong();

    private AmazonS3RateFriendlyProxy(AmazonS3 s3) {
        this.s3 = s3;
    }

    @Override
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        limitRate(MAX_S3_REQUEST_RATE, leapTime, lock);
        return method.invoke(s3, args);
    }
}

赞(0）回复(0）举报 2023-06-04

我来回答

java 如何以编程方式降低/控制S3请求速率？

1条答案

相关问题

热门标签

最新问答