AWS S3 v2 GetObject无法取消编组响应java.net.SocketTimeoutException

分类：编程问答

如何解决AWS S3 v2 GetObject无法取消编组响应java.net.SocketTimeoutException

我正在尝试从S3获取一个大对象，并将输出直接输出到HttpServletResponse输出流。

但是由于SocketTimeoutException，它总是在此对象上失败。

以下是使用S3Client获取对象的代码：

    public void downloadMergedResults(final String jobId,final OutputStream os) {
        get("jobs/" + jobId + "/results",os);
    }

    private boolean get(final String key,final OutputStream outputStream) {
        log.trace("GET {}",key);
        try {
            client.getObject(GetObjectRequest.builder().bucket(bucket).key(key).build(),ResponseTransformer.toOutputStream(outputStream));
            return true;
        } catch (S3Exception e) {
            if (e.statusCode() == 404) {
                log.info("Item with key: {} does not exist.",key);
            }
            else {
                log.error("Could not get item {} from S3. Error: {} {}",key,e.getMessage(),e.awsErrorDetails());
            }
        }
        return false;
    }

这是我在springboot应用程序中使用它的方式：

    public Callable<ResponseEntity<StreamingResponseBody>> allData(HttpServletResponse res,@PathVariable("jobId") String jobId) {

        Progress progress = s3client.downloadCachedProgress(jobId);
        if (progress == null) {
            log.info("{} Could not find progress file",jobId);
            return () -> new ResponseEntity<>(HttpStatus.NOT_FOUND);
        }

        res.setContentType(MediaType.APPLICATION_JSON_VALUE);
        res.setContentLengthLong(OPEN_ARRAY.length + progress.getDataSizeBytes() + (SEPARATOR.length * (progress.getMediaCount() - 1)) + CLOSE_ARRAY.length);

        return () -> {
            StreamingResponseBody streamingResponseBody = out -> {
                try {
                    s3client.downloadMergedResults(jobId,out);
                } catch(Exception e) {
                    log.error("{} Error downloading results: {}",jobId,e);
                }
            };
            return new ResponseEntity<>(streamingResponseBody,HttpStatus.OK);
        };
    }

这是卷曲的意思：

curl: (18) transfer closed with 524076376 bytes remaining to read

这是我的应用程序中的堆栈跟踪：

software.amazon.awssdk.core.exception.SdkClientException: Unable to unmarshall response (java.net.SocketTimeoutException). Response Code: 200,Response Text: OK
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:108) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:72) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:189) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:121) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:147) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:62) ~[aws-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:3539) ~[s3-2.13.44.jar!/:na]
        at org.me.blobstorage.S3ClientWrapper.get(S3ClientWrapper.java:362) ~[classes!/:na]
        at org.me.blobstorage.S3ClientWrapper.downloadMergedResults(S3ClientWrapper.java:253) ~[classes!/:na]
        at org.me.RequestHandler.lambda$null$4(RequestHandler.java:241) ~[classes!/:na]
        at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:111) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:98) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at org.springframework.web.context.request.async.WebAsyncManager.lambda$startCallableProcessing$4(WebAsyncManager.java:328) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_265]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_265]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_265]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_265]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_265]
Caused by: software.amazon.awssdk.core.exception.NonRetryableException: java.net.SocketTimeoutException
        at software.amazon.awssdk.core.exception.NonRetryableException$BuilderImpl.build(NonRetryableException.java:92) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.transformResponse(BaseSyncClientHandler.java:187) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.handle(BaseSyncClientHandler.java:168) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:97) ~[sdk-core-2.13.44.jar!/:na]
        ... 40 common frames omitted
Caused by: org.apache.catalina.connector.ClientAbortException: java.net.SocketTimeoutException
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:776) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:681) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:386) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:364) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at software.amazon.awssdk.utils.IoUtils.copy(IoUtils.java:113) ~[utils-2.13.44.jar!/:na]
        at software.amazon.awssdk.utils.IoUtils.copy(IoUtils.java:99) ~[utils-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.sync.ResponseTransformer.lambda$toOutputStream$2(ResponseTransformer.java:159) ~[sdk-core-2.13.44.jar!/:na]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.transformResponse(BaseSyncClientHandler.java:180) ~[sdk-core-2.13.44.jar!/:na]
        ... 42 common frames omitted
Caused by: java.net.SocketTimeoutException: null
        at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:134) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1257) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:814) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:584) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:522) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:538) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:73) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:190) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.coyote.Response.doWrite(Response.java:601) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:339) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
        ... 51 common frames omitted

以下是我尝试初始化S3Client的不同方法：

1：

        client = S3Client.builder()
                .region(region)
                .credentialsProvider(credentialsProvider)
                .build();

2：

        client = S3Client.builder()
                .region(region)
                .overrideConfiguration(ClientOverrideConfiguration.builder()
                        .apiCallTimeout(Duration.ofHours(1))
                        .build())
                .credentialsProvider(credentialsProvider)
                .build();

3：

        client = S3Client.builder()
                .region(region)
                .overrideConfiguration(ClientOverrideConfiguration.builder()
                        .apiCallTimeout(Duration.ofHours(1))
                        .build())
                .httpClient(UrlConnectionHttpClient.builder()
                        .connectionTimeout(Duration.ofHours(1))
                        .socketTimeout(Duration.ofHours(1)).build())
                .credentialsProvider(credentialsProvider)
                .build();

4：

        client = S3Client.builder()
                .region(region)
                .overrideConfiguration(ClientOverrideConfiguration.builder()
                        .apiCallTimeout(Duration.ofHours(1))
                        .build())
                .httpClientBuilder(UrlConnectionHttpClient.builder()
                        .connectionTimeout(Duration.ofHours(1))
                        .socketTimeout(Duration.ofHours(1)))
                .credentialsProvider(credentialsProvider)
                .build();

我的Spring Boot Applications.properties设置请求超时：

spring.mvc.async.request-timeout=-1

我的引导类上有@EnableAsync。

通过nginx代理请求，位置上的设置为：

upstream myupstream {
  server [SERVER-REDACTED]:[PORT-REDACTED];
  keepalive 15;
}


server {
...
  location ~ ^/(v1/bigsearch|actuator)/ {
    proxy_pass http://myupstream;
    proxy_http_version 1.1;
    proxy_set_header Connection "Keep-Alive";
    proxy_set_header Proxy-Connection "Keep-Alive";
    client_max_body_size 2G;
  }
...
}

相关的nginx.conf条目本身：

http {

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 1d;
    types_hash_max_size 2048;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    proxy_read_timeout 1d;
    proxy_send_timeout 1d;

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    gzip on;
    gzip_disable "msie6";

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

问题

我还没有做什么才能完成这项工作？

解决方法

和往常一样，我连续几天困扰着一个问题，将其发布在SO上，然后加以解决。

即使错误表现为SocketTimeoutException，实际上问题出在nginx代理缓冲上。

过了一会儿，我感到很重要，下载始终达到1G，直到我进行专门搜索之后，我才在这里发现实际问题和解决方案：https://github.com/owncloud/client/issues/5706

将这些设置添加到nginx.conf文件即可解决该问题：

proxy_request_buffering off;
proxy_buffering off;

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。

相关推荐

导入项目后报错问题
依赖报错 idea导入项目后依赖报错，解决方案：https://blog.csdn.net/weixin_42420249/article/details/81191861 依赖版本报错：更换其他版本 无法下载依赖可参考：https://blog.csdn.net/weixin_42628809/a
idea不能识别yaml文件
使用mybatis plus常见错误
错误1：代码生成器依赖和mybatis依赖冲突 启动项目时报错如下 2021-12-03 13:33:33.927 ERROR 7228 [ main] o.s.b.d.LoggingFailureAnalysisReporter : *************************** APPL
gradle常见问题与错误
错误1：gradle项目控制台输出为乱码 # 解决方案：https://blog.csdn.net/weixin_43501566/article/details/112482302 # 在gradle-wrapper.properties 添加以下内容 org.gradle.jvmargs=-Df
Mybatis Plus传入参数0不起作用
错误还原：在查询的过程中，传入的workType为0时，该条件不起作用 &lt;select id=&quot;xxx&quot;&gt; SELECT di.id, di.name, di.work_type, di.updated... &lt;where&gt; &lt;if test=&qu
linux中make编译源码包失败
报错如下，gcc版本太低 ^ server.c:5346:31: 错误：‘struct redisServer’没有名为‘server_cpulist’的成员 redisSetCpuAffinity(server.server_cpulist); ^ server.c: 在函数‘hasActiveC
微服务启动错误：Command line is too long. Shorten command line for
解决方案1 1、改项目中.idea/workspace.xml配置文件，增加dynamic.classpath参数 2、搜索PropertiesComponent，添加如下 &lt;property name=&quot;dynamic.classpath&quot; value=&quot;tru
vue常见错误
删除根组件app.vue中的默认代码后报错：Module Error (from ./node_modules/eslint-loader/index.js): 解决方案：关闭ESlint代码检测，在项目根目录创建vue.config.js，在文件中添加 module.exports = { lin
spark常见错误
查看spark默认的python版本 [root@master day27]# pyspark /home/software/spark-2.3.4-bin-hadoop2.7/conf/spark-env.sh: line 2: /usr/local/hadoop/bin/hadoop: No s
matplotlib报错：AttributeError: module 'backend_interagg' has no attribute 'FigureCanvas'. Did you mean: 'FigureCanvasAgg'?
使用本地python环境可以成功执行 import pandas as pd import matplotlib.pyplot as plt # 设置字体 plt.rcParams[&#39;font.sans-serif&#39;] = [&#39;SimHei&#39;] # 能正确显示负号 p
gitlab登录失败，报错：This challenge page was accidentally cached by an intermediary and is no longer available.
设置时间 控制面板
后端开发常见错误
错误1：Request method ‘DELETE‘ not supported 错误还原：controller层有一个接口，访问该接口时报错：Request method ‘DELETE‘ not supported 错误原因：没有接收到前端传入的参数，修改为如下 参考 错误2：cannot r
docker常见错误
错误1：启动docker镜像时报错：Error response from daemon: driver failed programming external connectivity on endpoint quirky_allen 解决方法：重启docker -&gt; systemctl r
idea常见错误
错误1：private field ‘xxx‘ is never assigned 按Altʾnter快捷键，选择第2项 参考：https://blog.csdn.net/shi_hong_fei_hei/article/details/88814070 错误2：启动时报错，不能找到主启动类 #
pip安装依赖失败
报错如下，通过源不能下载，最后警告pip需升级版本 Requirement already satisfied: pip in c:\users\ychen\appdata\local\programs\python\python310\lib\site-packages (22.0.4) Coll
maven常见错误
错误1：maven打包报错 错误还原：使用maven打包项目时报错如下 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.2.0:resources (default-resources)
cloud项目中常见错误
错误1：服务调用时报错 服务消费者模块assess通过openFeign调用服务提供者模块hires 如下为服务提供者模块hires的控制层接口 @RestController @RequestMapping(&quot;/hires&quot;) public class FeignControl
springboot常见错误
错误1：运行项目后报如下错误 解决方案 报错2：Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project sb 解决方案：在pom.
springmvc拦截器中使用redisTemplate报空指针异常
参考 错误原因 过滤器或拦截器在生效时，redisTemplate还没有注入 解决方案：在注入容器时就生效 @Component //项目运行时就注入Spring容器 public class RedisBean { @Resource private RedisTemplate&lt;String
vite报错：npm ERR! command C:WINDOWSsystem32cmd.exe /d /s /c C:UsersychenAppDataLocalTempnpx-dde1c848.cmd
使用vite构建项目报错 C:\Users\ychen\work&gt;npm init @vitejs/app @vitejs/create-app is deprecated, use npm init vite instead C:\Users\ychen\AppData\Local\npm-
pythonJavaScriptjavaHTMLPHPreactjsC#AndroidCSSNode.jssqlrpython-3.xMysqLjQueryc++pandasFlutterangularIOSdjangolinuxswifttypescript路由器JSON路由器设置无线路由器h3c华三华三路由器设置华三路由器电脑软件教程arraysdocker软件图文教程Cvue.jslaravelspring-boot