如何使用REST模板Spring MVC获得大型JSON而没有Java中的内存问题

当我从REST服务获取大JSON时,我收到java.lang.OutOfMemoryError：Java堆空间…

这是在〜1000调用REST服务之后发生的

    HttpEntity<ProductRequest> requestUpdate = new HttpEntity<>(request,httpHeaders);

    ResponseEntity<String> messageEntity = restTemplate.exchange(getEndPointAddress(),HttpMethod.POST,requestUpdate,String.class);

    Map<String,Object> map = gson.fromJson(Helper.cleanJson(messageEntity.getBody()),Map.class);

<pre>
java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:2694) ~[?:1.7.0_181]
    at java.lang.String.<init>(String.java:203) ~[?:1.7.0_181]
    at java.lang.StringBuilder.toString(StringBuilder.java:405) ~[?:1.7.0_181]
    at org.springframework.util.StreamUtils.copyToString(StreamUtils.java:80) ~[stormjar.jar:4.0.3]
    at org.springframework.http.converter.StringHttpMessageConverter.readInternal(StringHttpMessageConverter.java:86) ~[stonjar.jar:4.0.3]
    at org.springframework.http.converter.StringHttpMessageConverter.readInternal(StringHttpMessageConverter.java:41) ~[stonjar.jar:4.0.3]
    at org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:193) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:104) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:884) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:868) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:622) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) ~[stonjar.jar:4.0.3]
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) ~[stonjar.jar:4.0.3]
    at com.myorg.wcf.PMiddle.retrieveP(PMiddle.java:95) ~[ssar.jar:4.0.3]
    at com.myorg.RetrieveP.getPJsonFromService(RetrieveP.java:188) ~[ssar.jar:4.0.3]
    at com.myorg.RetrieveP.execute(RetrieveP.java:107) ~[ssar.jar:4.0.3]
    at org.apache.storm.daemon.executor$fn__5044$tuple_action_fn__5046.invoke(executor.clj:727) ~[storm-core-1.1.0.jar:1.1.0]
    at org.apache.storm.daemon.executor$mk_task_receiver$fn__4965.invoke(executor.clj:459) ~[storm-core-1.1.0.jar:1.1.0]
    at org.apache.storm.disruptor$clojure_handler$reify__4480.onEvent(disruptor.clj:40) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.daemon.executor$fn__5044$fn__5057$fn__5110.invoke(executor.clj:846) ~[ss-core-1.1.0.jar:1.1.0]
    at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) [ss-core-1.1.0.jar:1.1.0]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.7.0_181]</pre>

最佳答案

RestTemplate和基础Spring Bean不是为流处理设计的,它们在内存中构建了整个ResponsEntity.根据您的示例,您没有按预期使用RestTemplate.您正在读取String而不是委派响应对象Map< String,Object>反序列化到基础转换器bean.

如果JSON响应太大而无法放入您的堆中,则可以用URLConnection或任何其他HTTP客户端库替换RestTemplate,从而可以直接访问响应InputStream.这将允许您手动读取和解析响应,例如by using com.google.gson.stream.JsonReader：

URLConnection conn = // create and open
JsonReader reader = new JsonReader(new InputStreamReader(conn.getInputStream(),"UTF-8"));
// read manually using reader

您将可以通过避免容纳整个JSON主体的一个大String对象来以这种方式节省内存.