我在内存中有一个相当大的Java对象，它表示一个图，有顶点和边。每个顶点都有一个ArrayList的其他顶点连接到它（并有一个HashMap数据结构，以及用于其他目的）。这个图可以有几千个顶点和更多的边。
当尝试使用Java的内置序列化（implements Serializable等）来序列化图时，我总是遇到StackOverflowError。将图的其他属性设置为transient没有帮助，将堆栈大小设置得更大也没有帮助（即，-Xss1g或-Xss512m）。
我不认为我需要定制writeObject方法，因为ArrayList和HashMap已经有了它们自己的实现，它们在序列化时被调用。
我的问题是：有没有一种方法可以序列化已经在内存中的大型Java对象，而不需要获得StackOverflowError？

编辑：这里是堆栈跟踪：

Exception in thread "main" java.lang.StackOverflowError
at java.lang.reflect.Method.invoke(Method.java:575)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
// Many more lines after this

下面是我的Graph类的概述：

public class Graph implements Serializable {

/**
 * 
 */
private static final long serialVersionUID = -2632163054149021990L;
private ArrayList<Vertex> vertices;

private HashMap<Integer, Set<Vertex>> map;

public Graph(int rowMax, int colMax)
{
    map = new HashMap<Integer, Set<Vertex>>();

    this.vertices = new ArrayList<Vertex>();
}

public void connectVertices(Vertex u, Vertex v)
{
    u.addNeighbor(v);
    v.addNeighbor(u);
}

// other unrelated methods after this

下面是我的Vertex类：

public class Vertex implements Serializable {

/**
 * 
 */
private static final long serialVersionUID = 8520500010710631610L;
public int row;
public int col;
private ArrayList<Vertex> neighbors; // may change this to Set<Vertex>

public Vertex(int i, int j)
{
    this.row = i;
    this.col = j;
    this.neighbors = new ArrayList<Vertex>();
}

public boolean addNeighbor(Vertex v)
{
    this.neighbors.add(v);
    return true;
}

// unrelated methods after this

编辑2：同样，对于较小的图，但有“邻居”，没有这个问题。

我知道这已经晚了好几年，但我是通过谷歌发现的，当时我正试图自己解决这个问题。
我想要一个解决方案，它需要的代码量远远少于编写自定义序列化代码。
问题是节点连接到节点连接到节点。网络的很大一部分可以从任何给定的节点到达-对于无向图来说，它是每个节点。
当你尝试序列化一个Node时，每个可到达的Node都会出现在堆栈上。
最简单的解决方案是不直接将节点连接到其邻居，而是添加一层间接层：

class Node{

    private final Map<NodeRef, Connection> forwardConnections = new HashMap<>();
    private final Map<NodeRef, Connection> reverseConnections = new HashMap<>();

    ...
}

class Connection{

    private final NodeRef source;
    private final NodeRef dest;
    private final ConnectionMetadata meta;

    public Connection(Node source, Node dest, ConnectionMetadata meta){
        this.source = new NodeRef(source);
        this.dest = new NodeRef(dest);
    }

    public Node getSource(){
        return source.resolve();
    }

    public Node getDest(){
        return dest.resolve();
    }

    public ConnectionMetadata getMeta(){
        return meta;
    }

}

public class NodeRef{

    private transient Node node;
    private final Network network;
    private final int[] uid;

    public NodeRef(Node node){
        this.node = node;
        this.network = node.getNetwork();
        this.uid = node.getUID();
    }

    //Won't be called during deserialisation
    public Node resolve(){
        if(node == null){
            node = network.resolve(uid);
        }
        return node;
    }

    //Will be called when connections maps are deserialised.
    public boolean equals(Object o){
        //you might also want to check that the networks are equal
        return (o instanceof NodeRef) && Arrays.equals(uid, ((NodeRef)o).uid);
    }

    //Will be called when connections maps are deserialised.
    public int hashCode(){
        return Arrays.hashCode(uid);
    }
}

4条答案

按热度按时间

j2cgzkjk1#

如果图的深度太大，默认序列化无法处理，则序列化将抛出StackOverflowError。这是由于默认序列化在解析图时递归地序列化每个节点。
平面结构将工作良好（例如，具有2000个子节点的父节点），但是深层结构将失败（例如，具有2000个后代级别的节点）。
例如，以下内容将堆栈溢出：

public class Node implements Serializable
{
    private ArrayList<Node> nodes = new ArrayList<Node>();

    public static void main(String[] args) throws Exception
    {
        Node node = new Node();
        int depth = 3000;

        // Add nodes chained down to specified depth
        Node last = node;
        for (int i = 0; i < depth; i++)
        {
            Node temp = new Node();
            last.nodes.add(temp);
            last = temp;
        }

        System.out.println("starting");

        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        ObjectOutputStream out = new ObjectOutputStream(bos);
        // Below line will cause a stack overflow.
        out.writeObject(node);

        System.out.println("done");
    }
}

您需要减少图形的深度以限制递归序列化调用的数量，或者编写自定义序列化来解决这个问题。自定义序列化本质上需要是非递归的，不幸的是，乍一看似乎实现起来并不容易。

赞(0）回复(0）举报 12个月前

epggiuax2#

我也遇到过类似的问题。经过大量的搜索，我发现了一个Kryo的叉子，旨在处理深度嵌套的对象。通过https://github.com/EsotericSoftware/kryo/issues/103，克隆和mvn clean installhttps://github.com/romix/kryo/tree/kryo-2.23-continuations。目前是com.esotericsoftware.kryo:kryo:2.23-SNAPSHOT。
(Side注：有很多关于SO的问题可以用这个来回答。我应该张贴答案的副本（如https://stackoverflow.com/a/43327778/513038），或评论链接到这个答案，或标记的问题重复指向这一个，或什么？）

ruoxqz4g3#

nbysray54#

你能试试https://github.com/alipay/fury吗？这是一个由jit驱动的快速序列化。jit将使序列化调用堆栈变平，这将支持更多的deep/llarge图。我测试了一个100 W节点的图，工作正常：

import io.fury.Fury;
import io.fury.ThreadSafeFury;

import java.io.Serializable;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Set;

public class Graph implements Serializable {

  private static final long serialVersionUID = -2632163054149021990L;
  private ArrayList<Vertex> vertices;

  private HashMap<Integer, Set<Vertex>> map;

  public Graph(int rowMax, int colMax) {
    map = new HashMap<>();
    this.vertices = new ArrayList<>();
  }

  public void connectVertices(Vertex u, Vertex v) {
    if (!map.containsKey(u.col)) {
      map.put(u.col, new HashSet<>());
      vertices.add(u);
    }
    if (!map.containsKey(v.col)) {
      map.put(v.col, new HashSet<>());
      vertices.add(v);
    }
    map.get(u.col).add(v);
    u.addNeighbor(v);
    v.addNeighbor(u);
  }

  private static final ThreadSafeFury fury = Fury.builder().withRefTracking(true)
    .requireClassRegistration(false).buildThreadSafeFury();

  public static void main(String[] args) {
    Graph graph = new Graph(10000, 100);
    for (int i = 0; i < 10000; i++) {
      for (int j = 0; j < 100; j+=2) {
        Vertex vertex1 = new Vertex(i, j);
        Vertex vertex2 = new Vertex(i, j+1);
        graph.connectVertices(vertex1, vertex2);
      }
    }
    byte[] bytes = fury.serialize(graph);
    System.out.println(bytes.length);
    System.out.println(fury.deserialize(bytes));
  }
}

序列化Java对象时不出现StackOverflowError

4条答案

相关问题

热门标签

最新问答