2012-09-05 70 views
1

据我所知,Kryo在每创建一个className < - >numberID地图。这张地图太狭窄了。因为在你的对象模型中,实例往往属于同一个类,下一个writeObject将会再次构建并序列化一个类似的地图(并且再次,再次,再次)。我知道可以通过手动注册这些类来共享地图,但这是一种繁琐的手动硬编码。我希望地图可以通过第一次写入对象来启动,就像通常那样,但会话中的所有后续写入操作都会重新使用并扩展它。这样,注册将在运行时自动发生,无需额外的运行时间开销,而更频繁使用的对象自然会收到低ID号。之后,地图可以作为解密密钥分别存储在附件中。反序列化器将首先加载这张地图。你如何喜欢这个想法,以及如何实施?Kryo自动班级注册

我的问题与此类似 Strategy for registering classes with kryo其中用户可以使用List将单个writeObject下的所有写入组合在一起。正如我所建议的那样,这将比单独存储地图更简单。但是,他似乎不想这样做。在我的情况下,这种组合是不可能的,因为大的java模型,我避免通过分片序列化将它完全保存在内存中。在我的场景中,用户打开一个项目,进行更改并刷新它们。所以,该项目可以维护一个类的映射并将其用于所有序列化。

更新!我意识到有类/对象注册和autoReset。他们似乎被创造出来适合这项任务。但是,我不明白这些东西如何解决它。 Autoreset=false确实使第二次写入确实小得多。但是,在这种情况下,我无法反序列化对象。正如你例子看,第二反序列化失败:

public class A { 
    String f1; 
    A(String a) { 
     f1 = a; 
    } 
    List list = new ArrayList(); 
    public String toString() { 
     return "f1 = " + f1 + ":" + f1.getClass().getSimpleName(); 
    } 

    public static void main(String[] args) { 
     test(true); 
     test(false); 
    } 


    static void write(String time, Kryo kryo, ByteArrayOutputStream baos, Object o) { 
     Output output = new Output(baos); 
     kryo.writeClassAndObject(output, o); 
     output.close(); 
     System.err.println(baos.size() + " after " + time + " write"); 
    } 

    private static void test(boolean autoReset) { 
     Kryo kryo = new Kryo(); 
     kryo.setAutoReset(autoReset); 
     kryo.setInstantiatorStrategy(new StdInstantiatorStrategy()); 
     System.err.println("-------\ntesting autoreset = " + autoReset); 
     ByteArrayOutputStream baos = new ByteArrayOutputStream(); 
     A a = new A("a"), b = new A("b"); 
     write("first", kryo, baos, a); 
     write("second", kryo, baos, b); 
     A o1 = restore("first", baos, kryo); 
     A o2 = restore("second", baos, kryo); // this fails 
     System.err.println((o1.f1.equals(o2.f1)) ? "SUCCESS" : "FAILURE"); 

    } 

    private static A restore(String time, ByteArrayOutputStream baos, Kryo k) { 
     ByteArrayInputStream in = new ByteArrayInputStream(baos.toByteArray()); 
     Input input = new Input(in); 
     A o = (A) k.readClassAndObject(input); 
     System.err.println("reading object " + time + " time, got " + o); 
     return o; 
    } 

输出是

------- 
testing autoreset = true 
41 after first write 
82 after second write 
reading object first time, got f1 = a:String 
reading object second time, got f1 = a:String 
SUCCESS 
------- 
testing autoreset = false 
41 after first write 
52 after second write 
reading object first time, got f1 = a:String 
reading object second time, got null 
Exception in thread "main" java.lang.NullPointerException 
    at kryo_test.AutoresetDemo.test(AutoresetDemo.java:40) 
    at kryo_test.AutoresetDemo.main(AutoresetDemo.java:18) 

UPDATE2它也可能发生自动复位=虚假记载对象除了类名的引用。确实值得自动复位。

Update3我发现很难序列化类的映射(即类 - >注册),因为注册包含引用kryo对象并保持某种状态的序列化程序。然后,很难在很多kryo对象之间共享地图。

回答

2

好吧,这里是KRYO 2.20

public class GlobalClassKryo extends Kryo { 

    public static class ExternalizableClassResolver implements ClassResolver { 

     //local serializers 
     final Map<Class, Registration> fromClass = new HashMap(); 
     final Map<Integer, Registration> fromId = new HashMap(); 

     public static class GlobalRegistration { int id; Class type; Class<? extends Serializer> serializer; } 

     public final Map<Integer, GlobalRegistration> globalIds; 
     public final Map<Class, GlobalRegistration> globalClasses; 

     // I synchronize because I have one reader and one writer thread and 
     // writer may break the reader when adds something into the map. 
     public ExternalizableClassResolver() {this (
       Collections.synchronizedMap(new HashMap()), 
       Collections.synchronizedMap(new HashMap()) 
      ) ;} 

     public ExternalizableClassResolver(Map<Integer, GlobalRegistration> ids, Map<Class, GlobalRegistration> classes) { 
      globalIds = ids; 
      globalClasses = classes; 
     } 

     public ExternalizableClassResolver (DataInput in) throws ClassNotFoundException, IOException { 
      this(); 
      int id; 
      while ((id = in.readInt()) != -1) { 
       GlobalRegistration e = new GlobalRegistration(); 
       globalIds.put(e.id = id, e); 
       e.type = Class.forName(in.readUTF()); 
       e.serializer = (Class<? extends Serializer>) Class.forName(in.readUTF()); 
       globalClasses.put(e.type, e); 
      } 
     } 

     public void save(DataOutput out) throws IOException { 
      for (GlobalRegistration entry : globalIds.values()) { 
        out.writeInt(entry.id); 
        out.writeUTF(entry.type.getName()); 
        out.writeUTF(entry.serializer.getName()); 
      } 
      out.writeInt(-1); 
     } 

     static final boolean TRACE = false; 
     void log(String msg) { 
      System.err.println(kryo != null ? Utils.fill(kryo.getDepth(), ' ') + msg : msg); 
     } 
     @Override 
     public Registration writeClass(Output output, Class type) { 
      if (type == null) {output.writeInt(0, true); return null;} 
      Registration registration = kryo.getRegistration(type); 
      output.writeInt(registration.getId(), true); 
      return registration; 
     } 
     @Override 
     public Registration readClass(Input input) { 
      int classID = input.readInt(true); 
      if (classID == 0) return null; 
      Registration registration = fromId.get(classID); 
      if (registration == null) { 
       registration = tryGetFromGlobal(globalIds.get(classID), classID + ""); 
      } 
      if (registration == null) throw new KryoException("Encountered unregistered class ID: " + classID); 
      return registration; 
     } 

     public Registration register(Registration registration) { 
      throw new KryoException("register(registration) is not allowed. Use register(type, serializer)"); 
     } 

     public Registration getRegistration(int classID) { 
      throw new KryoException("getRegistration(id) is not implemented"); 
     } 

     Registration tryGetFromGlobal(GlobalRegistration globalClass, String title) { 
      if (globalClass != null) { 
       Serializer serializer = kryo.newSerializer(globalClass.serializer, globalClass.type); 
       Registration registration = register(globalClass.type, serializer, globalClass.id, "local"); 
       if (TRACE) log("getRegistration(" + title + ") taken from global => " + registration); 
       return registration; 
      } else 
       if (TRACE) log("getRegistration(" + title + ") was not found"); 
      return null; 
     } 
     public Registration getRegistration(Class type) { 
      Registration registration = fromClass.get(type); 
      if (registration == null) { 
       registration = tryGetFromGlobal(globalClasses.get(type), type.getSimpleName()); 
      } else 
       if (TRACE) log("getRegistration(" + type.getSimpleName() + ") => " + registration); 

      return registration; 
     } 

     Registration register(Class type, Serializer serializer, int id, String title) { 
      Registration registration = new Registration(type, serializer, id); 
      fromClass.put(type, registration); 
      fromId.put(id, registration); 

      if (TRACE) log("new " + title + " registration, " + registration); 

      //why dont' we put into fromId? 
      if (registration.getType().isPrimitive()) fromClass.put(getWrapperClass(registration.getType()), registration); 
      return registration; 
     } 

     int primitiveCounter = 1; // 0 is reserved for NULL 
     static final int PRIMITIVE_MAX = 20; 

     //here we register anything that is missing in the global map. 
     // It must not be the case that something available is registered for the second time, particularly because we do not check this here 
     // and use registered map size as identity counter. Normally, check is done prior to callig this method, in getRegistration 
     public Registration register(Class type, Serializer serializer) { 

      if (type.isPrimitive() || type.equals(String.class)) 
       return register(type, serializer, primitiveCounter++, "primitive"); 

      GlobalRegistration global = globalClasses.get(type); 

      if (global != null) 
        throw new RuntimeException("register(type,serializer): we have " + type + " in the global map, this method must not be called"); 

      global = new GlobalRegistration(); 
      globalIds.put(global.id = globalClasses.size() + PRIMITIVE_MAX, global); 
      globalClasses.put(global.type = type, global); 
      global.serializer= serializer.getClass(); 

      return register(global.type, serializer, global.id, "global"); 
     } 

     public Registration registerImplicit(Class type) { 
      throw new RuntimeException("registerImplicit is not needed since we register missing automanically in getRegistration"); 
     } 

     @Override 
     public void reset() { 
      // super.reset(); //no need to reset the classes 
     } 

     Kryo kryo; 
     public void setKryo(Kryo kryo) { 
      this.kryo = kryo; 
     } 
    } 




    public ExternalizableClassResolver ourClassResolver() { 
     return (ExternalizableClassResolver) classResolver; 
    } 

    public GlobalClassKryo(ClassResolver resolver) { 
     super(resolver, new MapReferenceResolver()); 
     setInstantiatorStrategy(new StdInstantiatorStrategy()); 
     this.setRegistrationRequired(true); 
    } 
    public GlobalClassKryo() { 
     this(new ExternalizableClassResolver()); 
    } 

    @Override 
    public Registration getRegistration (Class type) { 
     if (type == null) throw new IllegalArgumentException("type cannot be null."); 

     if (type == memoizedClass) return memoizedClassValue; 
     Registration registration = classResolver.getRegistration(type); 
     if (registration == null) { 
      if (Proxy.isProxyClass(type)) { 
       // If a Proxy class, treat it like an InvocationHandler because the concrete class for a proxy is generated. 
       registration = getRegistration(InvocationHandler.class); 
      } else if (!type.isEnum() && Enum.class.isAssignableFrom(type)) { 
       // This handles an enum value that is an inner class. Eg: enum A {b{}}; 
       registration = getRegistration(type.getEnclosingClass()); 
      } else if (EnumSet.class.isAssignableFrom(type)) { 
       registration = classResolver.getRegistration(EnumSet.class); 
      } 
      if (registration == null) { 
       //registration = classResolver.registerImplicit(type); 
       return register(type, getDefaultSerializer(type)); 
      } 
     } 
     memoizedClass = type; 
     memoizedClassValue = registration; 
     return registration; 
    } 

    public Registration register(Class type, Serializer serializer) { 
     return ourClassResolver().register(type, serializer);} 

    public Registration register(Registration registration) { 
     throw new RuntimeException("only register(Class, Serializer) is allowed");} 

    public Registration register(Class type) { 
     throw new RuntimeException("only register(Class, Serializer) is allowed");} 

    public Registration register(Class type, int id) { 
     throw new RuntimeException("only register(Class, Serializer) is allowed");} 

    public Registration register(Class type, Serializer serializer, int id) { 
     throw new RuntimeException("only register(Class, Serializer) is allowed"); 
    } 

    static void write(String title, Kryo k, ByteArrayOutputStream baos, Object obj) { 
     Output output = new Output(baos); 
     k.writeClassAndObject(output, obj); 
     output.close(); 
     System.err.println(baos.size() + " bytes after " + title + " write"); 
    } 
    static class A { 
     String field = "abcABC"; 
     A a = this; 
     //int b = 1; // adds 1 byte to serialization 
     @Override 
     public String toString() { 
      return field 
        + " " + list.size() 
        //+ ", " + b 
        ; 
     } 

     // list adds 3 bytes to serialization, two 3-byte string items add additionally 10 bytes in total 
     ArrayList list = new ArrayList(100); // capacity is trimmed in serialization 
     { 
      list.add("LLL"); 
      list.add("TTT"); 

     } 
    } 

    private static void test() throws IOException, ClassNotFoundException { 
     GlobalClassKryo k = new GlobalClassKryo(); 

     ByteArrayOutputStream baos = new ByteArrayOutputStream(); 

     write("first", k, baos, new A()); // write takes 24 byts 

     //externalize the map 
     ByteArrayOutputStream mapOut = new ByteArrayOutputStream(); 
     DataOutputStream dataOut = new DataOutputStream(mapOut); 
     k.ourClassResolver().save(dataOut); 
     dataOut.close(); 

     //deserizalize the map 
     DataInputStream serialized = new DataInputStream(new ByteArrayInputStream(mapOut.toByteArray())); 
     ExternalizableClassResolver resolver2 = new ExternalizableClassResolver(serialized); 

     //use the map 
     k = new GlobalClassKryo(resolver2); 
     write("second", k, baos, new A()); // 24 bytes 

     Input input = new Input(new ByteArrayInputStream(baos.toByteArray())); 
     Object read = k.readClassAndObject(input); 
     System.err.println("output " + read); 
    } 

    public static void main(String[] args) throws IOException, ClassNotFoundException { 
     Kryo k = new Kryo(); 
     ByteArrayOutputStream baos = new ByteArrayOutputStream(); 
     write("first", k, baos, new A()); // write takes 78 bytes 
     write("second", k, baos, new A()); // +78 bytes 
     System.err.println("----------------"); 

     test(); 
    } 
} 

溶液中生成的物流是清洁类名。不幸的是,与默认的Java序列化(2x或更多)相比,Kryo的速度太慢了,尽管它产生的流更加密集。 Kryo独自使我的样本序列化几乎缩小了10倍。你会发现在这个答案中提出的解决方案增加了3倍的额外因素。但是,在序列化兆字节的字段中,与Java Kryo一起存储到磁盘时,对于java序列化和2倍减速,我只获得2倍压缩。

+2

很高兴Kryo灵活地做你想做的事。虽然速度听起来不对,你有没有看到放缓的地方? – NateS