pyspark - Spark - Register model objects with Kyro - Caused by: java.lang.IllegalArgumentException: Class is not registered: -

i registering classes has business logic , model classes kyro in spark . below exception

> job aborted due stage failure: task 14 in stage 1.0 failed 4 times, > recent failure: lost task 14.3 in stage 1.0 (tid 90, **): > java.lang.illegalargumentexception: class not registered: object[] > note: register class use: kryo.register(object[].class);  @ > com.esotericsoftware.kryo.kryo.getregistration(kryo.java:442)     @ > com.esotericsoftware.kryo.util.defaultclassresolver.writeclass(defaultclassresolver.java:79) >   @ com.esotericsoftware.kryo.kryo.writeclass(kryo.java:472)     @ > com.esotericsoftware.kryo.kryo.writeclassandobject(kryo.java:565)     @ > org.apache.spark.serializer.kryoserializerinstance.serialize(kryoserializer.scala:296) >   @ > org.apache.spark.executor.executor$taskrunner.run(executor.scala:239) >   @ > java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142) >   @ > java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617) >   @ java.lang.thread.run(thread.java:745) >  > driver stacktrace: 

kyro registrtor :

public class kyroserializer implements kryoregistrator {          @override         public void registerclasses(kryo kryo) {             kryo.register(people.class);             kryo.register(lookup.class);          }     } 

model class:

    class people implements serializable{         private static final long serialversionuid = 1l; ...... }      public class lookup implements serializable{         private static final long serialversionuid = 1l;          private string code1;         private string code2;  } 

finally spark context :

sc.set("spark.kryo.registrator", kyroserializer.class.getname())

from exception, seems kryo has not registered class object object[] (array entries of type object). please try change code follows:

public class kyroserializer implements kryoregistrator {      @override     public void registerclasses(kryo kryo) {         kryo.register(object[].class);     // add line class         kryo.register(people.class);         kryo.register(lookup.class);     } } 

in addition, register custom registrator class spark this

sc.set("spark.kryo.registrator", kyroserializer.class.getcanonicalname()); 

if works you, though, ignore last remark.