JsonORCFileReaderWriterFactory crashes for null/non-existing values #369
Comments
attilagenc
pushed a commit
to attilagenc/secor
that referenced
this issue
Oct 13, 2017
|
Fix is provided in this (#370) pull request. |
HenryCaiHaiying
added a commit
that referenced
this issue
Oct 16, 2017
#369 fix handling null values for orc file generation
attilagenc
pushed a commit
to attilagenc/secor
that referenced
this issue
Oct 19, 2017
|
In my tests batch size was always 1 and my previous fix was to set index value to 0, correct fix should be to use "rowIndex" parameter as index. New pull request fixes this mistake (#371) |
HenryCaiHaiying
added a commit
that referenced
this issue
Oct 22, 2017
JsonORCFileReaderWriterFactory crashes for null/non-existing values #369
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When you try to use JsonORCFileReaderWriterFactory to generate ORC Files and input message does not contain a value for a column specified in template, secor crashes.
Here is a sample crash and stack trace:
INFO: Cleaning up!
2017-10-13 16:04:54.990 [Thread-3] ERROR (ConsumerMain.java:65) - Thread Thread[Thread-3,5,main] failed
java.lang.RuntimeException: Failed to apply upload policy
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:117)
at com.pinterest.secor.consumer.Consumer.run(Consumer.java:107)
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.io.Text.set(Text.java:225)
at org.apache.orc.impl.StringRedBlackTree.add(StringRedBlackTree.java:62)
at org.apache.orc.impl.WriterImpl$StringTreeWriter.writeBatch(WriterImpl.java:1459)
at org.apache.orc.impl.WriterImpl$StructTreeWriter.writeRootBatch(WriterImpl.java:2098)
at org.apache.orc.impl.WriterImpl.addRowBatch(WriterImpl.java:2811)
at com.pinterest.secor.io.impl.JsonORCFileReaderWriterFactory$JsonORCFileWriter.close(JsonORCFileReaderWriterFactory.java:168)
at com.pinterest.secor.common.FileRegistry.deleteWriter(FileRegistry.java:197)
at com.pinterest.secor.common.FileRegistry.deleteWriters(FileRegistry.java:218)
at com.pinterest.secor.common.FileRegistry.deleteWriters(FileRegistry.java:208)
at com.pinterest.secor.uploader.Uploader.uploadFiles(Uploader.java:104)
at com.pinterest.secor.uploader.Uploader.checkTopicPartition(Uploader.java:225)
at com.pinterest.secor.uploader.Uploader.applyPolicy(Uploader.java:257)
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:115)
... 1 more
2017-10-13 16:04:54.990 [Thread-4] ERROR (ConsumerMain.java:65) - Thread Thread[Thread-4,5,main] failed
java.lang.RuntimeException: Failed to apply upload policy
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:117)
at com.pinterest.secor.consumer.Consumer.run(Consumer.java:107)
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.io.Text.set(Text.java:225)
at org.apache.orc.impl.StringRedBlackTree.add(StringRedBlackTree.java:62)
at org.apache.orc.impl.WriterImpl$StringTreeWriter.writeBatch(WriterImpl.java:1459)
at org.apache.orc.impl.WriterImpl$StructTreeWriter.writeRootBatch(WriterImpl.java:2098)
at org.apache.orc.impl.WriterImpl.addRowBatch(WriterImpl.java:2811)
at com.pinterest.secor.io.impl.JsonORCFileReaderWriterFactory$JsonORCFileWriter.close(JsonORCFileReaderWriterFactory.java:168)
at com.pinterest.secor.common.FileRegistry.deleteWriter(FileRegistry.java:197)
at com.pinterest.secor.common.FileRegistry.deleteWriters(FileRegistry.java:218)
at com.pinterest.secor.common.FileRegistry.deleteWriters(FileRegistry.java:208)
at com.pinterest.secor.uploader.Uploader.uploadFiles(Uploader.java:104)
at com.pinterest.secor.uploader.Uploader.checkTopicPartition(Uploader.java:225)
at com.pinterest.secor.uploader.Uploader.applyPolicy(Uploader.java:257)
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:115)
... 1 more
The text was updated successfully, but these errors were encountered: