-
Notifications
You must be signed in to change notification settings - Fork 543
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JsonORCFileReaderWriterFactory crashes for null/non-existing values #369
Comments
attilagenc
pushed a commit
to attilagenc/secor
that referenced
this issue
Oct 13, 2017
Fix is provided in this (#370) pull request. |
HenryCaiHaiying
added a commit
that referenced
this issue
Oct 16, 2017
#369 fix handling null values for orc file generation
attilagenc
pushed a commit
to attilagenc/secor
that referenced
this issue
Oct 19, 2017
In my tests batch size was always 1 and my previous fix was to set index value to 0, correct fix should be to use "rowIndex" parameter as index. New pull request fixes this mistake (#371) |
HenryCaiHaiying
added a commit
that referenced
this issue
Oct 22, 2017
JsonORCFileReaderWriterFactory crashes for null/non-existing values #369
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When you try to use JsonORCFileReaderWriterFactory to generate ORC Files and input message does not contain a value for a column specified in template, secor crashes.
Here is a sample crash and stack trace:
INFO: Cleaning up!
2017-10-13 16:04:54.990 [Thread-3] ERROR (ConsumerMain.java:65) - Thread Thread[Thread-3,5,main] failed
java.lang.RuntimeException: Failed to apply upload policy
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:117)
at com.pinterest.secor.consumer.Consumer.run(Consumer.java:107)
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.io.Text.set(Text.java:225)
at org.apache.orc.impl.StringRedBlackTree.add(StringRedBlackTree.java:62)
at org.apache.orc.impl.WriterImpl$StringTreeWriter.writeBatch(WriterImpl.java:1459)
at org.apache.orc.impl.WriterImpl$StructTreeWriter.writeRootBatch(WriterImpl.java:2098)
at org.apache.orc.impl.WriterImpl.addRowBatch(WriterImpl.java:2811)
at com.pinterest.secor.io.impl.JsonORCFileReaderWriterFactory$JsonORCFileWriter.close(JsonORCFileReaderWriterFactory.java:168)
at com.pinterest.secor.common.FileRegistry.deleteWriter(FileRegistry.java:197)
at com.pinterest.secor.common.FileRegistry.deleteWriters(FileRegistry.java:218)
at com.pinterest.secor.common.FileRegistry.deleteWriters(FileRegistry.java:208)
at com.pinterest.secor.uploader.Uploader.uploadFiles(Uploader.java:104)
at com.pinterest.secor.uploader.Uploader.checkTopicPartition(Uploader.java:225)
at com.pinterest.secor.uploader.Uploader.applyPolicy(Uploader.java:257)
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:115)
... 1 more
2017-10-13 16:04:54.990 [Thread-4] ERROR (ConsumerMain.java:65) - Thread Thread[Thread-4,5,main] failed
java.lang.RuntimeException: Failed to apply upload policy
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:117)
at com.pinterest.secor.consumer.Consumer.run(Consumer.java:107)
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.io.Text.set(Text.java:225)
at org.apache.orc.impl.StringRedBlackTree.add(StringRedBlackTree.java:62)
at org.apache.orc.impl.WriterImpl$StringTreeWriter.writeBatch(WriterImpl.java:1459)
at org.apache.orc.impl.WriterImpl$StructTreeWriter.writeRootBatch(WriterImpl.java:2098)
at org.apache.orc.impl.WriterImpl.addRowBatch(WriterImpl.java:2811)
at com.pinterest.secor.io.impl.JsonORCFileReaderWriterFactory$JsonORCFileWriter.close(JsonORCFileReaderWriterFactory.java:168)
at com.pinterest.secor.common.FileRegistry.deleteWriter(FileRegistry.java:197)
at com.pinterest.secor.common.FileRegistry.deleteWriters(FileRegistry.java:218)
at com.pinterest.secor.common.FileRegistry.deleteWriters(FileRegistry.java:208)
at com.pinterest.secor.uploader.Uploader.uploadFiles(Uploader.java:104)
at com.pinterest.secor.uploader.Uploader.checkTopicPartition(Uploader.java:225)
at com.pinterest.secor.uploader.Uploader.applyPolicy(Uploader.java:257)
at com.pinterest.secor.consumer.Consumer.checkUploadPolicy(Consumer.java:115)
... 1 more
The text was updated successfully, but these errors were encountered: