On Tue, Dec 11, 2018 at 8:14 PM sayuan ***@***.***> wrote: When trimFiles[1] happen, it need the offset[2] of each massage. [1]: https://github.com/pinterest/secor/blob/master/src/main/java/com/pinterest/secor/uploader/Uploader.java#L258 [2]: https://github.com/pinterest/secor/blob/master/src/main/java/com/pinterest/secor/uploader/Uploader.java#L177 If one partition has multiple (partitioned) files, it will retrieve the incorrect offset. https://github.com/pinterest/secor/blob/master/src/main/java/com/pinterest/secor/io/impl/DelimitedTextFileReaderWriterFactory.java#L91 Therefore, it will filter out logs incorretctly and cause the log drop. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#476>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHCX_6yQZhW45JDWJnNFfXkrYUiDsv4mks5u4ILAgaJpZM4ZOs9C> .

On Wed, Dec 12, 2018 at 1:59 AM sayuan ***@***.***> wrote: Here is the list of Reader which doesn't manage the offset correctly: # git grep -l offset++ src/main/java/com/pinterest/secor/io/impl src/main/java/com/pinterest/secor/io/impl/AvroFileReaderWriterFactory.java src/main/java/com/pinterest/secor/io/impl/AvroParquetFileReaderWriterFactory.java src/main/java/com/pinterest/secor/io/impl/JsonORCFileReaderWriterFactory.java src/main/java/com/pinterest/secor/io/impl/ProtobufParquetFileReaderWriterFactory.java src/main/java/com/pinterest/secor/io/impl/ThriftParquetFileReaderWriterFactory.java``` — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#476 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHCX_2v4CcGfE_jj5kl6WcKvsmbKlufbks5u4NOWgaJpZM4ZOs9C> .