-
Notifications
You must be signed in to change notification settings - Fork 543
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Trimming partitioned files cause log drop #476
Comments
I think this problem is specific to DelimitedTextFileReaderWriter which
didn't have a proper management of kafka offsets.
…On Tue, Dec 11, 2018 at 8:14 PM sayuan ***@***.***> wrote:
When trimFiles[1] happen, it need the offset[2] of each massage.
[1]:
https://github.com/pinterest/secor/blob/master/src/main/java/com/pinterest/secor/uploader/Uploader.java#L258
[2]:
https://github.com/pinterest/secor/blob/master/src/main/java/com/pinterest/secor/uploader/Uploader.java#L177
If one partition has multiple (partitioned) files, it will retrieve the
incorrect offset.
https://github.com/pinterest/secor/blob/master/src/main/java/com/pinterest/secor/io/impl/DelimitedTextFileReaderWriterFactory.java#L91
Therefore, it will filter out logs incorretctly and cause the log drop.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#476>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHCX_6yQZhW45JDWJnNFfXkrYUiDsv4mks5u4ILAgaJpZM4ZOs9C>
.
|
Here is the list of Reader which doesn't manage the offset correctly:
|
I think the problem is DelimitedTextFileReaderWriter didn't store the
original kafka offset in the file so it couldn't skip to the right offset
when trimming happens.
…On Wed, Dec 12, 2018 at 1:59 AM sayuan ***@***.***> wrote:
Here is the list of Reader which doesn't manage the offset correctly:
# git grep -l offset++ src/main/java/com/pinterest/secor/io/impl
src/main/java/com/pinterest/secor/io/impl/AvroFileReaderWriterFactory.java
src/main/java/com/pinterest/secor/io/impl/AvroParquetFileReaderWriterFactory.java
src/main/java/com/pinterest/secor/io/impl/JsonORCFileReaderWriterFactory.java
src/main/java/com/pinterest/secor/io/impl/ProtobufParquetFileReaderWriterFactory.java
src/main/java/com/pinterest/secor/io/impl/ThriftParquetFileReaderWriterFactory.java```
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#476 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHCX_2v4CcGfE_jj5kl6WcKvsmbKlufbks5u4NOWgaJpZM4ZOs9C>
.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When
trimFiles
[1] happen, it need theoffset
[2] of each massage.[1]: https://github.com/pinterest/secor/blob/master/src/main/java/com/pinterest/secor/uploader/Uploader.java#L258
[2]: https://github.com/pinterest/secor/blob/master/src/main/java/com/pinterest/secor/uploader/Uploader.java#L177
If one partition has multiple (partitioned) files, it will retrieve the incorrect offset.
https://github.com/pinterest/secor/blob/master/src/main/java/com/pinterest/secor/io/impl/DelimitedTextFileReaderWriterFactory.java#L91
Therefore, it will filter out logs incorretctly and cause the log drop.
The text was updated successfully, but these errors were encountered: