Trimmed files are uploaded without configured file extension #358
Comments
|
Below is the current logic in MessageWriter.java. This probably needs to
be consistent with trim routine. I don't have too much problem of keeping
both snapp.parquet, but it seems it will break the backward compatibility
of existing path generation. So if you like snappy.parquet, go ahead with
a PR but make sure to introduce a new config flag to keep the backward
compatiblity. Or the workaround is to make secor.file.extenstion config
param == 'snappy.parquet'.
if (mConfig.getCompressionCodec() != null &&
!mConfig.getCompressionCodec().isEmpty()) {
mCodec = CompressionUtil.createCompressionCodec(mConfig.getCompressionCodec());
mFileExtension = mCodec.getDefaultExtension();
}
if (mConfig.getFileExtension() != null &&
!mConfig.getFileExtension().isEmpty()) {
mFileExtension = mConfig.getFileExtension();
…On Tue, Aug 8, 2017 at 2:05 PM, Igor Calabria ***@***.***> wrote:
With a little investigation, I found out that the files with missing
extensions were always trimmed. Looking at this snippet from
com.pinterest.secor.uploader.Uploader#trim you can clearly see that
extension is only set if a compression codec is configured:
String extension = "";
if (mConfig.getCompressionCodec() != null && !mConfig.getCompressionCodec().isEmpty()) {
codec = CompressionUtil.createCompressionCodec(mConfig.getCompressionCodec());
extension = codec.getDefaultExtension();
}
reader = createReader(srcPath, codec);
KeyValue keyVal;
while ((keyVal = reader.next()) != null) {
if (keyVal.getOffset() >= startOffset) {
if (writer == null) {
String localPrefix = mConfig.getLocalPath() + '/' +
IdUtil.getLocalMessageDir();
dstPath = new LogFilePath(localPrefix, srcPath.getTopic(),
srcPath.getPartitions(), srcPath.getGeneration(),
srcPath.getKafkaPartition(), startOffset,
extension);
writer = mFileRegistry.getOrCreateWriter(dstPath,
codec);
}
writer.write(keyVal);
copiedMessages++;
}
}
This may seem like a harmless bug, but it affects people that use a glob
pattern to process files. Example: s3a://secor-backups/some-
topic/dt=2017-08-08/*.parquet. If there's a trimmed file in there, it
will be missed by the pattern.
The solution is pretty straight forward(set extension by calling
mConfig.getFileExtension()). I'll gladly submit a patch, but I need to
know the desired behavior of extensions names when compression is enabled.
Do we keep both extensions? file.parquet.snappy or just the compressed
extension file.snappy ? My vote goes to keeping both extensions since it
is default with spark and other big data tools.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#358>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHCX_wUBMaO7oMcPnrHwCiYLfjzurUAaks5sWM2ygaJpZM4OxSv6>
.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With a little investigation, I found out that the files with missing extensions were always trimmed. Looking at this snippet from
com.pinterest.secor.uploader.Uploader#trimyou can clearly see that extension is only set if a compression codec is configured:This may seem like a harmless bug, but it affects people that use a glob pattern to process files. Example:
s3a://secor-backups/some-topic/dt=2017-08-08/*.parquet. If there's a trimmed file in there, it will be missed by the pattern.The solution is pretty straight forward(set extension by calling
mConfig.getFileExtension()). I'll gladly submit a patch, but I need to know the desired behavior of extensions names when compression is enabled. Do we keep both extensions?file.parquet.snappyor just the compressed extensionfile.snappy? My vote goes to keeping both extensions since it is default with spark and other big data tools.The text was updated successfully, but these errors were encountered: