[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File writer that includes message metadata #362

Closed
jlm3575 opened this issue Sep 11, 2017 · 1 comment
Closed

File writer that includes message metadata #362

jlm3575 opened this issue Sep 11, 2017 · 1 comment

Comments

@jlm3575
Copy link
jlm3575 commented Sep 11, 2017

Is there currently a supported way to include message metadata in the delimited text output (such as offset, topic, partition, and timestamp)? After searching through the code a bit, it seems like that isn't supported without some customization on our end.

We'd like to output these fields as part of each logged message value for searching (such as indexing on it in elastic) and programatic replay (finding a particularly small window of offsets in a file).

Have I just overlooked an obvious setting that can accomplish this, and if not, would it be something useful enough to submit in a pull request?

Thanks!

@HenryCaiHaiying
Copy link
Contributor

We usually don't keep those metadata on per message level. Some of the metatdata (e.g. topic/partition, starting offset was kept per file level).

If you want to do it on per message level, you can implement the MessageTransformer class to do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants