Hi, nice work! Congratulations on the Outstanding Paper award!

Having thoroughly reviewed the arXiv version, I find myself with a question about the results of "NovelEval". I concur with the idea of creating continuously updated IR test sets to ensure that the questions, passages to be ranked, and relevance annotations haven't been preemptively learned by the latest LLMs. However, without the detailed training information for GPT-4, it becomes challenging to confirm if "NovelEval" truly offers a "fair evaluation".

It's widely known that InstructGPT involves RLHF, which relies on a reward model that's pre-trained on human-labeled ranking loss. But it's unclear whether this same approach is used in the SFT or in the undisclosed training strategy for subsequent GPT models. I'm left wondering if GPT-4's superiority stems from its overall capacity or if it's simply due to the use of "Ranking loss" in its training. If it's the latter, then the seemingly magical Ranking becomes much more straightforward.

Looking forward to your reply.