I've evaluated other LLMs like Claude, PaLM, etc. In evaluation, I used the following code to convert the multi-round message used for ChatGPT into a single round prompt:

prompt = ""
for turn in messages:
    if turn['role'] == 'system':
        prompt += f"{turn['content']}\n\n"
    elif turn['role'] == 'user':
        prompt += f"{turn['content']}\n\n"
prompt += "The ranking results of the 20 passages (only identifiers) is:"

I used the models to re-rank the top-20 BM25 passages on TREC DL-19, here are the results I got for reference:

I haven't yet evaluated the latest LLMs like LLaMA-2, Claude-2, etc.

Since the APIs of the different LLMs vary significantly, I've conducted individual evaluations for each model. It would be great if the relevant code could be uploaded to help the use of LLMs other than ChatGPT :)