Hi @wendlerc it really depends on the implementation and hardware. I did the Scandinavian segment in less than an hour on a v100 (just to give you some approximate) and that was using 16bit using the naive transformers implementation.

@Muennighoff might have a better estimate or suggestions for how to do it fast.

I think I might have a training-free method to turn any autoregressive model into a sentence embedding model and would like to do a quick evaluation.

If there is a representative subset of tasks that people particularly care about that would also be helpful, or if anybody wants to get on board & help running evals.

I have specifically designed the Scandinavian embedding benchmark to be small. For testing purposes, I would probably go for that one. If you have your method wrapped in an encode(sentences: list[str], ...) -> np.array method I can def. help you there.

Alternatively, you use the table on the front page to select ~10 random datasets with relatively few samples.

Will implement the encode method tomorrow & keep you posted.

As promised, here is a quick and dirty implementation:

https://github.com/wendlerc/llama2-embeddings

Batch size is currently hard-coded and also a maximum sequence length in the tokenization step.
The original code was meant for batch size 1, so I am not sure whether I broke something by updating it to support larger batches. Also currently I log a ton of unnecessary activations for this task.

{'EmotionClassification': {'mteb_version': '1.2.0', 'dataset_revision': '4f58c6b202a23cf9a4da393831edf4f9183cad37', 'mteb_dataset_name': 'EmotionClassification', 'validation': {'accuracy': 0.21234999999999998, 'f1': 0.18402314056204205, 'accuracy_stderr': 0.03751336428527839, 'f1_stderr': 0.02261561372267123, 'main_score': 0.21234999999999998, 'evaluation_time': 4434.28}, 'test': {'accuracy': 0.20655, 'f1': 0.17523029518237007, 'accuracy_stderr': 0.03902463965240423, 'f1_stderr': 0.021088847119969625, 'main_score': 0.20655, 'evaluation_time': 4398.19}}}

I guess maybe the method needs some more work.