Fix index error when profiling token generation with `-ct 1` #1898

lvhan028 · 2024-07-02T07:36:46Z

Traceback (most recent call last):
  File "profile_generation.py", line 482, in <module>
    main()
  File "profile_generation.py", line 435, in main
    output = _process_map(profile_target, (args.model_path, ))
  File "profile_generation.py", line 377, in _process_map
    raise ret
IndexError: index 0 is out of bounds for axis 0 with size 0

zhulinJulia24 · 2024-07-02T07:52:09Z


(lmdeploy) [zhulin1@SH-IDC1-10-140-0-187 lmdeployLvhan]$ python benchmark/profile_generation.py /nvme/qa_test_models/meta-llama/Meta-Llama-3-8B-Instruct --tp 1 -c 8 256 -ct 256 2048 -pt 1
Traceback (most recent call last):
  File "/home/zhulin1/lmdeployLvhan/benchmark/profile_generation.py", line 492, in <module>
    main()
  File "/home/zhulin1/lmdeployLvhan/benchmark/profile_generation.py", line 396, in main
    assert len(args.prompt_tokens) == len(args.completion_tokens), \
AssertionError: mismatched size between `prompt-tokens` and `completion-tokenes`, 1 vs 2

AllentDan · 2024-07-02T08:22:04Z

benchmark/profile_generation.py

-    ]
-
-    throughput = np.round(token_latency_stats.size / elapsed_time, 2)
+    if output_seqlen > 1:


Do we need to support this? Prompts with output_len<4 were filtered.

fix profile_generation benchmark

ade2aa6

lvhan028 added the Bug:P1 label Jul 2, 2024

lvhan028 requested review from AllentDan and zhulinJulia24 July 2, 2024 07:36

AllentDan reviewed Jul 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix index error when profiling token generation with `-ct 1` #1898

Fix index error when profiling token generation with `-ct 1` #1898

Fix index error when profiling token generation with -ct 1 #1898

Are you sure you want to change the base?

Fix index error when profiling token generation with -ct 1 #1898

Conversation

Choose a reason for hiding this comment

Fix index error when profiling token generation with `-ct 1` #1898

Fix index error when profiling token generation with `-ct 1` #1898