Too little data to calculate quantiles! #252

balcklive · 2024-12-18T08:02:51Z

问题描述 / Issue Description

请简要描述您遇到的问题。 / Please briefly describe the issue you encountered.

使用的工具 / Tools Used

Perf / 模型推理压测工具

执行的代码或指令 / Code or Commands Executed

请提供您执行的主要代码或指令。 / Please provide the main code or commands you executed. 例如 / For example:

from evalscope.perf.main import run_perf_benchmark

task_cfg = {"url": "http://localhost:52415/v1/chat/completions",
            "parallel": 4,
            "model": "qwen/Qwen1.5-7B-Chat",
            "number": 15,
            "api": "openai",
            "dataset": "openqa",
            "stream": True}
run_perf_benchmark(task_cfg)

python benchmark.py

错误日志 / Error Log

请粘贴完整的错误日志或控制台输出。 / Please paste the full error log or console output. 例如 / For example:

2024-12-18 15:59:48,057 - datasets - INFO - PyTorch version 2.5.1 available.
2024-12-18 15:59:48,307 - evalscope - INFO - Save the result to: ./outputs/20241218_155948/Qwen1.5-7B-Chat
2024-12-18 15:59:48,308 - evalscope - INFO - Starting benchmark...
2024-12-18 15:59:48,308 - evalscope - INFO - {
    "model": "qwen/Qwen1.5-7B-Chat",
    "model_id": "Qwen1.5-7B-Chat",
    "attn_implementation": null,
    "api": "openai",
    "tokenizer_path": null,
    "port": 8877,
    "url": "http://localhost:52415/v1/chat/completions",
    "headers": {
        "Authorization": "Bearer EMPTY"
    },
    "connect_timeout": 120,
    "read_timeout": 120,
    "api_key": "EMPTY",
    "number": 15,
    "parallel": 4,
    "rate": -1,
    "log_every_n_query": 10,
    "debug": false,
    "wandb_api_key": null,
    "name": null,
    "outputs_dir": "./outputs/20241218_155948/Qwen1.5-7B-Chat",
    "max_prompt_length": 9223372036854775807,
    "min_prompt_length": 0,
    "prompt": null,
    "query_template": null,
    "dataset": "openqa",
    "dataset_path": null,
    "frequency_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": null,
    "n_choices": null,
    "seed": 42,
    "stop": [],
    "stop_token_ids": [],
    "stream": true,
    "temperature": null,
    "top_p": null
}
2024-12-18 15:59:48,450 - evalscope - INFO - Connection successful.
2024-12-18 15:59:50,825 - evalscope - INFO - Save the data base to: ./outputs/20241218_155948/Qwen1.5-7B-Chat/benchmark_data.db
Processing: 0it [00:00, ?it/s]2024-12-18 15:59:50,831 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '盗贼天赋盗贼怎么加天赋?知道告诉一下下啦~~'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,832 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': 'Ps2比 画面好吗?差距很大吗? Ps2比psp画面好吗?差距很大吗?'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,832 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '加装方向助力我有一普桑想加装一套方向助力可以吗？？怎么改？'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,833 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '比斯巨兽的皮干什么用的？多少钱？'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,834 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '神州行手机号码成为空号了怎么办，卡里还有几十圆钱，还可以再用吗'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,834 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '最近脸上长了好多脂肪粒，这个该怎么破？ 记得前段时间你好像也有～'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,835 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '菠营养很丰富,那每天吃多少为益? 菠菜能多吃吗?菠菜营养很丰富,那每天吃多少为益?'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,836 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '哈尔滨院附近的旅店多吗？价位大概是什么样的 哈尔滨金融学院附近的旅店多吗？价位大概是什么样的'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,836 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '天津大悦城附近有什么好吃的？3个人吃'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,837 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '怎样向中的女孩子说出第一句话？说什么？ 怎样向暗恋中的女孩子说出第一句话？说什么？'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,838 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '我想买一辆童车，请问什么牌子的好？济南的专卖童车的地方在那儿？'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,838 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '小伙伴们 这个叫什么名字 求科普[玫瑰][玫瑰][玫瑰]'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,839 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '怎么把物品捐给和尚道士加声望，他们不理我'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,839 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '玩什么好合欢105级玩什么法宝好是PK的哦  合欢105级    玩什么法宝好    是PK的哦'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,840 - evalscope - ERROR - Request: {'messages': [{'role': 'user', 'content': '请问编辑数码照片的那个比较好用？ 请问编辑数码照片的软件那个比较好用？'}], 'model': 'qwen/Qwen1.5-7B-Chat', 'max_tokens': 2048, 'seed': 42, 'stop': [], 'stream': True, 'stream_options': {'include_usage': True}, 'stop_token_ids': []} failed, state_code: 400, data: {"object": "error", "message": "[{'type': 'extra_forbidden', 'loc': ('body', 'seed'), 'msg': 'Extra inputs are not permitted', 'input': 42, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'stream_options'), 'msg': 'Extra inputs are not permitted', 'input': {'include_usage': True}, 'url': 'https://errors.pydantic.dev/2.5/v/extra_forbidden'}]", "type": "BadRequestError", "param": null, "code": 400}
2024-12-18 15:59:50,840 - evalscope - INFO - {
  "Time taken for tests (s)": 0.013,
  "Number of concurrency": 4,
  "Total requests": 10,
  "Succeed requests": 0,
  "Failed requests": 10,
  "Throughput(average tokens/s)": -1,
  "Average QPS": -1,
  "Average latency (s)": -1,
  "Average time to first token (s)": -1,
  "Average time per output token (s)": -1,
  "Average input tokens per request": -1,
  "Average output tokens per request": -1,
  "Average package latency (s)": -1,
  "Average package per request": -1
}
Processing: 15it [00:01, 14.75it/s]
2024-12-18 15:59:51,845 - evalscope - INFO - 
Benchmarking summary:
+-----------------------------------+-------------------------------------------------------------+
| Key                               | Value                                                       |
+===================================+=============================================================+
| Time taken for tests (s)          | 0.015                                                       |
+-----------------------------------+-------------------------------------------------------------+
| Number of concurrency             | 4                                                           |
+-----------------------------------+-------------------------------------------------------------+
| Total requests                    | 15                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Succeed requests                  | 0                                                           |
+-----------------------------------+-------------------------------------------------------------+
| Failed requests                   | 15                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Throughput(average tokens/s)      | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average QPS                       | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average latency (s)               | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average time to first token (s)   | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average time per output token (s) | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average input tokens per request  | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average output tokens per request | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average package latency (s)       | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Average package per request       | -1                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Expected number of requests       | 15                                                          |
+-----------------------------------+-------------------------------------------------------------+
| Result DB path                    | ./outputs/20241218_155948/Qwen1.5-7B-Chat/benchmark_data.db |
+-----------------------------------+-------------------------------------------------------------+
2024-12-18 15:59:51,845 - evalscope - INFO - Too little data to calculate quantiles!

运行环境 / Runtime Environment

操作系统 / Operating System:
- Euler 21.10 LTS
Python版本 / Python Version:
- 3.10

其他信息 / Additional Information

如果有其他相关信息，请在此处提供。 / If there is any other relevant information, please provide it here.

The text was updated successfully, but these errors were encountered:

Yunnglin · 2024-12-18T11:41:07Z

你请求的url不支持额外的参数请求，包括 seed 和 stream_options，你可以看一些怎么去修改模型服务的配置

balcklive · 2024-12-19T01:45:36Z

是的，我的模型服务不接受seed和stream_options作为参数，这个只能修改模型服务吗？是否可以在evalscope这边做设置取消这两个参数呢？

Yunnglin · 2024-12-19T01:52:33Z

请问你是用的什么框架启动的服务呢

balcklive · 2024-12-19T02:00:01Z

是别人开发的一个自定义框架，服务端这边不是很方便去修改

Yunnglin added the perf label Dec 18, 2024

Yunnglin mentioned this issue Dec 19, 2024

fix perf seed #254

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Too little data to calculate quantiles! #252

Too little data to calculate quantiles! #252

balcklive commented Dec 18, 2024

Yunnglin commented Dec 18, 2024

balcklive commented Dec 19, 2024

Yunnglin commented Dec 19, 2024

balcklive commented Dec 19, 2024

Too little data to calculate quantiles! #252

Too little data to calculate quantiles! #252

Comments

balcklive commented Dec 18, 2024

问题描述 / Issue Description

使用的工具 / Tools Used

执行的代码或指令 / Code or Commands Executed

错误日志 / Error Log

运行环境 / Runtime Environment

其他信息 / Additional Information

Yunnglin commented Dec 18, 2024

balcklive commented Dec 19, 2024

Yunnglin commented Dec 19, 2024

balcklive commented Dec 19, 2024