Hardware Requirements (Minimum)

• CPU architecture: x86_64

• Disk: 200G

Software Requirements

• Operating system: Windows 11/10

3.1.1 Supported Model Deployment Packages

The full package name format is: HiModel_<target>_<model_type>[-vl]_<param_size>_p256_<context_size>_<batch_size>_<core_num>_<device_size>_<sys_version>_<build_time>.gguf

• target indicates the hardware platform, such as xh1 or xh2.

• model_type indicates the model type.

  • deepseek: the deployment package is a DeepSeek-r1 model.

  • qwen2.5: the deployment package is a QWen2.5 model.

  • qwen3: the deployment package is a QWen3 model.

  • Other model types...

• [-vl] indicates multimodal.

• param_size indicates the model parameters, such as 4b.

• context_size indicates the context size, such as 2K.

• batch_size indicates the batch inference capability, such as 1b or 2b.

• core_num indicates the number of IPU cores the model is built for, such as 2core.

• device_size indicates how many AI chips are needed to load and run the model, such as 1chip or 2chips (legacy format: 1d, 2d).

• sys_version indicates the hardware version the model was compiled against, such as v0.7.0.

• build_time indicates the build time, such as 20260105154602.

Example: HiModel_xh2_qwen3_32b_256_32k_b1_4chips_2cores_v0.7.0_20251228.gguf

Please keep the model file name intact when using the service.

3.1.2 Model Download

Method 1:

Users can download models from the designated model website: https://llm.bee-link.cn/

Method 2 (Recommended):

Use the model management feature of the HLAWUtility tool to download models directly to the local model directory:

Enter model management as shown below. In the model list, a disk icon on the far left means the model is available locally, while a cloud icon means it is a cloud model that has not been downloaded. Find the desired model and click View Details in the Operation column, or download it directly. The download task is passed to the task center for downloading.

3.1.3 Manually Configuring SD3 Models (Skip If Not Needed)

Due to the current characteristics of SD3 models, they can only be configured by manually downloading them and creating a JSON information file.

Example SD3 image generation model directory:

models/
$ tree
.
├── SD3
│ ├── clip.hmm
│ ├── clip_l.hmm
│ ├── mmdit.hmm
│ ├── t5.hmm
│ └── vae.hmm
└── sd3.json
...

Create a JSON file to describe the SD3 model information:

{
"ModelType" : "Sd3",
"XHVersion" : 2,
"ClipModel" : "./SD3/clip.hmm",
"ClipLModel" : "./SD3/clip_l.hmm",
"T5Model" : "./SD3/t5.hmm",
"MMDitModel" : "./SD3/mmdit.hmm",
"VaeModel" : "./SD3/vae.hmm"
}

Save it as a json file containing the sd3 keyword in the configured model directory.

3.2.1 Manual Configuration

The configuration file is located in the config directory of the installation directory (default: C:\Program Files\AI\HLIECpp), named sys.cfg.

[server]
port=7901
# 设备ID, 以数字表示，默认只有0，多个设备用逗号隔开, 如 0,1
devices=0
# 实例副本数量，会启动多个进程，端口会递增+1， 用于负载均衡，多个设备同时运行同一种模型
replicas=1
# 用于多实例场景下节省内存使用，true时多实例的模型加载会有一定时间等待，达成串行效果
memory_saving=false [log]
# debug, info, warn, info
level=info # 4-debug, 3-warn, 2-error 1-info
tcim_level=2 # max files to keep
max_files=7
location=C:\ProgramData\AI\HLIECpp\logs\ [llm_params]
#llama-server的参数透传，请参考下面或者llamacpp的设置, 请使用完整option: --xxx = xx， 没有值的请留空
--jinja=
--reasoning-format=none
#--reasoning_budget = 0 [model]
# 模型目录
cur_model=D:\models
# 默认LLM/WHISPER模型，不带格式后缀
default=
default_mmproj=
# 是否默认加载SD3
enable_sd3=true
# 针对LLM配置为true时，相关Completions/Embeddings会根据接口中的model参数去自动加载模型
auto_load=true
# LLM模型懒加载
llm_lazy_mode=false

After modifying the configuration, restart the HLIECpp Service. Open Windows Services, find HLIECpp Service, right-click it, and restart.

3.2.2 Configuration with the Tool

After opening the application, switch to the service configuration item, modify the required configuration, click Save and Confirm, and restart the service.

3.3.1 Dynamically Loading a Model

*###start
POST*http://localhost:7901/models/load
Content-Type:application/json {
"model": "HiModel_xh2_qwen3_32b_256_32k_b1_4chips_2cores_v0.7.0_20251228"
}

Note that port 7901 mainly uses the default main process configuration. The following scenarios apply:

1) In a multi-card scenario, to dynamically create multiple process instances, you can continue sending requests to this port. The main process forks slave processes to load models.

2) If an LLM model is already running and you want to start sd3 text-to-image, you can also

*###startSD
POST*http://localhost:7901/models/load
Content-Type:application/json {
"devices": "1",
"model_type": "sd",
"params": {
"vae_model": "D:/workspace/models/SD3/vae.hmm",
"t5_model": "D:/workspace/models/SD3/t5.hmm",
"clip_model": "D:/workspace/models/SD3/clip.hmm",
"clip_l_model": "D:/workspace/models/SD3/clip_l.hmm",
"mmdit_model": "D:/workspace/models/SD3/mmdit.hmm",
}
}

The devices field above specifies card 1. If you do not want to specify it manually, configure "devices": "undefined", and the service automatically allocates from the existing devices.

3.3.2 Unloading a Model

*POST*http://10.65.35.16:7901/models/unload
Content-Type:application/json {
"model_type": "llm"
}

Note that port 7901 is configured according to each inference service process instance.

3.3.3 Stopping Service Instance Processes

*###stop
POST*http://localhost:7901/hlie/v1/stop
Content-Type:application/json {
"instance_id": 0,
"process_id": 0
}

If instance_id is 0, all models of the main process are stopped and unloaded.

If instance_id > 0 and process_id > 0, the main process finds the child process and kills it.

Note that the interface does not kill the main process; it only unloads all models of the main process. The main process stays resident.

3.3.4 Getting Information for All Service Instances

*###status
GET*http://localhost:7901/hlie/v1/status
Content-Type:application/json

4.6.1 Deep Thinking Configuration

You can add parameters in the llm_params area of the configuration file.

Setting --reasoning_budget = 0 avoids deep thinking.

#之前的配置 [llm_params]
#llama-server的参数透传，请参考下面或者llamacpp的设置, 请使用完整option: --xxx = xx， 没有值的请留空
--jinja=
--reasoning-format=none
--reasoning_budget=0 #继续之后的内容

4.6.2 Key llama.cpp Sampling Parameters

4.6.3 Recommended Configurations for Different Scenarios

1. Meeting minutes (stable, concise)

[llm_params]
#llama-server的参数透传，请参考下面或者llamacpp的设置, 请使用完整option: --xxx = xx， 没有值的请留空
--jinja =
--reasoning-format = none
--reasoning_budget = 0
--repeat-penalty = 1. 2
--presence-penalty = 0. 2
--frequency-penalty = 0. 2
--repeat-last-n = 256
--top-k = 40
--top-p = 0. 9
--temp = 0. 5

1. Writing drafts (creative and diverse)

[llm_params]
#llama-server的参数透传，请参考下面或者llamacpp的设置, 请使用完整option: --xxx = xx， 没有值的请留空
--jinja =
--reasoning-format = none
--reasoning_budget = 0
--repeat-penalty = 1. 05
--presence-penalty = 0. 8
--frequency-penalty = 0. 5
--repeat-last-n = 512
--top-k = 80
--top-p = 0. 95
--temp = 0. 8

1. Chatbot (natural, interactive)

[llm_params]
#llama-server的参数透传，请参考下面或者llamacpp的设置, 请使用完整option: --xxx = xx， 没有值的请留空
--jinja =
--reasoning-format = none
--repeat-penalty = 1. 1
--presence-penalty = 0. 6
--frequency-penalty = 0. 3
--repeat-last-n = 256
--top-k = 50
--top-p = 0. 9
--temp = 0. 7

4.7.1 Model Loading

• Curl command:

curl -X POST "http://localhost:7901/hlie/v1/start" \
-H "Content-Type: application/json" \
-d '{ "model_type": "whisper", "model": "HiModel_xh2_whisper-medium_1.5b_256_1chip_2cores_v1.0.0_20260228" }'

• AI Assistant:

4.7.2 HTTP Inference

curl-XPOST "http://127.0.0.1:7901/v1/audio/transcriptions" \
-H "Content-Type: multipart/form-data" \
-F"file=@E:/test_data/audio/long_2.wav"

4.7.3 WebSocket Inference

You can use a related script or service to connect to "ws://127.0.0.1:7999".

Then send the audio as a float32 byte stream in order and end with the string EOS for inference.

Reference Python script:

#!/usr/bin/env python3 # -*- coding: utf-8 -*- *"""*
* Whisper.cpp / miniaudio Compatible WebSocket Audio Stream Client* *功能:*
* 1. 读取 wav 音频*
* 2. 转 mono*
* 3. 重采样到 16k*
* 4. 转 float32 PCM [-1,1]*
* 5. websocket 流式发送* * 目标:*
* 与 whisper.cpp 中 miniaudio 的行为尽可能一致* * 数据格式:*
* float32 PCM*
* 16000Hz*
* mono* *发送协议:*
* binary:*
* float32 PCM bytes* * text:*
* EOS*
* """* import asyncio
import time
from pathlib import Path import numpy as np
import soundfile as sf
import websockets
from loguru import logger
from scipy import signal # ============================================================================= # Config # ============================================================================= TARGET_SAMPLE_RATE = 16000 # 1000ms chunk
CHUNK_DURATION_MS = 1000 # websocket uri
WS_URI = "ws://10.65.32.191:7999" # audio file
AUDIO_FILE = "D:/workspace/wtt//llm/hi.cpp/samples/zero_shot_prompt.wav" # ============================================================================= # Logger # ============================================================================= logger. remove () logger. add ( sink=lambda msg : print (msg, end=""), level="INFO", format=( "<green>{time:YYYY-MM-DD HH:mm:ss.SSS}</green> | " "<level>{level: </level> | " "{message}" ),) logger. add ( "asr_websocket_test.log", level="DEBUG", rotation="10 MB", retention=5, encoding="utf-8",) # ============================================================================= # Audio Process # ============================================================================= def load_audio (file_path : str) -> np. ndarray : *"""*
* 加载音频并转换成:* * float32*
* mono*
* 16000Hz*
* [-1,1]* * 尽量模拟 miniaudio 行为*
* """* * *logger. info (f"Loading audio:{file_path}") # soundfile: # int16 wav -> int16 # float wav -> float data, sample_rate = sf. read (file_path) logger. info ( f"Original audio info: " f"sample_rate={sample_rate}, " f"shape={data.shape}, " f"dtype={data.dtype}" ) # ========================================================================= # Convert to mono # ========================================================================= if data. ndim > 1 : logger. info ( f"Convert multi-channel -> mono " f"(channels={data.shape[1]})" ) # IMPORTANT: # whisper.cpp/miniaudio: # left + right # # NOT: # (left + right) / 2 # # 这里使用 sum 更接近 whisper.cpp data = np. sum (data, axis=1) # ========================================================================= # Resample to 16k # ========================================================================= if sample_rate != TARGET_SAMPLE_RATE : logger. info ( f"Resample audio: " f"{sample_rate}Hz ->{TARGET_SAMPLE_RATE}Hz" ) num_samples = int ( len (data) * TARGET_SAMPLE_RATE / sample_rate ) data = signal. resample ( data, num_samples,) sample_rate = TARGET_SAMPLE_RATE # ========================================================================= # Convert to float32 # ========================================================================= if data. dtype == np. int16 : logger. info ("Convert int16 -> float32") data = data. astype (np. float32) / 32768.0 elif data. dtype == np. int32 : logger. info ("Convert int32 -> float32") data = data. astype (np. float32) / 2147483648.0 elif data. dtype != np. float32 : logger. info (f"Convert{data.dtype} -> float32") data = data. astype (np. float32) # ========================================================================= # Clip # ========================================================================= # 防止 stereo 相加后超过 [-1,1] data = np. clip (data, -1.0, 1.0) duration = len (data) / TARGET_SAMPLE_RATE logger. success ( f"Audio ready: " f"samples={len(data)}, " f"duration={duration:.2f}s, " f"dtype={data.dtype}" ) return data # ============================================================================= # WebSocket Stream # ============================================================================= async def stream_audio ( websocket, pcmf32 : np. ndarray,): *"""*
* websocket 流式发送 float32 PCM*
* """* * *chunk_size = int ( TARGET_SAMPLE_RATE * CHUNK_DURATION_MS / 1000 ) total_chunks = ( len (pcmf32) + chunk_size - 1 ) // chunk_size logger. info ( f"Start streaming: " f"chunk_duration={CHUNK_DURATION_MS}ms, " f"chunk_size={chunk_size}, " f"total_chunks={total_chunks}" ) start_time = time. time () for chunk_index, start in enumerate ( range (0, len (pcmf32), chunk_size), start=1,): end = start + chunk_size chunk = pcmf32 [start : end] # binary float32 pcm binary = chunk. astype (np. float32). tobytes () await websocket. send (binary) chunk_duration = ( len (chunk) / TARGET_SAMPLE_RATE ) logger. info ( f"[Chunk{chunk_index}/{total_chunks}] " f"samples={len(chunk)}, " f"bytes={len(binary)}, " f"duration={chunk_duration:.3f}s" ) # 模拟真实实时发送 await asyncio. sleep (chunk_duration) elapsed = time. time () - start_time logger. success ( f"Audio stream completed (elapsed={elapsed:.2f}s)" ) # ============================================================================= # Main # ============================================================================= async def simulate_frontend_stream ( file_path : str, uri : str,): if not Path (file_path). exists (): logger. error ( f"Audio file not found:{file_path}" ) return # load audio pcmf32 = load_audio (file_path) logger. info (f"Connecting websocket:{uri}") try : async with websockets. connect ( uri, max_size=20 * 1024 * 1024, ping_interval=20, ping_timeout=20,) as websocket : logger. success ("WebSocket connected") # stream audio await stream_audio ( websocket, pcmf32,) # EOS logger. info ("Sending EOS") await websocket. send ("EOS") # wait result logger. info ("Waiting ASR result...") result = await asyncio. wait_for ( websocket. recv (), timeout=30.0,) logger. success ( f"ASR Result:\n{result}" ) except asyncio. TimeoutError : logger. error ( "Timeout waiting ASR result" ) except websockets. ConnectionClosed as e : logger. error ( f"WebSocket closed: " f"code={e.code}, " f"reason={e.reason}" ) except Exception as e : logger. exception ( f"Unexpected error:{e}" ) finally : logger. info ("Test finished") # ============================================================================= # Entry # ============================================================================= if __name__ == "__main__" : asyncio. run ( simulate_frontend_stream ( AUDIO_FILE, WS_URI,) )

4.8.1 Model Loading

• Curl command:

curl -X POST "http://localhost:7901/hlie/v1/start" \
-H "Content-Type: application/json" \
-d '{ "model_type": "cosyvoice", "model": "HiModel_xh2_cosyvoice3_0.5b_1chip_2cores_v1.1.0-20260409" }'

• AI Assistant:

4.8.2 HTTP Inference

curl -X POST "http://localhost:7901/v1/audio/speech" \
-H "Content-Type: application/json" \
--max-time 600 \
-d '{ "input": "君不见，黄河之水天上来，奔流到海不复回。君不见，高堂明镜悲白发，朝如青丝暮成雪！人生得意须尽欢，莫使金樽空对月。天生我材必有用，千金散尽还复来。烹羊宰牛且为乐，会须一饮三百杯。岑夫子，丹丘生，将进酒，杯莫停。与君歌一曲，请君为我倾耳听。钟鼓馔玉不足贵，但愿长醉不复醒。古来圣贤皆寂寞，惟有饮者留其名。陈王昔时宴平乐，斗酒十千恣欢谑。主人何为言少钱，径须沽取对君酌。五花马、千金裘，呼儿将出换美酒，与尔同销万古愁！", "prompt_wav_path": "E:/test_data/tts/zero_shot_0.wav", "prompt_text": "八百标兵奔北坡，北坡炮兵并排跑，炮兵怕把标兵碰，标兵怕碰炮兵炮。" }'

Among the above parameters, the audio of prompt_wav_path must be aligned one-to-one with the text of prompt_text. Of course, both parameters can also be omitted, for example:

In the following way, the system's built-in default prompt is used during inference.

curl -X POST "http://localhost:7901/v1/audio/speech" \
-H "Content-Type: application/json" \
--max-time 600 \
-d '{ "input": "君不见，黄河之水天上来，奔流到海不复回。君不见，高堂明镜悲白发，朝如青丝暮成雪！人生得意须尽欢，莫使金樽空对月。天生我材必有用，千金散尽还复来。烹羊宰牛且为乐，会须一饮三百杯。岑夫子，丹丘生，将进酒，杯莫停。与君歌一曲，请君为我倾耳听。钟鼓馔玉不足贵，但愿长醉不复醒。古来圣贤皆寂寞，惟有饮者留其名。陈王昔时宴平乐，斗酒十千恣欢谑。主人何为言少钱，径须沽取对君酌。五花马、千金裘，呼儿将出换美酒，与尔同销万古愁！" }'

Service Configuration

The above scenario requires the corresponding card configuration in the HLIECpp sys.cfg:

[server]
port=7901
# 设备ID, 以数字表示，多个设备用逗号隔开
devices=0, 1
# 实例副本数量，会启动多个进程，端口会递增+1， 用于负载均衡，多个设备同时运行同一种模型 ### 其他配置
### ......

7.6.1 Deep Thinking Configuration

sys.cfg

#之前的配置 [llm_params]
#llama-server的参数透传，请参考下面或者llamacpp的设置, 请使用完整option: --xxx = xx， 没有值的请留空
#--jinja =
#--reasoning_budget = 0 #继续之后的内容

7.6.2 Key llama.cpp Sampling Parameters

7.6.3 Recommended Configurations for Different Scenarios

1. Meeting minutes (stable, concise)

[llm_params]
#llama-server的参数透传，请参考下面或者llamacpp的设置, 请使用完整option: --xxx = xx， 没有值的请留空
--jinja =
--reasoning-format = none
--reasoning_budget = 0
--repeat-penalty = 1. 2
--presence-penalty = 0. 2
--frequency-penalty = 0. 2
--repeat-last-n = 256
--top-k = 40
--top-p = 0. 9
--temp = 0. 5

1. Writing drafts (creative and diverse)

[llm_params]
#llama-server的参数透传，请参考下面或者llamacpp的设置, 请使用完整option: --xxx = xx， 没有值的请留空
--jinja =
--reasoning-format = none
--reasoning_budget = 0
--repeat-penalty 1. 05
--presence-penalty 0. 8
--frequency-penalty 0. 5
--repeat-last-n 512
--top-k 80
--top-p 0. 95
--temp 0. 8

1. Chatbot (natural, interactive)

[llm_params]
#llama-server的参数透传，请参考下面或者llamacpp的设置, 请使用完整option: --xxx = xx， 没有值的请留空
--jinja =
--reasoning-format = none
--repeat-penalty 1. 1
--presence-penalty 0. 6
--frequency-penalty 0. 3
--repeat-last-n 256
--top-k 50
--top-p 0. 9
--temp 0. 7

7.7.1 Model Loading

• Curl command:

curl -X POST "http://localhost:7901/hlie/v1/start" \
-H "Content-Type: application/json" \
-d '{ "model_type": "whisper", "model": "HiModel_xh2_whisper-medium_1.5b_256_1chip_2cores_v1.0.0_20260228" }'

• AI Assistant:

7.7.2 HTTP Inference

curl-XPOST "http://127.0.0.1:7901/v1/audio/transcriptions" \
-H "Content-Type: multipart/form-data" \
-F"file=@E:/test_data/audio/long_2.wav"

7.7.3 WebSocket Inference

You can use a related script or service to connect to "ws://127.0.0.1:7999".

Then send the audio as a float32 byte stream in order and end with the string EOS for inference.

Reference Python script:

#!/usr/bin/env python3 # -*- coding: utf-8 -*- *"""*
* Whisper.cpp / miniaudio Compatible WebSocket Audio Stream Client* *功能:*
* 1. 读取 wav 音频*
* 2. 转 mono*
* 3. 重采样到 16k*
* 4. 转 float32 PCM [-1,1]*
* 5. websocket 流式发送* * 目标:*
* 与 whisper.cpp 中 miniaudio 的行为尽可能一致* * 数据格式:*
* float32 PCM*
* 16000Hz*
* mono* *发送协议:*
* binary:*
* float32 PCM bytes* * text:*
* EOS*
* """* import asyncio
import time
from pathlib import Path import numpy as np
import soundfile as sf
import websockets
from loguru import logger
from scipy import signal # ============================================================================= # Config # ============================================================================= TARGET_SAMPLE_RATE = 16000 # 1000ms chunk
CHUNK_DURATION_MS = 1000 # websocket uri
WS_URI = "ws://10.65.32.191:7999" # audio file
AUDIO_FILE = "D:/workspace/wtt//llm/hi.cpp/samples/zero_shot_prompt.wav" # ============================================================================= # Logger # ============================================================================= logger. remove () logger. add ( sink=lambda msg : print (msg, end=""), level="INFO", format=( "<green>{time:YYYY-MM-DD HH:mm:ss.SSS}</green> | " "<level>{level: </level> | " "{message}" ),) logger. add ( "asr_websocket_test.log", level="DEBUG", rotation="10 MB", retention=5, encoding="utf-8",) # ============================================================================= # Audio Process # ============================================================================= def load_audio (file_path : str) -> np. ndarray : *"""*
* 加载音频并转换成:* * float32*
* mono*
* 16000Hz*
* [-1,1]* * 尽量模拟 miniaudio 行为*
* """* * *logger. info (f"Loading audio:{file_path}") # soundfile: # int16 wav -> int16 # float wav -> float data, sample_rate = sf. read (file_path) logger. info ( f"Original audio info: " f"sample_rate={sample_rate}, " f"shape={data.shape}, " f"dtype={data.dtype}" ) # ========================================================================= # Convert to mono # ========================================================================= if data. ndim > 1 : logger. info ( f"Convert multi-channel -> mono " f"(channels={data.shape[1]})" ) # IMPORTANT: # whisper.cpp/miniaudio: # left + right # # NOT: # (left + right) / 2 # # 这里使用 sum 更接近 whisper.cpp data = np. sum (data, axis=1) # ========================================================================= # Resample to 16k # ========================================================================= if sample_rate != TARGET_SAMPLE_RATE : logger. info ( f"Resample audio: " f"{sample_rate}Hz ->{TARGET_SAMPLE_RATE}Hz" ) num_samples = int ( len (data) * TARGET_SAMPLE_RATE / sample_rate ) data = signal. resample ( data, num_samples,) sample_rate = TARGET_SAMPLE_RATE # ========================================================================= # Convert to float32 # ========================================================================= if data. dtype == np. int16 : logger. info ("Convert int16 -> float32") data = data. astype (np. float32) / 32768.0 elif data. dtype == np. int32 : logger. info ("Convert int32 -> float32") data = data. astype (np. float32) / 2147483648.0 elif data. dtype != np. float32 : logger. info (f"Convert{data.dtype} -> float32") data = data. astype (np. float32) # ========================================================================= # Clip # ========================================================================= # 防止 stereo 相加后超过 [-1,1] data = np. clip (data, -1.0, 1.0) duration = len (data) / TARGET_SAMPLE_RATE logger. success ( f"Audio ready: " f"samples={len(data)}, " f"duration={duration:.2f}s, " f"dtype={data.dtype}" ) return data # ============================================================================= # WebSocket Stream # ============================================================================= async def stream_audio ( websocket, pcmf32 : np. ndarray,): *"""*
* websocket 流式发送 float32 PCM*
* """* * *chunk_size = int ( TARGET_SAMPLE_RATE * CHUNK_DURATION_MS / 1000 ) total_chunks = ( len (pcmf32) + chunk_size - 1 ) // chunk_size logger. info ( f"Start streaming: " f"chunk_duration={CHUNK_DURATION_MS}ms, " f"chunk_size={chunk_size}, " f"total_chunks={total_chunks}" ) start_time = time. time () for chunk_index, start in enumerate ( range (0, len (pcmf32), chunk_size), start=1,): end = start + chunk_size chunk = pcmf32 [start : end] # binary float32 pcm binary = chunk. astype (np. float32). tobytes () await websocket. send (binary) chunk_duration = ( len (chunk) / TARGET_SAMPLE_RATE ) logger. info ( f"[Chunk{chunk_index}/{total_chunks}] " f"samples={len(chunk)}, " f"bytes={len(binary)}, " f"duration={chunk_duration:.3f}s" ) # 模拟真实实时发送 await asyncio. sleep (chunk_duration) elapsed = time. time () - start_time logger. success ( f"Audio stream completed (elapsed={elapsed:.2f}s)" ) # ============================================================================= # Main # ============================================================================= async def simulate_frontend_stream ( file_path : str, uri : str,): if not Path (file_path). exists (): logger. error ( f"Audio file not found:{file_path}" ) return # load audio pcmf32 = load_audio (file_path) logger. info (f"Connecting websocket:{uri}") try : async with websockets. connect ( uri, max_size=20 * 1024 * 1024, ping_interval=20, ping_timeout=20,) as websocket : logger. success ("WebSocket connected") # stream audio await stream_audio ( websocket, pcmf32,) # EOS logger. info ("Sending EOS") await websocket. send ("EOS") # wait result logger. info ("Waiting ASR result...") result = await asyncio. wait_for ( websocket. recv (), timeout=30.0,) logger. success ( f"ASR Result:\n{result}" ) except asyncio. TimeoutError : logger. error ( "Timeout waiting ASR result" ) except websockets. ConnectionClosed as e : logger. error ( f"WebSocket closed: " f"code={e.code}, " f"reason={e.reason}" ) except Exception as e : logger. exception ( f"Unexpected error:{e}" ) finally : logger. info ("Test finished") # ============================================================================= # Entry # ============================================================================= if __name__ == "__main__" : asyncio. run ( simulate_frontend_stream ( AUDIO_FILE, WS_URI,) )

7.8.1 Model Loading

• Curl command:

curl -X POST "http://localhost:7901/hlie/v1/start" \
-H "Content-Type: application/json" \
-d '{ "model_type": "cosyvoice", "model": "HiModel_xh2_cosyvoice3_0.5b_1chip_2cores_v1.1.0-20260409" }'

• AI Assistant:

7.8.2 HTTP Inference

curl -X POST "http://localhost:7901/v1/audio/speech" \
-H "Content-Type: application/json" \
--max-time 600 \
-d '{ "input": "君不见，黄河之水天上来，奔流到海不复回。君不见，高堂明镜悲白发，朝如青丝暮成雪！人生得意须尽欢，莫使金樽空对月。天生我材必有用，千金散尽还复来。烹羊宰牛且为乐，会须一饮三百杯。岑夫子，丹丘生，将进酒，杯莫停。与君歌一曲，请君为我倾耳听。钟鼓馔玉不足贵，但愿长醉不复醒。古来圣贤皆寂寞，惟有饮者留其名。陈王昔时宴平乐，斗酒十千恣欢谑。主人何为言少钱，径须沽取对君酌。五花马、千金裘，呼儿将出换美酒，与尔同销万古愁！", "prompt_wav_path": "E:/test_data/tts/zero_shot_0.wav", "prompt_text": "八百标兵奔北坡，北坡炮兵并排跑，炮兵怕把标兵碰，标兵怕碰炮兵炮。" }'

Among the above parameters, the audio of prompt_wav_path must be aligned one-to-one with the text of prompt_text. Of course, both parameters can also be omitted, for example:

In the following way, the system's built-in default prompt is used during inference.

curl -X POST "http://localhost:7901/v1/audio/speech" \
-H "Content-Type: application/json" \
--max-time 600 \
-d '{ "input": "君不见，黄河之水天上来，奔流到海不复回。君不见，高堂明镜悲白发，朝如青丝暮成雪！人生得意须尽欢，莫使金樽空对月。天生我材必有用，千金散尽还复来。烹羊宰牛且为乐，会须一饮三百杯。岑夫子，丹丘生，将进酒，杯莫停。与君歌一曲，请君为我倾耳听。钟鼓馔玉不足贵，但愿长醉不复醒。古来圣贤皆寂寞，惟有饮者留其名。陈王昔时宴平乐，斗酒十千恣欢谑。主人何为言少钱，径须沽取对君酌。五花马、千金裘，呼儿将出换美酒，与尔同销万古愁！" }'

Hardware Requirements (Minimum)

• CPU architecture: x86_64 / aarch64

• Disk: 200G

• AI hardware: M50

Software Requirements

• Operating system: Ubuntu 20.04

• Driver version:

  • M50-v1.0.0

2.3.1 Installing Dependencies

A Python 3 environment of >= 3.10 is recommended, preferably 3.10 or 3.12.

pip3install-rscripts/requirements-convert_hm_to_gguf.txt

2.3.2 Compiling and Quantizing Models

For model compilation and quantization, refer to the related documentation. The document download path is in the documentation center (Model Resource Center/doc).

2.3.3 Running the Conversion Script

1. Download the official configuration files.

For example, download from ModelScope to your_local_dir (in the same folder as the hmm files).

modelscopedownload--modeldeepseek-ai/DeepSeek-R1-0528-Qwen3-8B--exclude '*.safetensors'--local_diryour_local_dir

1. Run the conversion. (For qwen2.5, qwen3, and similar models; for qwenvl and deepseek, refer to steps 3 and 4.)

python3convert_hm_to_gguf.pyyour_local_dir/

1. qwen2.5-vl conversion

Text model conversion

# 文本模型指定vit 模型的width & height
python3convert_hm_to_gguf.pyyour_local_dir/

Vision model conversion

python3convert_hm_to_gguf.pyyour_local_dir/--mmproj--outfileyour_local_dir/mmproj.gguf

1. bge-m3 / bge-reranker-v2-ms model conversion

Rename the hmm file to embedding.hmm. The rest of the process is the same as above.

modelscopedownload--modelBAAI/bge-m3--exclude '*.safetensors'--local_dirbge-m3
# 将hmm文件重命名为embedding.hmm
python3convert_hm_to_gguf.pybge-m3

1. deepseek conversion

Currently, the deepseek-qwen3 conversion reports an error:

convert_hm_to_gguf.py ", line 1029, in get_vocab_base_pre
 raise NotImplementedError(
 NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()

You can solve it by updating the tokenizer.json file as follows:

from transformers import AutoTokenizer * # 替换成你目标模型的正确 HuggingFace 模型名称，比如：*
model_name = "deepseek-ai/[DeepSeek-R1-0528-Qwen3-8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B)" * # 请确认这个名称是否正确！* * # 下载到本地缓存 ~/.cache/huggingface...*
tokenizer = AutoTokenizer. from_pretrained (model_name) # 执行下面命令将*.json 保存到your_local_dir
tokenizer. save_pretrained (your_local_dir)

2.4.1 Querying the Reference Model List Published in the Cloud

Filters such as size, batch, device, and target are supported, for example: target=xh2.

CTX: the maximum context length of the model.

BATCH: whether it is a multi-batch model.

DEVICE: whether it is a single-card or multi-card model. For example, 2 means a dual-card model that needs to run on DM.2 or a multi-chip PCIe card.

FILES: how many gguf files; multi-card models usually have 2.

BUILD: which toolchain version was used to compile the model.

user@user-System-Product-Name:~/dev/tools$./model-cliquery
Auto-detectedtarget:xh2
Fetchingfrom:https://llm.bee-link.cn//models?build_version=1.0.0&format=gguf&open_type=cloud&target=xh2 Found 23models:
IDNAMESIZEBUILDCTXBATCHDEVICETARGETCORESFILES
----------------------------------------------
1767880490946qwen3-vl4b 1.0.0 16384 1 1xh2 2 2
1767880490947qwen314b 1.0.0 32768 1 2xh2 2 1
1767880490948qwen332b 1.0.0 32768 1 4xh2 2 1
1767880490949qwen2.57b 1.0.0 8192 1 1xh2 2 1
1767880490950qwen2.57b 1.0.0 32768 1 1xh2 2 1
1767880490951deepseek8b 1.0.0 32768 1 1xh2 2 1
1767880490952gpt20b 1.0.0 65536 1 1xh2 2 1
1767880490955gte 1.5b 1.0.0 2048 1 1xh2 2 1
1767880490956bge 0.5b 1.0.0 32768 1 1xh2 2 1
1767880490957bge-reranker-v2-m3 0.5b 1.0.0 32768 1 1xh2 2 1
1767880490958qwen34b 1.0.0 32768 1 1xh2 2 1
1767880490959qwen38b 1.0.0 32768 1 1xh2 2 1
1767880490960qwen38b 1.0.0 16384 4 1xh2 2 1
1767880490961qwen314b 1.0.0 16384 1 1xh2 2 1
1767880490962qwen3-vl4b 1.0.0 32768 1 1xh2 2 2
1767880490963qwen3-vl8b 1.0.0 32768 1 1xh2 2 2
1767880490964qwen2.5-vl7b 1.0.0 8192 1 1xh2 2 2
1767880490965qwen330b-a3b 1.0.0 32768 1 1xh2 2 1
1767880490966qwen3-vl30b-a3b 1.0.0 8192 1 1xh2 2 2
1772188146032qwen3-250730b-a3b 1.0.0 32768 1 1xh2 2 1
1772188325612qwen3-250730b-a3b 1.0.0 32768 1 2xh2 2 1
1772799664016gpt20b 1.0.0 65536 1 2xh2 2 1
1772878235701gpt20b 1.0.0 131072 1 1xh2 2 1 Todownload:./model-clipull<ID>

The default model download path is the ../models directory relative to model-cli.

You can also specify the download path with --model-path.

2.4.2 Local Model Management

Requires llama-server to be started with --models-dir.

• list: lists the models that have been scanned/loaded.

• run: loads a model.

Loading takes time; the actual return only means the load command was received. Use ps to check the loading result.

• ps: views loaded models.

• stop: unloads a loaded model.

2.5.1 Running Modes

# model_name 为gguf模型文件绝对路径
./bin/llama-server-mmodel_name.gguf
# 加载多模态
./bin/llama-server-mmodel_name.gguf--mmprojmmproj.gguf

./bin/llama-server--models-dirmodels

./bin/model-clilist # 列出llama-server 启动时扫描到的模型
./bin/model-clirunmodelname # 加载模型
./bin/model-clistopmodelname # 卸载模型

# 安装指令
sudobashservice/install_llama-server.sh
# 服务管理
sudosystemctlstatusllama-server.service # 查看服务状态
sudosystemctlstartllama-server.service # 启动服务
sudosystemctlstopllama-server.service # 停止服务
sudosystemctlrestartllama-server.service # 重启服务
sudojournalctl-ullama-server.service-f # 实时查看服务日志 # 服务卸载
sudobashservice/uninstall_llama-server.sh

2.5.2 Common Runtime Parameters

For specific usage parameters, refer to the llama-server binary. To support concurrent access at the same time, you can use --np 4 (currently limited to a maximum of 4; only multi-batch models are truly concurrent, while single-batch models execute alternately).

./bin/llama-server--models-dirmodles

./bin/llama-server-m../models/gte_Qwen2-1.5B-Q4_0.gguf--embedding--poolinglast-b 8096-ub 8096

./bin/llama-server-m../models/Qwen3-Reranker-8B-Q4_0.gguf--rerank

export drv_DEVICE_ID=0或者双卡export drv_DEVICE_ID=0,1

user@user-System-Product-Name:~$./dev/llama.cpp/release/xh2/x86_64/bin/llama-server--list-devices
[NPU]LoadingNPUSDK...
[NPU]Tryingtoload:/usr/local/-sdk/hal/lib/libhal_xh2a.so
[NPU]Successfullyloaded:/usr/local/-sdk/hal/lib/libhal_xh2a.so
[NPU]hm_sys_get_device_inforeturned: 1devices
[NPU]Device 0: device_id=0
[NPU]get_mem_inforeturned: 0, mem_total=24448, mem_used=0, mem_avail=24448
[NPU]Finaldevicememory: total=24448MB, free=24448MB
[NPU]Initializationcomplete, 1devicesavailable
Availabledevices:
0:NPULQ50 (ID: 0) (24448MiB, 24448MiBfree)

./bin/llama-server-mmodel_name--port 18080

./bin/llama-server-mmodel_name--reasoning-budget 0【2.1.0之前接口】 ./bin/llama-server-mmodel_name-reaoff【2.1.0版本接口，qwen3.5等需要使用这个参数】

./bin/llama-server-mmodel_name--jinja--chat-template-filebin/qwen3_nonthinking.jinja

--log-file写日志到文件
--log-timestamps为打印uptime的时间，
--verbosity控制等级
- 0:genericoutput
- 1:error
- 2:warning
- 3:info
- 4:debug example
--log-file/home/useradmin/test.log--log-prefix--log-timestamps--verbosity 2

{
"temperature" : 0.8,
"top_p" : 0.9,
"repetition_penalty" : 1.1,
"chat_template_kwargs" : {"enable_thinking" : false}
}

{
"temperature" : 0.8,
"top_p" : 0.9,
"repetition_penalty" : 1.1,
"reasoning" :{"effort" : "low"}
}

-a,--aliasSTRING set alias formodelname (tobeusedbyRESTAPI)

export LLAMA_CHECKPOINT_STORAGE=1
#表示存在device上，0 表示host上
#日志信息：set checkpoint_storage_ to kDevice from env LLAMA_CHECKPOINT_STORAGE export LLAMA_CHECKPOINT_INTERVAL=1024
#表示每隔多少个tokens 保存一次，默认1024， 最大40960

./bin/llama-server-m~/dev/models/HiModel_xh2_qwen3.6_27b_mtp_256_128k_1chip_2cores_v1.3.0_20260603.gguf--spec-typedraft-mtp--spec-draft-n-max 4

curl-XPOST "http://127.0.0.1:17701/v1/audio/transcriptions" \
-H "Content-Type: multipart/form-data" \
-F"model=Qwen3-ASR" \
-F"file=@E:/test_data/audio/test.wav"

curl-XPOST "http://127.0.0.1:17701/v1/audio/transcriptions" \
-F"file=@E:/test_data/audio/test.wav;type=audio/wav" \
-F"model=Qwen3-ForcedAligner-0.6B" \
-F"response_format=json" \
-F"asr_force_align=true" \
-F"text=HE BEGAN A CONFUSED COMPLAINT AGAINST THE WIZARD WHO HAD VANISHED BEHIND THE CURTAIN ON THE LEFT" \
-F"language=English"

2.5.3 Console Statistics

2.5.4 Testing

2.6.1 hlaw_ollama

This tool is a customized version of ollama. You can use the install script in the service directory to install it as a service. See the HLAWOllama Deployment and Usage guide for details. This tool is only provided as a demo.

2.6.2 llama-gguf

llama-gguf xx.gguf r is similar to the gguf-dump capability.

2.6.3 llama-gguf-split

This tool can split prefill.hmm and decoder.hmm from customized models. Currently, it only supports pd-classified models such as qwenvl. Combined with the readhmm tool shipped with the toolchain, model information can be read. This tool is mainly used for model tracing and problem location.

It can also be used to quickly test and verify models with different contexts based on compiled models without converting to gguf. Take the qwen3_8b model as an example; the split model has three parts:

1. qwen3_8b_00001-of-00003.gguf: contains vocab + embedding.

2. qwen3_8b_00002-of-00003.gguf: the prefill.hmm.

3. qwen3_8b_00003-of-00003.gguf: the decoder.hmm.

When compiling models with different contexts, vocab + embedding are shared; only the last two parts change. Overwrite 00002 and 00003 with the new prefill and decode models respectively. To load the model, only specify qwen3_8b_00001-of-00003.gguf with -m.

2.6.4 llama-cli

The command line version; basic functions are the same as the open-source version.

2.6.5 VL Model Usage Notes

Read this section before using VL models.

1. Currently, the VIT model only supports fixed resolutions. For example, the default provides 448x448 resolution, so all images sent to llama-server are resized to 448x448.

   When calling the OpenAI-compatible API, it is recommended that the caller performs the resize. This has two advantages:

   1. Reduces HTTP transfer overhead.

   2. The resize in llama.cpp is executed on the CPU by default and may not be the most efficient.

2. Currently, the resize logic in llama is proportional compression followed by padding, pasted to the top-left corner, with the remainder filled with (114,114,114). For inspection-box tasks, the returned boxes are relative to the resized image and need to be converted back. In addition, the bbox returned by object detection is usually between 0-1 or in 0-1000 resolution.

2.6.6 Recommendations for qwen3.5-Series Models

1. A 24G card can run qwen3.5 35B-A3B with np=1.

2. It is recommended to add --presence_penalty 1.5 to the startup parameters of qwen3.5-series models.

Official recommendations:

2.1.1 Creating a Python 3.11 Environment on Ubuntu

Use conda to create a Python 3.11 virtual environment; specify the virtual environment name after -n:

condacreate-nvllm_env python=3.11-y

Activate the virtual environment:

condaactivatevllm_env

2.2.1 Step 0: Confirm the tcim runtime is installed and the firmware version of the inference card matches

If tcim_runtime is not installed, download the matching runtime from the following link (select the version matching your driver): https://dr.bee-link.cn/download/dXBsb2Fkcy9MTE0vUnVudGltZS1TREsvaG91bW9fdGNpbV9ydW50aW1lX3hoMl9saW51eF94ODZfNjQtMV80XzBfdGFyLmd6/h/ca681089f70cbf5962aedfd56d866ba1 Install the tcim runtime v1.x package following the steps below. Also make sure the firmware version on the inference card matches the driver version. (If the runtime is already installed and the firmware version on the inference card is confirmed to match, this step can be skipped.)

First enter the Python virtual environment created above and make sure the operating system has gcc and g++ installed, with C++11 support; otherwise, the tcim runtime installation will fail.

If not installed, run the following command:

sudoaptinstallg++

Then run the following command to install the tcim runtime package:

Linux系统:注意根据cpu架构，选择对应版本
x86平台：
1.3.0的runtime包
pipinstallhttp://10.10.1.53:8082/artifactory/drv_software/release_xh2_v1.3.0/2026/05/19/0015/ubuntu2004/drv_tcim_runtime_xh2_linux_x86_64-1.3.0.tar.gz-ihttps://mirrors.aliyun.com/pypi/simple/ arm平台：
1.3.0的runtime包
pipinstalldrv_tcim_runtime_xh2_linux_aarch64-1..0.tar.gz-ihttps://mirrors.aliyun.com/pypi/simple/

2.2.2 Step 1: Install vLLM and Set Related Variables

Note: HLIEvLLM must be installed first, and then HLIEvLLMPlugin. This step installs the software packages that HLIEvLLM depends on and takes some time. It normally takes about 10-20 minutes to finish.

a. Install HLIEvLLM on Linux.

Starting from HLIEvLLM 1.3.0, the tcmalloc-minimal and omp libraries recommended by vllm are used first. According to the official instructions, install the following dependencies first:

依赖项1：
sudoapt-getinstalllibnuma-dev 依赖项2：
sudoapt-getinstall-y--no-install-recommendslibtcmalloc-minimal4
sudofind/usr-namelibtcmalloc_minimal*so
返回的路径要用到下面的环境变量设置中 依赖项3:
sudoapt-getinstalllibomp5libomp-dev
sudofind/usr-namelibomp*so* 设置下面的环境变量，以便找到相关的库
export IOMP_PATH=/上面findomp命令返回的路径/libomp.so.5
export TC_PATH=/上面findtcmalloc命令返回的路径/libtcmalloc_minimal.so.4
export LD_PRELOAD="$TC_PATH:$IOMP_PATH:$LD_PRELOAD"
echo $LD_PRELOAD
x86平台LD_PRELOAD变量输出示例如下：
/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4:/usr/lib/x86_64-linux-gnu/libomp.so.5: arm平台LD_PRELOAD变量输出示例如下：
/usr/lib/aarch64-linux-gnu/libtcmalloc_minimal.so.4:/usr/lib/aarch64-linux-gnu/libomp.so.5:

b. Download the vllm version.

Extract the archive and enter the extracted folder, for example "HLIEvLLM-1.3.0". Run the following command to install vllm (it takes about 10 minutes because dependencies are downloaded):

On Ubuntu 22.04 and earlier, the default gcc version is lower than 12.3 and must be upgraded; otherwise, vllm compilation reports an error. Upgrade it as follows (if gcc is greater than or equal to 12.3, no upgrade is needed):

# 1. 添加官方源（针对 Ubuntu 20.04/22.04）
sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
sudo apt update # 2. 安装 gcc-13 和 g++-13,如果下载较慢，ctrl—c终止，然后重新下载试试，速度不低于200KB/s就行。速度如果一直较慢，请设置clash 翻墙代理，正常不到10分钟可以安装完成
sudo apt install -y gcc-13 g++-13 # 3. 设置为系统默认版本（关键）
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 100
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-13 100 # 4. 验证版本（输出应为 13.+）
gcc --version
g++ --version

First confirm that the gcc and g++ versions are greater than or equal to 12.3. Then enter the extracted folder, for example "HLIEvLLM-1.3.0", and run the following command to install vllm: After installing the v1.4.0 preview version, remember to downgrade fastapi to 0.136.3 or 0.136.1; the official version will fix this problem. After the preview release, the fastapi version was upgraded, and the latest fastapi is not compatible. See section 3.1 of chapter 3.

VLLM_TARGET_DEVICE=cpupipinstall-e.--extra-index-urlhttps://download.pytorch.org/whl/cpu-ihttps://mirrors.aliyun.com/pypi/simple/

If you need to run audio-related models, such as the Whisper models below, install the additional dependencies. Enter the extracted folder, for example "HLIEvLLM-1.3.0", and run the following command:

VLLM_TARGET_DEVICE=cpupipinstall-e ".[audio]"--extra-index-urlhttps://download.pytorch.org/whl/cpu-ihttps://mirrors.aliyun.com/pypi/simple/

2.2.3 Step 2: Install the vllm Inference Plugin Package

Note: If an old whl plugin package is installed, be sure to uninstall it first by running pip uninstall HLIEvLLMPlugin.

If the installation encounters an unsupported instruction vpdpbusd' error, set the following two environment variables before installing:

export CFLAGS="-mno-avx"

export CXXFLAGS="-mno-avx"

The whl package installation command is as follows:

VLLM_TARGET_DEVICE=pipinstallhlievllmplugin-1.-py3-none-any.whl--no-build-isolation--extra-index-urlhttps://download.pytorch.org/whl/cpu-ihttps://mirrors.aliyun.com/pypi/simple/

After installation, view the installation location and source path through the Location: value in the output of the following command:

pip show HLIEvLLMPlugin

Example path: /home/useradmin/anaconda3/envs/vllm_env_120rc0/lib/python3.11/site-packages/vllm_/

Starting a Text Offline Inference Test

(vllm_env_szj_v110a4)user@PC:/HLIEvLLM-1.*.* $**python3drv_examples/LLM_example_chat.py

Starting a Multimodal Offline Inference Test

(vllm_env_szj_v110a4)user@PC:~/szj_project/HLIEvLLM-1.*.*$python3drv_examples/LLM_qwen35_vl_example.py

Speech-Text Forced Alignment Model

Use the following command to run the Qwen3-ForcedAligner-0.6B model for speech and text alignment, mapping each text token to the corresponding time segment of the audio. The reference text can be obtained through the speech model.

Note: the qwen-asr library must be installed in the hlievllm environment. The qwen-asr library downgrades transformers to 4.57.6. After using the example, manually upgrade the transformers library to avoid affecting other models.

pythondrv_examples/qwen3_forcealigner_offline.py--model-dir/path/to/Qwen3-ForcedAligner-0.6B--audio/path/to/test.mp3--text "参考文本"--pretty

Text Models

Start the OpenAI server with an API key.

vllmserve/home/user/szj_project/models/v1.2.0/qwen3.5_35B_a3b/--port 12321--host 0.0.0.0--no-enable-prefix-caching--served-model-nameqwen35--max-num-batched-tokens 262144--max-num-seqs 1--max-model-len 262144--enable-auto-tool-choice--tool-call-parserqwen3_coder--additional-config='{"device_ids":[0]}'

vllmserve/home/user/szj_project/models/qwen3_30b_a3b_v1.2.0--port 12321--host 0.0.0.0--no-enable-prefix-caching--max-num-seqs 1--served-model-nameqwen3_30b_a3b--enable-auto-tool-choice--tool-call-parserhermes--max-num-batched-tokens 32768--max-model-len 32768

vllmserve/home/user/szj_project/models/v1.2.0/qwen3_8b_16k_4batch--port 12321--host 0.0.0.0--no-enable-prefix-caching--max-num-seqs 4--served-model-nameqwen3_8b_b4--enable-auto-tool-choice--tool-call-parserhermes--max-num-batched-tokens 16384--max-model-len 16384

vllmserve/home/user/szj_project/models/v1.2.0/deepseek_8b--port 12321--host 0.0.0.0--no-enable-prefix-caching--max-num-seqs 1--served-model-namedeepseek_8b--enable-auto-tool-choice--tool-call-parserhermes--max-num-batched-tokens 32768--max-model-len 32768

Multimodal qwen3.5/qwen3.6 VL Series (Multiple Images Supported)

The M50 24G card supports qwen3.5/qwen3.6 35B-A3B vision-language models.

Note: visual_model_path is not the same gguf file as the language model. It is usually named with the mmproj prefix. This file path cannot use relative paths or the ~ symbol for the working directory. Use a complete absolute path, as shown below:

a)单chip模型启动示例：
vllmserve/home/user/szj_project/models/v1.3.0/qwen35_35b_a3b/--port 12321--host 0.0.0.0--no-enable-prefix-caching--served-model-nameqwen35_vl--max-num-batched-tokens 262144--max-num-seqs 1--max-model-len 262144--enable-auto-tool-choice--tool-call-parserqwen3_coder--additional-config='{"enable_cacheprompt":"False","visual_model_path":"/home/user/szj_project/models/v1.3.0/qwen35_35b_a3b/mmproj-qwen3.5-35b-a3b-p256-256K-b1-2core-1d-xh2-v1.3.0-20260420210756.gguf"}' b)多chip模型启动示例-以4chip语言模型+1chip视觉模型启动为例：
vllmserve/home/lenovo/release/models/130/qwen3.6_27b_128k_4chip--port 12321--host 0.0.0.0--no-enable-prefix-caching--served-model-nameqwen36_vl--max-num-batched-tokens 131072--max-num-seqs 1--max-model-len 131072--enable-auto-tool-choice--tool-call-parserqwen3_coder--additional-config='{"enable_cacheprompt":"True","visual_model_path":"/home/lenovo/release/models/130/qwen3.6_27b_128k_4chip/mmproj_xh2_qwen3.6_27b_2cores_vit_448x448_v1.3.0_20260603.gguf","device_ids":[0,1,2,3],"vl_device_ids":[3]}' 其中：
"device_ids": [0,1,2,3],指定语言模型跑在哪些卡上，示例是4chip语言模型，跑在devid 0/1/2/3这4张卡
"vl_device_ids": [3]，指定视觉模型跑在哪些卡上，示例是1chipvit模型，跑在devid 3这张卡上
注意：卡的数量，要跟模型实际chip数一致。

For testing, you can attach images through chatbox or use the curl command as shown below, where "url" should be replaced with your local image or the corresponding base64 data, "model" name and key should be replaced with your own values, and the final EOF must be included.

curl-XPOST [http://127.0.0.1:12321/v1/chat/completions](http://127.0.0.1:7801/v1/chat/completions)-H "Authorization: Bearer sk-123"-H "Content-Type: application/json"-d@- <<EOF
{
 "model": "qwen35_vl",
 "messages": [
 {"role": "system", "content": "你是一个智能助理"},
 {
 "role": "user",
 "content": [
 {
 "type": "image_url",
 "image_url": {
  "url": "data:image/jpeg;base64,$(base64 -w 0 /home/user/szj_project/-examples-xh2/models/llm/qwen3.5/image_dir/frame85.jpg)"
 },
 "detail":"high"
 },
 {
 "type": "text",
 "text": "请描述图片内容。"
 }
 ]
 }
 ],
 "stream": true
}
EOF

Multimodal gemma4 Model

Startup command: divided into single-chip and multi-chip models.

Note: to enable thinking mode, add parameters at startup; see the example below.

单芯模型启动示例：
vllmserve/home/user/szj_project/models/v1.4.0/gemmar4/gemma-4-26B-A4B-it--port 12321--host 0.0.0.0--no-enable-prefix-caching--served-model-namegemma4--max-num-batched-tokens 32768--max-num-seqs 1--max-model-len 32768--enable-auto-tool-choice--tool-call-parsergemma4--additional-config='{"enable_cacheprompt":"False","visual_model_path":"/home/user/szj_project/models/v1.4.0/gemmar4/gemma-4-26B-A4B-it/gemma-mmproj.gguf","device_ids":[0]}' 多芯模型启动示例：语言模型双芯，vit视觉模型单芯
vllmserve/home/user/vllm_v140_release_teset/models/gemma4_mutlichip/--port 12321--host 0.0.0.0--no-enable-prefix-caching--served-model-namegemma4--max-num-batched-tokens 32768--max-num-seqs 1--max-model-len 32768--enable-auto-tool-choice--tool-call-parsergemma4--additional-config='{"enable_cacheprompt":"True","visual_model_path":"/home/user/vllm_v140_release_teset/models/gemma4_mutlichip/mmproj_xh2_gemma4-26b-a4b_2cores_vit_448x448_v1.4.0_20260609.gguf","device_ids":[0,1],"vl_device_ids":[3]}' 开启Thinking模式
vllmserve/home/user/szj_project/models/v1.4.0/gemmar4/gemma-4-26B-A4B-it--port 12321--host 0.0.0.0--no-enable-prefix-caching--served-model-namegemma4--max-num-batched-tokens 32768--max-num-seqs 1--max-model-len 32768--enable-auto-tool-choice--tool-call-parsergemma4--reasoning-parsergemma4--default-chat-template-kwargs '{"enable_thinking": true}'--additional-config='{"enable_cacheprompt":"False","visual_model_path":"/home/user/szj_project/models/v1.4.0/gemmar4/gemma-4-26B-A4B-it/gemma-mmproj.gguf","device_ids":[0]}'--chat-templateexamples/tool_chat_template_gemma4.jinja

Speculative Decoding Test

vllmserve/home/user/szj_project/models/v1.4.0/qwen3_14b_speculate--host 0.0.0.0--port 12321--no-enable-prefix-caching--max-num-seqs 1--served-model-nameqwen3_14b--enable-auto-tool-choice--tool-call-parserhermes--max-num-batched-tokens 32768--max-model-len 32768--additional-config='{"enable_cacheprompt":"True"}'--speculative-config '{"model": "/home/user/szj_project/models/v1.4.0/qwen3_0.6B_speculate", "num_speculative_tokens": 5, "method": "draft_model"}'

vllmserve/home/user/szj_project/models/v1.3.0/qwen35_9b_mtp/gguf_main_verify_model/--max-model-len 262144--speculative-config '{"model":"/home/user/szj_project/models/v1.3.0/qwen35_9b_mtp/gguf_draft_model","method": "mtp", "num_speculative_tokens": 5}'--port 12321--host 0.0.0.0--no-enable-prefix-caching--served-model-nameqwen35--max-num-batched-tokens 262144--max-num-seqs 1--max-model-len 262144--enable-auto-tool-choice--tool-call-parserqwen3_coder

vllmserve/home/user/szj_project/models/v1.3.0/qwen36_27b_mtp--max-model-len 262144--speculative-config '{"model":"/home/user/szj_project/models/v1.3.0/qwen36_27b_mtp","method": "mtp", "num_speculative_tokens": 5}'--port 12321--host 0.0.0.0--no-enable-prefix-caching--served-model-nameqwen35--max-num-batched-tokens 262144--max-num-seqs 1--max-model-len 262144--enable-auto-tool-choice--tool-call-parserqwen3_coder

gpt_oss 20B

Startup command:

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1vllmserve/home/user/szj_project/models/v1.0.0/gpt_oss_20b_64K--port 12321--host 0.0.0.0--no-enable-prefix-caching--served-model-namegpt_oss--max-num-batched-tokens 65536--max-num-seqs 1--max-model-len 65536--enable-auto-tool-choice--tool-call-parseropenai

If the following error occurs: openai_harmony.HarmonyError: error downloading or loading vocab file: failed to download or load vocab file, solve it as follows: (In our self-test, x86 did not encounter this error, but ARM did. gpt-oss needs the following files; the default URL may fail to download and then report an error. The following method can solve it.)

mkdir-ptiktoken_encodings
curl-L-otiktoken_encodings/o200k_base.tiktoken "https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken"
curl-L-otiktoken_encodings/cl100k_base.tiktoken "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken"
export TIKTOKEN_ENCODINGS_BASE=${PWD}/tiktoken_encodings

ASR Models

vllmserve/path/to/ASR-Model \
--served-model-nameasr-model \
--no-enable-prefix-caching \
--load-formatdummy \
--additional-config='{"visual_model_path": "/path/to/ASR-Model/mmproj-asr-model.gguf"}'

curlhttp://127.0.0.1:8000/v1/audio/transcriptions-H "Content-Type: multipart/form-data"-F"file=@test.wav"-F"model=asr-model"-F"language=zh"-F"response_format=json"-F stream=false

Embedding Models

Startup command:

vllmserve/home/user/szj_project/models/v1.1.0/qwen3_embedding--port 12321--host 0.0.0.0--no-enable-prefix-caching--served-model-nameqwen3_embed--max-num-batched-tokens 8192--max-num-seqs 1--max-model-len 8192

Corresponding curl test command:

curlhttp://127.0.0.1:12321/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-123" \
-d '{"input": "Your text string goes here",
 "model":"qwen3_embed"}'

Rerank Models

Service Startup

vllmserve/data/useradmin/models/v1.2.0/qwen3-reranker--served-model-nameqwen3-reranker-8b--max-model-len 8192--host 0.0.0.0--port 7804--api-keysk-123--hf-overrides='{"architectures": ["Qwen3ForSequenceClassification"], "classifier_from_token": ["no", "yes"], "is_original_qwen3_reranker": true}'

/v1/score request

# 单个查询对单个文档
curl-X 'POST' 'http://127.0.0.1:7804/v1/score'-H 'accept: application/json'-H 'Content-Type: application/json'-H "Authorization: Bearer sk-123"-d '{
 "model": "qwen3-reranker-8b",
 "queries": "What is the capital of France?",
 "documents": "The capital of France is Paris."
}' # 单个查询对多个文档
curl-X 'POST' 'http://127.0.0.1:7804/score'-H 'accept: application/json'-H 'Content-Type: application/json'-d '{
 "model": "qwen3-reranker-8b",
 "queries": "What is the capital of France?",
 "documents": [
 "The capital of Brazil is Brasilia.",
 "The capital of France is Paris."
 ]
}'

/v1/rerank request

curl-X 'POST' 'http://127.0.0.1:7804/v1/rerank'-H 'accept: application/json'-H 'Content-Type: application/json'-H "Authorization: Bearer sk-123"-d '{
 "model": "qwen3-reranker-8b",
 "query": "What is the capital of France?",
 "documents": [
 "The capital of Brazil is Brasilia.",
 "The capital of France is Paris.",
 "Horses and cows are both animals"
 ]
}'

Alignment Models

vllmserve/path/to/Qwen3-ForcedAligner-0.6B \
--served-model-name-model \
--host 0.0.0.0 \
--port 12325 \
--api-keysk-123 \
--runnerpooling \
--enforce-eager \
--hf-overrides '{"architectures": ["Qwen3ASRForcedAlignerForTokenClassification"]}' \
--max-model-len 411 \
--load-formatdummy \
--trust-request-chat-template \
--additional-config '{"visual_model_path": "/path/to/Qwen3-ForcedAligner-0.6B/mmproj-Qwen3-ForcedAligner-Q4_0.gguf"}'

PROMPT='参考文本' curl-s-XPOSThttp://localhost:7803/v1/audio/transcriptions \
-H "Authorization: Bearer sk-123" \
-F"file=@/path/to/test.mp3" \
-F"model=asr-model" \
-F"prompt=${PROMPT}" \
-F"language=chinese" \
-F"response_format=verbose_json"

OCR Models

Startup command:

vllmserve/home/user/models/glm_ocr--host 0.0.0.0--port 12321--no-enable-prefix-caching--served-model-nameglm-ocr--max-num-batched-tokens 2048--max-num-seqs 1--max-model-len 2048--additional-config '{"visual_model_path":"/home/user/models/glm_ocr/mmproj_xh2_glm-ocr_0.8b_2cores_v1.3.0_20260519.gguf"}'

Corresponding curl test command:

curl--location--requestPOST 'http://127.0.0.1:12321/v1/chat/completions'--header 'Content-Type: application/json'--data-raw '{"model":"glm-ocr","messages":[{"role":"user","content":[{"type":"image_url","image_url":{"url":"https://cdn.bigmodel.cn/markdown/1770033643596image.png?attname=image.png"}},{"type":"text","text":"请识别图片中的所有文字。"}]}],"temperature":0,"max_tokens":256}'

Local image test command:

curl-XPOSThttp://127.0.0.1:12321/v1/chat/completions-H "Authorization: Bearer sk-123"-H "Content-Type: application/json"-d@- <<EOF
{
 "model": "glm-ocr",
 "messages": [
 {
 "role": "user",
 "content": [
 {
 "type": "image_url",
 "image_url": {
 "url": "data:image/jpeg;base64,$(base64 -w 0 /tmp/glm_ocr_test.jpg)"
 }
 },
 {
 "type": "text",
 "text": "请识别图片中的所有文字。"
 }
 ]
 }
 ]
}
EOF

Hardware Requirements (Minimum)

• CPU architecture: x86_64, aarch64

• Disk: 200G

• AI hardware: M50

Software Requirements

• Operating system: Ubuntu 20.04+

• Driver version:

  • M50-v1.3.0

2.2.1 Directory Structure

An example of the directory structure of the inference service installation package is shown below.

.
├── hliepython
│ ├── config
│ │ ├── app_config.py
│ │ └── __init__.py
│ ├── config.yaml
│ ├── __init__.py
│ ├── main.py
│ ├── model_server
│ │ ├── engines
│ │ │ ├── asr_engine.py
│ │ │ ├── base_engine.py
│ │ │ ├── embedding_engine.py
│ │ │ ├── files
│ │ │ ├── image_engine.py
│ │ │ ├── __init__.py
│ │ │ ├── protocol
│ │ │ ├── text_engine.py
│ │ │ ├── tts_engine.py
│ │ │ └── utils.py
│ │ ├── inference_adapter.py
│ │ ├── utils
│ │ │ ├── convert_gguf_to_hmm.py
│ │ │ ├── gguf_view.py
│ │ │ ├── read_gguf.py
│ │ │ └── test.py
│ │ └── worker.py
│ └── utils
├── PKG-INFO
├── pyproject.toml
└── setup.cfg

2.2.2 Service Deployment

HLIEPython depends on the HLIEvLLM service. Install HLIEvLLM before installing HLIEPython.

Refer to the HLIEvLLM Deployment and Usage Guide to install HLIEvLLM.

(1) Download the HLIEPython installation package from the inference framework section of the Beelink AI website (https://llm.bee-link.cn/frameworks.php), and extract it

Extract the archive and enter the extracted folder, for example "HLIEPython-3.0.0". Run the following command to install HLIEPython:

pipinstall-e.

(2) Modify the startup configuration

Modify the config.yaml configuration file to configure the models to be started statically.

vim hliepython/config.yaml

Note

• [Required] Set the --max-num-batched-tokens and --max-model-len parameters to be consistent with the maximum length supported by the model.

• Static loading and dynamic loading cannot be used at the same time in the current version; otherwise, static and dynamic model ID conflicts occur.

• The YAML file format is strict. If startup fails with config.yaml-related errors, check whether the config.yaml format is correct, for example: indentation, spaces, and so on.

# 服务运行配置
server:
main_port: 7800
log_level: "info"
model_root_dir: "/home/useradmin/workspaces/models/v1.2.0" # 模型配置
models:
# Qwen3.5-9B 模型
-name: "Qwen3.5-9B"
instances:
-port: 7801
model_path: "qwen3.5_9B"
model_type: "text" # text、embedding、image、asr、tts
vllm_config:
max-model-len: 65536
max-num-seqs: 1
no-enable-prefix-caching: true
max-num-batched-tokens: 65536
additional_config:
device_ids: [0]
enable_cacheprompt: "False" # # Qwen3.5_VL-35B 模型
# - name: "Qwen3.5-VL_35B"
# instances:
# - port: 7802
# model_path: "qwen3.5_35b"
# model_type: "text"
# vllm_config:
# max-model-len: 262144
# max-num-seqs: 1
# no-enable-prefix-caching: true
# max-num-batched-tokens: 262144
# enable_mm_embeds: true
# additional_config:
# device_ids: [0]
# enable_cacheprompt: "False"
# "vit_model_name": "mmproj-qwen3.5-35b-a3b-p256-256K-b1-2core-1d-xh2-v1.3.0-20260420210756.gguf" # # whisper-medium 模型
# - name: "whisper-medium"
# instances:
# - port: 7803
# model_path: "whisper-medium"
# model_type: "asr"
# additional_config:
# device_ids: [0]
# enable_cacheprompt: ""

Configuration field description:

server: configures the startup items of the server side

• main_port: the main process port number

• log_level: the log level (cannot be modified in this version)

• vllm_log_level: the log level of the underlying hlievllm

• model_root_dir: the root directory for storing models

models: configures the models started statically

• name: the model name used in requests, consistent with the model ID returned by the /v1/models interface

• instances: model instances

  • port: the child process port number

  • model_path: the gguf model directory relative to model_root_dir; each model needs an independent directory for its gguf file

  • model_type: the model type; this version supports text, embedding, asr, tts

  • device_ids: device ids

  • vllm_config: configures the core inference engine parameters of the underlying vllm startup (Note: the vLLM parameter allowlist is not maintained here; whatever is configured is passed through. If a parameter is not supported by the AsyncEngineArgs of the current vLLM version, the underlying vLLM reports an error according to its own logic.)

  • additional_config: additional configuration for hliepython and vllm

    • device_ids: configures the device id used at startup

    • enable_cacheprompt: determines whether KV cache reuse is enabled

    • vit_model_name: adds a multimodal VIT model

Core parameters supported by the vLLM inference engine:

2.2.3 Service Startup

python-mhliepython.main

Checking the Server Status

curl-XGEThttp://0.0.0.0:7800/hlie/v1/status

Checking the Model Startup Status

curl-XGEThttp://0.0.0.0:7800/hlie/v1/models

Response parameter description:

• data: the model information list

• id: the model id, displayed consistently with the local model directory name

• port: the port occupied by the current model; 0 if not loaded

• status: whether the current model is loaded; 0 if not loaded, 1 if loaded

• object: the fixed value "model"

• owned_by: the owning inference engine, fixed value "HLIEPython"

• created: the creation time of the model information

• aliases: model aliases (not enabled in this version)

• tags: model tags (not enabled in this version)

Dynamic Loading Example

Loading a model

Load the corresponding model according to the startup configuration parameters.

curl-XPOSThttp://0.0.0.0:7800/hlie/v1/models-H "Content-Type: application/json"-d '{ "model_name": "Qwen3-0.6B", "server_config": {"device_ids": [0], "model_type" : "text", "vllm_config": {"max-model-len": 16384, "max-num-seqs": 4, "max-num-batched-tokens": 16384}, "additional_config": {"enable_cacheprompt":"True"}}}'

Sending an inference request

curl-XPOSThttp://0.0.0.0:7801/v1/chat/completions-H "Authorization: Bearer sk-123"-H "Content-Type: application/json"-d '{"model": "Qwen3-0.6B", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "讲个笑话"}], "stream": true}'

Unloading a model

curl-XDELETEhttp://0.0.0.0:7800/hlie/v1/models/Qwen3-0.6B

LLM

curl-XPOSThttp://0.0.0.0:7801/v1/chat/completions-H "Authorization: Bearer sk-123"-H "Content-Type: application/json"-d '{"model": "Qwen3.5-9B", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "讲个笑话"}], "stream": false}'

VLM

Compare local images and network images

curl-XPOSThttp://0.0.0.0:7802/v1/chat/completions-H "Authorization: Bearer sk-123"-H "Content-Type: application/json"-d '{
 "model": "Qwen3.5-VL_4B",
 "messages": [
 {
 "role": "user",
 "content": [
 {
 "type": "image_url",
 "image_url": {
 "url": "file:///path/to/images/test.jpg"
 }
 },
 {
 "type": "image_url",
 "image_url": {
 "url": "https://ask.qcloudimg.com/http-save/yehe-6503436/l8m9wq9ivy.jpeg"
 }
 },
 {
 "type": "text",
 "text": "请描述这两张图片的内容。"
 }
 ]
 }
 ],
 "stream": false
}'

ASR

curl-XPOSThttp://0.0.0.0:7803/v1/audio/transcriptions-H "Authorization: Bearer sk-123"-H "Content-Type: multipart/form-data"-F"model=whisper-medium"-F"language=zh"-F"response_format=json"-F"file=@test.wav"-F"stream=false"

WebSocket streaming interface:

Test the WebSocket interface demo program: [demo.py]

recv: {"type":"ready","model":"Qwen3-ASR-1.7B","port":7804,"message":"WebSocket ASR 已连接"}
recv: {"type":"ack","message":"配置已更新","language":"zh"} === 模式1：一次性发送整段音频 ===
[FULL] recv: {"type":"transcript","text":"languageAnd so, my fellow Americans, ask not what your country can do for you. Ask what you can do for your country.","chunk_index":1,"final":false} === 模式2：分片实时发送 ===
[CHUNK] recv: {"type":"ack","message":"音频缓冲已清空"}
[CHUNK] recv: {"type":"transcript","text":"languageAnd so, and so.","chunk_index":1,"final":false}
[CHUNK] recv: {"type":"transcript","text":"languageAnd so, my fellow Americans.","chunk_index":2,"final":false}
[CHUNK] recv: {"type":"transcript","text":"languageAnd so, my fellow Americans.","chunk_index":3,"final":false}
[CHUNK] recv: {"type":"transcript","text":"languageAnd so, my fellow Americans, ask.","chunk_index":4,"final":false}
[CHUNK] recv: {"type":"transcript","text":"languageAnd so, my fellow Americans, ask not.","chunk_index":5,"final":false}
[CHUNK] recv: {"type":"transcript","text":"languageAnd so, my fellow Americans, ask not what your country can do for you—ask what you can do for your country.","chunk_index":6,"final":false}
[CHUNK] recv: {"type":"transcript","text":"languageAnd so, my fellow Americans, ask not what your country can do for you—ask what you can do for your country.","chunk_index":7,"final":false}
[CHUNK] recv: {"type":"transcript","text":"languageAnd so, my fellow Americans, ask not what your country can do for you.","chunk_index":8,"final":false}
[CHUNK] recv: {"type":"transcript","text":"languageAnd so, my fellow Americans, ask not what your country can do for you. Ask what you can do for your country.","chunk_index":9,"final":false}
[CHUNK] recv: {"type":"transcript","text":"languageAnd so, my fellow Americans, ask not what your country can do for you. Ask what you can do for your country.","chunk_index":10,"final":false}
[CHUNK] recv: {"type":"transcript","text":"languageAnd so, my fellow Americans, ask not what your country can do for you. Ask what you can do for your country.","chunk_index":11,"final":false}
[CHUNK] recv: {"type":"transcript","text":"languageAnd so, my fellow Americans, ask not what your country can do for you. Ask what you can do for your country.","chunk_index":12,"final":false}
recv: {"type":"done","text":"languageAnd so, my fellow Americans, ask not what your country can do for you. Ask what you can do for your country.","final":true}

Note: the response data of the current interface is incremental and does not return only the independent result of the current small chunk. This is because ASR strongly depends on context; single-chunk results have large jitter, severe word segmentation problems, and poor punctuation.

The ws implementation is:

1. When a chunk is received, append it to the server-side buffer.

2. Re-transcribe using the complete audio accumulated up to the current point.

3. Return the current complete hypothesis text.

You can return different modes by setting the return_type field of the request to full or delta.

TTS

curl-XPOSThttp://0.0.0.0:7804/v1/audio/speech \
-H "Authorization: Bearer sk-123" \
-H "Content-Type: application/json" \
-d '{
 "model": "Fun-CosyVoice3-0.5B",
 "input": "你好呀！的同学们。",
 "voice": "default",
 "response_format": "wav"
 }' \
--outputspeech.wav

Embedding

curlhttp://0.0.0.0:7805/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-123" \
-d '{
 "input": "Your text string goes here",
 "model":"qwen3_embed"
 }'

Program Requirements

• Software requirements: the program is compatible with Python 3.7 and later. Make sure an appropriate Python version is installed to run the code.

• Dependency libraries: the program requires the following Python libraries:

  • openai: used to access the OpenAI API for inference calls.

Install the required dependencies with the following command:

pip install openai

Configuration Modification

The current inference service is compatible with the OpenAI SDK. You can run the following Demo program to access the inference service. Note: modify base_url in the program to the ip of the machine where the inference service runs.

Example Program

import os
from openai import OpenAI base_url = "0.0.0.0:7801" def create_chat_completion (messages, use_stream=False): client = OpenAI ( base_url=f"http://{base_url}/v1", api_key="sk-xxx",) response = client. chat. completions. create ( model="QWen2.5-7b", messages=messages, stream=use_stream, max_completion_tokens=100, top_p=0.8, frequency_penalty=1.4,) if use_stream : print ("Streaming response:") for chunk in response : print (chunk. choices [0]. delta. content or "", end="", flush=True) print () else : print ("Non-streaming response:") print (response. choices [0]. message. content) def function_chat (): chat_messages = [ { "role" : "system", "content" : "You are a helpful assistant.", }, { "role" : "user", "content" : "讲一个笑话", } ] create_chat_completion (messages=chat_messages, use_stream=False) if __name__ == "__main__" : function_chat ()

Run the script:

python3 openai_sdk.py