LiteLLM + vLLM模型调用引擎架构

二、Docker 安装 vLLMdocker-compose.ymlversion: 3.7 services: vllm-qwen: image: vllm/vllm-openai:latest container_name: vllm-qwen runtime: nvidia environment: - NVIDIA_VISIBLE_DEVICESall volumes: - ./models:/models command: --model /models/qwen/Qwen2.5-0.5B-Instruct --host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.5 --max-model-len 1024 ports: - 8000:8000 litellm: image: ghcr.io/berriai/litellm:main-latest container_name: litellm volumes: - ./config.yaml:/app/config.yaml command: --config /app/config.yaml ports: - 4000:4000 depends_on: - vllm-qwen把模型放到modelsLiteLLM 配置config.yamlmodel_list: - model_name: qwen litellm_params: model: openai//models/qwen/Qwen2.5-0.5B-Instruct # 使用 vLLM 返回的完整模型 ID api_base: http://vllm-qwen:8000/v1 api_key: none启动服务docker compose up -d此过程比较慢因为下载的比较大。测试 vLLMcurl http://localhost:8000/v1/models测试 LiteLLMcurl http://localhost:4000/v1/models整体测试curl http://localhost:4000/v1/chat/completions -H Content-Type: application/json -d {\model\:\qwen\,\messages\:[{\role\:\user\,\content\:\你好\}]}python代码测试from openai import OpenAI client OpenAI( api_keyanything, base_urlhttp://10.61.104.181:4000/v1 ) response client.chat.completions.create( modelqwen, messages[ {role: user, content: 你好讲个笑话} ] ) print(response.choices[0].message.content)增加多个模型暂未尝试