Hi I am new people join Modular community. Firstly I try to deploy a model according to quick start guidance ( Quickstart | Modular ). Considering my computer resources is limited (6GB memory), I changed the model to a small model from Huggingface. And the MAX server startup successful. But when I use curl to access the model, MAX server cannot give me anticipated response.
Here I will share the steps:
1 max serve --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --weight-path=janhq/TinyLlama-1.1B-Chat-v1.0-GGUF/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf
This step is successful and the system enters status:Server ready on http://0.0.0.0:8000
2 When I use following command to access the model: (ref. TinyLlama-Chat-v1.0-1.1B-Q4_K_M Model | MAX Builds )
(1)Request: curl -N http://0.0.0.0:8000/v1/chat/completions -H “Content-Type: application/json” -d ‘{
“model”: “TinyLlama/TinyLlama-1.1B-Chat-v1.0”,
“messages”: [
{“role”: “system”, “content”: “You are a helpful assistant.”},
{“role”: “user”, “content”: “Who won the World Series in 2020?”}
]
}’
(2)Response:
curl: (56) Recv failure: Connection reset by peer
The only my environment is difference with Quickstart is that I have to access internet through a proxy deployment in my computer, I donot know whether this difference makes the problem.
Did any other guys / experts meet this problem? Could you please give me more suggestions?
BR Walker