MobileLLaMA-1.4B-Base

PyTorch ops → torch_npu (Ascend ACL) → CANN (Ascend Compute Library)

python -m vllm.entrypoints.openai.api_server \
    --model mtgv/MobileLLaMA-1.4B-Base \
    --dtype float16 \
    --trust-remote-code

Prompt:    The capital of France is
Output:   Paris.

Prompt:    Machine learning is a subset of
Output:   AI, and is the application of the principles of AI to the problem of training algorithms.
Machine learning is an area of AI that uses the principles of AI to create algorithms that can be used
to make decisions about human behavior.

Prompt:    The Eiffel Tower is located in
Output:   Paris, France. It is one of the world's most recognizable structures, and it has been used as
a symbol of Paris since it was built. The Eiffel Tower is 324 meters tall, and it is made of wrought
iron.

Prompt:    Python is a programming language used for
Output:   various purposes. It is a high-level programming language and is very easy to learn.
Python has a wide range of applications in different fields, such as web development, scientific
computation, data science, and artificial intelligence.

Prompt:    Hello, how are you?
Output:   I hope you are doing well. I hope you're having a nice day. This week has been a bit crazy,
I've been busy working on a couple of different projects.

pip install vllm vllm-ascend

from vllm import LLM, SamplingParams

model_path = "/path/to/MobileLLaMA-1.4B-Base"

llm = LLM(
    model=model_path,
    tokenizer=model_path,
    max_model_len=2048,
    tensor_parallel_size=1,
    gpu_memory_utilization=0.9,
)

sampling_params = SamplingParams(
    max_tokens=256,
    temperature=0.7,
    top_p=0.9,
)

outputs = llm.generate(["Hello, how are you?"], sampling_params)
for output in outputs:
    print(output.outputs[0].text)

python -m vllm.entrypoints.openai.api_server \
    --model /path/to/MobileLLaMA-1.4B-Base \
    --max-model-len 2048 \
    --tensor-parallel-size 1 \
    --port 8000

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/path/to/MobileLLaMA-1.4B-Base",
        "prompt": "Hello, how are you?",
        "max_tokens": 100,
        "temperature": 0.7
    }'