hf_mirrors/Mellum2-12B-A2.5B-Thinking-bf16
Model Introduction
Files & versions
Analysis

Mellum2 12B A2.5B Thinking - BF16 MLX

This is an MLX conversion of JetBrains/Mellum2-12B-A2.5B-Thinking, stored in bfloat16 without weight quantization.

Mellum2 Thinking is a reasoning-augmented Mixture-of-Experts assistant model. It has 64 experts with 8 active per token, a 131,072-token context window, and emits reasoning in <think>...</think> blocks before the final answer.

Conversion details

  • Source: JetBrains/Mellum2-12B-A2.5B-Thinking
  • Format: MLX safetensors
  • Precision: bfloat16
  • Quantization: none
  • License: Apache-2.0
  • EOS token: <|im_end|> (token ID 28)

The upstream config.json and generation_config.json identify token ID 0 as the EOS token, while the tokenizer identifies <|im_end|> (ID 28) as EOS. This conversion uses token ID 28 so MLX generation stops at the end of the assistant turn.

Usage

Install MLX-LM:

pip install -U mlx-lm

Chat with the model:

mlx_lm.chat \
  --model mlx-community/Mellum2-12B-A2.5B-Thinking-bf16 \
  --max-tokens 8192 \
  --temp 0.6 \
  --top-p 0.95

Or generate a single response:

mlx_lm.generate \
  --model mlx-community/Mellum2-12B-A2.5B-Thinking-bf16 \
  --prompt "Explain why 1024 is a power of two." \
  --max-tokens 2048 \
  --temp 0.6 \
  --top-p 0.95

Benchmarks

Benchmarked with oMLX 0.5.7 and MLX-LM 0.31.3 on an Apple M5 Max MacBook Pro with 128 GB unified memory, using the built-in Python-code workload and 1,024 generated tokens. Results vary with hardware, runtime versions, context length, and generation settings.

Prompt tokensTTFTPrefillGenerationEnd-to-endPeak memory
4,096812.3 ms5,042.4 tok/s93.8 tok/s11.7 s23.2 GB
8,1921,707.8 ms4,796.9 tok/s90.6 tok/s13.0 s23.4 GB
16,3844,286.3 ms3,822.4 tok/s79.6 tok/s17.2 s23.6 GB
32,76810,906.1 ms3,004.5 tok/s71.6 tok/s25.2 s23.9 GB
65,53626,813.7 ms2,444.1 tok/s58.0 tok/s44.5 s24.6 GB
131,07284,021.3 ms1,560.0 tok/s35.8 tok/s112.6 s26.0 GB

At a 4,096-token prompt, batch generation achieved 60.6, 89.0, and 116.1 aggregate generation tok/s at 2, 4, and 8 concurrent requests, respectively, compared with 93.8 tok/s for one request.

Model provenance

For the original model card, training details, benchmark results, and usage guidance, see the upstream JetBrains checkpoint.