Slow generation on macos M1.

[r-leyshon]

I'm attempting to run the example code in the README and finding the below steps to take approx 200 seconds. I was hoping to investigate the use of outlines with my chatbot but that sort of delay would render it useless.

generator = outlines.generate.choice(model, ["Positive", "Negative"])
answer = generator(prompt)

Running on Apple M1 Pro, macos seqouia 15.2, outlines==0.1.11, outlines_core==0.1.26 . I also tried older installations of outlines and found no discernible difference to execution times.

[Martins6]

Could you please provide the full example that you used, @r-leyshon ?

try some specific config such as the following:

    model = models.transformers(
        "microsoft/Phi-3-mini-4k-instruct",
        model_kwargs={"device_map": "mps", "torch_dtype": torch.float16},
    )

[Martins6]

Also, as I have a 16G M3 Pro, I like to use 1.7B models at most, such as SmolLM2-1.7B-Instruct

[r-leyshon]

Thanks @Martins6 that extra config sped things up significantly, down to about 10s. This was running the example from the docs for reference:

import time

import outlines
import torch

model = outlines.models.transformers(
    "microsoft/Phi-3-mini-4k-instruct",
    model_kwargs={"device_map": "mps", "torch_dtype": torch.float16},    
    )

prompt = """You are a sentiment-labelling assistant.
Is the following review positive or negative?

Review: This restaurant is just awesome!
"""

t0 = time.time()
generator = outlines.generate.choice(model, ["Positive", "Negative"])
answer = generator(prompt)
print(answer)
t1 = time.time()
print(f"Time to generate: {t1 - t0}")

[Martins6]

You welcome! Keep rocking @r-leyshon