How to Set Up an MLX Server on Mac

This guide shows how to set up an MLX server on Mac with mlx_lm.server, call it like the OpenAI API, keep it running with launchd, and reach it safely from elsewhere. It assumes mlx-lm is installed as in our MLX LM tutorial.

What mlx_lm.server gives you

mlx_lm.server ships with the mlx-lm package. It loads a model and exposes an HTTP API that is intended to be similar to OpenAI's chat API. From the official docs and source at the time of writing:

• Default address: 127.0.0.1, port 8080. Change them with --host and --port.
• Endpoints: POST /v1/chat/completions, POST /v1/completions, GET /v1/models and GET /health.
• Request fields include messages, max_tokens (default 512), temperature, top_p, stop, stream and logprobs, plus MLX-specific ones such as adapters and draft_model.

Two caveats matter for planning. The docs say the server is not recommended for production because it only implements basic security checks. And it has no API key setting of its own. So treat it as a private backend: keep it on localhost and put access control in front of it.

Start the server and test it with curl

Activate your virtual environment and start the server with a model from mlx-community. The first run downloads the model into the Hugging Face cache.

source ~/mlx-env/bin/activate
mlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit

In a second terminal, send a chat request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Say this is a test."}],
    "max_tokens": 100,
    "temperature": 0.7
  }'

The response follows the familiar shape: choices[0].message, a finish_reason of stop or length, and a usage block with token counts. Streaming, model listing and the health check work too:

# stream tokens as server-sent events
curl -N http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Count to five."}], "stream": true}'

curl http://127.0.0.1:8080/v1/models
curl http://127.0.0.1:8080/health

/health returns {"status": "ok"} when the server is ready, which makes it easy to use in monitoring scripts.

Use the MLX server with OpenAI-compatible clients

Most tools that accept a custom OpenAI base URL can talk to the server. With the official Python SDK (pip install openai):

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="unused")
resp = client.chat.completions.create(
    model="mlx-community/Mistral-7B-Instruct-v0.3-4bit",
    messages=[{"role": "user", "content": "One tip for faster builds?"}],
)
print(resp.choices[0].message.content)

The SDK requires an API key string, but the MLX server ignores it. If you add a reverse proxy with a bearer token (below), put the real token here. The model field is optional for mlx-lm. If you pass a different repo id, the server loads that model instead, so stick to one model unless you have memory for switching.

Run it as a launchd service

On a Mac mini or Mac Studio that serves a team, you want the server to start on its own and restart if it crashes. On macOS that is launchd's job. A LaunchAgent runs in your user session, which has normal access to the GPU and to your Hugging Face cache.

First run the model once by hand so the download is complete. Then save this as ~/Library/LaunchAgents/com.example.mlx-server.plist, replacing /Users/you with your home folder. launchd does not expand ~, so use full paths.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.example.mlx-server</string>
  <key>ProgramArguments</key>
  <array>
    <string>/Users/you/mlx-env/bin/mlx_lm.server</string>
    <string>--model</string>
    <string>mlx-community/Mistral-7B-Instruct-v0.3-4bit</string>
    <string>--host</string>
    <string>127.0.0.1</string>
    <string>--port</string>
    <string>8080</string>
  </array>
  <key>WorkingDirectory</key>
  <string>/Users/you/mlx</string>
  <key>RunAtLoad</key>
  <true/>
  <key>KeepAlive</key>
  <true/>
  <key>StandardOutPath</key>
  <string>/Users/you/mlx/server.log</string>
  <key>StandardErrorPath</key>
  <string>/Users/you/mlx/server.err.log</string>
</dict>
</plist>

WorkingDirectory matters: local model and adapter paths sent in requests are resolved relative to the folder the server started in. Load and check the service:

mkdir -p ~/mlx
plutil -lint ~/Library/LaunchAgents/com.example.mlx-server.plist
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.example.mlx-server.plist
launchctl print gui/$(id -u)/com.example.mlx-server | head -20
tail -f ~/mlx/server.err.log

To restart after changing the model, or to stop it:

launchctl kickstart -k gui/$(id -u)/com.example.mlx-server   # restart
launchctl bootout gui/$(id -u)/com.example.mlx-server        # stop and unload

A LaunchAgent starts when that user logs in. If the Mac reboots, the service comes back once the user session starts again, so on a headless server consider enabling automatic login for the service account (FileVault prevents automatic login). If you change the plist, bootout and bootstrap it again.

Keep it on localhost and use an SSH tunnel

The simplest secure setup is to leave --host at 127.0.0.1 and reach the server through SSH. From your laptop:

ssh -N -L 8080:127.0.0.1:8080 you@your-mac-address

Now http://127.0.0.1:8080 on your laptop reaches the server on the Mac, encrypted, and only for people with SSH access. Use key-based SSH login. Our guide to accessing a remote Mac covers SSH and the other methods.

Or add a reverse proxy with authentication

If other services or teammates need the API without SSH, put a reverse proxy in front that terminates TLS and checks a token. Here is an nginx example that requires an Authorization: Bearer header, which is exactly what OpenAI clients send. With Homebrew nginx, save it in /opt/homebrew/etc/nginx/servers/mlx.conf:

map $http_authorization $mlx_ok {
    default 0;
    "Bearer REPLACE_WITH_LONG_RANDOM_TOKEN" 1;
}

server {
    listen 8443 ssl;
    server_name mlx.example.com;
    ssl_certificate     /opt/homebrew/etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /opt/homebrew/etc/nginx/certs/privkey.pem;

    location / {
        if ($mlx_ok = 0) { return 401; }
        proxy_pass http://127.0.0.1:8080;
        proxy_buffering off;
        proxy_read_timeout 600s;
    }
}

brew install nginx
openssl rand -hex 32
nginx -t && brew services start nginx

Use the output of openssl rand -hex 32 as the token, supply your own TLS certificate, and keep the MLX server bound to 127.0.0.1 so the proxy is the only way in. proxy_buffering off lets streamed tokens through as they arrive. Test from outside:

curl https://mlx.example.com:8443/v1/models \
  -H "Authorization: Bearer $MLX_TOKEN"

Store the token in an environment variable or secret manager, not in code. Open only the proxy port in any firewall, and rotate the token if it leaks.

Tuning flags worth knowing

• --max-tokens sets the default response cap when a request does not send one.
• --kv-bits 4 or 8 quantises the KV cache for long contexts. The docs note it disables batching, so requests are served one at a time. Pair it with a smaller --prefill-step-size.
• --prompt-cache-size and --prompt-cache-bytes control how many reused prompt caches the server keeps.
• --decode-concurrency and --prompt-concurrency control parallel work for batchable requests.
• --allowed-origins defaults to *. Set it to your web app's origin if a browser calls the API.
• --adapter-path serves a LoRA adapter on top of the base model.

mlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --max-tokens 1024 --kv-bits 8 --prefill-step-size 512 \
  --allowed-origins https://app.example.com

Run mlx_lm.server --help for the current list, since flags change between releases.

Choosing a plan for your model size

For a shared endpoint, size for the model plus the context of the requests in flight. A 4-bit model needs roughly 0.5–0.7 GB per billion parameters, and by default macOS lets the GPU use roughly two-thirds to three-quarters of unified memory.

Model class (4-bit)	Suggested machine
3B–8B assistants, internal tools	Mac mini M4 or M6, 16GB
Up to about 14B, or 8B with longer context	Mac mini M6 24GB or Mac mini M5 Pro 24GB
About 30B class	Mac Studio M5 Max, 36GB
70B class	Mac Studio M5 Ultra, 96GB

These are rough guides that depend on context length and concurrency. More memory is available on request. For background, read running local LLMs on a Mac Studio, or see the Apple silicon AI server page.

Prices in USD per month for each machine's starting configuration. More memory, storage or several machines? Ask us.

Each Hatmac machine is a dedicated physical Mac assigned to one customer. After payment we set it up and email you, and the access details appear under My Macs in your account. Hatmac is independent and not affiliated with Apple.