Running your own engine

The Valiha web app (apps/web, on Cloudflare Workers) never analyses a song itself. It hands each analysis to an engine: a machine running services/engine as a small HTTP server (chordchart-engine serve, on 127.0.0.1:26835) behind a public HTTPS address, a Cloudflare Tunnel in Valiha's own setup. The engine downloads the recording from the web app, runs the analysis, uploads the stems and posts its progress and the result back. The formats it sends are in Data formats.

How an analysis runs

1. Someone starts an analysis: a new song or "Analyze again" in the studio, or POST /v1/songs and POST /v1/songs/{id}/analyze in the API. An open job of that song is cancelled first. The web app creates a job (status queued) with the song's options and sends {"jobId": "..."} to the Cloudflare Queue valiha-jobs.
2. The queue consumer picks an engine: enabled engines, most recently seen first. It skips an engine that failed in the last minute and asks GET /health of one not seen in the last 2 minutes. It sends the job with POST <baseUrl>/jobs (20 s timeout). A 202 or a 409 (the engine already has it) counts as taken; the job becomes dispatched.
3. When no engine takes it, the job waits with the message "No engine is online" and the queue tries again after 15, 30, 60, 120 and 240 seconds. After the sixth try the job fails with that message.
4. The engine downloads the audio, analyses it and posts running callbacks with its progress. After the analysis it uploads the stems of the job's set, one PUT each, then runs the note models the job asked for.
5. The engine posts the done callback with the analysis, the melody and the voices. The web app stores a new analysis version: the analysis object, the ChordPro text made from it, the key, tempo, meter and length, the stems its stems route actually stored, the melody, the voices and the job's options. The song becomes ready and its webhook, if any, is sent.
6. A failed callback marks the job and the song failed, with the engine's message, and drops the stems the job uploaded.

Two more cases:

• Lost jobs. When someone looks at a song, a job or the admin's job list, any job silent for 10 minutes is checked in the background. If an engine holds it, the web app asks GET /jobs/<jobId>: a 404 (the engine restarted) puts the job back in the queue, and after 5 dispatches it fails with "The engines lost this job too many times"; an engine answer of failed fails it with the engine's error; done without a callback fails it with "The engine finished but its result did not arrive". A job still queued is sent to the queue again.
• Cancelling. Cancel in the studio, or a new analysis of the same song, closes the open job as cancelled and sends DELETE /jobs/<jobId> to the engine holding it. A queued job is forgotten there; a running one finishes, and its callbacks are ignored.

The owners of the valiha organization can also pick an engine for an analysis in the studio. Such a job is offered to that engine only, and waits with "{name} is not online" while that engine does not answer. The choice is never sent to the engine; an unknown or disabled name runs as Automatic.

For local development, VALIHA_DISPATCH=direct makes the web app dispatch right away instead of through the queue; a job no engine takes then fails at once. VALIHA_PUBLIC_URL is the origin the web app gives engines for the audio, stems and callback addresses, so the engine must be able to reach it.

The engine HTTP contract

Every route but GET /health needs Authorization: Bearer <ENGINE_TOKEN>, the engine's token. A missing or wrong token answers 401. The server refuses to start without a token. JSON in and out.

{
  "ok": true,
  "name": "mac",
  "version": "0.1.0",
  "device": "mps",
  "busy": false,
  "queued": 0,
  "aligner": "mms",
  "models": {"stems6": false, "game": true, "choir": true},
  "options": {"stems": [2, 4, 6], "melody": ["game", "choir"]}
}

device is mps, cuda or cpu. models says which model files are on the engine's disk: stems6 (the 6 part separation model, downloaded by the first 6 part song, about 55 MB), game (GAME's ONNX files and onnxruntime) and choir (the choir model's weights and h5py). An older engine sends no models and no options: read them as false, and as stems 4 with melody ["game"].

One job runs at a time. The others wait in memory, in the order they came; a restart forgets them.

The job

What the web app posts to /jobs:

{
  "jobId": "cmh2x1k0p0001",
  "audioUrl": "https://<web app>/api/jobs/cmh2x1k0p0001/audio?token=<callback token>",
  "audioName": "isaorana.mp3",
  "lyrics": "[Andininy 1]\nTompo ô, Ianao no hirako",
  "language": "mg",
  "callbackUrl": "https://<web app>/api/jobs/cmh2x1k0p0001/callback",
  "callbackToken": "<64 hex characters>",
  "stemsUrl": "https://<web app>/api/jobs/cmh2x1k0p0001/stems",
  "options": {"stems": 4, "melody": ["game"]}
}

The options:

Without GAME's files, or when GAME fails, the engine writes the melody with its older method ("model": "classic"). Without the choir model's weights, "choir" is skipped. A note model that fails never fails the job: its result is null.

What the engine calls on the web app

The stems route refuses a wrong token (401), a closed job (410), a name that is not a stem name (404), a type that is not audio/* (415), a missing Content-Length (411) and a stem over 40 MB (413). A stem that arrives after its job closed is deleted again and answers 410. The callback route answers 401 for a wrong token and 400 for a body it cannot read. The audio download answers 503 with Retry-After: 10 when the storage cannot be read.

The engine tries a stem upload 3 times in all (pauses of 2 and 5 seconds, 120 s timeout each) and a callback 4 times in all (pauses of 1, 2 and 4 seconds, 30 s timeout each). A 4xx answer other than 408 and 429 is not retried. A stem that does not arrive is skipped and logged; it never fails the job. The web app trusts only the stems its route stored, not the list in the callback.

Callbacks

{"status": "running", "progress": 0.42, "stage": "chords", "message": "Recognizing the chords"}

{
  "status": "done",
  "analysis": {"version": 2, "audio": {"path": "isaorana.mp3", "sha1": "3f2a9c41d0e8b7a6f5e4d3c2b1a0918273645546", "duration": 203.418}},
  "engine": {"name": "mac", "device": "mps", "seconds": 212.4},
  "stems": ["vocals", "drums", "bass", "other"],
  "melody": {"version": 1, "model": "game", "notes": []},
  "voices": null
}

{"status": "failed", "error": "The lyrics could not be matched to the vocals"}

The done example is cut short: analysis is the whole analysis object of pipeline.analyze, with stems removed and audio.path replaced by the original file name, and melody and voices are the whole objects, or null when not asked or not written. stems, melody and voices are optional for older engines. Limits: message 500 characters, error 2000, stems 16 names.

Progress callbacks go out at most once every 2 seconds. stage is one of download, separate, beats, chords, align, write:

The download takes the first 5% of the progress bar. The end of the bar goes to the stem uploads (3%), then to GAME (10%, or 3% for the older melody method) and to the choir model (15%), each only when the job runs it.

Setting up an engine

The engine is a Python project managed with uv. The heavy steps are cached per audio file (~/Library/Caches/chordchart on a Mac, ~/.cache/chordchart on Linux), so analysing the same song again with new lyrics takes seconds.

On a Mac

From the repo root, once:

pnpm engine:setup    # uv sync, then every model (uv run chordchart-engine setup)

Make a token (32 random bytes in hex) and put it, with the engine's name, in services/engine/.env (gitignored):

openssl rand -hex 32

ENGINE_TOKEN=<the token>
ENGINE_NAME=mac

Then two terminals, from the repo root:

pnpm engine:serve    # the server, reads ENGINE_TOKEN and ENGINE_NAME from services/engine/.env
pnpm engine:tunnel   # the tunnel valiha-mac, with services/engine/worker/tunnel-mac.yml
curl https://valiha-engine-mac.marindrano.com/health

The Mac's tunnel has its own config file, services/engine/worker/tunnel-mac.yml. Other projects run their own tunnels on that Mac from ~/.cloudflared/config.yml: never edit that file and never run cloudflared service install there. The details are in the engine workers README.

On a Linux PC

The setup for an Arch based PC (Omarchy, EndeavourOS, Arch) lives in setup/linux, one script per step. From a clone of the repo:

./setup/linux/setup.sh <name> --tunnel-credentials ~/Downloads/<tunnel id>.json

<name> is the engine's name, lowercase letters, digits and dashes (default linux). --web also sets up the web app on that PC. The script installs the system packages and uv, runs uv sync and checks the graphics card, writes the token to services/engine/.env (a token already there is kept), downloads every model, sets up the tunnel, and starts two systemd user services, valiha-engine and valiha-tunnel, then checks /health. It can run again at any time. An NVIDIA card (GTX 16xx, RTX 20xx or newer, driver 580 or newer) makes analyses several times faster; without one the engine uses the CPU. Read the Linux setup README first.

The tunnel lives in the Cloudflare account that owns marindrano.com. Its owner creates it and sends the credentials file privately:

cloudflared tunnel --config /dev/null create valiha-<name>
cloudflared tunnel --config /dev/null route dns valiha-<name> valiha-engine-<name>.marindrano.com
cloudflared tunnel --config /dev/null route dns valiha-<name> engine-<name>.valiha.marindrano.com

Every day:

systemctl --user status valiha-engine valiha-tunnel
journalctl --user -u valiha-engine -f          # the engine's log, live
curl http://127.0.0.1:26835/health
curl https://valiha-engine-<name>.marindrano.com/health

Register https://valiha-engine-<name>.marindrano.com. The second name, engine-<name>.valiha.marindrano.com, fails over HTTPS until the zone gets a certificate for *.valiha.marindrano.com.

Settings

services/engine/.env (or the environment, which wins):

Restart the engine after a change (systemctl --user restart valiha-engine on Linux). For a new token on Linux, delete the ENGINE_TOKEN line, run the setup again and enter the new token in the admin page (or in VALIHA_ENGINES, when the engine is listed there).

Without the web app

The same pipeline runs from the command line and writes the formats of Data formats:

cd services/engine
uv run chordchart-engine analyze song.mp3 --lyrics lyrics.txt --language mg --out song.analysis.json --parts 4
uv run chordchart-engine melody --analysis song.analysis.json --out song.melody.json
uv run chordchart-engine doctor    # what the engine sees: device, ffmpeg, models

--stems-dir <folder> also puts the stems in a folder of your choice.

Registering an engine

Who can open the admin

The admin (/admin) is for signed-in owners of the valiha organization. People whose email is in VALIHA_OWNER_EMAILS (a list in the code when it is not set) become owners of the valiha and starapp organizations when they sign in. Anyone else signed in sees "Only the owners of the Valiha organization can open the admin."; a visitor who is not signed in is sent to sign in. The admin procedures refuse everyone else anyway. Owners find "Admin" in the account menu.

The Engines page

Admin, Engines (/admin/engines): "The machines that analyze songs. Valiha sends each job to an enabled one that answers." The table shows Name, Kind, Address, Device, Last seen ("Never" before the first answer) and Enabled ("On" or "Off"), with "Ping" and "Edit" on each row.

To add one, press "Add an engine" and fill in:

Then "Save". To change an engine, press "Edit": the dialog is titled "Edit {name}" and the token has to be entered again ("Enter the token again to save your changes."). "Remove" asks "Remove {name}?", and "Remove the engine" deletes it.

The Modal kind exists in the list, but services/modal/app.py is a modal run command today, not a server of the contract above, so it cannot take jobs from the web app.

Ping

"Ping" calls GET /health on the engine (8 s timeout) and shows "Online, {device}" (or "Online"), or "Offline. {error}". A good answer records the device and the time it was seen; a bad one records the error, shown later as "Last error: {error}". The dispatcher keeps those up to date too, and skips an engine for a minute after an error.

Enable or disable

The switch "Send jobs to this engine" turns an engine on or off. A disabled engine gets no jobs, is not asked what it can do, and is not offered as a choice; a job that names it runs as Automatic. Disabling is the way to take an engine out for a while, since a removed engine listed in VALIHA_ENGINES comes back.

VALIHA_ENGINES

Engines can also come from the web app's environment (.env.local in development): VALIHA_ENGINES is a JSON array of {name, kind, baseUrl, token}, the token at least 16 characters.

VALIHA_ENGINES='[{"name":"mac","kind":"mac","baseUrl":"https://valiha-engine-mac.marindrano.com","token":"<ENGINE_TOKEN of the mac>"}]'

The list is upserted by name into the engine table on the first use in each Worker isolate (the engine list, a dispatch, the studio's engine check). An engine that exists gets its kind, address and token from the list again, so edit those in the list, not in the admin; whether it is enabled is left alone. A list that is not valid JSON, or not a valid engine list, is logged and ignored.