Data formats

This page describes the data a partner gets from the Valiha /v1 API: the chart, the ChordPro text, the melody, the voices and the stems. It also describes the engine's analysis object, which the API does not send but which everything else is made from. The requests themselves are in the API reference.

What a song carries

GET /v1/songs/{id} answers a PublicSong. Its fields come from the song's current analysis:

GET /v1/songs/{id}/chordpro answers the same text as chart.chordpro, as text/plain. Each stem downloads with GET /v1/songs/{id}/stems/{name}.

chart is null until the first analysis is ready. When a song is analysed again, the API keeps showing the previous chart, stems, melody and voices until the new analysis is done.

The raw analysis object is not part of the API. A partner works from the derived chart above. The analysis object section is for people who run their own engine or read the engine's analysis.json directly.

Key, tempo and meter

Each one is null while the song has no analysis.

The chart in JSON

chart.sections and chart.chords let you place chords without parsing ChordPro. For a short made up song in G, they look like this:

{
  "sections": [
    {"kind": "intro", "label": "Intro", "start": 0, "end": 4},
    {"kind": "verse", "label": "Andininy 1", "start": 4, "end": 12},
    {"kind": "chorus", "label": "Fiv.", "start": 12, "end": 20},
    {"kind": "outro", "label": "Outro", "start": 20, "end": 24}
  ],
  "chords": [
    {"time": 0, "beat": 0, "name": "G"},
    {"time": 8, "beat": 16, "name": "C"},
    {"time": 10, "beat": 20, "name": "D"},
    {"time": 12, "beat": 24, "name": "G"},
    {"time": 16, "beat": 32, "name": "Em"},
    {"time": 18, "beat": 36, "name": "C"},
    {"time": 20, "beat": 40, "name": "D"},
    {"time": 22, "beat": 44, "name": "G"}
  ]
}

Sections:

• kind is one of verse, chorus, bridge, intro, outro, instrumental, other.
• label is the header from the lyrics ("Andininy 1", "Fiv."), or "" when there is none. The chart builder adds sections of its own for chords played while nobody sings: "Intro" and "Outro", "Chords" for a song without timed lyrics, and instrumental sections with an empty label between sung sections.
• start and end are seconds, rounded to the millisecond.

Chords:

• time is the start of the chord in seconds, rounded to the millisecond. A chord lasts until the next one starts; the last one lasts to the end of the song.
• beat is the index of the beat the chord starts on. Melody notes use the same beat numbers, see melody.json.
• name is a letter name: G, Em, D/F#, Bm7b5. A stretch with no chord is N.C..

Both lists are computed from the stored analysis with the default chart settings: standard depth, a chord at every change, slash chords from the bass line, letter names, no transposition. Edits made in the studio (hand fixes, another depth, a transposition) are not applied to them. Those edits are in chart.chordpro, see Studio directives.

ChordPro

Valiha writes standard ChordPro. The first chart of a song is made from the analysis with the default settings; a chart edited in the studio is saved as the text of the edited chart.

{title: Isaorana anie Ianao}
{key: G}
{tempo: 120}
{time: 4/4}

{comment: Intro}
[G]

{start_of_verse: Andininy 1}
[G]Tompo ô, Ianao no hirako
[C]Haleloia, [D]haleloia
{end_of_verse}

{start_of_chorus: Fiv.}
[G]Isaorana [Em]anie [C]Ianao
{end_of_chorus}

{comment: Outro}
[D] [G]

How it is written:

• The header holds title (the song title, without { or }), key, tempo (rounded to a whole number) and time (beats per bar over 4). A directive is left out when its value is missing.
• Verses, choruses and bridges are environments: {start_of_verse: <label>} ... {end_of_verse}, the label left out when there is none. Every other section is a {comment: <label>} line followed by its lines.
• Each chord is written in brackets just before the syllable being sung when it starts. A chord that starts after the last word of a line, in a short gap, goes at the end of the line.
• Chords played while nobody sings (intro, interludes, outro) are chord-only lines like [D] [G], four bars per line at most.
• Sections are separated by an empty line. The text ends with a newline.

Studio directives

A chart saved from the studio's chart editor carries two more directives right after the header. ChordPro readers ignore them: the spec keeps the x_ prefix for extensions.

{title: Isaorana anie Ianao}
{key: G}
{tempo: 120}
{time: 4/4}
{x_valiha_chart: depth=standard frequency=change notation=letters transpose=0 slash=1}
{x_valiha_fixes: 20=2:maj/6 21=2:maj/6 22=2:maj/6 23=2:maj/6 32=4:min7 33=4:min7 34=4:min7 35=4:min7}

{comment: Intro}
[G]

{start_of_verse: Andininy 1}
[G]Tompo ô, Ianao no hirako
[C]Haleloia, [D/F#]haleloia
{end_of_verse}

{start_of_chorus: Fiv.}
[G]Isaorana [Em7]anie [C]Ianao
{end_of_chorus}

{comment: Outro}
[D] [G]

x_valiha_chart holds the chart settings. It is always written on a studio save.

x_valiha_fixes is written only when someone fixed chords by hand. Each pair is <beat index>=<chord key>. A chord key is N (no chord) or <root>:<quality>, with an optional /<bass>: the root and the bass are pitch classes (0 is C, 11 is B) and the quality is one of min, maj, dim, aug, min6, maj6, min7, minmaj7, maj7, 7, dim7, hdim7, sus2, sus4. So 2:maj/6 is D/F# and 4:min7 is Em7.

The chord names in the body follow the settings: with notation=solfege they read Do, Ré; with notation=numbers they read 1, 4, 6m; a transposition moves them. The key directive is always a letter name, transposed.

GET /v1/songs/{id}/chordpro and chart.chordpro return the stored text as it is, with these directives when the chart was edited. The studio's own download (Export, "Download .cho") and copy leave them out. To strip them yourself:

const shareable = chordpro.replace(/^\{x_valiha_[^}\n]*\}[ \t]*(\r?\n|$)/gm, "");

melody.json

The main sung melody, version 1. It is in PublicSong.melody when the song's analysis asked for "game" in options.melody and the melody could be written; else it is null.

{
  "version": 1,
  "engine": {"version": "0.1.0"},
  "model": "game",
  "sha1": "3f2a9c41d0e8b7a6f5e4d3c2b1a0918273645546",
  "tuning": -12.4,
  "notes": [
    {"start": 4.012, "end": 4.398, "midi": 67, "beat": 8.0, "length": 0.75, "conf": 1.0, "syllable": [0, 0, 0, 0]},
    {"start": 4.402, "end": 4.851, "midi": 69, "beat": 8.75, "length": 1.25, "conf": 1.0, "syllable": [0, 0, 0, 1]},
    {"start": 21.3, "end": 21.92, "midi": 71, "beat": 42.5, "length": 1.5, "conf": 1.0}
  ],
  "timings": {"load": 0.7, "slice": 0.1, "encoder": 1.2, "segmenter": 9.6, "estimator": 0.4, "total": 12.0, "notes": 0.05}
}

A note:

beat uses the same beat numbers as chart.chords[].beat, so a note and a chord on the same beat line up. The beat times themselves (analysis.beats) and the lyrics structure that syllable points into are not in the API. For timing against the recording, use start and end.

voices.json

Every voice the choir model hears in the vocals, version 1. Experimental. It is in PublicSong.voices when the song's analysis asked for "choir" in options.melody, the engine has the model's weights and the run worked; else it is null.

{
  "version": 1,
  "model": "choir",
  "sha1": "3f2a9c41d0e8b7a6f5e4d3c2b1a0918273645546",
  "tuning": -12.4,
  "notes": [
    {"start": 4.01, "end": 4.52, "midi": 67, "velocity": 0.812},
    {"start": 4.01, "end": 4.47, "midi": 59, "velocity": 0.634},
    {"start": 4.53, "end": 4.9, "midi": 69, "velocity": 0.775}
  ],
  "timings": {"load": 1.8, "features": 5.1, "model": 10.4, "notes": 0.1, "total": 17.4}
}

start and end are seconds rounded to the millisecond, midi is an integer and velocity is how loud the model heard the note, 0 to 1. Voice notes have no beat grid and no syllables.

Stems

The stems are the separated parts of the recording. How many there are depends on the stem set the analysis was made with:

Every name, in the order the lists use: vocals, drums, bass, guitar, piano, other, music.

Three PublicSong fields go with them:

• options, for example {"stems": 4, "melody": ["game"]}: what the song's analysis is asked to make, as sent last. stems is 2, 4 or 6; melody lists the note models, "game" (the main melody) and "choir" (every voice), and [] asks for none. Without options a song uses {"stems": 4, "melody": ["game"]}.
• stems: the names of the stems the current analysis has, in the order above, [] when none. Only stems that reached the server are listed.
• melody and voices: see above.

options says what is asked; stems, melody and voices say what the current analysis has. They can differ, for example while a new analysis with other options runs, or when a note model was skipped.

Each stem is AAC (44.1 kHz stereo, about 160 kbps) in an .m4a file, served as audio/mp4. Range requests work (206 Partial Content, If-Range), and so do ETag and If-None-Match (304). A name that is not a stem name, or a stem the current analysis does not have, answers 404.

curl -H "Authorization: Bearer vk_live_..." \
  -o vocals.m4a https://valiha.marindrano.com/v1/songs/<id>/stems/vocals

From notes to MIDI

The studio has the MIDI files ready: in a song's Export tab (and its Melody tab), under Download, "MIDI of the melody" saves <title> melody.mid and "MIDI of all voices" saves <title> voices.mid. Without the studio, you can write the same file from PublicSong.

What the studio writes (apps/web/src/lib/midi-file.ts):

• A Standard MIDI File, type 1, 480 ticks a beat (a quarter note).
• One tempo for the whole song: the song's tempo, 120 BPM when it is not known.
• A first track named Tempo with the tempo and the time signature (beats per bar over 4), then one track per part, named Melody or Voices, each on its own channel (channel 10, the drums, is skipped), with program 0 (piano).
• Notes are placed by their times in seconds (start, end), not by their beats, so the file lines up with the recording in any sequencer.
• Velocity: 90 for melody notes (they have none), round(1 + 126 x velocity) for voice notes.
• A note that starts on a key still sounding ends the earlier note there.
• The studio also moves the notes by the chart's transposition, and its voices file leaves out notes shorter than 30 ms.

This script uses the Python standard library only. Given the same notes, tempo and meter, it writes the same bytes as the studio.

"""Writes a Standard MIDI File from the melody or the voices of a song (GET /v1/songs/{id})."""
import json
import math
import struct
import sys
import unicodedata

TICKS = 480  # ticks a beat, as in the studio's export


def js_round(x):
    return math.floor(x + 0.5)


def vlq(value):
    """A MIDI variable length number: 7 bits a byte, the high bit set on all but the last."""
    value = max(0, int(value))
    out = [value & 0x7F]
    value >>= 7
    while value:
        out.insert(0, (value & 0x7F) | 0x80)
        value >>= 7
    return bytes(out)


def meta(kind, data):
    return bytes([0xFF, kind]) + vlq(len(data)) + data


def track(events):
    """A track chunk from (tick, bytes) pairs in time order, with the end of track."""
    body, last = b"", 0
    for tick, data in events:
        body += vlq(tick - last) + data
        last = tick
    body += b"\x00\xff\x2f\x00"
    return b"MTrk" + struct.pack(">I", len(body)) + body


def midi_file(notes, bpm, beats_per_bar=4, name="Melody"):
    bpm = bpm if bpm and bpm > 0 else 120
    per_second = bpm / 60 * TICKS

    def tick(seconds):
        return js_round(max(0.0, seconds) * per_second)

    head = track([
        (0, meta(0x03, b"Tempo")),
        (0, meta(0x51, js_round(60_000_000 / bpm).to_bytes(3, "big"))),
        (0, meta(0x58, bytes([beats_per_bar, 2, 24, 8]))),  # beats_per_bar/4
    ])
    rows = []
    for n in notes:
        v = n.get("velocity")  # voices have one (0..1), melody notes do not
        velocity = 90 if v is None else js_round(1 + 126 * min(1.0, max(0.0, v)))
        rows.append({"on": tick(n["start"]), "off": tick(n["end"]),
                     "key": min(127, max(0, js_round(n["midi"]))), "vel": min(127, max(1, velocity))})
    rows.sort(key=lambda r: (r["on"], r["key"]))
    sounding = {}
    for r in rows:  # a note on a key still sounding ends the one before it there
        before = sounding.get(r["key"])
        if before and before["off"] > r["on"]:
            before["off"] = r["on"]
        sounding[r["key"]] = r
    events = []
    for r in rows:
        if r["off"] <= r["on"]:
            if any(o is not r and o["key"] == r["key"] and o["on"] == r["on"] for o in rows):
                continue  # two notes on one key at one tick: the first one goes
            r["off"] = r["on"] + 1
        events.append((r["on"], 1, bytes([0x90, r["key"], r["vel"]])))  # note on, channel 1
        events.append((r["off"], 0, bytes([0x80, r["key"], 64])))  # note off
    events.sort(key=lambda e: (e[0], e[1]))  # at one tick, note offs first
    label = unicodedata.normalize("NFC", name).encode("latin-1", "replace")
    part = track([(0, meta(0x03, label)), (0, bytes([0xC0, 0]))] + [(t, d) for t, _, d in events])
    return b"MThd" + struct.pack(">IHHH", 6, 1, 2, TICKS) + head + part


if __name__ == "__main__":
    song_file, which, out_file = sys.argv[1:4]  # song.json, melody or voices, out.mid
    song = json.load(open(song_file, encoding="utf-8"))
    notes = (song.get(which) or {}).get("notes") or []
    per_bar = int((song.get("timeSignature") or "4/4").split("/")[0])
    data = midi_file(notes, song.get("bpm") or 120, per_bar, which.capitalize())
    open(out_file, "wb").write(data)
    print(f"{len(notes)} notes written to {out_file}")

Save it as to_midi.py, then:

curl -s -H "Authorization: Bearer vk_live_..." https://valiha.marindrano.com/v1/songs/<id> > song.json
python3 to_midi.py song.json melody melody.mid
python3 to_midi.py song.json voices voices.mid

The analysis object

The engine writes one analysis.json per run (chordchart-engine analyze ... --out song.analysis.json, or the network worker, see Running your own engine). The web app stores it with each analysis and makes the chart, the ChordPro text and the facts from it. What it stores is exactly what the engine wrote, with two changes: stems is removed (local file paths; the stem files themselves are uploaded) and audio.path is replaced by the original file name.

This page describes version 2. Version 1 analyses have every field but beatBass, waveform, stemWaveforms and stems.

{
  "version": 2,
  "engine": {"version": "0.1.0", "device": "mps"},
  "audio": {"path": "isaorana.mp3", "sha1": "3f2a9c41d0e8b7a6f5e4d3c2b1a0918273645546", "duration": 203.418},
  "language": "mg",
  "tempo": 120.0,
  "beatsPerBar": 4,
  "beats": [0.512, 1.012, 1.511, 2.013, 2.512],
  "downbeats": [0, 4],
  "key": {"tonic": 7, "mode": "major", "name": "G", "confidence": 0.81},
  "chordVocab": ["C:min", "C", "C:dim", "C:aug"],
  "probScale": 10000,
  "beatProbs": [[0, 12, 0, 0]],
  "beatBass": [[0, 0, 0, 0, 0, 0, 0, 10000, 0, 0, 0, 0]],
  "sections": [],
  "waveform": [0, 3, 41, 87],
  "stemWaveforms": {"vocals": [0, 0, 22, 64], "drums": [0, 2, 30, 51]},
  "timings": {"models": 0.1, "decode": 1.4, "separate": 48.2, "beats": 3.1, "chords": 6.0, "bass": 2.2, "lyrics": 9.7, "stems": 0.3}
}

The arrays are cut short here: chordVocab always has 170 entries, every beatProbs row 170 and every beatBass row 12.

Beats, bars and tempo

Beat i lasts from beats[i] to beats[i + 1]; the last beat lasts one median beat interval, cut at the end of the song. downbeats names the beats where bars start, so with "downbeats": [0, 4] the second bar starts at beats[4]. Everything else that happens on the beat (beatProbs, beatBass, a melody note's beat, a chord's beat in the API) uses these beat indexes.

Key

tonic is a pitch class, 0 (C) to 11 (B). mode is major or minor. name is the tonic in sharps plus m for minor: G, Em, F#m. confidence is the correlation between the pitch classes of the song's chords and the Krumhansl-Kessler profile of that key, rounded to 0.001.

Chords: chordVocab, probScale and beatProbs

chordVocab is the 170 chord classes of the chord model, in model order. Class i < 168 has root ROOTS[floor(i / 14)] and quality QUALITIES[i % 14]:

ROOTS      C C# D D# E F F# G G# A A# B
QUALITIES  min maj dim aug min6 maj6 min7 minmaj7 maj7 7 dim7 hdim7 sus2 sus4

A major chord is written as its root alone, any other as root:quality, so the list starts "C:min", "C", "C:dim", "C:aug". Class 168 is "X" (a chord the model cannot name) and 169 is "N" (no chord).

probScale is 10000. beatProbs[b][i] is the probability of class i on beat b, as an integer out of probScale; a row sums to about probScale.

The most likely chord of each beat, in JavaScript:

// The most likely chord class of each beat, with its probability (0..1)
function bestChords(analysis) {
  return analysis.beatProbs.map((row, beat) => {
    let best = 0;
    for (let i = 1; i < row.length; i++) if (row[i] > row[best]) best = i;
    return {
      beat,
      time: analysis.beats[beat],
      chord: analysis.chordVocab[best], // "G", "E:min", "X" (unknown) or "N" (no chord)
      p: row[best] / analysis.probScale,
    };
  });
}

And in Python:

import json

analysis = json.load(open("song.analysis.json", encoding="utf-8"))
scale = analysis["probScale"]
for beat, (time, row) in enumerate(zip(analysis["beats"], analysis["beatProbs"])):
    best = max(range(len(row)), key=row.__getitem__)
    print(beat, time, analysis["chordVocab"][best], row[best] / scale)

The studio's chart does not take the best class beat by beat. It merges the classes for the chosen depth and finds the best chord sequence with a cost for every change, so its chords change less often than the beat by beat answer.

Bass notes: beatBass

beatBass[b] is 12 integers, the pitch class distribution of the bass note sounding on beat b, read from the bass stem (index 0 is C, 11 is B). A row sums to exactly probScale when a bass note sounds and is all zeros when the bass is silent or too weak. The largest value is the bass note; when it differs from the chord root, the chart writes a slash chord.

// The bass note of each beat as a pitch class (0 = C ... 11 = B), null when silent
const bassOf = (analysis) =>
  (analysis.beatBass ?? []).map((row) =>
    row.some((v) => v > 0) ? row.indexOf(Math.max(...row)) : null,
  );

Lyrics: sections, lines, words and syllables

The sections come from the lyrics text sent with the song: a header in brackets such as [Andininy 1] or [Fiv.] starts a section, a blank line starts a new stanza, and a stanza without a header that appears twice counts as a chorus. Then each word and syllable is aligned to the vocals.

{
  "label": "Andininy 1",
  "kind": "verse",
  "lines": [
    {
      "text": "Tompo ô, Ianao no hirako",
      "words": [
        {"text": "Tompo", "charStart": 0, "start": 4.0, "end": 4.794, "conf": 0.9,
         "syllables": [{"charStart": 0, "start": 4.0}, {"charStart": 2, "start": 4.422}]},
        {"text": "ô,", "charStart": 6, "start": 4.844, "end": 5.217, "conf": 0.9,
         "syllables": [{"charStart": 6, "start": 4.844}]},
        {"text": "Ianao", "charStart": 9,
         "syllables": [{"charStart": 9}, {"charStart": 11}]}
      ]
    }
  ]
}

start, end and conf are absent when a word could not be aligned, like Ianao above; its syllables then have only charStart. The chart puts each chord before the syllable sung closest to the chord's start, at that syllable's charStart.

Waveforms

waveform is 1200 integers from 0 to 100: the peak level of the full mix in 1200 equal slices of the song, linear, 100 being the loudest sample of the song. stemWaveforms has the same for each part of the stem set (vocals, drums, bass, other with 4 parts), 600 values each, on the same scale as the mix, so a quiet stem stays low.

Timings

timings gives the seconds spent in each stage: models, decode, separate, beats, chords, bass, lyrics, stems. The object an engine posts back to the web app also has write. A song analysed again reads its heavy steps from the engine's cache, so their timings are small.