valis-shot: checking a sound without listening

valis-shot measures a sound file. Use it when you cannot listen, or want a number to compare. It is a port of ideas from earshot, written in C++ under src/shot/. It is a work in progress.

This page covers the command itself. Measuring a sound is the overview of every measurement Valis offers, and where valis-shot sits among the tools.

Everything it reports is measured, not heard. A number or a model can say a sound is odd. It cannot say a sound is good.

The four questions

Start with measure. Use is for the "does it sound right" question. Use listen only when you want to choose the competing sentences yourself.

measure

Prints one JSON object. Read signal.notes first: plain words about silence, clipping, DC offset or values that are not numbers. Then signal.pitch (the note and how many cents off), loudness and envelope.

compare

spectralDistanceDb is 0 for the same sound, a few dB for a small change, tens of dB for a different sound. A sound 6 dB louder reads about 5.6. loudnessDifferenceLu is the loudness change, second file minus first.

When the first file is 20 s or longer, sections says where the difference lies. The first file is cut at its own section boundaries (the sections of measure) and the same stretch of each file is compared: spectralDistanceDb, levelDifferenceDb (second minus first, RMS) and, for two stereo files, widthDifferenceDb (see stereoImage in measure). mostChangedSection is the index of the largest spectral distance. Both files are cut at the same times, so renders that are not the same length or whose parts have moved compare unlike music; a section that starts after the second file ends is left out.

is

Give it a file and one sentence. It scores the sentence against twenty common rivals (other instruments, a synthesizer preset, noise, a sine) and answers:

{"sentence":"a trumpet","verdict":"no","share":0.196,"rank":2,"of":20,
 "closest":[{"sentence":"a synthesizer keyboard preset","share":0.600},{"sentence":"a trumpet","share":0.196}, ...]}

verdict is yes at a share of 0.5 or more, maybe from 0.2, else no. Those cut-offs are a judgement. The useful part is closest: what the sound was mistaken for. A synthetic-sounding instrument shows up as a synthesizer preset.

listen

The same models without the rivals: the eight most likely AudioSet labels, and a cosine and share for each sentence you give. Read the share, not the cosine; a cosine alone means little. Give two or three real alternatives.

Matching a recording with the circuit's own knobs

fit_params (MCP) or OpDispatcher::fitParams plays one note, scores the render against a recording by spectral distance, and moves the circuit's val:Param values to lower it. It reports the distance before and after and the value found for each knob, and changes nothing. The search is a seeded evolution strategy, so the same call gives the same answer; a render takes a fraction of a second, so 60 evaluations is a few seconds to a minute. In the test it takes an SH-101 whose filter peak is wrong by 6500 Hz from 19.6 dB away to 0.2 dB and finds 1517 Hz where the answer is 1500. Give it slots to search a few knobs; searching all of them needs more evaluations. A small distance means the spectra agree, not that the sound is right.

With 24 evaluations or more it first tries a seeded spread of starting points over the whole range and climbs from the best two or three, because one climb from a poor start can stop at the edge of a knob's range (tried on the dub bass, it stopped at 1.8 dB where the true settings score 0; the spread reaches 0.45 dB in 120 renders). It releases the note at 80% of a whole recording, and not at all inside the window when only the start of a longer recording is matched; hold names the time in seconds if you know it. If the best match is still over 1 dB away the result says so in notes.

Aiming a mix at a loudness

A song's mix aims at a loudness itself. Give the song val:loudness, or pass --lufs N to valis-song mix (or lufs to the mix_song tool): the last gain brings the mix to that integrated loudness, held back so the true peak stays at or under -1 dBTP, and the reply says when the ceiling, not the target, set the gain. It uses the same measureLoudness as valis-shot measure. See songs. The Python helper that did this before is kept in scripts/archive/lufs.py.

Setting up the neural questions (is and listen)

cmake -S . -B build -DVALIS_WITH_ONNX=ON      # fetches ONNX Runtime 1.31.1, pinned and SHA-256 checked
cmake --build build
scripts/shot/fetch-models.sh                   # about 1.1 GB, once, into ~/.local/share/valis/models

Nothing downloads unless you run the script. A build without VALIS_WITH_ONNX, or without the models, answers is and listen with that reason and nothing else breaks. The first call in a process loads the models (a few seconds, about a gigabyte of memory); later calls are fast. Set VALIS_MODELS to keep the models elsewhere.

What it said about "Lights On, Nobody Home"

is on each rendered stem, with the sentence that should fit:

The plucked and struck parts pass. The trumpets, violins, harp and shakers do not, and the models say what they sound like instead. That agreed with a listener who called the first trumpets a cheap keyboard. The next work is in TODO.md.

Limits

• The models hear about ten seconds at a time. For a long file AST reads the loudest ten seconds and CLAP averages windows.
• CLAP stops at 14 kHz, so it cannot hear "air"; measure and compare can.
• CLAP does not know Valis. A score says how much a sound resembles a described sound, not whether a circuit is well made.
• The audio front ends have no reference implementation to compare with (that needs Python and torch). The tests use known answers instead: AST must name a sine Sine wave and hiss White noise, and CLAP must put a low tone nearer "a low, deep rumble" and a high tone nearer "a high pitched whistle". That rules out a gross mistake, not a small shift.
• If a model disagrees with your ears, trust your ears.

How it is built

The work is in include/valis/ShotReport.h and src/shot/ShotReport.cpp. The command line tool, the operation surface (OpDispatcher::measure, compare, soundsLike, listen) and the MCP server all call it, so there is one implementation.

Tests are in tests/shot/, each against a signal whose answer is known: a 997 Hz sine at -23 dBFS reads -23.0 LUFS, and a tone at a quarter of the sample rate has a true peak 3 dB above its sample peak. On the v3 mix, loudness is -12.3 LUFS with 7.5 LU range and a -0.97 dBTP true peak, louder than streaming services need.

The CLAP temperature used for shares, 38.6647, is exp(logit_scale_a) read from the published weights.

Weights

The weights are not in git and not downloaded by a build or a test. scripts/shot/fetch-models.sh fetches them from pinned commits and checks each file's SHA-256; a bad download is discarded. The tests skip themselves when the weights are absent.

The original models carry the licences shown. The ONNX repositories state none of their own and are conversions of those weights.

Not ported, and why

The two worth doing next, the free-mode modal bank and a parameter search, are in TODO.md.