Test of 22 open-weights TTS systems

Every system was run locally on one Ubuntu box with a single RTX 3090 — no API, no cloud, nothing rented — to find out how well open-weights text-to-speech actually speaks Romanian.

Reading the table

Sorted by the blind score. The player in each row is the same sentence for every system — “Bună dimineața! Afară sunt paisprezece grade și e înnorat.” — so you can hear them against each other without leaving this page.

“All 17 sentences” is the wall-clock time to synthesise the whole set, not one sentence. Piper Liana medium really does produce all seventeen in 9.6 seconds — about 95 seconds of speech. “Runs on CPU?” gives the speed against playback for the models that need no GPU: 10× faster than playback means a minute of audio takes six seconds to make. Everything here is one run on one RTX 3090 — no batching, no warm cache, no seed search — so it tells you whether a model fits on your card and how long you wait, not its best achievable quality.

† FireRed publishes no parameter count. 2 120 632 897 is counted from the safetensors header of fireredtts3_base, all fp32; the RedAE codec adds 943 776 322. Following the vendor's own download instruction pulls 20.8 GB, because the base and instruct variants share one repository.
‡ Higgs' weights are only 7.61 GiB. The rest is preallocation: the two-stage engine stands up two vLLM instances reserving 0.6 and 0.25 of whatever the card has free. Read it as needs a 24 GB card, not needs 21 GB.
§ TADA's manifest does not reconcile with itself — it records 1 244.4 s of total generation while the per-item figures in the same file sum to about 124 s. Both are reproduced as recorded, so read that cell as unresolved rather than as a measurement.

The score is one native listener, blind, on letter codes. Not a MOS panel. Where two scores are within a point, treat the ordering as an opinion. “Not publishable” in the listen column is a provenance rule, not a quality verdict: those systems were only ever run as clones of a reference clip this repository does not redistribute, so there is nothing to publish. ZONOS2 scored 9 and has no audio here for exactly that reason, while two systems scoring 1 and 2 are published in full.

Tried to run but failed, for various reasons

• Voxtral 4B TTS — environment built and all 17 sentences tokenised, then the vLLM-Omni orchestrator timed out at 600 s, twice. Not a stack fault: Higgs uses the same stack on the same machine and came up in 122 s.
• MegaTTS3 — its WaveVAE encoder is not public, so the shared reference cannot be encoded for it at all. It could not have been compared fairly under any schedule.
• Confucius4-TTS — 14 languages declared, Romanian not among them.
• Orpheus 3B — English expressivity ceiling; gated, in a 56.7 GB repository.
• Step-Audio-EditX — a speech editing model; declares no languages.
• scenema-audio — audio extracted from a video-generation model; no language claim.
• PilotTTS — zero downloads, no licence tag, no public inference code.

The last six were never run, so nothing about them was measured and nothing about them is claimed. Details for all seven are in METHOD.md.

Chatterbox ran, but not in Romanian. It rejects language_id='ro' with a ValueError, so it was run under Italian — the listener's verdict was “Italian, not Romanian”. Its score measures that substitution.