We conducted both a human preference ranking and an automated evaluation using NISQA, a widely recognized model for assessing speech quality and naturalness.

Expert human evaluators were enlisted from Mechanical Turk to review 50 diverse transcripts spanning general customer support, healthcare, payment information, maintenance, and other common conversational scenarios. Each transcript was evaluated three times by different annotators that passed the baseline test to compare both TTS providers. The ranking was based on quality preference defined by the voice clarity, naturalness, and emotional sensitivity. The results showed that out of 50 transcripts, ElevenLabs was preferred 14 times while Cartesia was preferred 36 times. Similarly, NISQA ratings awarded Cartesia a score of 4.7 for overall quality, compared to a score of 4.38 for ElevenLabs.

Independent platforms have produced similar evaluation results. According to LabelBox, a leading data labeling provider, Cartesia ranked first 27.93% of the time, compared to ElevenLabs at only 10.68%, across six different providers (Figure 1). In LabelBox's Speech Generation rankings on September 9, 2024, Cartesia also has the highest ELO rating, with ElevenLabs coming in at fourth place (Figure 2). Note also that ElevenLabs is a customer of LabelBox while Cartesia is not. Cartesia also holds the highest win rate of 75% on Artificial Analysis' Text-to-Speech Arena, a leading provider of independent model evaluations (Figure 3). Their Text-to-Speech Arena conducts a blind human preference test across different speech providers.

Figure 1: Cartesia ranks in front of other TTS providers like AWS, Eleven Labs, Deepgram, and Google.

Figure 2: Cartesia ranks in front of other TTS providers like AWS, Open AI, Eleven Labs, Google, and Deepgram in Elo ratings. Elo is a dynamic rating system used in competitive games to rank players. In this context, it's applied to models. Higher Elo ratings indicate better performance based on head-to-head rankings.

Figure 3: Cartesia compared to other TTS providers like Open AI, ElevenLabs, Google, Amazon, Azure, and more. Cartesia ranks first in Artificial Analysis' Text-to-Speech Arena.