TokAN: Accent Normalization Using Self-Supervised Speech Tokens


Abstract

Accent normalization (AN) seeks to convert non-native (L2) accented speech into standard (L1) speech while preserving speaker identity. The current techniques either require naturally recorded parallel L1–L2 speech for training, or suffer from quality degradation when supervised by synthesized targets. In this paper, we present TokAN, a token-based accent normalization framework that operates on self-supervised discrete speech tokens extracted from a L1–L2 jointly trained vector-quantization (VQ) tokenizer, without the need of synthetic supervisory speech. An autoregressive encoder-decoder model performs token-to-token conversion, translating L2-accented token sequences into the tokens of standard voice. We also introduce reinforcement learning (RL) post-training based on Group Relative Policy Optimization (GRPO), using word error rate and accent classifier confidence as complementary rewards. A non-autoregressive flow-matching synthesizer recovers the Mel-spectrogram from the converted tokens, conditioned on the source speaker embedding. We also develop a flow-matching duration predictor that supports total-duration-aware synthesis, making TokAN applicable to duration-critical tasks such as voice dubbing and live casting. Experiments on seven English accents demonstrate that TokAN reduced the word error rate from 12.40% to 9.89% after supervised fine-tuning, and further to 9.23% after RL post-training, consistently outperforming frame-to-frame, direct flow-matching, and prompt-based token-conversion baselines in terms of accent reduction and intelligibility.

System Overview

Overview of the TokAN pipeline.
Overview of TokAN. (a) The SSL tokenizer encodes L2-accented speech into discrete tokens via a jointly trained VQ codebook. (b) The autoregressive token converter transforms L2-accented tokens into L1-accented ones. (c) The speech synthesizer generates the Mel-spectrogram from the converted tokens, conditioned on the source speaker embedding and an optional total-duration constraint. A pre-trained vocoder synthesizes the final waveform.

You can download all audio files on this page by cloning this github repository.

General System Comparison

Each column corresponds to a different speaker utterance and each row to a different system. The Source row contains the original accented audio; subsequent rows show the accent-normalized outputs of the compared systems. TokAN-1 and TokAN-2 are the proposed token-based system after GRPO post-training; Resynthesis is the TokAN tokenize-and-synthesize pipeline without conversion, included as a reference.

Naming convention: -1 systems are duration-free (no total-duration constraint); -2 systems are duration-controlled (preserving the source total duration).

ABA (Arabic) YBAA (Arabic) ZHAA (Arabic) SKA (Arabic) RRBI (Hindi) TNI (Hindi) ASI (Hindi) SVBI (Hindi) ERMS (Spanish) EBVS (Spanish) NJS (Spanish) MBMPS (Spanish) LXC (Chinese) BWC (Chinese) NCC (Chinese) TXHC (Chinese) HJK (Korean) YDCK (Korean) YKWK (Korean) HKK (Korean) HQTV (Vietnamese) PNV (Vietnamese) THV (Vietnamese) TLV (Vietnamese) BDL (American) RMS (American) SLT (American) CLB (American) ABA (Arabic) YBAA (Arabic) ZHAA (Arabic) SKA (Arabic) RRBI (Hindi) TNI (Hindi) ASI (Hindi) SVBI (Hindi) ERMS (Spanish) EBVS (Spanish) NJS (Spanish) MBMPS (Spanish) LXC (Chinese) BWC (Chinese) NCC (Chinese) TXHC (Chinese) HJK (Korean) YDCK (Korean) YKWK (Korean) HKK (Korean) HQTV (Vietnamese) PNV (Vietnamese) THV (Vietnamese) TLV (Vietnamese) BDL (American) RMS (American) SLT (American) CLB (American)
Text

The girl faced him her eyes shining with sudden fear

It lived in perpetual apprehension of that quarter of the compass

I never allow what can't be changed to annoy me

Suddenly his fingers closed tightly over the handkerchief

But he reconciled himself to it by an act of faith

He made sure that the magazine was loaded and resumed his paddling

Philip bent low over Pierre

He looked like one who had passed through an uncomfortable hour or two

I had been born with no organic chemical predisposition toward alcohol

Philip knew that she was not an Indian

I was brought up the way most girls in Hawaii are brought up

He was sure now of but few things

He caught himself with a jerk

Do you know that you are shaking my confidence in you

One by one the boys were captured

Obviously it was a disease that could be contracted by contact

But there came no promise from the bow of the canoe

He was smooth shaven and his hair and eyes were black

They are his tongue by which he makes his knowledge articulate

Now you understand

His immaculate appearance was gone

Fresh meat they failed to obtain

It occurred to me that there would have to be an accounting

He moved away as quietly as he had come

All operations have been carried on from Montreal and Toronto.

There's not an iota of truth in it.

Scarcely had he uttered the name when Pierre's closing eyes shot open.

Mab, she said.

A bush chief had died a natural death

They edged nearer and stood shoulder to shoulder facing their world

Bassett was a fastidious man

It wasn't a large lake and almost round

Man could not conquer them

There is not an iota of truth in it certainly not

The very thought of the effort to swim over was nauseating

It was beating and waiting in the ambush of those black pits

That is the strange part of it

But how are you going to do it

From my earliest recollection my sleep was a period of terror

They likewise are disinclined to being eaten

She is essentially the life giving life conserving female of the species

O'Brien had been a clean living young man with ideals

A rising tide of fat had submerged them

The land exchanged its austere robes for the garb of a smiling wanton

Lots of men take women buggy riding

Down there the earth was already swelling with life

Miss Brodie's smile was slightly sarcastic

But already he had composed himself

Without them he could not run his empire

What an excited whispering and conferring took place

To my dearest and always appreciated friend I submit myself

When I can't see beauty in woman I want to die

I want to know how all this is possible.

Of course much grumbling went on, and little outbursts were continually occurring.

Now go ahead and tell me in a straightforward way what has happened.

I was sick once -- typhoid.

Source
FramAN
CosyAccent-1
CosyAccent-2
VEVO
Resynthesis
TokAN-1
TokAN-2

↔ Scroll horizontally to explore additional columns.

Comparison on Long Samples

In this panel, the testing samples are relatively long compared to the text, due to accentedness and potential disfluencies.

ABA (Arabic) YBAA (Arabic) ZHAA (Arabic) SKA (Arabic) RRBI (Hindi) TNI (Hindi) SVBI (Hindi) ERMS (Spanish) EBVS (Spanish) NJS (Spanish) MBMPS (Spanish) LXC (Chinese) BWC (Chinese) NCC (Chinese) TXHC (Chinese) YDCK (Korean) YKWK (Korean) HKK (Korean) HQTV (Vietnamese) PNV (Vietnamese) THV (Vietnamese) TLV (Vietnamese) ABA (Arabic) YBAA (Arabic) RRBI (Hindi) SVBI (Hindi) ERMS (Spanish) EBVS (Spanish) MBMPS (Spanish) LXC (Chinese) BWC (Chinese) TXHC (Chinese) YDCK (Korean) HKK (Korean) PNV (Vietnamese) TLV (Vietnamese)
Text

They closed now until his fingers were like cords of steel

I had been born with no organic chemical predisposition toward alcohol

There is not an iota of truth in it certainly not

He was sure now of but few things

He heard a sound which brought him quickly into consciousness of day

A rising tide of fat had submerged them

He was sure now of but few things

They are his tongue by which he makes his knowledge articulate

He was smooth shaven and his hair and eyes were black

He was sure now of but few things

They closed now until his fingers were like cords of steel

Scarcely had he uttered the name when Pierre's closing eyes shot open

I was brought up the way most girls in Hawaii are brought up

His immaculate appearance was gone

She is essentially the life giving life conserving female of the species

There is not an iota of truth in it certainly not

Of course much grumbling went on and little outbursts were continually occurring

They are his tongue by which he makes his knowledge articulate

But Martin smiled a superior smile

Down there the earth was already swelling with life

But he reconciled himself to it by an act of faith

The girl faced him her eyes shining with sudden fear

The other felt a sudden wave of irritation rush through him

They edged nearer and stood shoulder to shoulder facing their world

Besides had he not whipped the big owl in the forest

Lots of men take women buggy riding

He looked like one who had passed through an uncomfortable hour or two

Now go ahead and tell me in a straightforward way what has happened

Now go ahead and tell me in a straightforward way what has happened

It lived in perpetual apprehension of that quarter of the compass

One by one the boys were captured

The land exchanged its austere robes for the garb of a smiling wanton

All operations have been carried on from Montreal and Toronto

Without them he could not run his empire

When I can't see beauty in woman I want to die

They are his tongue by which he makes his knowledge articulate

Source
FramAN
CosyAccent-1
CosyAccent-2
VEVO
Resynthesis
TokAN-1
TokAN-2

↔ Scroll horizontally to explore additional columns.

Duration Control

These samples are intended to check the influence of the duration condition. The condition values are ×0.5, ×1.0, and ×1.5 of the source average duration. The duration values are generated without post-processing scaling. The samples were not deployed in the subjective evaluation.

ABA (Arabic) YBAA (Arabic) ZHAA (Arabic) SKA (Arabic) RRBI (Hindi) TNI (Hindi) ASI (Hindi) SVBI (Hindi) ERMS (Spanish) EBVS (Spanish) NJS (Spanish) MBMPS (Spanish) LXC (Chinese) BWC (Chinese) NCC (Chinese) TXHC (Chinese) HJK (Korean) YDCK (Korean) YKWK (Korean) HKK (Korean) HQTV (Vietnamese) PNV (Vietnamese) THV (Vietnamese) TLV (Vietnamese) BDL (American) RMS (American) SLT (American) CLB (American) ABA (Arabic) YBAA (Arabic) ZHAA (Arabic) SKA (Arabic) RRBI (Hindi) TNI (Hindi) ASI (Hindi) SVBI (Hindi) ERMS (Spanish) EBVS (Spanish) NJS (Spanish) MBMPS (Spanish) LXC (Chinese) BWC (Chinese) NCC (Chinese) TXHC (Chinese) HJK (Korean) YDCK (Korean) YKWK (Korean) HKK (Korean) HQTV (Vietnamese) PNV (Vietnamese) THV (Vietnamese) TLV (Vietnamese) BDL (American) RMS (American) SLT (American) CLB (American)
Text

The girl faced him her eyes shining with sudden fear

It lived in perpetual apprehension of that quarter of the compass

I never allow what can't be changed to annoy me

Suddenly his fingers closed tightly over the handkerchief

But he reconciled himself to it by an act of faith

He made sure that the magazine was loaded and resumed his paddling

Philip bent low over Pierre

He looked like one who had passed through an uncomfortable hour or two

I had been born with no organic chemical predisposition toward alcohol

Philip knew that she was not an Indian

I was brought up the way most girls in Hawaii are brought up

He was sure now of but few things

He caught himself with a jerk

Do you know that you are shaking my confidence in you

One by one the boys were captured

Obviously it was a disease that could be contracted by contact

But there came no promise from the bow of the canoe

He was smooth shaven and his hair and eyes were black

They are his tongue by which he makes his knowledge articulate

Now you understand

His immaculate appearance was gone

Fresh meat they failed to obtain

It occurred to me that there would have to be an accounting

He moved away as quietly as he had come

All operations have been carried on from Montreal and Toronto.

There's not an iota of truth in it.

Scarcely had he uttered the name when Pierre's closing eyes shot open.

Mab, she said.

A bush chief had died a natural death

They edged nearer and stood shoulder to shoulder facing their world

Bassett was a fastidious man

It wasn't a large lake and almost round

Man could not conquer them

There is not an iota of truth in it certainly not

The very thought of the effort to swim over was nauseating

It was beating and waiting in the ambush of those black pits

That is the strange part of it

But how are you going to do it

From my earliest recollection my sleep was a period of terror

They likewise are disinclined to being eaten

She is essentially the life giving life conserving female of the species

O'Brien had been a clean living young man with ideals

A rising tide of fat had submerged them

The land exchanged its austere robes for the garb of a smiling wanton

Lots of men take women buggy riding

Down there the earth was already swelling with life

Miss Brodie's smile was slightly sarcastic

But already he had composed himself

Without them he could not run his empire

What an excited whispering and conferring took place

To my dearest and always appreciated friend I submit myself

When I can't see beauty in woman I want to die

I want to know how all this is possible.

Of course much grumbling went on, and little outbursts were continually occurring.

Now go ahead and tell me in a straightforward way what has happened.

I was sick once -- typhoid.

Source
Duration x0.5
Duration x1.0
Duration x1.5

↔ Scroll horizontally to explore additional columns.