Accent normalization (AN) seeks to convert non-native (L2) accented speech into standard (L1) speech while preserving speaker identity. The current techniques either require naturally recorded parallel L1–L2 speech for training, or suffer from quality degradation when supervised by synthesized targets. In this paper, we present TokAN, a token-based accent normalization framework that operates on self-supervised discrete speech tokens extracted from a L1–L2 jointly trained vector-quantization (VQ) tokenizer, without the need of synthetic supervisory speech. An autoregressive encoder-decoder model performs token-to-token conversion, translating L2-accented token sequences into the tokens of standard voice. We also introduce reinforcement learning (RL) post-training based on Group Relative Policy Optimization (GRPO), using word error rate and accent classifier confidence as complementary rewards. A non-autoregressive flow-matching synthesizer recovers the Mel-spectrogram from the converted tokens, conditioned on the source speaker embedding. We also develop a flow-matching duration predictor that supports total-duration-aware synthesis, making TokAN applicable to duration-critical tasks such as voice dubbing and live casting. Experiments on seven English accents demonstrate that TokAN reduced the word error rate from 12.40% to 9.89% after supervised fine-tuning, and further to 9.23% after RL post-training, consistently outperforming frame-to-frame, direct flow-matching, and prompt-based token-conversion baselines in terms of accent reduction and intelligibility.
You can download all audio files on this page by cloning this github repository.
Each column corresponds to a different speaker utterance and each row to a different system. The Source row contains the original accented audio; subsequent rows show the accent-normalized outputs of the compared systems. TokAN-1 and TokAN-2 are the proposed token-based system after GRPO post-training; Resynthesis is the TokAN tokenize-and-synthesize pipeline without conversion, included as a reference.
Naming convention: -1 systems are duration-free (no total-duration constraint); -2 systems are duration-controlled (preserving the source total duration).
| ABA (Arabic) | YBAA (Arabic) | ZHAA (Arabic) | SKA (Arabic) | RRBI (Hindi) | TNI (Hindi) | ASI (Hindi) | SVBI (Hindi) | ERMS (Spanish) | EBVS (Spanish) | NJS (Spanish) | MBMPS (Spanish) | LXC (Chinese) | BWC (Chinese) | NCC (Chinese) | TXHC (Chinese) | HJK (Korean) | YDCK (Korean) | YKWK (Korean) | HKK (Korean) | HQTV (Vietnamese) | PNV (Vietnamese) | THV (Vietnamese) | TLV (Vietnamese) | BDL (American) | RMS (American) | SLT (American) | CLB (American) | ABA (Arabic) | YBAA (Arabic) | ZHAA (Arabic) | SKA (Arabic) | RRBI (Hindi) | TNI (Hindi) | ASI (Hindi) | SVBI (Hindi) | ERMS (Spanish) | EBVS (Spanish) | NJS (Spanish) | MBMPS (Spanish) | LXC (Chinese) | BWC (Chinese) | NCC (Chinese) | TXHC (Chinese) | HJK (Korean) | YDCK (Korean) | YKWK (Korean) | HKK (Korean) | HQTV (Vietnamese) | PNV (Vietnamese) | THV (Vietnamese) | TLV (Vietnamese) | BDL (American) | RMS (American) | SLT (American) | CLB (American) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
The girl faced him her eyes shining with sudden fear |
It lived in perpetual apprehension of that quarter of the compass |
I never allow what can't be changed to annoy me |
Suddenly his fingers closed tightly over the handkerchief |
But he reconciled himself to it by an act of faith |
He made sure that the magazine was loaded and resumed his paddling |
Philip bent low over Pierre |
He looked like one who had passed through an uncomfortable hour or two |
I had been born with no organic chemical predisposition toward alcohol |
Philip knew that she was not an Indian |
I was brought up the way most girls in Hawaii are brought up |
He was sure now of but few things |
He caught himself with a jerk |
Do you know that you are shaking my confidence in you |
One by one the boys were captured |
Obviously it was a disease that could be contracted by contact |
But there came no promise from the bow of the canoe |
He was smooth shaven and his hair and eyes were black |
They are his tongue by which he makes his knowledge articulate |
Now you understand |
His immaculate appearance was gone |
Fresh meat they failed to obtain |
It occurred to me that there would have to be an accounting |
He moved away as quietly as he had come |
All operations have been carried on from Montreal and Toronto. |
There's not an iota of truth in it. |
Scarcely had he uttered the name when Pierre's closing eyes shot open. |
Mab, she said. |
A bush chief had died a natural death |
They edged nearer and stood shoulder to shoulder facing their world |
Bassett was a fastidious man |
It wasn't a large lake and almost round |
Man could not conquer them |
There is not an iota of truth in it certainly not |
The very thought of the effort to swim over was nauseating |
It was beating and waiting in the ambush of those black pits |
That is the strange part of it |
But how are you going to do it |
From my earliest recollection my sleep was a period of terror |
They likewise are disinclined to being eaten |
She is essentially the life giving life conserving female of the species |
O'Brien had been a clean living young man with ideals |
A rising tide of fat had submerged them |
The land exchanged its austere robes for the garb of a smiling wanton |
Lots of men take women buggy riding |
Down there the earth was already swelling with life |
Miss Brodie's smile was slightly sarcastic |
But already he had composed himself |
Without them he could not run his empire |
What an excited whispering and conferring took place |
To my dearest and always appreciated friend I submit myself |
When I can't see beauty in woman I want to die |
I want to know how all this is possible. |
Of course much grumbling went on, and little outbursts were continually occurring. |
Now go ahead and tell me in a straightforward way what has happened. |
I was sick once -- typhoid. |
| Source | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| FramAN | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| CosyAccent-1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| CosyAccent-2 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| VEVO | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Resynthesis | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| TokAN-1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| TokAN-2 |
↔ Scroll horizontally to explore additional columns.
In this panel, the testing samples are relatively long compared to the text, due to accentedness and potential disfluencies.
| ABA (Arabic) | YBAA (Arabic) | ZHAA (Arabic) | SKA (Arabic) | RRBI (Hindi) | TNI (Hindi) | SVBI (Hindi) | ERMS (Spanish) | EBVS (Spanish) | NJS (Spanish) | MBMPS (Spanish) | LXC (Chinese) | BWC (Chinese) | NCC (Chinese) | TXHC (Chinese) | YDCK (Korean) | YKWK (Korean) | HKK (Korean) | HQTV (Vietnamese) | PNV (Vietnamese) | THV (Vietnamese) | TLV (Vietnamese) | ABA (Arabic) | YBAA (Arabic) | RRBI (Hindi) | SVBI (Hindi) | ERMS (Spanish) | EBVS (Spanish) | MBMPS (Spanish) | LXC (Chinese) | BWC (Chinese) | TXHC (Chinese) | YDCK (Korean) | HKK (Korean) | PNV (Vietnamese) | TLV (Vietnamese) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
They closed now until his fingers were like cords of steel |
I had been born with no organic chemical predisposition toward alcohol |
There is not an iota of truth in it certainly not |
He was sure now of but few things |
He heard a sound which brought him quickly into consciousness of day |
A rising tide of fat had submerged them |
He was sure now of but few things |
They are his tongue by which he makes his knowledge articulate |
He was smooth shaven and his hair and eyes were black |
He was sure now of but few things |
They closed now until his fingers were like cords of steel |
Scarcely had he uttered the name when Pierre's closing eyes shot open |
I was brought up the way most girls in Hawaii are brought up |
His immaculate appearance was gone |
She is essentially the life giving life conserving female of the species |
There is not an iota of truth in it certainly not |
Of course much grumbling went on and little outbursts were continually occurring |
They are his tongue by which he makes his knowledge articulate |
But Martin smiled a superior smile |
Down there the earth was already swelling with life |
But he reconciled himself to it by an act of faith |
The girl faced him her eyes shining with sudden fear |
The other felt a sudden wave of irritation rush through him |
They edged nearer and stood shoulder to shoulder facing their world |
Besides had he not whipped the big owl in the forest |
Lots of men take women buggy riding |
He looked like one who had passed through an uncomfortable hour or two |
Now go ahead and tell me in a straightforward way what has happened |
Now go ahead and tell me in a straightforward way what has happened |
It lived in perpetual apprehension of that quarter of the compass |
One by one the boys were captured |
The land exchanged its austere robes for the garb of a smiling wanton |
All operations have been carried on from Montreal and Toronto |
Without them he could not run his empire |
When I can't see beauty in woman I want to die |
They are his tongue by which he makes his knowledge articulate |
| Source | ||||||||||||||||||||||||||||||||||||
| FramAN | ||||||||||||||||||||||||||||||||||||
| CosyAccent-1 | ||||||||||||||||||||||||||||||||||||
| CosyAccent-2 | ||||||||||||||||||||||||||||||||||||
| VEVO | ||||||||||||||||||||||||||||||||||||
| Resynthesis | ||||||||||||||||||||||||||||||||||||
| TokAN-1 | ||||||||||||||||||||||||||||||||||||
| TokAN-2 |
↔ Scroll horizontally to explore additional columns.
These samples are intended to check the influence of the duration condition. The condition values are ×0.5, ×1.0, and ×1.5 of the source average duration. The duration values are generated without post-processing scaling. The samples were not deployed in the subjective evaluation.
| ABA (Arabic) | YBAA (Arabic) | ZHAA (Arabic) | SKA (Arabic) | RRBI (Hindi) | TNI (Hindi) | ASI (Hindi) | SVBI (Hindi) | ERMS (Spanish) | EBVS (Spanish) | NJS (Spanish) | MBMPS (Spanish) | LXC (Chinese) | BWC (Chinese) | NCC (Chinese) | TXHC (Chinese) | HJK (Korean) | YDCK (Korean) | YKWK (Korean) | HKK (Korean) | HQTV (Vietnamese) | PNV (Vietnamese) | THV (Vietnamese) | TLV (Vietnamese) | BDL (American) | RMS (American) | SLT (American) | CLB (American) | ABA (Arabic) | YBAA (Arabic) | ZHAA (Arabic) | SKA (Arabic) | RRBI (Hindi) | TNI (Hindi) | ASI (Hindi) | SVBI (Hindi) | ERMS (Spanish) | EBVS (Spanish) | NJS (Spanish) | MBMPS (Spanish) | LXC (Chinese) | BWC (Chinese) | NCC (Chinese) | TXHC (Chinese) | HJK (Korean) | YDCK (Korean) | YKWK (Korean) | HKK (Korean) | HQTV (Vietnamese) | PNV (Vietnamese) | THV (Vietnamese) | TLV (Vietnamese) | BDL (American) | RMS (American) | SLT (American) | CLB (American) | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text |
The girl faced him her eyes shining with sudden fear |
It lived in perpetual apprehension of that quarter of the compass |
I never allow what can't be changed to annoy me |
Suddenly his fingers closed tightly over the handkerchief |
But he reconciled himself to it by an act of faith |
He made sure that the magazine was loaded and resumed his paddling |
Philip bent low over Pierre |
He looked like one who had passed through an uncomfortable hour or two |
I had been born with no organic chemical predisposition toward alcohol |
Philip knew that she was not an Indian |
I was brought up the way most girls in Hawaii are brought up |
He was sure now of but few things |
He caught himself with a jerk |
Do you know that you are shaking my confidence in you |
One by one the boys were captured |
Obviously it was a disease that could be contracted by contact |
But there came no promise from the bow of the canoe |
He was smooth shaven and his hair and eyes were black |
They are his tongue by which he makes his knowledge articulate |
Now you understand |
His immaculate appearance was gone |
Fresh meat they failed to obtain |
It occurred to me that there would have to be an accounting |
He moved away as quietly as he had come |
All operations have been carried on from Montreal and Toronto. |
There's not an iota of truth in it. |
Scarcely had he uttered the name when Pierre's closing eyes shot open. |
Mab, she said. |
A bush chief had died a natural death |
They edged nearer and stood shoulder to shoulder facing their world |
Bassett was a fastidious man |
It wasn't a large lake and almost round |
Man could not conquer them |
There is not an iota of truth in it certainly not |
The very thought of the effort to swim over was nauseating |
It was beating and waiting in the ambush of those black pits |
That is the strange part of it |
But how are you going to do it |
From my earliest recollection my sleep was a period of terror |
They likewise are disinclined to being eaten |
She is essentially the life giving life conserving female of the species |
O'Brien had been a clean living young man with ideals |
A rising tide of fat had submerged them |
The land exchanged its austere robes for the garb of a smiling wanton |
Lots of men take women buggy riding |
Down there the earth was already swelling with life |
Miss Brodie's smile was slightly sarcastic |
But already he had composed himself |
Without them he could not run his empire |
What an excited whispering and conferring took place |
To my dearest and always appreciated friend I submit myself |
When I can't see beauty in woman I want to die |
I want to know how all this is possible. |
Of course much grumbling went on, and little outbursts were continually occurring. |
Now go ahead and tell me in a straightforward way what has happened. |
I was sick once -- typhoid. |
| Source | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Duration x0.5 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Duration x1.0 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Duration x1.5 |
↔ Scroll horizontally to explore additional columns.