Live is great but I don’t think it’d be feasible for most languages to be a real 1:1 translation in live.
Even a 10s delay allows for the whole sentence/phrase to be captured and translated in entirety. A lot of languages can drastically change meaning due to a word on the other side of the sentence.
It’s already a thing with near-zero delay. MS Teams does it (dunno about the translation) and the QSMP Minecraft server has a bunch of livestreamers from different countries who use it for realtime translation.
What actually happens is that the current sentence gets “corrected” several times as you keep speaking. It’s a bit jittery and if the word order differs significantly then the translated sentence might be a bit wonky for a few seconds, and there are a few misses but overall it works really well; at least well enough that people who don’t speak each others’ language can have a conversation in their native tongues with essentially no more delay than reading speed. I can easily follow a livestream in a foreign language with the live subtitles (which was not the case a mere 6 months ago for any language other than English).
Ironically, this might be an area where machine learning could be beneficial.
I’ve been watching a few projects that are attempting to live translate videos. We are very close
I absolutely hate to watch subtitles appear word for word. So no, please no live captions.
Live is great but I don’t think it’d be feasible for most languages to be a real 1:1 translation in live.
Even a 10s delay allows for the whole sentence/phrase to be captured and translated in entirety. A lot of languages can drastically change meaning due to a word on the other side of the sentence.
It’s already a thing with near-zero delay. MS Teams does it (dunno about the translation) and the QSMP Minecraft server has a bunch of livestreamers from different countries who use it for realtime translation.
[EDIT: Live demo from today. Shit’s impressive.]
What actually happens is that the current sentence gets “corrected” several times as you keep speaking. It’s a bit jittery and if the word order differs significantly then the translated sentence might be a bit wonky for a few seconds, and there are a few misses but overall it works really well; at least well enough that people who don’t speak each others’ language can have a conversation in their native tongues with essentially no more delay than reading speed. I can easily follow a livestream in a foreign language with the live subtitles (which was not the case a mere 6 months ago for any language other than English).
Amazing clip you posted seven months ago here. Doesn’t seem like it could even be any better now.