So I've made some progress with my synthetic voice. With about ~1000 voice clips the results are quite decent. Still, more can be done to improve the quality. 
https://twitter.com/andrewbrown/status/1414637875677876224