Creating a Virtual Singer VSTi
Hello,
I am one of the developers of Cantamus (https://cantamus.app/).
We have a quite nice sounding synthesis technology for the singing voice and we are exploring the idea of integrating it into Musescore.
After reading the Musescore handbook I came to the conclusion the only way to do that would be a VSTi. But there I haven't found much documentation about it beyond this: https://musescore.org/en/handbook/4/working-vst-and-vsti
We need to know how Musescore is sending the lyrics to the VSTi. In particular, it would be ideal for us if we could access the whole score in MusicXML format (or any other format we can transform). I know this can be done in plugins but there is no information about VSTis.
We believe this integration would be a dream come true for many vocal score composers.
Any help would be highly appreciated!
Comments
You can use Birchlabs JuicySFPlugin to manually assign the voices, but it's a pain in the neck to use. That is the way it is now with Musescore 4 unfortunately. But your app sounds amazing.
I'm kind of looking something similar, so far as I can tell there seem to be a few approaches:
Create a custom instruments.xml with a custom Staff Text + CC mapping to change phonemes and a MuseScore plugin that automatically generates the phoneme Staff Text based on the lyrics (https://musescore.org/en/handbook/developers-handbook/references/instru…). But, I'm not quite clear how well MS4 supports it. https://musescore.org/en/node/334701#MIDI_mapping_and_MIDI_CC
MIDI 2.0 also has Flex Data Messages which can be used to send lyrics. However, phonemization probably needs more than a single syllable to work nicely -- the plugin based approach would be able to look at the full text. Similarly, I'm not quite clear how well VST3 supports it -- VST3.7.9 does seem to have a kTextTypeID and kPhonemeTypeID, but the latter also has a TODO attached to it.
There seems to be something called MusePlaybackEvents in MS4, but there doesn't seem to be much documentation on it. https://musescore.org/en/node/325860, https://github.com/musescore/MuseScore/pull/9610, https://github.com/musescore/MuseScore/projects/40, https://github.com/musescore/MuseScore/tree/master/src/framework/mpe
From these, approach 2. seems the least amount of friction involved. i.e. MS4 sends lyrics data using text events to VST and then VST processes them as necessary.
In reply to I'm kind of looking… by egonelbre
I didn't find this earlier, but MIDI 1 also supports via meta events. which could be restricted to a specific channel as necessary.
That's an amazing idea!
Do you have some news about it? Is the integration proceeding?
That would really be a breakthrough!
I will say this since this an old post from Jun 2023, now its March 2024. VST's are not the way how Musescore Studio is developing things. If you want to get dedicated sound synthesis to be able to be in Musescore Studio then you should do a collab with developers of Muse Sounds by Muse Group. So you'll need to contact them. From what I heard on your Cantamus website, this synthesis technology really competes with Synthesizer V (what was used to create Hatsune Miku music recordings). If anything you have to email the Muse Group. Someone from Musescore studio devs said this on Discord on March 2024 and I quote:
"
If they want to integrate this with MuseScore, they have two options:
Option 1.
Step 1. Release it as a VST. Users would need to enter the lyrics into the VST manually.
Step 2. Optionally, contribute code to MuseScore so that MuseScore can automatically transfer the lyrics to the VST.
Option 2. Contact the Muse Sounds team and see what they are willing to do.
To my knowledge they haven't done any of these things yet, but I don't know everything that happens in the Muse Sounds team, so maybe they did reach out. But I don't know and probably nobody will know because the Muse Sounds team is not known for being very generous with development updates.
Building this technology into MuseScore is not an option (unless they make all of it open-source), since it's not possible to package non-GPL3(-compatible) stuff with MuseScore. (That's also why Muse Sounds need to be installed separately using Muse Hub.)", Casper Jeukendrup, Musescore Studio developer.
It's only to your team to do what you can. Honestly I'd prefer if you worked in collaboration with Muse Sounds team from Muse Group as it will give a much better result, but its entirely up to you.
Kind regards.
In reply to I will say this since this… by TobyTwo32
That was very useful, thanks!
I always suspected that Option 1 was the way to go.
Since manually entering the lyrics is a no-go (at least for a final polished product) we would need to do step 2, maybe exploring what egonelbre suggested above. Despite that, it would be much simpler for use to have the whole score before hand because there are somethings that you can't do unless you know the next notes/syllables.
I don't see why implementing the functionality of sending the lyrics to VSTis could create a license problem as long as we simply contribute it as PR to the project. Besides, other VSTis could benefit from it.
Regarding option 2, it would be great if somebody took care of adding this. I just fail to see what Muse Sound has to do with it. This sounds like Musescore functionality to me not Muse Sound. I'd be happy to learn how those 2 groups really differentiate, to be honest, it is a blurry line for me.
Thanks again!
In reply to That was very useful, thanks… by hparra
My (weak) understanding is that Muse Sounds is the interface by which way sound libraries can have more direct access to the data in Musescore (see, for example, audio imperia chorus), and that if you want to work on the integration of external sound libraries with Musescore then the Muse Sounds team are the ones that interface that. I could be wrong, though.
This is very interesting to me. Are there any updates on this since your last comment in March of this year?
I have no special knowledge of how you could achieve this integration, but I just want to weigh in and say I hope you do pursue it. I am on the free trial of Cantamus right now, and it's amazing. Fantastic work!