Having a dig around on the mobile I couldn't find the music files anywhere. How was I to copy my music? Turns out that Google Music stores the music files in a sector of the memory card that you cannot access unless you have root access.
So you cannot move them around easily. Also watch out for the fact that Google Music applies its own indexing system so all your files are renamed to match a cloud based index database. They have numeric names now instead of the original ones. To work out what each song is you'll have to listen to them each in turn.