as Snopes cautions:

which a scammer could authorize charges in another person's name simply by possessing a voice recording of that person saying "yes", without also already possessing a good deal of personal and account information for that person,

And you think they are not trawling for any info that will help them triangulate. Tel No. with a recording. Tel No. with a name. Tel No. with credit card number. It still needs more data but they can sell the data they have to someone who has a different data set. They don't need every entry to ring the cash till. Only one or two would feed them for months.