Cheating
I was just going through Steve's code that does the musicseer evaluation and I discovered something bad: the results I had that showed that randomly-trained anchor models do as badly as a random sim metric is wrong. It turns out that steve code looks for the word "rand" in the name of a SIM file, and creates its own random SIM metric if it finds it. So my file SIM_ankrand12 was hitting this case and the actual contents of the file were being overridden with random numbers. So obviously this metric did as bad as random, it was random.
when i fix it, the real results aren't so good. The random anchors do almost as well as the "true" genre-based anchors, in some cases:
Mode & ank14v1Centroid & ankrand12Centroid & erdos & rand \\ \hline
Survey, all (6102 resp, 8.97 av.choices) & 3.9736 & 4.3296 & 3.8270 & 5.4193 \\ \hline
Survey, known (4739 resp, 3.59 av.choices) & 4.4577 & 4.7374 & 4.0704 & 5.4425 \\ \hline
Game, all (7124 resp, 11.10 av.choices) & 4.4532 & 4.6012 & 4.4940 & 5.4964 \\ \hline
Game, known (6244 resp, 4.72 av.choices) & 4.8654 & 4.9094 & 4.8661 & 5.4522 \\
What does this mean? Where does the improvement over randomness come from, when you train anchor models on random training labels? some thoughts:
- Perhaps the neural nets are learning something useful, even though they were given random training labels. I trained the random anchors by selecting several random artists and giving each net several songs by that artist. perhaps there is a bias in this process, and some anchors actually learn characteristics of a "dominant" artist in their training set.
- It may be an effect of the centroid. These results use the highly sophisticated "centroid" method of comparing distributions in anchor space, if you recall. Even on mfcc features that would probably do better than random. That should really be the experiment - modeling the distribution in mfcc space the same way as I model the anchor space distribution and comparing those. which, actually, is basically what we decided to do anyway, i.e. comparing to Beth's method.