For prediction model statistics
| chords of history | top-1 | top-3 | top-5 | top-10 | next-chord-change top-1 |
|---|---|---|---|---|---|
| 1 | 0.375 | 0.634 | 0.750 | 0.884 | 0.328 |
| 4 | 0.453 | 0.683 | 0.782 | 0.896 | 0.412 |
| 8 | 0.497 | 0.712 | 0.801 | 0.903 | 0.454 |
| 16 | 0.527 | 0.732 | 0.814 | 0.909 | 0.484 |
| 32 | 0.558 | 0.752 | 0.827 | 0.914 | 0.517 |
| 64 | 0.583 | 0.768 | 0.838 | 0.918 | 0.539 |
| 128 | 0.590 | 0.773 | 0.841 | 0.920 | 0.542 |
| label | classes | top-1 | repeat-last | margin |
|---|---|---|---|---|
| full | 49 | 0.590 | 0.289 | +0.301 |
| major/minor only | 25 | 0.620 | 0.320 | +0.300 |
| root only | 13 | 0.656 | 0.362 | +0.294 |
The above chart is about removing complexities from the chosen chord and counting those as the same chord or separating them.
| error type | share of errors |
|---|---|
| predicted the previous chord again | 33.4% |
| wrong root, unrelated | 23.6% |
| right root, wrong quality (e.g. 7 vs major) | 15.5% |
| root a whole tone away | 10.2% |
| fifth or fourth away, same quality | 9.1% |
| no-chord involved | 5.0% |
| relative major/minor | 3.3% |
It is very possible that the 80% goal is unreasonable. In many songs at any given point, a player is strumming the same chord over and over again so it makes sense that this is a prediction error. Additionally adding more training data didnt help after ~60k.
Tested different model sizes for generations
| size | params | nats/note (lower is better) | top-1 clean | top-5 clean | top-1 with +/-30 ms jitter | top-1 jitter + 80 ms sort | p95 decode, ms/token | train hours per epoch |
|---|---|---|---|---|---|---|---|---|
| XS | 10M | 1.665 | 0.748 | 0.923 | 0.490 | 0.730 | 0.90 | 3.6 |
| S | 26M | 1.592 | 0.758 | 0.928 | 0.499 | 0.741 | 1.42 | 8 |
| M | 53M | 1.521 | 0.772 | 0.934 | 0.512 | 0.754 | 1.85 | 14 |
| L | 114M | 1.479 | 0.778 | 0.936 | 0.521 | 0.762 | 3.25 | 25 |
| XL | 207M | 1.467 | 0.778 | 0.935 | 0.525 | 0.761 | 3.94 | 43 |
| XXL | 404M | not trained | – | – | – | – | 10.13 | about 90 |
This metric is given the previous information, how accurate is the generated token given the prediction.
The largest model ran out of memory on the 32gb gpu so I couldnt train it
Behind on training the XXL model but most likely I’ll skip it. It’ll be slower based on the trend, not helping accuracy.
Next week i’ll do pretraining for M or L model. Measure latency on laptop(or just keep it and test on the lab device and try to sample input and output on it.
Do more chord evaluation and latency, including when paired with the predictive model
