Rendered at 22:31:18 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
augment_me 55 minutes ago [-]
Crazy to not see a word about benchmarking/evaluation of the kernels.
IMO at this time when all performant GPU kernels are generated/hill-climbed, its the largest source of error we see.
Should be kernel run with a hot or cold cache in practice? Should your kernel be bitwise equivalent to a ref? What error margins should your kernel have to ref? How many times are you running the kernel to avoid thermal noise? Are you running on separate GPUs to avoid overfitting the current hardware? Can you cudagraph-compiling your kernel to remove launch overhead? Are you A/B B/A interleaving your kernel/reference runs to avoid running thermal bias?
IMO at this time when all performant GPU kernels are generated/hill-climbed, its the largest source of error we see.
Should be kernel run with a hot or cold cache in practice? Should your kernel be bitwise equivalent to a ref? What error margins should your kernel have to ref? How many times are you running the kernel to avoid thermal noise? Are you running on separate GPUs to avoid overfitting the current hardware? Can you cudagraph-compiling your kernel to remove launch overhead? Are you A/B B/A interleaving your kernel/reference runs to avoid running thermal bias?