Most of the CMOS Cores are static. The clock rate can goto DC. Skip speed up/down too elaberate for a first pass. Look at your numbers compared to hours of use. If 10ma gives you 250 hours of use is that good enough? 100ma gets you 25 hours. Still a lot of chess. Sleep mode should get you by.

Pause: Look at the way you want it to work. If pause only makes it stop you do not need to debounce. Otherwise poll it it does not have to respond as fast.

Remember how fast the CPU is. Poll or interupt 10 or 20ms of error will not matter in your app. I did a slot car timer once. I was able to poll 8 tracks to 10 ms, update the displays, and poll a button. You have a lot less to do.

After you get it working, do not forget the bells and whistles.
Fastest move, slowest move, average move