Have you had an opportunity to test your system coupled to an actual load? The resonant frequency will change considerably when the motor shaft is coupled to the load unless the load inertia is small compared to the rotor interia. I remember an OLD CNC mill we used that had viscous dampers mounted directly to the motor shaft.
If you are able to capture the raw encoder signals; you will probably see many fast edges when the motor stops. The shift register filter can reject some of them (frequency dependant) low clk frequencies will make the count error much greater because the motor shaft and the encoder really are vibrating. In an "ideal" system the count would increment and decrement precisely the same number of times yielding zero error. Not so alas........