You're correct about the wheel spacers since the spacer on the sprocket side needs to be longer to reach further into the hub to contact the bearing on that side.
Actually, you sould be removing those things anyway in order to clean the parts and apply fresh grease to the oil seal lips and those spacers, as well as the wheel axle, itself.
It's this grease that keeps the seals alive, improves the ability of all of those parts to seal dirt and water out, and keeps all of those steel parts from rusting together.