Some rear movement is normal since hard braking transfers much weight bias to the front, which is why the front brake has such importance with hard braking.
Also, the harder you want to ride a bike, the more you can expect it to move around, all things being equal.
Still, even with the added movement of aggressive riding over rough terrain, the rider still needs the correct feel from the bike to be satisfied.
I'd see if I have the rear shock spring preload set to allow 3.5 inches of sag when you sit on the seat.
From there, assuming you don't have the clickers or any other settings already screwed wildly off from where they were in standard trim, I'd probably be increasing the rear compression and rebound a bit.
Beside all of that, you'll not be able to get top-shelf performance out of the stock shock (or fork) because it doesn't have that in it the way the bike comes - it's basically setup for street riding and gentle off-road use.
That's because the Yamaha factory stopped short of doing that to save money manufacturing the bike, and, the bike will always feel, at least under certain conditions, a bit bouncy and imprecise.
If you really want to step it up, a suspension shop will have to revalve the suspension for what you want to do with it, as the stock suspension sorely needs it.