The brooms affect the ice in front of the rock, which changes the rock's behaviour as it moves over the swept patch. You have to exert quite a bit of force to push the rock directly.
And if it had no effect why do they do it?
Sometimes by accident, I'm sure. And probably more relevant, sometimes out of sheer laziness.
Look, I find extending a finger to give the stone a boop after release completely baffling...but there's no chance at all that it affected the trajectory of the thing. You might as well "cheat" by blowing on it.