Hello.
Don't worry, you won't break them if you apply the effort (strictly!) horizontally. Either direction (backward or forward). It obviously shouldn't be easy to fold them as you don't want your mirrors to flap on the go at high speed.
I've never folded mine myself, just found one folded out by some pedestrian to preen, touched/hit the side of a narrow doorway exiting a parking and had a car wash worker do it for some fine detailing.