Your wheel stays in the same position. Instead of a risk to hit the tyre with the fenders, you are now moving to risk to the far less forgiving wheel.
I admit, the chances are smaller of scraping/touching/... when a smaller/stretched tyre is used, but it surely does not eliminate the possibility.
I wouldn't do it
