This reverts commit fd5206cc55.
Fixing the test case would require some awkard %if use that I'm not
sure would even work, or splitting it into 2 copies that are almost
identical.
Instead, always add -m for clang, which allows it for all targets,
but not for gcc which does not.