This reverts commit a954558e87.
Thanks Yuanfang's help. I think I found the root cause of the buildbot
fail.
The failed test has both Memory and Immediate X86Operand. All data of
different operand kinds share the same memory space by a union
definition. So it has chance we get the wrong result if we don't check
the operand kind.
It's probably it happen to be the correct value in my local environment
so that I can't reproduce the fail.
Differential Revision: https://reviews.llvm.org/D116090