caio/clang-p2996
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
351e4fa0e2c98bd24dfd8b547f366e00a4d554b2
clang-p2996/llvm/test/CodeGen/X86/fast-isel-store.ll
Reid Kleckner 3a7a2e4a0a [FastISel] Sink local value materializations to first use 
Summary:
Local values are constants, global addresses, and stack addresses that
can't be folded into the instruction that uses them. For example, when
storing the address of a global variable into memory, we need to
materialize that address into a register.

FastISel doesn't want to materialize any given local value more than
once, so it generates all local value materialization code at
EmitStartPt, which always dominates the current insertion point. This
allows it to maintain a map of local value registers, and it knows that
the local value area will always dominate the current insertion point.

The downside is that local value instructions are always emitted without
a source location. This is done to prevent jumpy line tables, but it
means that the local value area will be considered part of the previous
statement. Consider this C code:
  call1();      // line 1
  ++global;     // line 2
  ++global;     // line 3
  call2(&global, &local); // line 4

Today we end up with assembly and line tables like this:
  .loc 1 1
  callq call1
  leaq global(%rip), %rdi
  leaq local(%rsp), %rsi
  .loc 1 2
  addq $1, global(%rip)
  .loc 1 3
  addq $1, global(%rip)
  .loc 1 4
  callq call2

The LEA instructions in the local value area have no source location and
are treated as being on line 1. Stepping through the code in a debugger
and correlating it with the assembly won't make much sense, because
these materializations are only required for line 4.

This is actually problematic for the VS debugger "set next statement"
feature, which effectively assumes that there are no registers live
across statement boundaries. By sinking the local value code into the
statement and fixing up the source location, we can make that feature
work. This was filed as https://bugs.llvm.org/show_bug.cgi?id=35975 and
https://crbug.com/793819.

This change is obviously not enough to make this feature work reliably
in all cases, but I felt that it was worth doing anyway because it
usually generates smaller, more comprehensible -O0 code. I measured a
0.12% regression in code generation time with LLC on the sqlite3
amalgamation, so I think this is worth doing.

There are some special cases worth calling out in the commit message:
1. local values materialized for phis
2. local values used by no-op casts
3. dead local value code

Local values can be materialized for phis, and this does not show up as
a vreg use in MachineRegisterInfo. In this case, if there are no other
uses, this patch sinks the value to the first terminator, EH label, or
the end of the BB if nothing else exists.

Local values may also be used by no-op casts, which adds the register to
the RegFixups table. Without reversing the RegFixups map direction, we
don't have enough information to sink these instructions.

Lastly, if the local value register has no other uses, we can delete it.
This comes up when fastisel tries two instruction selection approaches
and the first materializes the value but fails and the second succeeds
without using the local value.

Reviewers: aprantl, dblaikie, qcolombet, MatzeB, vsk, echristo

Subscribers: dotdash, chandlerc, hans, sdardis, amccarth, javed.absar, zturner, llvm-commits, hiraditya

Differential Revision: https://reviews.llvm.org/D43093

llvm-svn: 327581
2018-03-14 21:54:21 +00:00

752 lines
25 KiB
LLVM
Raw Blame History

; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
; RUN: llc -mtriple=x86_64-none-linux -fast-isel -fast-isel-abort=1 -mattr=+sse2 < %s | FileCheck %s --check-prefix=ALL32 --check-prefix=SSE32
; RUN: llc -mtriple=i686-none-linux -fast-isel -fast-isel-abort=1 -mattr=+sse2 < %s | FileCheck %s --check-prefix=ALL64 --check-prefix=SSE64
; RUN: llc -mtriple=x86_64-none-linux -fast-isel -fast-isel-abort=1 -mattr=+avx < %s | FileCheck %s --check-prefix=ALL32 --check-prefix=AVX32 --check-prefix=AVXONLY32
; RUN: llc -mtriple=i686-none-linux -fast-isel -fast-isel-abort=1 -mattr=+avx < %s | FileCheck %s --check-prefix=ALL64 --check-prefix=AVX64 --check-prefix=AVXONLY64
; RUN: llc -mtriple=x86_64-none-linux -fast-isel -fast-isel-abort=1 -mattr=+avx512f < %s | FileCheck %s --check-prefix=ALL32 --check-prefix=AVX32 --check-prefix=AVX51232 --check-prefix=KNL32
; RUN: llc -mtriple=i686-none-linux -fast-isel -fast-isel-abort=1 -mattr=+avx512f < %s | FileCheck %s --check-prefix=ALL64 --check-prefix=AVX64 --check-prefix=AVX51264 --check-prefix=KNL64
; RUN: llc -mtriple=x86_64-none-linux -fast-isel -fast-isel-abort=1 -mattr=+avx512vl,+avx512dq,+avx512bw < %s | FileCheck %s --check-prefix=ALL32 --check-prefix=AVX32 --check-prefix=AVX51232 --check-prefix=SKX32
; RUN: llc -mtriple=i686-none-linux -fast-isel -fast-isel-abort=1 -mattr=+avx512f,+avx512dq,+avx512bw < %s | FileCheck %s --check-prefix=ALL64 --check-prefix=AVX64 --check-prefix=AVX51264 --check-prefix=SKX64
define i32 @test_store_32(i32* nocapture %addr, i32 %value) {
; ALL32-LABEL: test_store_32:
; ALL32: # %bb.0: # %entry
; ALL32-NEXT: movl %esi, (%rdi)
; ALL32-NEXT: movl %esi, %eax
; ALL32-NEXT: retq
;
; ALL64-LABEL: test_store_32:
; ALL64: # %bb.0: # %entry
; ALL64-NEXT: movl {{[0-9]+}}(%esp), %eax
; ALL64-NEXT: movl {{[0-9]+}}(%esp), %ecx
; ALL64-NEXT: movl %eax, (%ecx)
; ALL64-NEXT: retl
entry:
store i32 %value, i32* %addr, align 1
ret i32 %value
}
define i16 @test_store_16(i16* nocapture %addr, i16 %value) {
; ALL32-LABEL: test_store_16:
; ALL32: # %bb.0: # %entry
; ALL32-NEXT: movw %si, (%rdi)
; ALL32-NEXT: movl %esi, %eax
; ALL32-NEXT: retq
;
; ALL64-LABEL: test_store_16:
; ALL64: # %bb.0: # %entry
; ALL64-NEXT: movzwl {{[0-9]+}}(%esp), %eax
; ALL64-NEXT: movl {{[0-9]+}}(%esp), %ecx
; ALL64-NEXT: movw %ax, (%ecx)
; ALL64-NEXT: retl
entry:
store i16 %value, i16* %addr, align 1
ret i16 %value
}
define <4 x i32> @test_store_4xi32(<4 x i32>* nocapture %addr, <4 x i32> %value, <4 x i32> %value2) {
; SSE32-LABEL: test_store_4xi32:
; SSE32: # %bb.0:
; SSE32-NEXT: paddd %xmm1, %xmm0
; SSE32-NEXT: movdqu %xmm0, (%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_4xi32:
; SSE64: # %bb.0:
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: paddd %xmm1, %xmm0
; SSE64-NEXT: movdqu %xmm0, (%eax)
; SSE64-NEXT: retl
;
; AVX32-LABEL: test_store_4xi32:
; AVX32: # %bb.0:
; AVX32-NEXT: vpaddd %xmm1, %xmm0, %xmm0
; AVX32-NEXT: vmovdqu %xmm0, (%rdi)
; AVX32-NEXT: retq
;
; AVX64-LABEL: test_store_4xi32:
; AVX64: # %bb.0:
; AVX64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX64-NEXT: vpaddd %xmm1, %xmm0, %xmm0
; AVX64-NEXT: vmovdqu %xmm0, (%eax)
; AVX64-NEXT: retl
%foo = add <4 x i32> %value, %value2 ; to force integer type on store
store <4 x i32> %foo, <4 x i32>* %addr, align 1
ret <4 x i32> %foo
}
define <4 x i32> @test_store_4xi32_aligned(<4 x i32>* nocapture %addr, <4 x i32> %value, <4 x i32> %value2) {
; SSE32-LABEL: test_store_4xi32_aligned:
; SSE32: # %bb.0:
; SSE32-NEXT: paddd %xmm1, %xmm0
; SSE32-NEXT: movdqa %xmm0, (%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_4xi32_aligned:
; SSE64: # %bb.0:
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: paddd %xmm1, %xmm0
; SSE64-NEXT: movdqa %xmm0, (%eax)
; SSE64-NEXT: retl
;
; AVX32-LABEL: test_store_4xi32_aligned:
; AVX32: # %bb.0:
; AVX32-NEXT: vpaddd %xmm1, %xmm0, %xmm0
; AVX32-NEXT: vmovdqa %xmm0, (%rdi)
; AVX32-NEXT: retq
;
; AVX64-LABEL: test_store_4xi32_aligned:
; AVX64: # %bb.0:
; AVX64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX64-NEXT: vpaddd %xmm1, %xmm0, %xmm0
; AVX64-NEXT: vmovdqa %xmm0, (%eax)
; AVX64-NEXT: retl
%foo = add <4 x i32> %value, %value2 ; to force integer type on store
store <4 x i32> %foo, <4 x i32>* %addr, align 16
ret <4 x i32> %foo
}
define <4 x float> @test_store_4xf32(<4 x float>* nocapture %addr, <4 x float> %value) {
; SSE32-LABEL: test_store_4xf32:
; SSE32: # %bb.0:
; SSE32-NEXT: movups %xmm0, (%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_4xf32:
; SSE64: # %bb.0:
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: movups %xmm0, (%eax)
; SSE64-NEXT: retl
;
; AVX32-LABEL: test_store_4xf32:
; AVX32: # %bb.0:
; AVX32-NEXT: vmovups %xmm0, (%rdi)
; AVX32-NEXT: retq
;
; AVX64-LABEL: test_store_4xf32:
; AVX64: # %bb.0:
; AVX64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX64-NEXT: vmovups %xmm0, (%eax)
; AVX64-NEXT: retl
store <4 x float> %value, <4 x float>* %addr, align 1
ret <4 x float> %value
}
define <4 x float> @test_store_4xf32_aligned(<4 x float>* nocapture %addr, <4 x float> %value) {
; SSE32-LABEL: test_store_4xf32_aligned:
; SSE32: # %bb.0:
; SSE32-NEXT: movaps %xmm0, (%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_4xf32_aligned:
; SSE64: # %bb.0:
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: movaps %xmm0, (%eax)
; SSE64-NEXT: retl
;
; AVX32-LABEL: test_store_4xf32_aligned:
; AVX32: # %bb.0:
; AVX32-NEXT: vmovaps %xmm0, (%rdi)
; AVX32-NEXT: retq
;
; AVX64-LABEL: test_store_4xf32_aligned:
; AVX64: # %bb.0:
; AVX64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX64-NEXT: vmovaps %xmm0, (%eax)
; AVX64-NEXT: retl
store <4 x float> %value, <4 x float>* %addr, align 16
ret <4 x float> %value
}
define <2 x double> @test_store_2xf64(<2 x double>* nocapture %addr, <2 x double> %value, <2 x double> %value2) {
; SSE32-LABEL: test_store_2xf64:
; SSE32: # %bb.0:
; SSE32-NEXT: addpd %xmm1, %xmm0
; SSE32-NEXT: movupd %xmm0, (%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_2xf64:
; SSE64: # %bb.0:
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: addpd %xmm1, %xmm0
; SSE64-NEXT: movupd %xmm0, (%eax)
; SSE64-NEXT: retl
;
; AVX32-LABEL: test_store_2xf64:
; AVX32: # %bb.0:
; AVX32-NEXT: vaddpd %xmm1, %xmm0, %xmm0
; AVX32-NEXT: vmovupd %xmm0, (%rdi)
; AVX32-NEXT: retq
;
; AVX64-LABEL: test_store_2xf64:
; AVX64: # %bb.0:
; AVX64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX64-NEXT: vaddpd %xmm1, %xmm0, %xmm0
; AVX64-NEXT: vmovupd %xmm0, (%eax)
; AVX64-NEXT: retl
%foo = fadd <2 x double> %value, %value2 ; to force dobule type on store
store <2 x double> %foo, <2 x double>* %addr, align 1
ret <2 x double> %foo
}
define <2 x double> @test_store_2xf64_aligned(<2 x double>* nocapture %addr, <2 x double> %value, <2 x double> %value2) {
; SSE32-LABEL: test_store_2xf64_aligned:
; SSE32: # %bb.0:
; SSE32-NEXT: addpd %xmm1, %xmm0
; SSE32-NEXT: movapd %xmm0, (%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_2xf64_aligned:
; SSE64: # %bb.0:
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: addpd %xmm1, %xmm0
; SSE64-NEXT: movapd %xmm0, (%eax)
; SSE64-NEXT: retl
;
; AVX32-LABEL: test_store_2xf64_aligned:
; AVX32: # %bb.0:
; AVX32-NEXT: vaddpd %xmm1, %xmm0, %xmm0
; AVX32-NEXT: vmovapd %xmm0, (%rdi)
; AVX32-NEXT: retq
;
; AVX64-LABEL: test_store_2xf64_aligned:
; AVX64: # %bb.0:
; AVX64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX64-NEXT: vaddpd %xmm1, %xmm0, %xmm0
; AVX64-NEXT: vmovapd %xmm0, (%eax)
; AVX64-NEXT: retl
%foo = fadd <2 x double> %value, %value2 ; to force dobule type on store
store <2 x double> %foo, <2 x double>* %addr, align 16
ret <2 x double> %foo
}
define <8 x i32> @test_store_8xi32(<8 x i32>* nocapture %addr, <8 x i32> %value) {
; SSE32-LABEL: test_store_8xi32:
; SSE32: # %bb.0:
; SSE32-NEXT: movups %xmm0, (%rdi)
; SSE32-NEXT: movups %xmm1, 16(%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_8xi32:
; SSE64: # %bb.0:
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: movups %xmm0, (%eax)
; SSE64-NEXT: movups %xmm1, 16(%eax)
; SSE64-NEXT: retl
;
; AVX32-LABEL: test_store_8xi32:
; AVX32: # %bb.0:
; AVX32-NEXT: vmovups %ymm0, (%rdi)
; AVX32-NEXT: retq
;
; AVX64-LABEL: test_store_8xi32:
; AVX64: # %bb.0:
; AVX64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX64-NEXT: vmovups %ymm0, (%eax)
; AVX64-NEXT: retl
store <8 x i32> %value, <8 x i32>* %addr, align 1
ret <8 x i32> %value
}
define <8 x i32> @test_store_8xi32_aligned(<8 x i32>* nocapture %addr, <8 x i32> %value) {
; SSE32-LABEL: test_store_8xi32_aligned:
; SSE32: # %bb.0:
; SSE32-NEXT: movaps %xmm0, (%rdi)
; SSE32-NEXT: movaps %xmm1, 16(%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_8xi32_aligned:
; SSE64: # %bb.0:
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: movaps %xmm0, (%eax)
; SSE64-NEXT: movaps %xmm1, 16(%eax)
; SSE64-NEXT: retl
;
; AVX32-LABEL: test_store_8xi32_aligned:
; AVX32: # %bb.0:
; AVX32-NEXT: vmovaps %ymm0, (%rdi)
; AVX32-NEXT: retq
;
; AVX64-LABEL: test_store_8xi32_aligned:
; AVX64: # %bb.0:
; AVX64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX64-NEXT: vmovaps %ymm0, (%eax)
; AVX64-NEXT: retl
store <8 x i32> %value, <8 x i32>* %addr, align 32
ret <8 x i32> %value
}
define <8 x float> @test_store_8xf32(<8 x float>* nocapture %addr, <8 x float> %value) {
; SSE32-LABEL: test_store_8xf32:
; SSE32: # %bb.0:
; SSE32-NEXT: movups %xmm0, (%rdi)
; SSE32-NEXT: movups %xmm1, 16(%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_8xf32:
; SSE64: # %bb.0:
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: movups %xmm0, (%eax)
; SSE64-NEXT: movups %xmm1, 16(%eax)
; SSE64-NEXT: retl
;
; AVX32-LABEL: test_store_8xf32:
; AVX32: # %bb.0:
; AVX32-NEXT: vmovups %ymm0, (%rdi)
; AVX32-NEXT: retq
;
; AVX64-LABEL: test_store_8xf32:
; AVX64: # %bb.0:
; AVX64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX64-NEXT: vmovups %ymm0, (%eax)
; AVX64-NEXT: retl
store <8 x float> %value, <8 x float>* %addr, align 1
ret <8 x float> %value
}
define <8 x float> @test_store_8xf32_aligned(<8 x float>* nocapture %addr, <8 x float> %value) {
; SSE32-LABEL: test_store_8xf32_aligned:
; SSE32: # %bb.0:
; SSE32-NEXT: movaps %xmm0, (%rdi)
; SSE32-NEXT: movaps %xmm1, 16(%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_8xf32_aligned:
; SSE64: # %bb.0:
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: movaps %xmm0, (%eax)
; SSE64-NEXT: movaps %xmm1, 16(%eax)
; SSE64-NEXT: retl
;
; AVX32-LABEL: test_store_8xf32_aligned:
; AVX32: # %bb.0:
; AVX32-NEXT: vmovaps %ymm0, (%rdi)
; AVX32-NEXT: retq
;
; AVX64-LABEL: test_store_8xf32_aligned:
; AVX64: # %bb.0:
; AVX64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX64-NEXT: vmovaps %ymm0, (%eax)
; AVX64-NEXT: retl
store <8 x float> %value, <8 x float>* %addr, align 32
ret <8 x float> %value
}
define <4 x double> @test_store_4xf64(<4 x double>* nocapture %addr, <4 x double> %value, <4 x double> %value2) {
; SSE32-LABEL: test_store_4xf64:
; SSE32: # %bb.0:
; SSE32-NEXT: addpd %xmm3, %xmm1
; SSE32-NEXT: addpd %xmm2, %xmm0
; SSE32-NEXT: movupd %xmm0, (%rdi)
; SSE32-NEXT: movupd %xmm1, 16(%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_4xf64:
; SSE64: # %bb.0:
; SSE64-NEXT: subl $12, %esp
; SSE64-NEXT: .cfi_def_cfa_offset 16
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: addpd {{[0-9]+}}(%esp), %xmm1
; SSE64-NEXT: addpd %xmm2, %xmm0
; SSE64-NEXT: movupd %xmm0, (%eax)
; SSE64-NEXT: movupd %xmm1, 16(%eax)
; SSE64-NEXT: addl $12, %esp
; SSE64-NEXT: retl
;
; AVX32-LABEL: test_store_4xf64:
; AVX32: # %bb.0:
; AVX32-NEXT: vaddpd %ymm1, %ymm0, %ymm0
; AVX32-NEXT: vmovupd %ymm0, (%rdi)
; AVX32-NEXT: retq
;
; AVX64-LABEL: test_store_4xf64:
; AVX64: # %bb.0:
; AVX64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX64-NEXT: vaddpd %ymm1, %ymm0, %ymm0
; AVX64-NEXT: vmovupd %ymm0, (%eax)
; AVX64-NEXT: retl
%foo = fadd <4 x double> %value, %value2 ; to force dobule type on store
store <4 x double> %foo, <4 x double>* %addr, align 1
ret <4 x double> %foo
}
define <4 x double> @test_store_4xf64_aligned(<4 x double>* nocapture %addr, <4 x double> %value, <4 x double> %value2) {
; SSE32-LABEL: test_store_4xf64_aligned:
; SSE32: # %bb.0:
; SSE32-NEXT: addpd %xmm3, %xmm1
; SSE32-NEXT: addpd %xmm2, %xmm0
; SSE32-NEXT: movapd %xmm0, (%rdi)
; SSE32-NEXT: movapd %xmm1, 16(%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_4xf64_aligned:
; SSE64: # %bb.0:
; SSE64-NEXT: subl $12, %esp
; SSE64-NEXT: .cfi_def_cfa_offset 16
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: addpd {{[0-9]+}}(%esp), %xmm1
; SSE64-NEXT: addpd %xmm2, %xmm0
; SSE64-NEXT: movapd %xmm0, (%eax)
; SSE64-NEXT: movapd %xmm1, 16(%eax)
; SSE64-NEXT: addl $12, %esp
; SSE64-NEXT: retl
;
; AVX32-LABEL: test_store_4xf64_aligned:
; AVX32: # %bb.0:
; AVX32-NEXT: vaddpd %ymm1, %ymm0, %ymm0
; AVX32-NEXT: vmovapd %ymm0, (%rdi)
; AVX32-NEXT: retq
;
; AVX64-LABEL: test_store_4xf64_aligned:
; AVX64: # %bb.0:
; AVX64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX64-NEXT: vaddpd %ymm1, %ymm0, %ymm0
; AVX64-NEXT: vmovapd %ymm0, (%eax)
; AVX64-NEXT: retl
%foo = fadd <4 x double> %value, %value2 ; to force dobule type on store
store <4 x double> %foo, <4 x double>* %addr, align 32
ret <4 x double> %foo
}
define <16 x i32> @test_store_16xi32(<16 x i32>* nocapture %addr, <16 x i32> %value) {
; SSE32-LABEL: test_store_16xi32:
; SSE32: # %bb.0:
; SSE32-NEXT: movups %xmm0, (%rdi)
; SSE32-NEXT: movups %xmm1, 16(%rdi)
; SSE32-NEXT: movups %xmm2, 32(%rdi)
; SSE32-NEXT: movups %xmm3, 48(%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_16xi32:
; SSE64: # %bb.0:
; SSE64-NEXT: subl $12, %esp
; SSE64-NEXT: .cfi_def_cfa_offset 16
; SSE64-NEXT: movaps {{[0-9]+}}(%esp), %xmm3
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: movups %xmm0, (%eax)
; SSE64-NEXT: movups %xmm1, 16(%eax)
; SSE64-NEXT: movups %xmm2, 32(%eax)
; SSE64-NEXT: movups %xmm3, 48(%eax)
; SSE64-NEXT: addl $12, %esp
; SSE64-NEXT: retl
;
; AVXONLY32-LABEL: test_store_16xi32:
; AVXONLY32: # %bb.0:
; AVXONLY32-NEXT: vmovups %ymm0, (%rdi)
; AVXONLY32-NEXT: vmovups %ymm1, 32(%rdi)
; AVXONLY32-NEXT: retq
;
; AVXONLY64-LABEL: test_store_16xi32:
; AVXONLY64: # %bb.0:
; AVXONLY64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVXONLY64-NEXT: vmovups %ymm0, (%eax)
; AVXONLY64-NEXT: vmovups %ymm1, 32(%eax)
; AVXONLY64-NEXT: retl
;
; AVX51232-LABEL: test_store_16xi32:
; AVX51232: # %bb.0:
; AVX51232-NEXT: vmovups %zmm0, (%rdi)
; AVX51232-NEXT: retq
;
; AVX51264-LABEL: test_store_16xi32:
; AVX51264: # %bb.0:
; AVX51264-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX51264-NEXT: vmovups %zmm0, (%eax)
; AVX51264-NEXT: retl
store <16 x i32> %value, <16 x i32>* %addr, align 1
ret <16 x i32> %value
}
define <16 x i32> @test_store_16xi32_aligned(<16 x i32>* nocapture %addr, <16 x i32> %value) {
; SSE32-LABEL: test_store_16xi32_aligned:
; SSE32: # %bb.0:
; SSE32-NEXT: movaps %xmm0, (%rdi)
; SSE32-NEXT: movaps %xmm1, 16(%rdi)
; SSE32-NEXT: movaps %xmm2, 32(%rdi)
; SSE32-NEXT: movaps %xmm3, 48(%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_16xi32_aligned:
; SSE64: # %bb.0:
; SSE64-NEXT: subl $12, %esp
; SSE64-NEXT: .cfi_def_cfa_offset 16
; SSE64-NEXT: movaps {{[0-9]+}}(%esp), %xmm3
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: movaps %xmm0, (%eax)
; SSE64-NEXT: movaps %xmm1, 16(%eax)
; SSE64-NEXT: movaps %xmm2, 32(%eax)
; SSE64-NEXT: movaps %xmm3, 48(%eax)
; SSE64-NEXT: addl $12, %esp
; SSE64-NEXT: retl
;
; AVXONLY32-LABEL: test_store_16xi32_aligned:
; AVXONLY32: # %bb.0:
; AVXONLY32-NEXT: vmovaps %ymm0, (%rdi)
; AVXONLY32-NEXT: vmovaps %ymm1, 32(%rdi)
; AVXONLY32-NEXT: retq
;
; AVXONLY64-LABEL: test_store_16xi32_aligned:
; AVXONLY64: # %bb.0:
; AVXONLY64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVXONLY64-NEXT: vmovaps %ymm0, (%eax)
; AVXONLY64-NEXT: vmovaps %ymm1, 32(%eax)
; AVXONLY64-NEXT: retl
;
; AVX51232-LABEL: test_store_16xi32_aligned:
; AVX51232: # %bb.0:
; AVX51232-NEXT: vmovaps %zmm0, (%rdi)
; AVX51232-NEXT: retq
;
; AVX51264-LABEL: test_store_16xi32_aligned:
; AVX51264: # %bb.0:
; AVX51264-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX51264-NEXT: vmovaps %zmm0, (%eax)
; AVX51264-NEXT: retl
store <16 x i32> %value, <16 x i32>* %addr, align 64
ret <16 x i32> %value
}
define <16 x float> @test_store_16xf32(<16 x float>* nocapture %addr, <16 x float> %value) {
; SSE32-LABEL: test_store_16xf32:
; SSE32: # %bb.0:
; SSE32-NEXT: movups %xmm0, (%rdi)
; SSE32-NEXT: movups %xmm1, 16(%rdi)
; SSE32-NEXT: movups %xmm2, 32(%rdi)
; SSE32-NEXT: movups %xmm3, 48(%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_16xf32:
; SSE64: # %bb.0:
; SSE64-NEXT: subl $12, %esp
; SSE64-NEXT: .cfi_def_cfa_offset 16
; SSE64-NEXT: movaps {{[0-9]+}}(%esp), %xmm3
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: movups %xmm0, (%eax)
; SSE64-NEXT: movups %xmm1, 16(%eax)
; SSE64-NEXT: movups %xmm2, 32(%eax)
; SSE64-NEXT: movups %xmm3, 48(%eax)
; SSE64-NEXT: addl $12, %esp
; SSE64-NEXT: retl
;
; AVXONLY32-LABEL: test_store_16xf32:
; AVXONLY32: # %bb.0:
; AVXONLY32-NEXT: vmovups %ymm0, (%rdi)
; AVXONLY32-NEXT: vmovups %ymm1, 32(%rdi)
; AVXONLY32-NEXT: retq
;
; AVXONLY64-LABEL: test_store_16xf32:
; AVXONLY64: # %bb.0:
; AVXONLY64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVXONLY64-NEXT: vmovups %ymm0, (%eax)
; AVXONLY64-NEXT: vmovups %ymm1, 32(%eax)
; AVXONLY64-NEXT: retl
;
; AVX51232-LABEL: test_store_16xf32:
; AVX51232: # %bb.0:
; AVX51232-NEXT: vmovups %zmm0, (%rdi)
; AVX51232-NEXT: retq
;
; AVX51264-LABEL: test_store_16xf32:
; AVX51264: # %bb.0:
; AVX51264-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX51264-NEXT: vmovups %zmm0, (%eax)
; AVX51264-NEXT: retl
store <16 x float> %value, <16 x float>* %addr, align 1
ret <16 x float> %value
}
define <16 x float> @test_store_16xf32_aligned(<16 x float>* nocapture %addr, <16 x float> %value) {
; SSE32-LABEL: test_store_16xf32_aligned:
; SSE32: # %bb.0:
; SSE32-NEXT: movaps %xmm0, (%rdi)
; SSE32-NEXT: movaps %xmm1, 16(%rdi)
; SSE32-NEXT: movaps %xmm2, 32(%rdi)
; SSE32-NEXT: movaps %xmm3, 48(%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_16xf32_aligned:
; SSE64: # %bb.0:
; SSE64-NEXT: subl $12, %esp
; SSE64-NEXT: .cfi_def_cfa_offset 16
; SSE64-NEXT: movaps {{[0-9]+}}(%esp), %xmm3
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: movaps %xmm0, (%eax)
; SSE64-NEXT: movaps %xmm1, 16(%eax)
; SSE64-NEXT: movaps %xmm2, 32(%eax)
; SSE64-NEXT: movaps %xmm3, 48(%eax)
; SSE64-NEXT: addl $12, %esp
; SSE64-NEXT: retl
;
; AVXONLY32-LABEL: test_store_16xf32_aligned:
; AVXONLY32: # %bb.0:
; AVXONLY32-NEXT: vmovaps %ymm0, (%rdi)
; AVXONLY32-NEXT: vmovaps %ymm1, 32(%rdi)
; AVXONLY32-NEXT: retq
;
; AVXONLY64-LABEL: test_store_16xf32_aligned:
; AVXONLY64: # %bb.0:
; AVXONLY64-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVXONLY64-NEXT: vmovaps %ymm0, (%eax)
; AVXONLY64-NEXT: vmovaps %ymm1, 32(%eax)
; AVXONLY64-NEXT: retl
;
; AVX51232-LABEL: test_store_16xf32_aligned:
; AVX51232: # %bb.0:
; AVX51232-NEXT: vmovaps %zmm0, (%rdi)
; AVX51232-NEXT: retq
;
; AVX51264-LABEL: test_store_16xf32_aligned:
; AVX51264: # %bb.0:
; AVX51264-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX51264-NEXT: vmovaps %zmm0, (%eax)
; AVX51264-NEXT: retl
store <16 x float> %value, <16 x float>* %addr, align 64
ret <16 x float> %value
}
define <8 x double> @test_store_8xf64(<8 x double>* nocapture %addr, <8 x double> %value, <8 x double> %value2) {
; SSE32-LABEL: test_store_8xf64:
; SSE32: # %bb.0:
; SSE32-NEXT: addpd %xmm7, %xmm3
; SSE32-NEXT: addpd %xmm6, %xmm2
; SSE32-NEXT: addpd %xmm5, %xmm1
; SSE32-NEXT: addpd %xmm4, %xmm0
; SSE32-NEXT: movupd %xmm0, (%rdi)
; SSE32-NEXT: movupd %xmm1, 16(%rdi)
; SSE32-NEXT: movupd %xmm2, 32(%rdi)
; SSE32-NEXT: movupd %xmm3, 48(%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_8xf64:
; SSE64: # %bb.0:
; SSE64-NEXT: subl $12, %esp
; SSE64-NEXT: .cfi_def_cfa_offset 16
; SSE64-NEXT: movapd {{[0-9]+}}(%esp), %xmm3
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: addpd {{[0-9]+}}(%esp), %xmm3
; SSE64-NEXT: addpd {{[0-9]+}}(%esp), %xmm2
; SSE64-NEXT: addpd {{[0-9]+}}(%esp), %xmm1
; SSE64-NEXT: addpd {{[0-9]+}}(%esp), %xmm0
; SSE64-NEXT: movupd %xmm0, (%eax)
; SSE64-NEXT: movupd %xmm1, 16(%eax)
; SSE64-NEXT: movupd %xmm2, 32(%eax)
; SSE64-NEXT: movupd %xmm3, 48(%eax)
; SSE64-NEXT: addl $12, %esp
; SSE64-NEXT: retl
;
; AVXONLY32-LABEL: test_store_8xf64:
; AVXONLY32: # %bb.0:
; AVXONLY32-NEXT: vaddpd %ymm3, %ymm1, %ymm1
; AVXONLY32-NEXT: vaddpd %ymm2, %ymm0, %ymm0
; AVXONLY32-NEXT: vmovupd %ymm0, (%rdi)
; AVXONLY32-NEXT: vmovupd %ymm1, 32(%rdi)
; AVXONLY32-NEXT: retq
;
; AVXONLY64-LABEL: test_store_8xf64:
; AVXONLY64: # %bb.0:
; AVXONLY64-NEXT: pushl %ebp
; AVXONLY64-NEXT: .cfi_def_cfa_offset 8
; AVXONLY64-NEXT: .cfi_offset %ebp, -8
; AVXONLY64-NEXT: movl %esp, %ebp
; AVXONLY64-NEXT: .cfi_def_cfa_register %ebp
; AVXONLY64-NEXT: andl $-32, %esp
; AVXONLY64-NEXT: subl $32, %esp
; AVXONLY64-NEXT: movl 8(%ebp), %eax
; AVXONLY64-NEXT: vaddpd 40(%ebp), %ymm1, %ymm1
; AVXONLY64-NEXT: vaddpd %ymm2, %ymm0, %ymm0
; AVXONLY64-NEXT: vmovupd %ymm0, (%eax)
; AVXONLY64-NEXT: vmovupd %ymm1, 32(%eax)
; AVXONLY64-NEXT: movl %ebp, %esp
; AVXONLY64-NEXT: popl %ebp
; AVXONLY64-NEXT: retl
;
; AVX51232-LABEL: test_store_8xf64:
; AVX51232: # %bb.0:
; AVX51232-NEXT: vaddpd %zmm1, %zmm0, %zmm0
; AVX51232-NEXT: vmovupd %zmm0, (%rdi)
; AVX51232-NEXT: retq
;
; AVX51264-LABEL: test_store_8xf64:
; AVX51264: # %bb.0:
; AVX51264-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX51264-NEXT: vaddpd %zmm1, %zmm0, %zmm0
; AVX51264-NEXT: vmovupd %zmm0, (%eax)
; AVX51264-NEXT: retl
%foo = fadd <8 x double> %value, %value2 ; to force dobule type on store
store <8 x double> %foo, <8 x double>* %addr, align 1
ret <8 x double> %foo
}
define <8 x double> @test_store_8xf64_aligned(<8 x double>* nocapture %addr, <8 x double> %value, <8 x double> %value2) {
; SSE32-LABEL: test_store_8xf64_aligned:
; SSE32: # %bb.0:
; SSE32-NEXT: addpd %xmm7, %xmm3
; SSE32-NEXT: addpd %xmm6, %xmm2
; SSE32-NEXT: addpd %xmm5, %xmm1
; SSE32-NEXT: addpd %xmm4, %xmm0
; SSE32-NEXT: movapd %xmm0, (%rdi)
; SSE32-NEXT: movapd %xmm1, 16(%rdi)
; SSE32-NEXT: movapd %xmm2, 32(%rdi)
; SSE32-NEXT: movapd %xmm3, 48(%rdi)
; SSE32-NEXT: retq
;
; SSE64-LABEL: test_store_8xf64_aligned:
; SSE64: # %bb.0:
; SSE64-NEXT: subl $12, %esp
; SSE64-NEXT: .cfi_def_cfa_offset 16
; SSE64-NEXT: movapd {{[0-9]+}}(%esp), %xmm3
; SSE64-NEXT: movl {{[0-9]+}}(%esp), %eax
; SSE64-NEXT: addpd {{[0-9]+}}(%esp), %xmm3
; SSE64-NEXT: addpd {{[0-9]+}}(%esp), %xmm2
; SSE64-NEXT: addpd {{[0-9]+}}(%esp), %xmm1
; SSE64-NEXT: addpd {{[0-9]+}}(%esp), %xmm0
; SSE64-NEXT: movapd %xmm0, (%eax)
; SSE64-NEXT: movapd %xmm1, 16(%eax)
; SSE64-NEXT: movapd %xmm2, 32(%eax)
; SSE64-NEXT: movapd %xmm3, 48(%eax)
; SSE64-NEXT: addl $12, %esp
; SSE64-NEXT: retl
;
; AVXONLY32-LABEL: test_store_8xf64_aligned:
; AVXONLY32: # %bb.0:
; AVXONLY32-NEXT: vaddpd %ymm3, %ymm1, %ymm1
; AVXONLY32-NEXT: vaddpd %ymm2, %ymm0, %ymm0
; AVXONLY32-NEXT: vmovapd %ymm0, (%rdi)
; AVXONLY32-NEXT: vmovapd %ymm1, 32(%rdi)
; AVXONLY32-NEXT: retq
;
; AVXONLY64-LABEL: test_store_8xf64_aligned:
; AVXONLY64: # %bb.0:
; AVXONLY64-NEXT: pushl %ebp
; AVXONLY64-NEXT: .cfi_def_cfa_offset 8
; AVXONLY64-NEXT: .cfi_offset %ebp, -8
; AVXONLY64-NEXT: movl %esp, %ebp
; AVXONLY64-NEXT: .cfi_def_cfa_register %ebp
; AVXONLY64-NEXT: andl $-32, %esp
; AVXONLY64-NEXT: subl $32, %esp
; AVXONLY64-NEXT: movl 8(%ebp), %eax
; AVXONLY64-NEXT: vaddpd 40(%ebp), %ymm1, %ymm1
; AVXONLY64-NEXT: vaddpd %ymm2, %ymm0, %ymm0
; AVXONLY64-NEXT: vmovapd %ymm0, (%eax)
; AVXONLY64-NEXT: vmovapd %ymm1, 32(%eax)
; AVXONLY64-NEXT: movl %ebp, %esp
; AVXONLY64-NEXT: popl %ebp
; AVXONLY64-NEXT: retl
;
; AVX51232-LABEL: test_store_8xf64_aligned:
; AVX51232: # %bb.0:
; AVX51232-NEXT: vaddpd %zmm1, %zmm0, %zmm0
; AVX51232-NEXT: vmovapd %zmm0, (%rdi)
; AVX51232-NEXT: retq
;
; AVX51264-LABEL: test_store_8xf64_aligned:
; AVX51264: # %bb.0:
; AVX51264-NEXT: movl {{[0-9]+}}(%esp), %eax
; AVX51264-NEXT: vaddpd %zmm1, %zmm0, %zmm0
; AVX51264-NEXT: vmovapd %zmm0, (%eax)
; AVX51264-NEXT: retl
%foo = fadd <8 x double> %value, %value2 ; to force dobule type on store
store <8 x double> %foo, <8 x double>* %addr, align 64
ret <8 x double> %foo
}
Reference in New Issue View Git Blame Copy Permalink