FusionLoops based on MZ's LoopNest #233

Open

lly-zero-one wants to merge 290 commits intobertmaher:pytorch_fusionfrom

lly-zero-one:loop_nest

This pull request is big! We're only showing the most recent 250 commits

Commits on Feb 18, 2020

fix the schedule_test

zheng-xq
authored and
Mikhail Zolotukhin
committed
fixed lowering

zheng-xq
authored and
Mikhail Zolotukhin
committed
LLVM code generation for simple loops

bertmaher
authored and
Mikhail Zolotukhin
committed
bugfixes

bertmaher
authored and
Mikhail Zolotukhin
committed
refcount fixing self-assignment

zheng-xq
authored and
Mikhail Zolotukhin
committed
Make LOG(FATAL) nonreturn

zheng-xq
authored and
Mikhail Zolotukhin
committed
Adding statement conversion for SplitWithTail

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add a reference tests for Split

zheng-xq
authored and
Mikhail Zolotukhin
committed
clang-format

bertmaher
authored and
Mikhail Zolotukhin
committed
A functinoal reference chck for schedule tests.

zheng-xq
authored and
Mikhail Zolotukhin
committed
clang-format

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add support for Float immediates.

Owen Anderson
authored and
Mikhail Zolotukhin
committed
Get absolute path for ASMJIT_DIR (#24)

bertmaher
authored and
Mikhail Zolotukhin
committed
Silence deprecation warnings from LLVM

bertmaher
authored and
Mikhail Zolotukhin
committed
Include legacy PassManager for debug printing

bertmaher
authored and
Mikhail Zolotukhin
committed
Set code model to medium to avoid indirect jumps in generated asm

bertmaher
authored and
Mikhail Zolotukhin
committed
Fix argument type of input float buffers

bertmaher
authored and
Mikhail Zolotukhin
committed
Add support for Casts in LLVM codegen.

Owen Anderson
authored and
Mikhail Zolotukhin
committed
Add a complete tensor+lower+llvm test

zheng-xq
authored and
Mikhail Zolotukhin
committed
Enable the failing test

zheng-xq
authored and
Mikhail Zolotukhin
committed
Enable export of compile_commands.json.

Owen Anderson
authored and
Mikhail Zolotukhin
committed
Floating point arithmetic

bertmaher
authored and
Mikhail Zolotukhin
committed
Test fp32 mul using compute expr

bertmaher
authored and
Mikhail Zolotukhin
committed
Broadcast add test using compute expr

bertmaher
authored and
Mikhail Zolotukhin
committed
Update to LLVM 9

Owen Anderson
authored and
Mikhail Zolotukhin
committed
Implementation of Broadcast for LLVM.

Owen Anderson
authored and
Mikhail Zolotukhin
committed
Add Buffer operator() overload, and some other minor features

zheng-xq
authored and
Mikhail Zolotukhin
committed
Cleanup use of ConstantInt API.

Owen Anderson
authored and
Mikhail Zolotukhin
committed
fix accidental experimental changes

zheng-xq
authored and
Mikhail Zolotukhin
committed
Change the Compute interface to bring the dim sizes and names together

zheng-xq
authored and
Mikhail Zolotukhin
committed
clang-format

zheng-xq
authored and
Mikhail Zolotukhin
committed
refactor Buffer into its own files

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add support for vector casts in LLVM CodeGen

Owen Anderson
authored and
Mikhail Zolotukhin
committed
Implement masked loads and stores.

Owen Anderson
authored and
Mikhail Zolotukhin
committed
Implement vector masked loads and stores.

Owen Anderson
authored and
Mikhail Zolotukhin
committed
Add a PaddedBuffer test util

zheng-xq
authored and
Mikhail Zolotukhin
committed
Improve the user interface for SimpleIREvaluator

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add a test for Block codegen.

Owen Anderson
authored and
Mikhail Zolotukhin
committed
Fix gtest include path

Owen Anderson
authored and
Mikhail Zolotukhin
committed
clang-format

bertmaher
authored and
Mikhail Zolotukhin
committed
Add expressions and support for Max and Min. (#5)

resistor
authored and
Mikhail Zolotukhin
committed
Rename compiler to tensorexpr and move files around to be more similar to other pytorch parts. (#6)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
Add missing include <math.h> (#7)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
Change isnan to std::isnan. It breaks my clang builds. (#8)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Change the SimpleIREvaluator frontend (#9)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Make LLVM dependency optional. (#10)

resistor
authored and
Mikhail Zolotukhin
committed
[wip] Basic fuser pass to select texpr subgraphs

bertmaher
authored and
Mikhail Zolotukhin
committed
Revert "[wip] Basic fuser pass to select texpr subgraphs"

bertmaher
authored and
Mikhail Zolotukhin
committed
Revert changes to the main pytorch CMakeLists.txt (for now).
Mikhail Zolotukhin
committed
Add a test for aten::_cast_Float lowering. (#12)

resistor
authored and
Mikhail Zolotukhin
committed
Hook tensorexp up to the main build, and switch to c10 logging

bertmaher
authored and
Mikhail Zolotukhin
committed
More ATen op tests. (#16)

resistor
authored and
Mikhail Zolotukhin
committed
Fix some missing returns

bertmaher
authored and
Mikhail Zolotukhin
committed
Include tests back to the 'all' target. (#14)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
Even more ATen op tests. (#18)

resistor
authored and
Mikhail Zolotukhin
committed
Test for relu ATen op. (#19)

resistor
authored and
Mikhail Zolotukhin
committed
Add intrinsics function support. (#20)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Remove fmax/fmin, as they are already covered by the Max/Min operators (#21)

zheng-xq
authored and
Mikhail Zolotukhin
committed
refactor CallNode and BaseCallNode, so we can have a common concrete base class for visitors. (#22)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add FunctionCall to use existing tensors (#23)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add the ability to use an existing tensor expression in other compute functions. (#24)

zheng-xq
authored and
Mikhail Zolotukhin
committed
fixing broken compilation on mac/clang

Protonu Basu
authored and
Mikhail Zolotukhin
committed
adding IRnode for Compare-Select Ops and their LLVM Codegen

Protonu Basu
authored and
Mikhail Zolotukhin
committed
Fix Werror. (#26)

resistor
authored and
Mikhail Zolotukhin
committed
Add tests for some transcendental ops. (#27)

resistor
authored and
Mikhail Zolotukhin
committed
Add Allocate and Free support. (#29)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Tensor expr fuser pass for extremely simple expressions

bertmaher
authored and
Mikhail Zolotukhin
committed
Make fusion work for arbitrary buffer/tensor combinations of inputs (#30)

bertmaher
authored and
Mikhail Zolotukhin
committed
fix Let02 test

Protonu Basu
authored and
Mikhail Zolotukhin
committed
Access inputs and intermediates uniformly through Tensors (#31)

bertmaher
authored and
Mikhail Zolotukhin
committed
adding LLVM Codegen for Let

Protonu Basu
authored and
Mikhail Zolotukhin
committed
Adding ComputeInline support. (#35)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Fix broken tests (#36)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Make tx fuser work with arbitrary ranks

bertmaher
authored and
Mikhail Zolotukhin
committed
[fuser] Broadcast args

bertmaher
authored and
Mikhail Zolotukhin
committed
Improve naming of arg broadcasting function

bertmaher
authored and
Mikhail Zolotukhin
committed
modifying CMakeLists.txt to enable ninja test && minor update for LLVM Codegen for Let (handling XQ's comment)

Protonu Basu
authored and
Mikhail Zolotukhin
committed
Test cases for tensorexpr fusion (#37)

bertmaher
authored and
Mikhail Zolotukhin
committed
CompareSelct Op: Addressing XQ and Owen's comments

Protonu Basu
authored and
Mikhail Zolotukhin
committed
Sketch sufficient support for constants to get constant alpha working. (#40)

resistor
authored and
Mikhail Zolotukhin
committed
Fix indices when inlining non-leaf calls (#39)

bertmaher
authored and
Mikhail Zolotukhin
committed
Fixing the inline ordering issue (#43)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Avoid creating redundant and/or improperly ordered Constant's in fused subgraphs. (#42)

resistor
authored and
Mikhail Zolotukhin
committed
Move fuser-styled tests to schedule_test (#44)

bertmaher
authored and
Mikhail Zolotukhin
committed
Add aten::sub to the new fuser. (#46)

resistor
authored and
Mikhail Zolotukhin
committed
Refactor CodeGen from SimpleIREval (#47)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Inline all the things (#45)

bertmaher
authored and
Mikhail Zolotukhin
committed
clang-format for atent_test.cpp

Protonu Basu
authored and
Mikhail Zolotukhin
committed
Eliminate a ton of warnings for my own sanity. (#48)

resistor
authored and
Mikhail Zolotukhin
committed
Add support for type promotion/demotion. (#50)

resistor
authored and
Mikhail Zolotukhin
committed
Flesh out new fuser coverage to several more ops. (#51)

resistor
authored and
Mikhail Zolotukhin
committed
Adding the first basic CudaCodeGen. (#52)

zheng-xq
authored and
Mikhail Zolotukhin
committed
aten tests for eq, ge, gt, le, lt

Protonu Basu
authored and
Mikhail Zolotukhin
committed
support for aten ops: eq

Protonu Basu
authored and
Mikhail Zolotukhin
committed
support for more aten ops: ge, gt, le, lt, ne

Protonu Basu
authored and
Mikhail Zolotukhin
committed
Minimal CMake change to link LLVM to libtorch

bertmaher
authored and
Mikhail Zolotukhin
committed
Fix issues causing assertion failures in llvm debug builds

bertmaher
authored and
Mikhail Zolotukhin
committed
Fatal on unimplement llvm codegen ops (Allocate, etc.)

bertmaher
authored and
Mikhail Zolotukhin
committed
Optionally compile tx fuser kernels with llvm

bertmaher
authored and
Mikhail Zolotukhin
committed
Test for 2D broadcasted with large dims to show vectorization

bertmaher
authored and
Mikhail Zolotukhin
committed
Updated isSupported for increased op coverage. (#54)

resistor
authored and
Mikhail Zolotukhin
committed
Refactor LLVMCodeGen to compile kernel in constructor

bertmaher
authored and
Mikhail Zolotukhin
committed
Cmake integration to PT codebase (#28)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
Remove old padded_buffer.{cpp,h}. (#56)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
Add support for code generation of Log10 intrinsics with LLVM. (#57)

resistor
authored and
Mikhail Zolotukhin
committed
Remove tests/test_utils.h: inline what's still used and nuke what's unused. (#58)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
Move Fuser tests (tests/tests.py) to test/test_tensorexpr.py. (#59)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
Remove old CMakeLists and README.txt
Mikhail Zolotukhin
committed
Add support for vectorized and unmasked loads and stores with LLVM. (#62)

resistor
authored and
Mikhail Zolotukhin
committed
Enable CodeGen-level optimizations in LLVM. (#63)

resistor
authored and
Mikhail Zolotukhin
committed
Add Bind/GPUBlock/GPUThread support. (#64)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Bind/run interface to CodeGen (#60)

bertmaher
authored and
Mikhail Zolotukhin
committed
Fix ambiguity in CreateExtractElementCall (0ull can be a Value*, I guess?) (#65)

bertmaher
authored and
Mikhail Zolotukhin
committed
Allow constants as lhs/rhs args (not just alpha) (#66)

bertmaher
authored and
Mikhail Zolotukhin
committed
Use correct tensor type for fuser output (#67)

bertmaher
authored and
Mikhail Zolotukhin
committed
clang-format

bertmaher
authored and
Mikhail Zolotukhin
committed
Rename 'compiler' namespace to 'tensorexpr'.
Mikhail Zolotukhin
committed
Include all built llvm targets (#68)

bertmaher
authored and
Mikhail Zolotukhin
committed
Switch back to linking only the native LLVM target. (#69)

resistor
authored and
Mikhail Zolotukhin
committed
Virtual dtors for IRVisitor/IRMutator (#70)

bertmaher
authored and
Mikhail Zolotukhin
committed
Add semicolon to make nvcc compile (#71)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Enable NVRTC for the GPU backend. (#74)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Fix non-CUDA testing. (#75)

resistor
authored and
Mikhail Zolotukhin
committed
Getting fused (a)Sin(h), (a)Cos(h),(a) Tan(h), abs working with the interpreter (#73)

protonu
authored and
Mikhail Zolotukhin
committed
remove the leak tests, as we will get rid of refcounting (#76)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Implement aten::min, max, and clamp (#72)

bertmaher
authored and
Mikhail Zolotukhin
committed
clang-format tensorexpr/tests.h (#77)

bertmaher
authored and
Mikhail Zolotukhin
committed
Refactor UniqueNameManager into its own files. (#79)

zheng-xq
authored and
Mikhail Zolotukhin
committed
refactor cuda_codegen (#80)

zheng-xq
authored and
Mikhail Zolotukhin
committed
simplify nvrtc major, minor versions (#81)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Allow CodeGen to take Var args (interpreter support only) (#78)

bertmaher
authored and
Mikhail Zolotukhin
committed
[LLVMCodeGen] Refactor kernel constructor to be less sprawling (#82)

bertmaher
authored and
Mikhail Zolotukhin
committed
(TE Interpreter)Support for floor, ceil, trunc, remainder, sqrt and improving tests (#83)

protonu
authored and
Mikhail Zolotukhin
committed
Add Cond and Mod to SimpleIREval (#84)

zheng-xq
authored and
Mikhail Zolotukhin
committed
[LLVMCodeGen] Support dynamic shapes by binding Var args (#86)

bertmaher
authored and
Mikhail Zolotukhin
committed
Add SplitWithMask core support. (#87)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add Cuda tests for SplitWithMask (#88)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Disable DEBUG_PRINT (#89)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Remove some debug prints (#90)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Fix the no-CUDA build. (#92)

resistor
authored and
Mikhail Zolotukhin
committed
Add support for multiple outputs from the fused subgraph. (#91)

resistor
authored and
Mikhail Zolotukhin
committed
Remove RefCounting (#93)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add some comments for KernelScope. Address comments. (#94)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Completely remove refcount.h (#95)

zheng-xq
authored and
Mikhail Zolotukhin
committed
fix the fuser pass (#97)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Rename Kernel to KernelArena (#98)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add support for fusion through ConstantChunk ops. (#96)

resistor
authored and
Mikhail Zolotukhin
committed
Fix implicit noexcept deduction warning. (#99)

resistor
authored and
Mikhail Zolotukhin
committed
Make llvm tests conditional on USE_LLVM (#100)

bertmaher
authored and
Mikhail Zolotukhin
committed
Refactor ComputeNode into ComputeValue, to be able to handle arbitrary (#101)

resistor
authored and
Mikhail Zolotukhin
committed
Improve Stmt pretty printing from TensorExprFuser (#102)

bertmaher
authored and
Mikhail Zolotukhin
committed
Add support for IfThenElse (#103)

resistor
authored and
Mikhail Zolotukhin
committed
Add end-to-end support and a PyTorch fuser example on CudaCodeGen (#104)

zheng-xq
authored and
Mikhail Zolotukhin
committed
fix rebase errors (#105)

zheng-xq
authored and
Mikhail Zolotukhin
committed
fixes to build on system without LLVM and CUDA (#107)

protonu
authored and
Mikhail Zolotukhin
committed
Add support for aten::cat to the new fuser. (#106)

resistor
authored and
Mikhail Zolotukhin
committed
Bail out of fusion if we don't have a complete tensor type (for now). (#108)

resistor
authored and
Mikhail Zolotukhin
committed
Standardize codegen call() interface and remove bind/run (#109)

bertmaher
authored and
Mikhail Zolotukhin
committed
Clean up sketchy handling of scalar args in llvm codegen (#110)

bertmaher
authored and
Mikhail Zolotukhin
committed
Test 2D dynamic shapes (#112)

bertmaher
authored and
Mikhail Zolotukhin
committed
clang-format (#113)

bertmaher
authored and
Mikhail Zolotukhin
committed
Add LLVM codegen for a lot of transcendental ops. (#115)

resistor
authored and
Mikhail Zolotukhin
committed
Fix bug with binary math intrinsics. (#116)

resistor
authored and
Mikhail Zolotukhin
committed
Use CUDA for 3-arg test (#117)

bertmaher
authored and
Mikhail Zolotukhin
committed
Refactor CudaCodeGen into generic registration, so we can have both the Cuda and non-Cuda builds. (#118)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add instructions on how to rebase on master.
Mikhail Zolotukhin
committed
Dynamic shape support in CUDA codegen (#120)

bertmaher
authored and
Mikhail Zolotukhin
committed
Disable GPU fuser. Revive the Cuda tests (#121)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add ExecutionCounter to detect whether the underlying code is executed. (#122)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Adding GPU index flatting to support arbitrary elementwise and broadcasting support. (#126)

zheng-xq
authored and
Mikhail Zolotukhin
committed
fix a bug kLog to Intrin::log (#124)

Krovatkin
authored and
Mikhail Zolotukhin
committed
Allow scalar variables as inputs (#125)

bertmaher
authored and
Mikhail Zolotukhin
committed
clang-format (#127)

bertmaher
authored and
Mikhail Zolotukhin
committed
Format python tests with `black` (#128)

bertmaher
authored and
Mikhail Zolotukhin
committed
Add support for fusion in nested blocks. (#129)

resistor
authored and
Mikhail Zolotukhin
committed
Teach the LLVM JIT to use dlsym to resolve symbols. (#130)

resistor
authored and
Mikhail Zolotukhin
committed
Factor out kernel codegen from tx fusion pass (#131)

bertmaher
authored and
Mikhail Zolotukhin
committed
Use standard JIT logging in TX fuser.
Mikhail Zolotukhin
committed
Move memory management classes (KernelArena, KernelScope, KernelScopedObject) to a separate file. (#132)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
(IR Interpreter) Adding more Operators: Erfc, Exmp1, frac, lgamma, neg, sigmoid, reciprocal, neg, relu (#133)

protonu
authored and
Mikhail Zolotukhin
committed
Add erfc to llvm codegen (#134)

bertmaher
authored and
Mikhail Zolotukhin
committed
Squash some warnings (#135)

bertmaher
authored and
Mikhail Zolotukhin
committed
(IR interpreter) addcmul (#137)

protonu
authored and
Mikhail Zolotukhin
committed
Remove IRNode. CodeGen accepts only Stmt. Add ExprEval utility wrapper. (#138)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add the benchmark from NNC (#141)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Fix verifier errors in LLVM codegen when conditional loads feed directly into concats. (#143)

resistor
authored and
Mikhail Zolotukhin
committed
Strength reduction peephole for pow(). (#144)

resistor
authored and
Mikhail Zolotukhin
committed
Fix incorrect pow(x, 0) case. (#145)

resistor
authored and
Mikhail Zolotukhin
committed
Use `const Value*` where possible (#146)

bertmaher
authored and
Mikhail Zolotukhin
committed
Make Broadcast work (#147)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Fixed CudaCodeGen output streams. Switch to __ldg by default (#148)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add ElementWise support (#150)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Fix an assertion failure when merging constants into aten::cat fusions. (#151)

resistor
authored and
Mikhail Zolotukhin
committed
adding LLVM support ops: sigmoid, relu, neg, addcmul, reciprocal, lgamma, expm1 (#149)

protonu
authored and
Mikhail Zolotukhin
committed
Add more operator support and tests (#140)

lly-zero-one
authored and
Mikhail Zolotukhin
committed
Fix accidental assignment in condition (#153)

bertmaher
authored and
Mikhail Zolotukhin
committed
Add elementwise benchmarks and comparisons. (#155)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Backport some of the fixes from the master PR. (#157)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
Adding broadcasting benchmarks (#158)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Fix the missing aten::pow support (#160)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Fix the missing aten::pow support (#161)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Fixing the failing test (#164)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Add NNC support for aten::slice and aten::unsqueeze. (#159)

resistor
authored and
Mikhail Zolotukhin
committed
Get strides working (#163)

bertmaher
authored and
Mikhail Zolotukhin
committed
Change the default block size from 1024 to 512. (#165)

zheng-xq
authored and
Mikhail Zolotukhin
committed
Check that dtype is float before calling std::isnan. (#167)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
Check that dtype is float before calling std::isnan. (#168)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
Remove asmjit backend. (#169)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
Cleanup fuser pass a little. (#170)

ZolotukhinM
authored and
Mikhail Zolotukhin
committed
Add the Binary/unary op and also tests (#154)

lly-zero-one
authored and
Mikhail Zolotukhin
committed