Weird runtime error when computing the gradients of the initial parameters

综合讨论区
#1

Hi!
I’m still trying to do a simple rolling ball simulation, but now I want to pass initial speed as a parameter to the simulation and pass the gradients of the speed further to pytorch controller and I have two questions:

  1. What does the error described bellow mean?
  2. How do I pass the gradients to the pytorch optimizers?

This task looks super similar to the billiard example from difftachi repo but in my scenario, I’m getting a weird error (actually two of them) that I cannot interpret or find any documentation on

Here is the code I’m trying to run (run_sim.py is the main file here)

And here is the stacktrace
the first one

[E 05/28/20 22:08:35.475] [ir.h:as@256] is<T>()


***********************************
* Taichi Compiler Stack Traceback *
***********************************
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::Logger::error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::GlobalPtrStmt* taichi::lang::IRNode::as<taichi::lang::GlobalPtrStmt>()
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::MakeAdjoint::visit(taichi::lang::GlobalStoreStmt*)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::MakeAdjoint::visit(taichi::lang::Block*)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::MakeAdjoint::visit(taichi::lang::StructForStmt*)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::MakeAdjoint::visit(taichi::lang::Block*)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::irpass::make_adjoint(taichi::lang::IRNode*, bool)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::irpass::compile_to_offloads(taichi::lang::IRNode*, taichi::lang::CompileConfig const&, bool, bool, bool, bool, bool)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::Kernel::lower(bool)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::Program::compile(taichi::lang::Kernel&)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::Kernel::compile()
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::Kernel::operator()()
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so(+0x6e4b14) [0x7f03ad2d7b14]
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so(+0x64fd70) [0x7f03ad242d70]
python3.8(PyCFunction_Call+0x56) [0x55f2dd8e7dc6]
python3.8(_PyObject_MakeTpCall+0x21f) [0x55f2dd8a97cf]
python3.8(+0x16f0b1) [0x55f2dd8d10b1]
python3.8(PyVectorcall_Call+0x6f) [0x55f2dd8a8fff]
python3.8(+0x19311a) [0x55f2dd8f511a]
python3.8(_PyObject_MakeTpCall+0x21f) [0x55f2dd8a97cf]
python3.8(_PyEval_EvalFrameDefault+0x4c89) [0x55f2dd94bc29]
python3.8(_PyEval_EvalCodeWithName+0x961) [0x55f2dd8f3e41]
python3.8(_PyFunction_Vectorcall+0x21e) [0x55f2dd8f47be]
python3.8(PyVectorcall_Call+0x6f) [0x55f2dd8a8fff]
python3.8(_PyEval_EvalFrameDefault+0x1f77) [0x55f2dd948f17]
python3.8(_PyEval_EvalCodeWithName+0x1dc) [0x55f2dd8f36bc]
python3.8(_PyObject_FastCallDict+0x1b8) [0x55f2dd8f4d18]
python3.8(_PyObject_Call_Prepend+0x63) [0x55f2dd8f4fe3]
python3.8(+0x1930ea) [0x55f2dd8f50ea]
python3.8(PyObject_Call+0x70) [0x55f2dd8a91d0]
python3.8(_PyEval_EvalFrameDefault+0x1f77) [0x55f2dd948f17]
python3.8(_PyFunction_Vectorcall+0xfb) [0x55f2dd8f469b]
python3.8(_PyEval_EvalFrameDefault+0x49a) [0x55f2dd94743a]
python3.8(_PyFunction_Vectorcall+0xfb) [0x55f2dd8f469b]
python3.8(+0x16f06b) [0x55f2dd8d106b]
python3.8(+0x147940) [0x55f2dd8a9940]
python3.8(_PyEval_EvalFrameDefault+0x21a3) [0x55f2dd949143]
python3.8(_PyEval_EvalCodeWithName+0x1dc) [0x55f2dd8f36bc]
python3.8(PyEval_EvalCodeEx+0x44) [0x55f2dd8f4564]
python3.8(PyEval_EvalCode+0x1c) [0x55f2dd8f458c]
python3.8(+0x2333a4) [0x55f2dd9953a4]
python3.8(+0x25fa64) [0x55f2dd9c1a64]
python3.8(PyRun_FileExFlags+0xa1) [0x55f2dd9c42e1]
python3.8(PyRun_SimpleFileExFlags+0x1c1) [0x55f2dd9c44d1]
python3.8(Py_RunMain+0x36d) [0x55f2dd9c50dd]
python3.8(Py_BytesMain+0x39) [0x55f2dd9c52d9]
/lib/x86_64-linux-gnu/libc.so.6: __libc_start_main
python3.8(+0x203493) [0x55f2dd965493]
Traceback (most recent call last):
  File "scripts/run_pytorch_ball_simulation.py", line 36, in <module>
    sim.run_simulation(
  File "/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/lang/tape.py", line 18, in __exit__
    self.grad()
  File "/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/lang/tape.py", line 27, in grad
    func.grad(*args)
  File "/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/lang/kernel.py", line 464, in __call__
    return self.compiled_functions[key](*args)
  File "/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/lang/kernel.py", line 428, in func__
    t_kernel()
RuntimeError: [ir.h:as@256] is<T>()

and the second one

***********************************
* Taichi Compiler Stack Traceback *             
***********************************        
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::Logger::error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)                        
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::MakeAdjoint::visit(taichi::lang::ContinueStmt*)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::MakeAdjoint::visit(taichi::lang::IfStmt*)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::MakeAdjoint::visit(taichi::lang::Block*)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::MakeAdjoint::visit(taichi::lang::StructForStmt*)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::MakeAdjoint::visit(taichi::lang::Block*)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::irpass::make_adjoint(taichi::lang::IRNode*, bool)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::irpass::compile_to_offloads(taichi::lang::IRNode*, taichi::lang::CompileConfig const&, bool, bool, bool, bool, bool)                                   
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::Kernel::lower(bool)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::Program::compile(taichi::lang::Kernel&)
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::Kernel::compile()
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so: taichi::lang::Kernel::operator()()
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so(+0x6e4b14) [0x7f43a4e52b14]
/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/core/../lib/taichi_core.so(+0x64fd70) [0x7f43a4dbdd70]
python3.8(PyCFunction_Call+0x56) [0x55f552213dc6]                  
python3.8(_PyObject_MakeTpCall+0x21f) [0x55f5521d57cf]              
python3.8(+0x16f0b1) [0x55f5521fd0b1]                                      
python3.8(PyVectorcall_Call+0x6f) [0x55f5521d4fff]
python3.8(+0x19311a) [0x55f55222111a]                                                      
python3.8(_PyObject_MakeTpCall+0x21f) [0x55f5521d57cf]
python3.8(_PyEval_EvalFrameDefault+0x4c89) [0x55f552277c29]
python3.8(_PyEval_EvalCodeWithName+0x961) [0x55f55221fe41]
python3.8(_PyFunction_Vectorcall+0x21e) [0x55f5522207be]                          
python3.8(PyVectorcall_Call+0x6f) [0x55f5521d4fff]
python3.8(_PyEval_EvalFrameDefault+0x1f77) [0x55f552274f17]        
python3.8(_PyEval_EvalCodeWithName+0x1dc) [0x55f55221f6bc]               
python3.8(_PyObject_FastCallDict+0x1b8) [0x55f552220d18]                   
python3.8(_PyObject_Call_Prepend+0x63) [0x55f552220fe3]
python3.8(+0x1930ea) [0x55f5522210ea]                                                      
python3.8(PyObject_Call+0x70) [0x55f5521d51d0]                                                                  
python3.8(_PyEval_EvalFrameDefault+0x1f77) [0x55f552274f17]                                                     
python3.8(_PyFunction_Vectorcall+0xfb) [0x55f55222069b]                                                           
python3.8(_PyEval_EvalFrameDefault+0x49a) [0x55f55227343a]                                                        
python3.8(_PyFunction_Vectorcall+0xfb) [0x55f55222069b]
python3.8(+0x16f06b) [0x55f5521fd06b]
python3.8(+0x147940) [0x55f5521d5940]
python3.8(_PyEval_EvalFrameDefault+0x21a3) [0x55f552275143]
python3.8(_PyEval_EvalCodeWithName+0x1dc) [0x55f55221f6bc]
python3.8(PyEval_EvalCodeEx+0x44) [0x55f552220564]
python3.8(PyEval_EvalCode+0x1c) [0x55f55222058c]
python3.8(+0x2333a4) [0x55f5522c13a4]
python3.8(+0x25fa64) [0x55f5522eda64]
python3.8(PyRun_FileExFlags+0xa1) [0x55f5522f02e1]
python3.8(PyRun_SimpleFileExFlags+0x1c1) [0x55f5522f04d1]
python3.8(Py_RunMain+0x36d) [0x55f5522f10dd]
python3.8(Py_BytesMain+0x39) [0x55f5522f12d9]
/lib/x86_64-linux-gnu/libc.so.6: __libc_start_main
python3.8(+0x203493) [0x55f552291493]
Traceback (most recent call last):   
  File "run_sim.py", line 36, in <module>
    sim.run_simulation(             
  File "/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/lang/tape.py", line 18, in __exit__
    self.grad()                                                                                                   
  File "/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/lang/tape.py", line 27, in grad                 
    func.grad(*args)                                                 
  File "/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/lang/kernel.py", line 464, in __call__
    return self.compiled_functions[key](*args)
  File "/opt/conda/envs/taichi/lib/python3.8/site-packages/taichi/lang/kernel.py", line 428, in func__
    t_kernel()                     
RuntimeError: [make_adjoint.cpp:visit@360] Not supported.
#2

Hi belkakari,

Thanks for reporting this! Could you file a bug under https://github.com/taichi-dev/taichi/issues? I think most people watch that list instead of here.

I have some sense for the second stacktrace, it’s because continue isn’t supported in kernels that needs autodiff, see https://github.com/taichi-dev/taichi/blob/c193e4b431d52ffa4e875940ece589d5be9599f0/taichi/transforms/make_adjoint.cpp#L360 I guess this isn’t hard to fix

For the first one, it seems like a violation of some internal invariants in the Taichi IR. We need to take a further look…

Cheers,
Ye

#3

Hi Ye,
Thanks for the explanation, removing the continue statement resolved the issue
I’ve created an issue in the repo https://github.com/taichi-dev/taichi/issues/1074

1 个赞