Hard crash when running complex model on GPU (Trac #678)

[pkienzle]

The attached model crashes sasview when it is run on an Radeon R9 Nano with windows, leaving the screen briefly blank, then not returning (!SasView 4.0 release).

Testing again under run.py, and the display went blank, forcing a hard reboot.

Not tested on other hardware/os.

Migrated from http://trac.sasview.org/ticket/678

{
    "status": "closed",
    "changetime": "2016-10-03T23:49:32",
    "_ts": "2016-10-03 23:49:32.608588+00:00",
    "description": "The attached model crashes sasview when it is run on an Radeon R9 Nano with windows, leaving the screen briefly blank, then not returning (!SasView 4.0 release).\n\nTesting again under run.py, and the display went blank, forcing a hard reboot.\n\nNot tested on other hardware/os.\n",
    "reporter": "pkienzle",
    "cc": "",
    "resolution": "wontfix",
    "workpackage": "SasView Bug Fixing",
    "time": "2016-10-03T22:54:11",
    "component": "SasView",
    "summary": "Hard crash  when running complex model on GPU",
    "priority": "major",
    "keywords": "",
    "milestone": "SasView 4.1.0",
    "owner": "",
    "type": "defect"
}

[pkienzle]

Trac update at 2016/10/03 22:54:32: pkienzle changed attachment from "" to "KatharinaDataRadomOrientation_SandiaD2O.ABS"

[pkienzle]

Trac update at 2016/10/03 22:55:33: pkienzle changed attachment from "" to "RandomToOrientedCoreShellShellChains.py"

[pkienzle]

Trac update at 2016/10/03 23:01:07: pkienzle commented:

OS/X result. Maybe don't use arrays?

Error(s) Occurred:
	Traceback (most recent call last):
  File "/Users/pkienzle/Source/sasview-new/src/sas/sascalc/data_util/calcthread.py", line 268, in _run
    self.compute(*args, **kwargs)
RuntimeError: clBuildProgram failed: build program failure - 

Build on <pyopencl.Device 'HD Graphics 4000' on 'Apple' at 0x1024400>:

/Users/pkienzle/.sasview/plugin_models/RandomToOrientedCoreShellShellChains.py:153:33: error: array initializer must be an initializer list
    float CrossSection[4][5] = {0.0f};
                                ^
/Users/pkienzle/.sasview/plugin_models/RandomToOrientedCoreShellShellChains.py:161:19: warning: unused variable 'phi'
            float phi = (b*45.0f)*M_PI_180;
                  ^

(options: -I /Users/pkienzle/anaconda/envs/bumps/lib/python2.7/site-packages/pyopencl/cl)
(source saved as /var/folders/yd/50j1j6_x4dl0gj5ns3thq0n00011yf/T/tmpED94r1.cl)


Further information might be available in the Console log (bottom right corner).

[pkienzle]

Trac update at 2016/10/03 23:12:18: pkienzle commented:

After "fixing" the initialization error, I get the following. This code is too big to run on the HD 4000, and needs to be forced onto the CPU.

Error(s) Occurred:
	Traceback (most recent call last):
  File "/Users/pkienzle/Source/sasview-new/src/sas/sascalc/data_util/calcthread.py", line 268, in _run
    self.compute(*args, **kwargs)
RuntimeError: clBuildProgram failed: build program failure - 

Build on <pyopencl.Device 'HD Graphics 4000' on 'Apple' at 0x1024400>:

Error: internal error.

(options: -I /Users/pkienzle/anaconda/envs/bumps/lib/python2.7/site-packages/pyopencl/cl)
(source saved as /var/folders/yd/50j1j6_x4dl0gj5ns3thq0n00011yf/T/tmpbgdZdw.cl)


Further information might be available in the Console log (bottom right corner).

[pkienzle]

Trac update at 2016/10/03 23:44:32:

• pkienzle changed _comment0 from:

This model as a significant amount of code that depends only on viewing angle and not on q so it could be precomputed. It requires significant memory which does not need to be in the local register space when executing the kernel for particular q values.

Extending the computation model so that an initialization kernel can be called before the polydispersity kernel is beyond the scope of this ticket.

to:

1475538315852187

• pkienzle commented:

This model has a significant amount of code that depends only on viewing angle and not on q so it could be precomputed. It requires significant memory which does not need to be in the local register space when executing the kernel for particular q values.

Extending the computation model so that an initialization kernel can be called before the polydispersity kernel is beyond the scope of this ticket.

[pkienzle]

Trac update at 2016/10/03 23:49:32:

• pkienzle commented:

Cannot fix.

When users can create and run arbitrary C code on the GPU, we cannot guarantee that the program will be well behaved. The best we can do is make it possible to run the code on the CPU when it is too complex for the GPU.

• pkienzle changed resolution from "" to "wontfix"
• pkienzle changed status from "new" to "closed"