Add parallel tuning on multiple remote GPUs using Ray

[isazi]

Working on a simple parallel runner that uses Ray to distribute the benchmarking of different configurations to remote Ray workers.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[stijnh]

The current parallel runner works. I've been able to run on multiple GPUs on DAS6-VU and DAS6-Leiden.

There are several remaining problems:

• The timings are incorrect as the host assumes that the total time is just the sum over individual configurations
• Use of tuning_options need to be refactored, as now it the entire object is sent to every node for each benchmark job
• Logging information can be improved
• The strategies are not parallel-aware yet (except brute-force)
• A guide needs to be added to the docs explaining how to launch a Ray cluster on DAS6

[benvanwerkhoven]

I have resolved a couple of issues with sequential tuning that had arisen from the changes to the 'eval_all' costfunc. I haven't actually tested the parallel runner yet. But there are couple of other things I need to attend to now.

[stijnh]

This branch is almost ready for merging. The only big remaining issue is that the timing are not tracked correctly (strategy_time and framework_time) and currently these values are incorrect

[benvanwerkhoven]

Sometimes with Python you run into an error and think: 'How on Earth has this error not surfaced years ago?'

It seems that observers has been None by default instead of an empty list since forever and miraculously it was never an issue. It seems that the code responsible for replacing a None value with an empty list is currently hidden in (and duplicated across) the backends, and because there is now code that (rightfully so) assumes observers is a list just before the backends are created this is suddenly an issue. The real issue is of course that the backends have somehow become responsible for sanitizing user input, which is not what a backend should do.

[sonarqubecloud]

Quality Gate passed

Issues
75 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud