Ansys and CFX Distributed Computing
Ansys Distributed Processing
DECS has a cluster of ten linux servers with dual quad-core processors and 64 GB of RAM. The linux servers have identical OS installs (Gentoo), they all get software from the same /opt, and they all mount the same file systems. This makes the configuration of Ansys for Distributed Computer Processing rather easy.
Ansys 13.0 comes with HP-MPI already installed, so there are only a few steps to follow to prepare the systems for parallel processing.
Setting up ssh
Ssh is used for remote login. Users are not permitted to have .rhosts files that are world-readable. In order to use ssh without having to enter a password, each user must set up DSA authentication. On any one of the linux machines, run the command:
ssh-keygen -t dsa
Accept the default location for the file to store the key($HOME/.ssh/id_dsa) and enter a passphrase. It is not recommended that the login password be used, something similar may be helpful. The password is encrypted and secure.
Next, copy the $HOME/.ssh/id_dsa.pub file generated by ssh-keygen to $HOME/.ssh/authorized_keys2. Check that the permission on this file is 644.
Setting up Ansys
You do not need to be running ssh to perform the following steps. Set your environment for ansys and run ans_admin130. Select the bar ANSYS/Workbench Configure Cluster, click on the button to configure hosts130.ans in your home directory, and select Distributed ANSYS. Click OK. This will open a configure cluster window. In the machine hostname, enter a server name and click Add. This will open another window, the window type is a pull-down menu, select linx64. In the box for the maximum number of processors, we recommend not entering more than four. We have ten machines and we do not have enough licenses to utilize all the CPUs. You can run four parallel processes on the standard teaching license, if you want to run more, you must buy High Performance Computing (HPC) tasks for each additional processor and the research license. Enter as many machine names as you like, there is runtime set up where you specify which machines and how many processors to use for a specific job.
Preparing the Run
Unless you are submitting batch runs from the command line, Ansys will open a Graphical User Interface (GUI) for each server you specify on a particular job. We set the DISPLAY environment variable in the user's .cshrc, ie:
setenv DISPLAY machinename:0.0 *nix
setenv DISPLAY localhost:0.0 PC
In order to connect to other servers without typing a password, the routine to run on the master server is called ssh-agent. It is a two-step process. On the master server, type the command:
eval `ssh-agent`
It will start the process and report the process number.
Then type the command:
ssh-add $HOME/.ssh/id_dsa
It will ask for the passphrase and if it is entered correctly, it will let you know that it is ready to go. If not, the system will tell you the passphrase does not match and you can try again.
Once you start ssh-agent, it will run forever, even if you log off or get bumped off the machine. However, once you log off the machine, you lose the authentication you received with ssh-add. Logging back in and running another ssh-add will not connect to the older ssh-agent, you have to start ssh-agent again. When you finish running the jobs, kill the ssh-agent process with the following command:
ssh-agent -k
At this time it is safe to log off.
Ansys Launch
When you run the launcher, select the High Performance Computing set up. Check the box for Use Distributed Computing, the MPI type should be HP_MPI. Check the box to Use Secure Shell, instead of Remote Shell. Then check the box Specify Multiple Hosts. The list of machines you entered should appear in Available Hosts: select one, click Add and it will appear in selected hosts. Doing this allows you to change your configuration for different runs. You will notice that the selected host only shows one processor. That can be changed by clicking on the hostname in the Selected Host list and then clicking Edit. You can change the number of processers you want to use on that host.
CFX Launch
The CFX launcher is cfx5launch. When the launcher window opens, click the CFX-Solver Manager. This opens a window for the Solver Manager configuration. The page icon at the left of the menu bar defines a new run. Insert the name of the input file in the Solver Input File box. With the Run Definition tab selected, select HP MPI Distributed Parallel from the drop-down menu. The list of hosts should start with the host you are currently using. The page icon to the right opens the list of all the servers you can select to add to this run. As hosts are added, each is assigned one partition or CPU. After you have selected the hosts you want, you can change the partition number by clicking on the + or - sign while the host name is highlighted. You should also set your working directory in the last box.