V4 Windows Cluster
From Cornell CAC Documentation
Contents |
Login Nodes
The login servers are your gateway to the clusters. On a login server, you can interactively compile programs, create and edit files, establish input data sets, submit and monitor jobs, and check resulting data from batch runs. Don't do intensive work there. Submit intensive jobs to the clusters listed below.
- winlogin.cac.cornell.edu (full hostname: winx64login)
- 2.33 GHz Quad Core Intel Xeon L5410 ("Harpertown")
- 8 GB RAM
- Windows Server 2008 Enterprise SP1
New Users
You have likely received an email with basic account information. When you first login, find the temporary password given to you in the welcome email, connect to the login node, and follow steps at #Changing a Password at First Login to pick a new password. In order to use the batch scheduler, you will also need to set up ssh keys as described in Setting Up for Moab.
First Linux Cluster Job walks through a first login and submission of a batch job.
For any questions about accounting for projects and user limits, see Project Management and Member Limits.
Software
An updated list of v4 Windows Software is online.
To run Matlab on the cluster, see Matlab In Batch.
Using Remote Desktop Connection to connect to winlogin
This method of connecting to winlogin is preferred because it provides you with a fully functional Windows desktop. At the login screen, if the domain is specified, it should be set to CTC_ITH, not the local name of the machine to which you are connecting.
Batch machines
Once you have established a remote desktop connection to a login machine, you can use a remote desktop connection to connect to a batch machine on which you have a running job. You cannot use telnet. These machines provide a command line interface, plus the normal graphical user interface for individual applications. To leave the batch machine, type logoff at a command prompt.
If you have a Windows batch job (NORMAL) and you want to log onto one of your compute nodes, first log on to winx64login, then use Remote Desktop Connect to log in to any node of your running job.
If you have a Linux batch job (v4 or v4-64g or v4dev) and you want to log onto one of your compute nodes, first log on to linuxlogin3, then use ssh to log on to any node of your running job (use checkjob jobid to determine what node(s) your job is running on).
Remote Desktop Connect Details:
Remote Desktop sessions do not expire, but they will end when machines are rebooted during down times.
- If you use a Windows machine:
Use the Remote Desktop Connection (older name Terminal Services Client) to connect to a login machine. This software is pre-installed with Windows XP and Vista. To run it, click Start, then All Programs, Accessories, Communications, and Remote Desktop Connection. Otherwise you need to download the client before you can use it. - If you use Mac OS X v.10.4.9 or later:
Use the free download from http://www.microsoft.com/mac/downloads.mspx. Works just like the Remote Desktop Connection in Windows XP. You can also use rdesktop (see below). Tip: if authentication fails, make sure your software updates are current. - If you use Unix or Linux or Mac:
You can access the login machines by using the cross-platform rdesktop client. If you are running Linux, typically it is part of the distribution. If you prefer to build it yourself, it is available for download from http://www.rdesktop.org/. Executables for old versions are available from http://www.jacco2.dds.nl/rdesktop/index.html.
File Transfer To Clusters
A single, central file server, storage01.cac.cornell.edu, serves all CAC user home directories. You can connect to this server in a variety of ways from any operating system to access your files.
From Linux
Secure Copy
Secure copy is a standard tool to copy files to and from remote hosts.
localhost$ scp localfile.dat username@linuxlogin3.cac.cornell.edu:remoteinput.dat localhost$ scp username@linuxlogin3.cac.cornell.edu:results.dat localresults.dat
Secure FTP
FTP is disabled for security reasons, but sftp's interface is nearly identical.
Samba Client
Type
smbclient //storage01.cac.cornell.edu/<user name> -U <user name>
Enter the password for your CAC account when prompted You will see the smb:\> prompt. You can now start transferring files between your local machine and CAC home directory similar to ftp client. Type help for more instructions.
-sh-3.2$ smbclient //storage01.cac.cornell.edu/<user name> -U <user name>
Password:
Domain=[CTC_ITH] OS=[Unix] Server=[Samba 3.0.28-1.el5_2.1]
smb: \> help
From Windows
Secure Copy
The people who make Putty provide a secure copy client called pscp. From the command prompt, type:
cmd> pscp localfile.dat username@linuxlogin3.cac.cornell.edu:remoteinput.dat
<enter your username's password when prompted>
cmd> pscp username@linuxlogin3.cac.cornell.edu:results.dat localresults.dat
Secure FTP
FTP is disabled for security reasons, but psftp's interface is nearly identical. From the command prompt, type:
cmd> psftp username@linuxlogin3.cac.cornell.edu
<enter your username's password when prompted>
psftp> put localresults.dat results.dat
psftp> quit
Home Directory Access
From computers on the Cornell network or using the Cornell VPN, it is possible to mount your home directory on your local machine.
Windows XP/Vista Users
- Open My Computer
- Click on Tools -> Map Network Drive
- Drive H: (if you are already using this drive letter, use another letter)
- Folder: \\storage01.cac.cornell.edu\<userid>
- Then:
- Select "Connect using a different user name:". This will allow you to enter the domain associated with CAC and your userid at CAC, rather than those associated with your own machine.
- User name: CTC_ITH\your_userid
- Password: your CAC password
- Troubleshooting: If you have already mapped the drive and subsequently have problems, disconnect the drive and remap it.
- Next, Change the DNS settings for TCP/IP:
- Start
- Control Panel
- Network and Internet Connections
- Network Connections
- Right click on a connection
- Properties
- Internet Protocol(TCP/IP)
- Properties
- Advanced
- DNS Tab
- Append this DNS suffix:
- Add
- cac.cornell.edu
MacOS X Users
- In the Finder, select Connect to Server... from the Go menu.
- Enter smb://storage01.cac.cornell.edu/<user name> in the Server Address field as shown below. You may need to use smb://<username>@storage01.cac.cornell.edu/<username>.
- Enter your CAC user name and password to log in.
Linux Users
You cannot mount the CAC home directory as NFS for security reasons. To mount it as a CIFS drive, you need to be root, which often means using the sudo command. Then execute
mount -o user=<username> -t cifs //storage01.cac.cornell.edu/<username> /mount/point
where <username> is your username, and /mount/point is the name of a directory you have already created on your local filesystem. Enter the password for CAC account when prompted. See man mount.cifs for available options for the mount command
If you see errors, such as "missing codepage or helper program," then you have not installed the mount and umount packages for CIFS on your local machine. If problems persist, send your initial command and the results of dmesg | tail.
Changing Your Password
Your password can be set or changed on any of the CAC login nodes. The password will be updated on all CAC resources. After you set or change your password, additional steps must be taken for batch and for using MPI. Passwords expire every six months. Do not share your password.
Rules for Creating Passwords
Do not share your password. Each user should be the only one to know the password for his or her account. Well-chosen passwords are essential to preserve the integrity of the system and individual user accounts. Never leave your password in plain text (unencrypted) in any of your files. Passwords stored in this way are easily stolen.
When you change your password, the new password must comply with our password complexity policy:
- Each password must have at least eight characters.
- Each password must contain at least three of the following four elements among its first eight characters:
- uppercase letters (English, A through Z)
- lowercase letters (English, a through z)
- special characters (for example, !, $, #, %)
- digits (0 through 9)
- Do not use a space in a password. A space will cause the command used to register your password with the batch system to fail and you will not be able to run batch jobs.
- Do not form a password by appending a digit to a word--this type of password is easily guessed.
- Each password must differ from the user's login name and any permutation of that login name. For comparison purposes, an upper case letter and its corresponding lower case letter are equivalent.
- New passwords should differ from the old by at least three characters.
If you need additional ideas for creating a new password, please see http://online.securityfocus.com/infocus/1554/. Items 2, 4 and 8 are useful tips for creating strong passwords.
Changing a Password at First Login
You have to change your password the first time you login. You do this once, and it changes your password for both the Windows and Linux clusters.
If you plan to use the batch system or MPI, follow the instructions associated with #After Changing a Password.
Note for Windows: Using SSH to login to the Windows login nodes will not prompt you to change your password if it's new or expired; for Windows in order to see these messages you must use a Remote Desktop Connection (Terminal Services Client).
Changing a Password After it Expires
Your password will expire after six months or 185 days. About a week before your password expires, you will be asked if you want to change it. You can do it then or wait until it expires. If your password has expired, you will be prompted to change it, consistent with the #Rules for Creating Passwords.
Be sure that you have no other open connections to any CAC resources:
- The only open interactive session should be the one in which you are changing the password. Failure to do so will lead to the system locking your account. Disconnecting is not enough.
- Log off all other sessions connected to login nodes.
- Log off all remote connections to other CAC machines.
- Disconnect locally mapped drives to the CAC file server. If you do not do this, the system will automatically lock your account.
After you change your password, you will be logged in.
See example in #Windows Example.
If you plan to use the batch system or MPI, follow the instructions associated with #After Changing A Password.
Changing a Password at Any Time
You can change your CAC password before it expires. You will want to do so if you feel that your password has been compromised in any way. For example, suppose you think that someone else knows your password or you are concerned that you issued your password in a nonsecure setting that would have led to sending it in clear text.
Log in to one of the CAC Login nodes using Remote Desktop Connection (Terminal Services Client) for winlogin or an SSH client for linuxlogin3. Be sure that you have no other open connections to any CAC resources as discussed in the previous section.
Windows: Issue the following key sequence, in order, holding down all keys until the sequence is complete: Ctrl Alt Delete. This will bring up a Windows Security screen. Select Change Password... and follow the instructions.
Linux: Issue the command passwd and follow the instructions.
After you change your password, you will be logged in.
If you plan to use the batch system or MPI, follow the instructions associated with #After Changing a Password.
After Changing a Password
Windows: After each password change:
- register your password with the batch system by running the Perl script described in Setting_Up_for_Moab#Set Up Security Cache
Linux: No additional steps needed
Note: If your password expires or is changed during a batch run, startup of MPI processes will fail if these steps are not completed.
Password Expiration Date
To see when your password expires:
Windows: Open a command prompt window on a login node, then issue the command
net user <your login id> /domain
and look for the line "Password expires".
Linux: There is no equivalent on the linux login nodes.
Locked Accounts
There have been instances in which user accounts have been locked. Some common causes of locked accounts and the solutions are:
- Mistyping your password several times in a row.
Solution: Wait about a 1/2 hour and then try again. Be sure that your caps lock key is not on! - Trying to login to a Windows login node by using SSH when you have a new or expired password.
Solution: Login to a Windows login node using Remote Desktop Connection or SSH to a linux login node. - Failing to log off all other sessions connected to login nodes.
Solution: Log off all remote connections. Disconnecting the sessions is not enough. - Failing to disconnect locally mapped drives to the CAC file server before changing your password.
Solution: Disconnect all locally mapped drives, wait a 1/2 hour until account is unlocked, and then re-map the drive with the new password.
If you can't log on or can't wait you can submit a ticket on our [issue tracking system].
Windows Example
Assume that you have an old password, 0ldpassw0rd!! and a new password, newpassw0rd!!. Here is what should happen:
Use Remote Desktop Connect or a Terminal services client to connect to winlogin1.tc.cornell.edu (or winlogin2). If this is your first login, you will be prompted to change your password. If it is not your first login, choose
Start | Windows Security | Change Password
Then in the interface, enter:
Old Password: 0ldpassw0rd!! New Password: newpassw0rd!! Confirm New Password: newpassw0rd!!
Contact consultants by submitting a ticket on our issue tracking system or calling 607.254.8686.
Batch Scheduler
V4 Windows resources are allocated and run by Microsoft Windows Compute Cluster Server (CCS).
- Submitting to Windows Cluster
- Windows Batch Examples
- Python Scripts for Multiple Tasks on a Node
- Submitting Jobs from a Program
Queues
- NORMAL - Main batch queue to get access to blades.
- Operating system: Windows HPC Server 2008
- Number of Nodes: 15 servers with total of 120 cores, plus 1 head node with 8 cores
- Processors: 2 quad-core processors, 2.5 GHz Intel Xeon E5420 ("Harpertown")
- Memory: 16 GB RAM/server
- Network: Force10 Gigabit Ethernet
- Named: winhpc[01-16].tc.cornell.edu
- Limits: Currently no time limits.
Compiling and Linking Codes
The Windows login node provides the Microsoft Visual Studio 2005 and 2008 integrated development environments (IDEs) into which the following compilers are integrated: Microsoft C/C++, Intel C/C++ 11.0, and Intel Fortran 11.0, among others. If desired, any one of these compilers can be invoked from the command line in a Windows command shell (cmd) as well. (The command-line compiler names are cl, icl, and ifort.) However, the Visual Studio environment is preferred because it doubles as a debugger.
Getting Started in Visual Studio
In Visual Studio, you begin by creating a solution that builds one or more projects. Roughly speaking, the projects within a solution are equivalent to the targets of a Linux makefile. Each separate project corresponds to one executable or one library that is to be built.
To start a new project that (e.g.) builds an executable, given one or more files of Fortran or C code:
- First select "File | New | Project...", then
- either "Intel Fortran | Console Application | Empty Project"
- or "Visual C++ | Win32 | Win32 Console Application"
- Next, add files containing source code to the project, using
- either "Project | Add | New Item..."
- or "Project | Add | Existing Item..."
A C/C++ project can later be converted to Intel C/C++ if desired.
Compilation is controlled through the Properties dialog, in the Project menu. This dialog is nothing more than a GUI for setting the command-line compiler flags. You can define multiple configurations for each project. The active configuration determines the particular set of flags that are passed to the compiler when the project is built. By default, there are two configurations named Debug and Release that are predefined for the Win32 platform, and Debug is the active configuration.
The problem is we don't generally want to use the Win32 platform; we want the x64 platform. Strangely, this choice does not appear anywhere in the Properties dialog until you click the Configuration Manager button (which is also available from "Build | Configuration Manager"). Under "Active solution platform", choose "New...", then "x64", then "OK" the changes you have made.
(Note: Unexpected problems with the Visual Studio interface can sometimes be cleared by starting it up with "devenv /resetsettings".)
Additional Steps for MPI Codes
For MPI codes, a few more steps are necessary in Properties. Set the Configuration to All Configurations and add the following:
- For x64 platform -
- C/C++ or Fortran | General | Additional Include Directories:
C:\Program Files\Microsoft HPC Pack 2008 SDK\Include - Linker | General | Additional Library Directories:
C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64 - Linker | Input | Additional Dependencies:
msmpi.lib
- C/C++ or Fortran | General | Additional Include Directories:
For Fortran codes, in addition to msmpi.lib, one of the msmpif*c.lib static libraries is required:
- msmpifec.lib supports Fortran compilers that put the string length of character variables at the very end of the parameter stack (e = end; most common case).
- msmpifmc.lib supports Fortran compilers that put the string length of character variables immediately after the corresponding string pointers on the parameter stack (m = mixed).
The Intel EM64T processors on the V4 Windows Cluster can run either 64-bit ("amd64") or 32-bit ("i386") applications, so it is possible to compile and link MPI codes for the Win32 platform, as well as x64. To build for Win32, only one minor change must be made to the above Properties:
- For Win32 platform -
- Linker | General | Additional Library Directories:
C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\i386
- Linker | General | Additional Library Directories:
Compiling at the Command Prompt
There seems to be no convenient "mpicc" wrapper designed for use with the Microsoft C compiler and msmpi.lib. But if you prefer to compile your MPI code at the command prompt instead of within Visual Studio, it's not hard at all to pass a couple of options to cl.exe. First, fire up a shell in which you can run "cl" (the command-line name of the Microsoft C compiler):
Start > All Programs > Microsoft Visual Studio 2008 > Visual Studio Tools > Visual Studio 2008 x64 Cross Tools Command Prompt
In the shell window that appears, compile your code using "cl", simply supplying it with the include path to the MPI header file via the capital-I flag, and, after the /link flag, the /libpath to msmpi.lib, like this:
cl helloworld.c /I "C:\Program Files\Microsoft HPC Pack 2008 SDK\Include" /link /libpath:"C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64" msmpi.lib
Be sure to pay careful attention to all spaces, colons, quotes, etc. in the above syntax. More information on the various options for cl can be found by typing "cl /?".
Software Hints
Using Subversion
What is Source Control?
Subversion and CVS are source control software, which means they help you collaborate with others on developing source code by saving versions of the code as you write it.
- Subversion installed on Linux and Windows Clusters
- CVS installed on Linux cluster
- Git not installed
Cornell's SourceForge
Cornell has moved towards Subversion now that CIT runs a free SourceForge Enterprise site. You can login with Cornell Single Sign-on. If you join Cornell's Forge and get an account, read the welcome email carefully because you need to register for a second password to use SVN.
SVN Versions at CAC
On Windows, the TortoiseSVN client is a newer version than the Linux cluster's version. If you check out code on Windows, then return to Linux, you will get an error of the form:
This client is too old to work with working copy...
You can downgrade the working copy to a common version using a script available on the Subversion FAQ
Changing Python Version
We have a few versions of Python on the system.
- C:\Python25epd - Enthought Python 2.5.4 is the default. It has a large list of helpful libraries that are already tested.
- C:\Python25 - For anyone who might prefer to use the python.org distribution of Python 2.5.1.
- C:\Python26 - For those who need 2.6 language features. Popular numeric libraries aren't available for this release.
- C:\Python30 - For those who need 3.0 language features. This new release changes the language. Most numeric libraries aren't available.
To choose a particular version of Python over the default, in a batch script or interactively, set the PYTHONHOME variable to the appropriate directory, then start up the command-line interpreter:
set PYTHONHOME=C:\Python30 C:\Python30\python myfile.py
If you fail to set PYTHONHOME, the executable will load its libraries from Python25epd.
In order to start IDLE, which is the integrated development environment (IDE) for all of the above versions of Python, enter:
%PYTHONHOME%\pythonw %PYTHONHOME%\Lib\idlelib\idle.pyw
