Last modified: 25 Oct 2022

Compute Nodes: Phase I

Return to a different web page:

Follow links below for table of contents for this page:

The VCL with blades (VCL 1) was used for this phase. After experiencing several limitations and annoyances with VMware Server and VMware ESXi, other virtual machine managers were considered.

Virtual Machine Manager Requirements

Key desirable criteria were:

Oracle's VirtualBox open-source VM manager or type II hypervisor has all of those characteristics. We started out by using VirtualBox 3.1.4, and as of October 2022, are now using 6.1.38 and 7.0.2.

The Plan

Now that a powerful and versatile VM manager was chosen, its critical features were tested:

This was done in a fairly incremental fashion, learning new features only as needed. Once we felt comfortable with a feature, we would normally try to script it somehow. Scripting was done in Perl, with the VM work done by the provided VBoxManage subcommand.

Configuring a Compute Node

This is a complex topic, perhaps better convered elsewhere. We shall focus only on the things that needed to happen to make the scripts work.

There are two major sources for scripts: /root/scripts and /usr/local/bin. "/root/scripts" is used for nearly all VM creation, management, monitoring and orchestration, while "/usr/local/bin" is used for user interaction with the VMs, primarily through ssh_vbm. Because ssh_vbm uses some scripts in "/root/scripts", the permissions for /root must be changed to:

chmod o+x /root

Over the years, configuring a new compute node has been made easier via the /root/scripts/make_prep_vcl_tar script, which creates a .tgz file from the current compute node and places it in /prep_vcl.tgz. That file is copied to the new compute node and untarred from the root directory "/":

cd /
tar -zxvf prep_vcl.tgz

which copies most of the files to the right locations. After that, one uses these commands on the new compute node (restarting between them):

cd /root/scripts
./prep_vcl 1
./prep_vcl 2

which install additional packages and can propagate the files to other new compute nodes within a new VCL. But often, this is only done for one of the nodes, and then a Clonezilla image is made of that node and restored on all of the others, using a small script (cn, modified on the first node prior to imaging it so it will work for all nodes) to change the node name and set up IP addresses for each other node.

  1. Everything must be run from root.
  2. These directories must exist (according to the scripts):
  3. Ports 50000-55000 must be opened for ingress in iptables.

    When preconfiguring a VM, we often use host ports in the 33890-33999 range, so they have to be opened as well. It is from the port range 50000-55000 that we reserve host ports for use by the VMs or user/team accounts. Such uses typically include VRDP ports, and NAT port mappings for ssh and/or guest RDP.

  4. VRDP external authentication must be set up.

    We need to allow the VRDP server to authenticate users (VBoxManage modifyvm vm_name ‐‐vrdpauthtype external). This restricts the display of a powered on VM to only the users who know the user name and password. To support teams, we allow multiple users to simultaneously view the display and type or use the mouse (‐‐vrdpmulticon on).

    A file called /etc/pam.d/vrpdauth must exist and have these contents:

      #%PAM-1.0
      auth        sufficient    pam_unix.so nullok try_first_pass debug
      auth        requisite     pam_succeed_if.so uid >= 500 quiet
      auth        required      pam_deny.so
    
      account     required      pam_unix.so broken_shadow
      account     sufficient    pam_localuser.so
      account     sufficient    pam_succeed_if.so uid < 500 quiet
      account     required      pam_permit.so
    
      password    requisite     pam_cracklib.so try_first_pass retry=3
      password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok
      password    required      pam_deny.so
    
      session     optional      pam_keyinit.so revoke
      session     required      pam_limits.so
      session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
      session     required      pam_unix.so
      

Preconfiguring a VM

Using the command VirtualBox from labadmin (usually via VNC or xrdp to the host), a VM and an empty virtual disk are manually created or configured. An OS is installed in the VM and configured as desired, though the configuration is often minimal. Here are the usual steps:

  1. install standard OS from media
  2. set/create administrator/itadmin/root account and password
  3. apply OS updates to latest level
  4. install any desired apps, configure and update them

    Security apps such as anti-virus and firewalls are common if the network is exposed to the internet (e.g., "bridged"), but not otherwise. Specialized needs may require other apps, such as SQL Server Express.

  5. configure any desired services, enable them (to survive a restart) and start them

    If licensing allows, install the VirtualBox Extension Pack on the host (only needs to be done once per host, per VirtualBox release). The default extension pack provides an RDP service (VRDP) that, when configured and enabled, can show the display of the VM from boot up (including the BIOS screen) to shutdown.

    VRDP provides many benefits to students, who can then:

    The issue for creating a VM is defining the "virtual remote desktop environment" (VRDE) to enable it, set up a unique host port, enable multiple simultaneous connections if desired (useful for teams or when the instructor wants access to student VMs), and provide a way to authenticate (usually, the "external" method is used since it is a simple addition of a file to Linux PAM).

    If you can't install the VirtualBox Extension Pack, then it becomes very important to configure a guest remote desktop service inside the guest operating system to make the user experience much more palatable. This is usually the TermService service on Windows or the xrdp service on Linux. Another possibility, more prevalent on Linux, is to configure, enable and start the sshd service. Make sure the firewall is open for the service's port(s).

  6. install host-guest services (VirtualBox Guest Additions)
  7. install any required license management client
  8. install any auto-configuration script(s)
  9. install any resource managements script(s) (e.g., getidleinfo)

If this is a new VM definition, selected parts of the definition are saved in the base VM information file, together with a unique name for later use.

Once the virtual disk holds an OS, that is the essence of what people think the VM is: the guest operating system. The virtual disk is then detached from the VM and the VirtualBox Media Manager, and copied (or cloned using VBoxManage clonehd if on same host/compute node) to storage and distributed to other compute nodes generally to their /classroom/vms/xxxx folder where "xxxx" is the course name, the research project name, or other name.

Note that a virtual disk is marked as "immutable" when it gets re-added to the media manager (VBoxManage openmedium disk xxx.vdi --type immutable). This does not mark the virtual disk itself as immutable -- that's really a characteristic of how a particular VM records how the disk is to be handled; i.e., as if it were unchangeable. But there is nothing that prevents the disk or a clone of it (VBoxManage clonehd old.vdi new.vdi) from being changed, except if it added to a VM as immutable.

Later VirtualBox releases added "multiattach" as a means of allowing the same virtual disk to be attached to multiple VMs at once, using copy-on-write for VM-unique changes (or "differencing"). This is much easier to manage, as VirtualBox handles the automatic differencing vs. taking snapshots manually. However, "immutable" will continue to be used in this documentation.

We use immutable disks to save space and provide a consistent, common class or template from which to create user-specific instances of a VM. Such a VM instance is unique to the user. Uniqueness is desired mainly for Windows, but making a unique guest OS causes much larger "difference images" than just using the guest OS would, since all the differences between this VM's virtual disk and the immutable virtual disk are recorded and unique security IDs are generated.

Actions such as licensing the OS, providing a unique OS security identifier (SID), generating unique workgroup names, assigning unique account passwords for the privileged itadmin and unprivileged ituser accounts, and setting account password restrictions cause the VM instance to be unique. A unique workgroup is used to provide isolation for a security domain, and unique account passwords prevent users of a guest OS from using other users' guest OS. Unique workgroups is not really an issue with NAT, but would be for other networking types.

Creating a VM

Overview

Scripts are used to create a VM from a template file of desired VM characteristics. A given base VM name is used to look up the corresponding base VM information. Specifying the course (class), research project name or other name provides a grouping mechanism used later on for control of the VMs, using specified user or team accounts as members. User/team accounts are created as needed and associated with a special OS group called vbvmuser. Each user/team account can own multiple VMs, as recorded in their "VirtualBox VMs" subdirectory.

The task of the scripts is to create a unique instance of a VM from preconfigured information — namely, the base VM information and the virtual disk with a guest OS installed on it (its name is specified in the base VM information file).

Creating a VM from a Template

The first deficiency noted with VirtualBox was that there was no way to create a VM from a template. Thus, the make_vb_vm script was born (resides in /usr/local/bin). It is a rather simple script that hard-codes what is thought to be common VM characteristics, and it takes a few arguments to override some defaults. Those arguments are: VM name, OS type audio state (on or none), the disk controller type, the IO APIC state, the PAE state and whether or not page fusion is desired.

Using a centralized script to create VMs allows changing of the core VM specs for all subsequent VM creations, but won't change any of the existing ones. The script needs to be modified to change the core specs; this was before we started to use external files to provide configuration information.

But what are the "core" specs?

The VM is created and registered using the createvm subcommand of VBoxManage, and then the shell VM is modified using the modifyvm subcommand, using all of the settings above. Once that is done, storage controllers are added for the specific storage controller using the storagectl subcommand, and an empty DVD drive and floppy drive are attached to the machine using storageattach.

That's all this script does. Virtual disks are attached and any other modifications are done later, using another script.

Creating a VM Instance

The VM instance creation process consists of these major steps:

  1. Read information about the base VM, reserved ports and standard network services
  2. Create a new or get a reference to an existing user account
  3. Force the virtual disk to be immutable for this VM, if the disk is marked as immutable
  4. Create a VM
  5. Attach the virtual disk with the guest OS on it
  6. Attach shared folders to later configure the guest OS
  7. Set up any NAT ingress ports and reserve them
  8. Start the VM
  9. Create a snapshot that will hold this unique VM instance
  10. Pass user and OS type information from host to guest
  11. For Windows, here are steps that are optional and likely to be unique:
    1. Autologon to Windows to run configuration scripts
    2. Run sysprep after it was configured
    3. Pass more information between host and guest to complete configuration
    4. Autologon to Windows to run post-sysprep scripts

    Using sysprep was common in the first few years of the VCL, but became less important as we migrated to using trial versions of Windows, where licensing issues were not as complicated and we didn't consume a lot of licensing keys per quarter.

  12. Perform a graceful shutdown
  13. Attach any additional disks
  14. Get a network port for remote display of this VM and reserve it
  15. Connect the virtual network cable
  16. Set up remote display for the VM
  17. Record the instance-specific information for the user

The VM instance creation process is implemented in the create_vb_vm_for_user script and the vb_vm.common common subroutine file. Both reside in each compute node's /root/scripts directory. The script calls make_vb_vm.

create_vb_vm_for_user creates a VirtualBox VM for a given course number (class)/research project/other use, user/team account, and base VM name. It relies heavily on the vb_vm.common common subroutine file. The subroutines were originally part of the script, but removed then included ("required") when other scripts were written to allow reuse of the subroutines in the newer scripts.

Some information is hard-coded at the top of the create_vb_vm_for_user script. It should be placed in a configuration file in the future. That information includes the paths to:

  1. the base VM information file

    /root/create_vb_vm_for_user.dat is the source of the template definitions of the VMs. It defines prototypical VMs, configured a certain way for the needs of various courses, thereby providing a common definition file. The file must be secured such that only root can read it, since it contains passwords. Once the template is defined (and probably debugged as well), it is often copied to all other cns for consistency and to make backup copies.

    The items in the template are in tab-delimited columns, denoting:

    Here is a sample file, with passwords appearing as 'xxxx':

    #base_name      os_type CPUs    RAM_in_MB       audio_state     admin_acct/pw   disk_controller     disk_file       cd_file nicnum/guestport/protocol       nic_settings ioapic_state    snapshot_folder_list    pae_state       page_fusion
    win10_tinfo457_sum2020  Windows10_64    2       4096    none    administrator/xxxx   SATA    /classroom/vms/tinfo457/win10_ent_installed_20200603.vdi#multiattach2/3389/tcp      --nic1 intnet --intnet1 %user%_intnet %dhcp/192.168.u/100% --nictype1 82540EM,--nic2 nat --nictype2 82540EM on              off     off
    fedora31_rdptest_sum2020        Fedora_64       1       2048    none    root/xxxx    SATA    /classroom/vms/rdptest/fedora31_rds_updated_20200824.vdi#multiattach2/22/tcp,2/3389/tcp     --nic1 intnet --intnet1 %user%_intnet %dhcp/192.168.u/100% --nictype1 82540EM,--nic2 nat --nictype2 82540EM on      /ssdm3.5t/snapshots#100 offoff
      
  2. the reserved ports file

    /root/reserved_ports records information about network ports. There is a range of network ports on the host that are open and reserved for use by the VMs. They are used to prevent re-allocation by other VMs; otherwise, there would be no centralized record of them (VirtualBox does record the information with each VM). This tab-delimited information is recorded:

    Here is a sample file:

      54455   20100405 16:27:53       _431team3       w2k8_web        http
      53838   20100405 16:27:54       _431team3       w2k8_web        https
      53092   20100405 16:36:10       _431team3       w2k8_web        ms-wbt-server
      53075   20100405 16:36:16       _431team3       w2k8_sql        ms-sql-s
      54630   20100405 16:44:03       _431team3       w2k8_sql        ms-wbt-server
      

    Most of the time, ports are reserved during the VM creation process. However, users can also reserve ports. This happens indirectly when they create new web servers to periodically update their screenshots, via the screenshot command sent via ssh and processed by ssh_vbm.

  3. the log file

    /root/create_vb_vm_for_user.userid.log is a per-user log (i.e., "userid" varies) that records with a date-time stamp what happened during all VM creations for that user.

  4. the user home directory

    This is where new user directories will be created. This is under the "course" directory which groups users in the same course (and section). This is all under "/classroom/home", so a user called "user123" in course "tcss431" would have a home directory in "/classroom/home/tcss431/user123".

    The .vcl file resides here. It records user name/password, host IP, and VM instance and optional port information for each VM. This file is sent to the user's or team's account on "cssgate.insttech.washington.edu", in the corresponding "/home/INSTTECH/acctname" home directory. This is done after "/root/scripts/harvest_vb_vm_info" is done over the entire course for all hosts, and then actually distributed to cssgate via "/root/scripts/gather_vb_vm_info".

    A log of ssh commands sent to the VMs, called ssh_vbm.log is stored in the user/team account's home directory.

    The VMs are stored in the home directory, under the VirtualBox VMs folder. Often, due to immutable/multi-attach disk files, the only thing saved in the VM's subdirectory are the snapshots, whose size may be significant. Snapshots for Windows disk images brought out of sysprep can easily consume 4GB just for the unique security IDs and other actions Windows takes to make the disk unique.

    Other files placed in the user/team account home directory are:

  5. the sysprep information directory

    Information for sysprepping a Windows image can be found here.

    However, in recent years sysprepping inside a VM has been abandoned for sysprepping the VM once and optionally starting the VM manually and making any necessary final adjustments from the virtual display. Also, trial versions have been also used instead of any sysprepping.

    It takes far too long to automate sysprepping (i.e., getting the VM out of sysprep via the out of box experience) to get this right, but that means doing the same thing n times for n sysprepped Windows VMs. Some of that is alleviated by using VBoxManage keyboardputscancode or /root/scripts/send_sc to send keystrokes to each VM within a host/cns (or serially across hosts via ssh) and scripts placed in C:\temp prior to sysprep.

    Here is an example of how to send keystrokes in parallel, which is much faster than serially:

    rshall '/root/scripts/send_sc -u ituser -v win10 -k enter `hostname`'
    
  6. the shared folder paths

    Paths to various shared folders are listed to allow permanent or temporary attachment to VMs.

  7. VCL information file

    Information about the VM name, user accounts, passwords, IP addresses, and ports are placed in the user's home directory in a file called .vcl.

Example of VM Creation

Presume that a preconfigured VM for Windows 10 (see win10_tinfo457_sum2020) already exists, and now want to use that VM for a class on a compute node. Currently, you'll have to manually copy the virtual disk (e.g., win10_ent_installed_20200603.vdi) that contains the OS and any additional virtual disks into its base VM directory (for now, /classroom/vms/tinfo457). Then you'll be ready to create one VM for one user (see create_batch for times when you need to create more).

Here is what one would enter as root for class "tinfo457", user "css_test" using "win10_tinfo457_sum2020", to create the VM instance "win10" owned by "css_test":

cd ~/scripts
./create_vb_vm_for_user -v tinfo457 css_test win10_tinfo457_sum2020 win10

Create Many VMs for a Class

A class requires the same set of VMs for each user or team account. After defining a template for each VM and creating the base virtual disk that the template uses, one can list the templates to use for each VM, including a placeholder for the user name and a specification for what the VM's name will be. For example, create_class_batch.dat might contain:

#tcss431        %u      win7_tcss431_sum2014    win7
#tcss431        %u      kali_tcss431_sum2014    kali
tcss431 %u      metasploitable_tcss431_win2017  metasp
tcss431 %u      kali2016_64_tcss431_win2017     kali2016
tcss431 %u      winxp_tcss431_win2017   winxp

Lines starting with "#" are commented and will be discarded. They function as a historical log or to remind the user of the format of the file, or allow other classes to be defined and used and return back to a commented out class later.

The tab-delimited values are the class name, a placeholder for the user name ("%u"), the template's "base name", and the name for the VM as it will appear to the user. Note that each VM name must be unique for the user, which is especially important when multiple VMs (e.g., multiple win10 VMs) are required. Generally, letters from a..z are appended to the name (e.g., win10a, win10b), and each VM needed has its own line.

create_class_batch.dat is never used on its own. It is coupled with create_batch_users.dat, which lists each user on a separate line of the file. These user ids (UW Net Ids) should be members of the class currently uncommented in create_class_batch.dat.

Once those two .dat files are defined, one can run the create_batch script to create VMs for all users, as follows:

cd /root/scripts
./create_batch -nosp -noap

The arguments passed tell create_vb_vm_for_user to ignore sysprep and after-sysprep commands, in case Windows VMs are being defined (sometimes there is a mixture of Windows and Linux VMs in the class batch file).

Typical Tasks for Creating VMs for a Class/Course

This is an attempt at describing the entire workflow for getting VMs ready for a course.

  1. Receive request from instructor for VMs needed for specific course

  2. Get feedback from instructor about VMs and the virtual network

    You need to get a feel for the resources needed for the VMs, and if it is even a feasible request given the current or anticipated usage of the VCLs. The instructor may not know, or rely on you to use reasonable defaults. Some important aspects:

    Cognito Forms provides a web service to create forms with conditional logic in them, and it is free for a low volume of form submissions.

    Such a form was created for the purpose of requesting VMs. Its URL https://www.cognitoforms.com/UniversityOfWashingtonTacoma1/RequestVMs was linked to and the uwnetid was passed in for the email address in cssgate:/var/html/www/secure/reqvmv2.php to prefill the email address with the UW Net ID, and https://cssgate.insttech.washington.edu/secure/reqvmv2.php was shortened by bit.ly to http://bit.ly/setreqvms .

    Unfortunately, the VM request form story does not end there. The email that Cognitos Forms sends is in HTML format, and one that is almost a compressed one-liner with a lot of extra formatting inside. It looks okay in a web browser, but email clients don't handle it well. Pegasus Mail strips out all of the form information, so forwarding the form entry details email to Request Tracker ends up as an empty ticket. Outlook extracts the text from the form entry HTML when forwarding, but the resulting text is not well-formatted, making it difficult to read the relevant information.

    Consequently, to make the VM request information to be useful, the extract_text_from_html.pl Perl script was written and is located in \\home.tacoma.uw.edu\srondeau$\scripts\perl. It provides much better extraction and control of output than free HTML to text converters that can run from the command line, and has been customized for handling the Cognitos RequestVMs form entry details email.

    For example, if from an email client such as Pegasus email, we save the body of the message (the HTML) to C:\reqvm as rvm.html and have Perl installed, we can run this command:

    perl \\home.tacoma.uw.edu\srondeau$\scripts\perl\extract_text_from_html.pl C:\reqvm\rvm.html
    

    which will result in a .out file with a name generated from the quarter, year, and course number from the form entry. The content of the file is all of the relevant VM request information, nicely formatted and readable.

    Needing to manually run a script on a particular file complicates the process and forces the user to move from handling email to running an external script then going back to handling email again to use the better-formatted output.

    The process can be and was made easier by automatically detecting when a file is added to a directory. One way to do that is by installing WatchDirectory Pro, and then creating a task to run a Windows batch file when something happens in a predefined directory (e.g., C:\reqvm).

    The events to watch for is any file (except those with a .out extension) that is created, modified or renamed in that directory. The task should be configured to run as a service. After that, one would need to change the service login to a user such as uwtacoma\srondeau to access the network share (\\home.tacoma.uw.edu\srondeau$), because the default LocalSystem service logon account cannot.

    Sometimes it is far easier to put complex commands in a batch or .cmd file instead of running them directly from WatchDirectory. That is what was done, and the file named C:\watchdirectory\watch_reqvm.cmd was created, containing:

    @echo off
    
    C:\perl\bin\perl \\home.tacoma.uw.edu\srondeau$\scripts\perl\extract_text_from_html.pl "%WD_FILE%"
    

    That WatchDirectory task needs to be set up once, and tested to make sure it does. Once it does, the workflow to process Request VMs form entry information and submit it as a ticket becomes the following, for each set of VMs requested per course:

    1. Wait for an email from Cognitos Forms, indicating a request for VMs.

    2. Open the email and save its contents to the C:\reqvm folder

    3. WatchDirectory then detects the file creation or change, depending on what file name was used when saving, and runs C:\watchdirectory\watch_reqvm.cmd, creating an appropriately-named .out file.

    4. In the email client, compose an email to request@mail.d.insttech.washington.edu
        Import or copy and paste the contents of the newly-created .out file into the body of the forwarded email.

      1. Cut the last line of the new body (a new subject line) and paste it as the email subject.
    5. Send the email.

  3. Use the information provided by the instructor or by experience to define each VM

    The VM definition goes in /root/create_vb_vm_for_user.dat. Often, this file is replicated across all compute nodes in a given VCL, which is useful if the demands of the course require that multiple hosts be used.

  4. Determine if more than one host needs to be used.

    Use the number of students, the number of VMs per student, and the cores needed per VM to come up with a rough estimate of the load on a host. If it exceeds the maximum physical cores (NOT hyperthread) available on the host (minus two for the host OS to use), then you should split the list of students across more than one host.

    You can do that manually, or use /root/scripts/split_batch.

    A good rule of thumb is that if more than three powerful (64-core) hosts are needed, then the instructor should be told to re-think the VM needs or they way they have students use them. The reason is that there are often multiple courses that require the VCL per quarter, and overconsuming the hosts will limit the number of courses we can satisfy. Possibilities for reducing the demand are: use of multi-student teams instead of individual students (multiple VMs assigned to each team vs. to each student), lowering number of VMs needed per student or team, lowering higher-core requirements, and/or lowering RAM requirements.

  5. Choose a host on which to create VM definitions and install the guest OS.

    Using a host that has SSDs will make the entire OS installation and updating process much more pleasant, as it can be very I/O-intensive.

    It is the virtual disk that is ultimately the most important thing to create, since it holds the operating system and its configuration. As a file, it is portable to other hosts, including those with different host operating systems or CPU vendors (within the x86/x86-64 family).

  6. Create each base VM needed by the course
    1. Get .iso image

      Often, you will need to download and copy to a host an .iso image of the operating system(s) desired, and put it in a folder on the host named /iso. Trial versions of Windows are often used to avoid licensing issues; Windows 10 lasts 90 days, while server versions often last 180 days. However, other avenues for Windows such as Azure Dev Tools for Education may provide .iso files as well, with other licensing terms.

      Generally, we avoid using UW-negotiated Windows licenses because we don't want to deplete the pool needed for lab workstations, and because the terms usually require that one version of Windows be previously installed prior to upgrading to the desired Windows version. That's not feasible to do for multiple VMs on the same host computer, which may not have ever had Windows installed on it natively.

      In 2020, we use 64-bit versions of the operating systems unless there is nothing else available. It will require somewhat more RAM to run, but it is the most capable and recent instruction architecture available.

    2. Define a VM with the desired settings

      It's best to define it as requested, to determine if there are unexpected resource demands and so that the future guest OS will properly detect the hardware and use the right number and type of drivers.

      Choose a good name for the VM, utilizing what OS will be installed, version information, and when it was installed or updated, so you know when a trial license may expire or if the resulting OS image can be reused later and updated. This name should be the name of virtual disk defined in /root/create_vb_vm_for_user.dat for this VM.

      Select the OS and architecture size (32 or 64-bit) corresponding to the guest OS to install, as that will be the best performing one.

      In 2020, we tend to define virtual hard disks as VDI files which dynamically expand to the maximum amount we think is useful, which is 150GB. Most operating systems and their updates will use far less, but since the dynamic choice doesn't consume any more disk than it needs (mostly), it uses the least disk resources.

      This is the point at which you should attach any .iso file to the disk controller, to be used to install the guest OS.

      Make sure the network is defined as desired as well, with the correct NIC networking methods per NIC, in the order desired. It is NOT necessary to define any NAT port mapping.

    3. Start the VM and install the guest OS

      Install to the virtual hard disk defined above, partitioning the disk as desired. The default settings are often used for Windows, while Logical Volume Management (LVM) is often used for Linux.

      Define an administrator/root account and use the password defined in /root/create_vb_vm_for_user.dat for the VM. Make sure you update the /root/scripts/mod_vcl_file.patterns file with the named VM's correct guest user and password — the one you want the students to see or use — in their .vcl file.

      Install any applications possible to install at OS installation time.

    4. Update the operating system

      It is good practice to always do so unless the needs of the course demand otherwise, as specified by the instructor. You may want to ask the instructor about that explicitly if the VMs will be used for vulnerability testing, as unpatched operating systems are desirable.

    5. Install and configure VirtualBox Guest Additions

      This provides a better user experience, synchronization with clocks and many useful features.

    6. Install and configure the remote desktop service, or ssh service

      A guest remote desktop service is only needed if the VirtualBox Extension Pack can't be installed and hence VRDP cannot be used. In that case, the guest remote desktop service would provide a familiar desktop experience for people who are accustomed to using GUI desktops. Windows, Ubuntu, and Kali are all desktop-driven, as are other Linux systems that install desktop environments.

      The guest service may need to be installed, configured and set to start when the OS starts. In addition, the guest operating system's firewall must be opened to allow for the port that the remote desktop service uses, which is usually 3389.

      You must also ensure that the network method attached to one of the virtual NICs is either NAT or Bridged. If NAT, then a host port needs to be reserved when the VM is created via the VM definition information, mapping to the guest port (3389 in this case) and ms-wbt-server service. If bridged, the port 63890 should be used to avoid being blocked at the UW network perimeter, and at run time the guest's IP address (if dynamic) will need to be determined.

      Alternatively, or perhaps in addition, install, enable, configure and start the ssh daemon or service. Make sure that at least one user can login via ssh from outside, since root logins are by default not permitted using passwords, and the root account could be the only login account on a Linux system. Similar restrictions as the remote desktop service uses apply to ssh ports and the for use of NAT or bridged networks by the NICs.

    7. Install and configure any other applications/services.

      These would be the ones that help manage the VCL. For 2020, that may mean software that can detect user input activity and record the last time there was some. That information can be used to determine whether or not to save the state of the VM to decrease the utilization of host resources, allowing more VMs to be assigned to the host.

      From ticket 6938 -- Keyboard and Mouse Input Activity Detection:

      • Windows guest remote desktop and console activity

        A custom AutoIT Script script called getidleinfo.au3 was created to see if keyboard/mouse I/O has occured. If there is a change from one sampling of an I/O counter to the next, there is input activity.

        Getting this to work properly and consistently has proven to be a challenge. The current (February 2021) method involves installing the AutoIT Script package on each Windows machine, installing the nssm service manager, and configuring a service called GetIdleInfo to run AutoIt3_x64.exe with getidleinfo.au3 as a parameter. The service seems to stay running longest when this is done; otherwise, it stops running within an hour.

        To make the installation easier and more consistent, each host has the necessary files in /fake/ro/tools, which is mapped to the Z:\ drive. One copies those files to the C:\tools directory. Then the script C:\tools\gii_install.cmd is run to set up everything.

        The current (February 2021) method of exporting the information is via the VirtualBox Guest Additions VBoxControl.exe program, which allows the guest OS to send data to the host OS. In this case, the guestproperty set /setlabs/gii/datetime is set with the last time input activity was detected. A corresponding host command via VBoxManage reads that guest property for a Windows VM when free_vm_resources is run via a cron job.

      • Linux ssh and console keyboard activity

        The script command records both terminal input and output.

        We don't care about what the actual user input is (which is why such input is written to /dev/null) but we do care about the timing information. Ignoring actual input avoids recording sensitive information such as passwords in clear text. We use the timing information to calculate the time of the last keyboard activity by a logged-on user, and may have to do something special to allow non-root user background processes to be detected so we don't kill active work no longer attached to a terminal.

      • Linux desktop activity

        The xinput command can be used to detect mouse and keyboard activity.

      • All Linux input activity

        The getidleinfo script handles everything; see a detailed explanation of what is happening and how to install it.

      • Activity information

        In the examples above, the activity information is either recorded in a guest property of the VM or on the virtual disk. Putting it on the virtual disk makes it cumbersome to see from the host. Using shared folders and making sure that Guest Additions are installed, the activity information could be made to write to the user's home directory on the host. However, shared folders have some drawbacks that make them somewhat unreliable.

    8. Install and configure any other services/applications.

      These would be the ones required by the course instructor.

    9. Create any special user accounts.

      Sometimes there are multiple admin or unprivileged accounts that should be created; often, these are given the same password as the default admin account for ease of use. If we are trying to impress upon the students the principle of least privilege, we may provide a login account which can be elevated on demand to executing admin commands (e.g., via sudo) but not be able to execute them by default.

    10. Test to make sure everything works.

    11. Shutdown the operating system

    12. Copy the virtual disk to the /classroom/vms directory for the course.

  7. Using class lists of students

    The class list of UW Net Ids (one per student) is placed in a file. For teams, put the team account names in a file. There is one UW Net Id or team account name per line. The class list file is usually named "qqqyyyy.course", where "qqq" is a quarter name abbreviation (e.g., "win", "spr", "sum", "aut"), "yyyy" is the year, and "course" is the course number and possibly section (all one word and in lower case). It can be copied into /root/create_batch_users.dat for later use by /root/scripts/create_batch.

    For example, let's say file "/root/aut2020.tinfo431" has these lines:

    janedoe
    johnsm
    mhubbard
    jackspr
      

    However, when you have to split that class list across multiple hosts, you can edit the original class list file and place the host name followed by a colon as a line before the list of students that will be placed on that host. For example, using hosts "cn2" and "cn3" for the above file, it now looks like:

    cn2:
    janedoe
    johnsm
    cn3:
    mhubbard
    jackspr
      

    Then you run /root/split_batch to create create_batch_users.dat files representing the desired students on each host. For example:

    /root/scripts/split_batch /root/aut2020.tinfo431
      
  8. Create or append to the class batch file.

    This is the /root/create_class_batch.dat described elsewhere. The course number, VM definition's base name and other relevant information for each VM used in the course must be present. If more than one host is needed for the course, that file must be replicated to each host used.

  9. On each host to be used, create the VMs for the students on that host.

    Use /root/scripts/create_batch as described above.

  10. Change the /root/scripts/mod_vcl_file.patterns file to reflect the needs of the course.

  11. Apply the mod_vcl_file changes (see above)

    It is easier to run the /root/scripts/apply_mod_vcl file for a course than to do the above manually, as it can change everyone's .vcl file a given course number. For example:

      cd /root/scripts
      ./apply_mod_vcl tinfo431
      
  12. Harvest the VCL information for the course.

    Here's an example, which needs to run on all of the cns involved (e.g., from cn2 first):

      cd /root/scripts
      ./harvest_vb_vm tinfo431
      ssh cn3 /root/scripts//harvest_vb_vm tinfo431
      
  13. Distribute the VCL information for the course.

    Here's an example, presuming /root/scripts/cns can generate all of the cns involved so gather_vb_vm can collect all of the information harvested previously:

      cd /root/scripts
      ./gather_vb_vm -d f tinfo431
      

    Note that special handling is required for students enrolled in more than one course that uses the VCL.

  14. Finalize any post-creation changes to the guest operating systems that are needed.

    This could be starting all of the VMs (such as a sysprepped Windows 10 VM) and bringing it to the point where the login screen is shown. This can be fairly involved, as all of the course's VMs need to be touched, and the same commands issued to each VM. See /root/scripts/send_sc.

    Often, if you change something in the VM, you will want to delete the User_Changes snapshot and re-take it. For example, for the entire course "tcss555" on the current host:

    /root/scripts/snapshot_vb_vm "delete User_Changes" tcss555
    /root/scripts/snapshot_vb_vm "take User_Changes" tcss555
    

    That makes it possible for the users to easily revert to "User_Changes" with all of your modifications intact, if they really mess up the guest OS configuration or filesystem. If you don't do the above, all of your modifications you made, including things like getting Windows out of sysprep, will be deleted after they revert, which is usually not desirable.

  15. Test a sample set of VMs for proper operation from manage_vc.

    Often it is useful to test one set per host used. Check for connectivity amongst VMs, to the internet (and for domain names), remote desktop and/or ssh usage from a remote client, etc.

    If something is wrong, you might need to delete all of the VMs on all of the hosts, redefine the base VMs, and redo most of the other steps above, to get it all working correctly.

  16. Finalize resource management for the VMs

    The primary ones are:


  17. Notify the instructor that the VMs are ready.

  18. Record information about VCL usage.

    VCL usage information is recorded per quarter via modifying this web page. This information is used for tracking which hosts have been used for what courses or research, as well as justifying the purchase of replacement host computers when one or more start failing and are out of warranty (if any).

  19. At the end of the quarter, ask the instructor when the VMs can be reclaimed so the hosts can be reused for the next quarter.

Creating a Virtual Network

Overview

One can start providing networking to a VM by defining a virtual NIC that is supported by the guest operating system, assigning it a networking method provided by VirtualBox, and configuring that networking method.

Using NAT Networking Method

If all that is needed is a connection to the internet, then the NAT networking method is used. NAT provides a private IP address and "translates" it to the host's public one to send information to and receive information from the internet.

Most network traffic initiated via the guest OS will be captured by the virtual NAT device, its source VM information will be remembered and then the public IP address will be substituted in the packet and sent out to the internet. When a response returns, the IP address of the originating VM will be looked up and the response packet will be sent back to it.

More complicated NAT networking involves using the VM as a server providing some service. That service could be ssh, remote desktop, or web. The only way some entity on the internet can access a VM is if a port mapping exists between the host and the guest OS. The mapping of host port to guest port is part of the VM definition, and is communicated to the user.

For example if one knows that on host IP 140.142.71.32, the host port 53466 maps to port 22 inside the guest VM, all one has to do is use the host IP and port to connect to the guest OS:

ssh -P 53466 itadmin@140.142.71.32

There are more complicated configurations and uses of NAT, but those are the most common.

Networking VMs

Networking VMs is often requested by an instructor for a course.

Controlling VM(s)

When a VM instance is created, it is left in a powered off state. At minimum, we need to be able to start a VM (power it on), but there are other actions we might like to take as well. The script in /root/scripts that one uses is control_vb_vm. It operates for all VMs on the current host. Starting a VM requires the use of the VBoxManage startvm subcommand. The other control commands supported by the script are VBoxManage controlvm control commands and VBoxManage getextradata commands; both of which require a running VM.

The script can operate on an entire course, all the VMs for a user within the course, a particular VM that a user owns, or all course VMs named the same, regardless of the user.

Here is the current complete list of actions:

  1. start

    If powered off, power it on. Once running, assign it to the same CPU core as was previously assigned.

  2. shutdown

    If powered on, attempt to perform a graceful shutdown by simulating pressing an ACPI power button. The VM may not power off, in which case you must issue the "poweroff" action.

  3. poweroff

    If powered on, essentially pull the plug on the VM. The OS won't be informed of the "loss of power" event, and the VM will stop executing and be left in the powered off state.

  4. show

    Show the VM information (uses VBoxManage showvminfo). One may have to use the "-v" option and pipe the script's standard error output to the same as standard output to be able to use a command like "less". For example:

      cd ~/scripts
      ./control_vb_vm 2>&1 -v show tinfo431 css_test w7c | less
      
  5. showxd

    Displays the "extra data" associated with the VM, which is predominantly NAT's ingress host port to guest port mapping information.

  6. resume

    If a VM runs out of real disk space, it will be put in a "paused" state. One could also do this manually. This action will resume a VM that is in a "paused" state.

  7. test

    Since the other actions always need to check to see if the VM to take action upon exists, and to determine its state, this is a quick way of doing nothing other than verify the state of VM.

  8. setcore

    Set this VM to use the core number specified. This was designed when only one CPU was common. We are not sure how VirtualBox assigns cores, but hope that it assigns them sequentially, and ignores assigning hyperthreads. It is an attempt to enhance performance by reducing context-switches.

  9. showcore

    Show the core number that this VM is using.

  10. savestate

    Save the state of the VM. Very useful if we need to take down the host and don't want to interfere with the state of each VM that the user is controlling. However, if the host is being updated and a new release of VirtualBox is installed, the saved state may not restore correctly. Also, beware of the extra snapshot storage needed to save the state.

Examples

Modifying VM(s)

When a VM is powered off, we may want to modify a VM's configuration. This is difficult to do with just the VBoxManage scripts for multiple users. The modify_vb_vm script makes it easier to do for entire classes, individual users within a class, or a particular VM for a user. It operates on all VMs on the current host.

There are many aspects of a VM to modify, so the action(s) that modifies the VM via the VBoxManage modifyvm command is simply passed as a quoted argument.

In some cases, the quoted argument would ideally contain something user-specific. For example, a private "internal network" might be named after the user (e.g., "user123_intnet"). You can use "%u%" in the action where you want the user id to be substituted.

Example

Turn off the audio for class tinfo431:

cd ~/scripts
./modify_vb_vm -v "--audio none" tinfo431

Deleting VM(s)

Like everything else, a VM has a lifecycle. Once its usefulness is over, it can either be very easy to delete a VM or fairly complex. Therefore, a script called delete_vb_vm was written to delete VMs on the current host.

Examples

Distributing VM Instance Information

Each user has a VCL information file that records unique information about created VMs. That information needs to get back to the users somehow, and since we don't allow them to login to a terminal session on the compute nodes, we must put that info elsewhere.

  1. setting course-wide admin account passwords

    Initially, each user would get a unique password per user account, but that proved difficult for the students, albeit secure. However, most VMs are inaccessible to people on the internet, including other students, and the need for unique passwords was further mitigated by unique and random port mappings. In more recent times, the admin account password is the same for each student in a course, but not necessarily for each VM, and it can be different from the one created when the base VM was created. Consequently, we change the .vcl file to reflect the desired admin account and password per VM, and that is done by the /root/scripts/mod_vcl_file script.

    mod_vcl_file requires a smaller file called /root/scripts/mod_vcl_file.patterns, which contains the pattern to match in the .vcl file and what to replace the pattern with, per named VM. That .patterns file will vary per course, and is usually changed whenever a new VM or set of VMs is created on the host.

    One must send the name of a user or team account to the standard input of mod_vcl_file so it knows which user. But this can only be done from the home directory of the course. If done manually, this will change each user's .vcl file in the "/classroom/home/tinfo457" directory:

    cd /classroom/home/tinfo457
    for u in `ls -1`; do echo $u | /root/scripts/mod_vcl_file; done
      

    That command is rather difficult to remember and type, so /root/scripts/apply_mod_vcl was created. For example:

    cd /root/scripts
    ./apply_mod_vcl tinfo431
      
  2. harvesting information

    Now that all of the .vcl information is correct, we should harvest that information, which basically collects in one place the VCL information files for all users in a course on a compute node. We can ask each compute node to do that, thereby collecting all of the information available, perhaps for an entire class whose VMs are spread out over several compute nodes.

    The harvest_vb_vm_info script collects all of the desired VCL information files for the compute node on which it is running. Here is an example which gathers all of the VCL info files for the tinfo431 class:

      cd ~/scripts
      ./harvest_vb_vm_info -v tinfo431
      

    If we want to do that over all compute nodes (verify that /root/scripts/cns is correctly set up first):

      cd ~/scripts
      ./do_all "cd scripts && ./harvest_vb_vm_info -v tinfo431"
      
  3. gathering information

    Now we want to gather all of the harvested information in one place, so we can put it on a system (viz,, cssgate.insttech.washington.edu) that the students can login to. Here is what that would look like:

      cd ~/scripts
      ./gather_vb_vm_info tinfo431
      

    This archives the information in a tar file and transmits it to cssgate (root password is required), where it is unarchived and placed in each user's home directory, creating the home directory beforehand if it doesn't already exist.

    Exceptions to normal distribution:

    1. Overwriting all old .vcl information for students in new quarter

      The VMs for students are usually created every quarter, and the previous quarter's information can be overwritten vs. appended to. Here is what that would look like:

      cd ~/scripts
      ./gather_vb_vm_info -d f tinfo431
          

      Note that there is no "-" before the "f", as it indicates an option to pass ("-d") to cssgate, not to gather_vb_vm_info.

    2. Appending select .vcl information for students in multiple courses in same quarter

      Students that are enrolled in more than one course that uses the VCL need to get the VMs used by all courses. There is currently (December 2020) no information to indicate which course uses what VMs, but there is a way to append VCL information for each course.

      The first thing is to identify which students are taking both courses (usually, it is only two courses). On a Linux system, we create a sorted list of UW Net Ids for each course, e.g., win2021.tinfo442.sorted and win2021.tinfo457.sorted, and find the lines in common:

      comm -12 win2021.tinfo442.sorted win2021.tinfo457.sorted
          

      and you should see the list of student UW Net Ids that are in both courses. Save that list via redirection to a file (e.g., /root/students_to_append) so it can be used on the host(s) for the second course — the first course (e.g., tinfo442) can be handled as indicated above.

      To append to existing VCL information from the host for the second course (e.g., tinfo457) as well as overwrite the VCL information for those students only taking one course (e.g., info457):

      cd ~/scripts
      ./gather_vb_vm_info -d f -a -f /root/students_to_append tinfo457
          

      Note that there is no hyphen (-) before the f in -d f as it indicates an option to pass to cssgate, not to gather_vb_vm_info. The -f option indicates the file of users whose .vcl files will be appended.

Monitoring VMs

When there are hundreds of VMs created for use, it can be very difficult to find out what was done (e.g., which compute nodes host which VMs) and what is going on (e.g., current state of the VM). In addition, it would be good to collect that information so that one could change the VM definitions or virtual network at will, perhaps even delegating that authority to the instructor (a process, under development in 2020, called "orchestration").

The first step, however, is to gather the information about current VMs. This could be integrated into create_vb_vm_for_user, but it doesn't handle what existed prior to the integration. As such, it may be best to collect information for all courses or for the "ituser" account.

We can look at /classroom/home for a list of subdirectories, which are usually course names (e.g., "tcss431") or special names like "faculty" or "sbrtest". Underneath the course subdirectories are usually the users' home directories, but other information can be found in them as well. Underneath a non-course directory like "sbrtest" might be only the "VirtualBox VMs" directory, but these special directories might not be that interesting; i.e., we would like to know what is going on for larger numbers of VMs, such as those used for a course.

Hence, monitor_vms was created. It can simply list all VMs on the current host by traversing the information found in /classroom/home or /home/ituser, or it can save that "basic" information (via VBoxManage list vms) to a FirebirdSQL database on cssgate. In addition, it can extract VM-specific information from each VM (via VBoxManage showvminfo vmname --machinereadable) and either print it or save it to another table in the database.

The reason to optionally put the information in a database is to avoid re-running monitor_vms every time, which can be fairly slow. That script does not record any historical information, either. In addition, it is easier to pull data out of a database than out of the output of monitor_vms.

A future use of this VCL information is in a web-based VM orchestration, system — ticket 6089, VCL Orchestration. In 2017 a student intern started work on this, and another student intern in 2020 moved it further along, but as of October 2022, it still isn't yet ready to use.

Use of monitor_vms

Here are some common use cases (relative to /root/scripts):

Overcommitting VMs

The Problem with Current VM Allocation

The VM allocation practice in 2020 and in years preceding it was to pre-define a VM with a certain number of cores and RAM, and provide one or more of these VMs per student or team, depending on the needs of a course. The capacity of the host was taken into account when determining how many VMs could be placed on the host; capacity is defined as available cores and RAM. A host's available cores is the total number of cores minus 2 (to support the host OS), and the available RAM is the amount of host RAM.

If there is one VM per student/team, the maximum number of students/teams that a host can support is:

(host cores minus 2)/(cores needed by that VM)

If there are a set of VMs needed per student/team, the number of students/teams calculation is:

(host cores minus 2)/(total cores needed by the set of VMs)

One takes the number of students in the course and divides by the number of similar hosts to determine how many hosts to allocate to the course.

These calculations presume that students could not or would not manage their usage by shutting down their VMs when not in use, so we must always allow for all VMs running and consuming host resources at the same time.

It has long been presumed that most VMs used by students lie idle most of the time, although as of December 2020 there are no statistics for this presumption. If that is the case, then host resources are being wasted on VMs that are doing essentially nothing. One can guesstimate that any given student interacts with the guest OS via the keyboard or mouse ("input activity") for an average of two hours per day, and maybe only on the day of or the day before class is held or an assignment is due. Furthermore, most VMs are not used for performing long-running computations, providing a service to other VMs, or serving as targets for penetration testing.

Let's say that a student has two VMs assigned to her, but only uses (interacts with) each for an average of two hours per weekday, for a total of 20 hours/wk. A quarter is about 10 weeks long, comprising of 10 wks * 7 days/wk * 24 hrs/day, or 1680 hours. The student's use is 20 hrs/wk * 10 wks = 200 hrs, or 200/1680 = 12% of the time. If we could recover the host resources consumed by her two VMs, the host would be available for other use 88% of the time.

Of course, it isn't as simple as that. Students probably don't use the VMs much for any course in the first 2-3 weeks, and tend to use them more in the week or so prior to an assignment deadline. Also, there may be times when the instructor wants all students in his course to use their VMs at the same time, which could be a problem if at that time, the host resources are committed to occasional use by students in other courses. Consequently, being able to recover host resources used by a VM isn't necessarily a panacea, but it can work a high percentage of the time.

Input Activity Detection

The problem with recovering host resources for a VM is knowing when the VM's host resources can be recovered. The hardest part is knowing if the student is actively interacting with the guest OS, by entering keystrokes on a console, terminal or desktop session, and/or moving the mouse, clicking mouse buttons or moving the mouse scroll wheel. This is user "input activity" which gets detected by the guest operating system as a user interacts with a terminal session or desktop.

Of course, there are other uses for computing that do not require interaction ("non-interactional uses"), such as long-running computations, providing services to client computers, and penetration testing. One may not want to free the VM's host resources in these cases despite the lack of interaction. There needs to be a mechanism to to "opt out" of freeing up the VM's host resources.

In 2020, after much investigation, googling and trial and error, it was determined that input activity for Windows and Linux distributions could be reliably detected. For Windows, it involves looking at a change in the IO reads of the csrss executable; "IO reads" covers both keystroke and mouse activity. For Linux distributions, keyboard activity from a console or ssh terminal session can be detected by the script command, while desktops that rely on X Windows or Xorg can utilize the xinput command to detect keyboard and mouse activity.

We want to place information about input activity, specifically, when it happened, in a file that is accessible to the host, not just inside file system of the guest operating system inside the VM. Doing so makes it easier to write a script to process all VMs for a course. This can be accomplished if each VM has a shared folder called guestlog mapped to the host user's home directory. A subdirectory called guestlog/vm_name, where "vm_name" is the name of the VM, is created at VM creation time. That shared folder is defined to be automounted in read/write mode by the guest OS. The definition of the shared folder and its mapped directory is created when the VM is created.

Here is more detail on how this was designed and implemented on each major OS:

  1. Windows

    A scheduled task is started at boot time. Its function is to get a relevant session id of the csrss.exe process that relates to an RDP-Tcp session or a Console session. That session id is used to query process information (viz., IO reads) to set the baseline for the current number of IO reads done, then periodically check the current IO reads to see if it has changed. If it has, there is input activity; if not, periodically rewrite an output file with the last date and time that input activity has been detected.

    This functionality is in the getidleinfo.au3 AutoIT Script file.

    For more details, see Keyboard and Mouse Input Activity Detection.

  2. Linux

    Three login cases must be handled for Linux distributions:

    Console and ssh sessions are terminal sessions that receive keystrokes. The script command can record those keystrokes and their timings. However, we don't care what the keystrokes are, so we can discard them to avoid recording plaintext passwords. The timing information is useful, as it co-occurs with the keystrokes and is normally intended for replay of the script (together with the keystrokes). If there is any timing info after a user logs in to a console or ssh session, there is terminal keystroke activity.

    The only issue is starting this universally after login. That can be handled by scripts placed in /etc/profile.d — one for ssh and one for console, specifically coded (by testing environment variables such as TERM and SSH_CONNECTION) to only allow those types of logins and no other.

    If the script command is running in a console session, startx can't detect the console and fails to start. To bypass this, in the /etc/bash.bashrc file, set the environment variable CONSOLE_OPT_OUT to the name of the user to be able to use startx. For example, for user "itadmin":

    export CONSOLE_OPT_OUT=itadmin
    

    Another means of bypassing the recording of timing information in a console is to use virtual terminal 6. That way, if something goes wrong with getidleinfo, there is a way to avoid having the script command run at login, because if something fails in /etc/profile.d/record_console_timing.sh, the login session is immediately exited.

    The getidleinfo script (in labadmin@cssgate:scripts) will check for requirements, install everything, create logon scripts called /etc/profile.d/record_ssh_timings.sh and /etc/profile.d/record_console_timings.sh, and set up a cron job to capture ssh or console timing information.

    It shouldn't always record console activity; it should ignore it when:

    Desktop input activity is handled by xinput. xinput needs the device number of the devices that relate to keyboard and mouse activity, and specifically for VirtualBox keyboard and mouse pointers, as well as the PS/2 mouse device. The device name to number mapping can be found by parsing the output of:

      xinput --list
    

    Then, by using the device number after the --test argument of xinput (one invocation per device), one can see keyboard and mouse activity.

    A Perl script called getidleinfo is placed in a Linux guest OS after installation, in the /usr/local/bin directory. getidleinfo has several options that can be used to install (-i), start (-s), or detect desktop events (-x).

    The getidleinfo installation option is run to prepare the VM for Linux input activity detection:

      getidleinfo -i
    

    With the exception of the event collection forked process, the other backgrounded processes are controlled by the timeout command. That is, those processes will die when the timeout period expires. The reason to do that is to allow the script to limit the size of the files so that the events and the idleinfo.out file of event timestamps don't grow indefinitely.

    For more details, see Keyboard and Mouse Input Activity Detection.

Freeing VM Resources on Host

If all we cared about is input activity detection on a VM, then one could write a script that checked for no input activity for certain period of time, and if there was none, to take some action on it that reduces the VM resources used on that host.

What action on a VM would reduce its resource use?

The safest action is to save the state of the VM, also known as hibernating the VM, like a laptop computer does to its OS when the lid is closed, to conserve battery power. A laptop computer basically saves the state of the RAM and the current CPU state to disk, so that it can be restored when the power is restored. The VirtualBox hypervisor provides a command (controlvm vm_name savestate) to save the state of a running VM. This would mean that all the VM user would need to do is start the VM again if something saved its state, and nothing would be lost, for the vast majority of cases. That's what makes it "safe".

The next best thing to saving the state of a VM is gracefully shutting down the guest OS. This allows buffers to be written to disk before the OS shuts down, preserving the integrity of the file system. In some cases, applications can detect the impending shutdown and do their own cleanup prior to shutdown. However, we really want this to happen as soon as possible, so we would likely force all applications to close and then shutdown and poweroff. If we wait until the user responds and handles a proper shutdown or saves her files, then it is not useful for freeing resources without user intervention.

For VirtualBox, one can shutdown the guest OS via the controlvm vm_name acpipowerbutton command, which requires the guest OS to be one that responds to ACPI commands; most modern OSes do. However, it is not an immediate command and doesn't always appear to work (although it could be waiting for a grace period before shutting down), so one can also issue a shutdown command via guestcontrol vm_name run ..., which goes directly to the OS. The problem with using guestcontrol is that it requires a valid privileged user and its password, which the student could have changed from the defaults provided, so it may fail.

Based on experience, it appears that an application failing to close combined with a lack of valid privileged credentials means that the VM resource will never be freed. In the ssh_vbm.log file for that user, there will be multiple indications that the acpipowerbutton action was taken, but the VM remains up. Consequently, the logic was changed to look at the last two consecutive entries in ssh_vbm.log for freeing the VM, for the VM of interest. If they are present and increasing in input idle activity, we attempt controlvm vm_name poweroff.

Put another way, on a shutdown action, we avoid a poweroff for as long as possible, after exhausting all other possibilities, but we do want to reclaim the VM resource. Hopefully, this strategy minimizes loss of data or data corruption.

Finally, one can simply "pull the plug" on the computer, which is the VirtualBox controlvm vm_name poweroff command. This could lead to file system corruption, but it is fairly rare given normal student OS use.

Each of the above "actions" will free up the VM's host cores and host RAM usage. That doesn't mean that additional resources aren't consumed. Saving the state will consume more disk space because RAM needs to be saved as well; that extra space will be released when the VM is started again. Shutting down can consume more disk space as well, but not much more. Powering off doesn't normally change the disk space.

Putting it All Together

We want to free up the host resources used by a VM We have a way of detecting the last time input activity was done. We know what to do to the VM to recover the host resources. All that remains is implementing the script itself.

A script called /root/scripts/free_vm_resources was created that does just that, but also allows for students or SET lab staff to opt-out of freeing VM resources for VMs or change the action taken to free the VM resources.

/root/scripts/free_vm_resources accepts arguments to free resources for either all VMs in a course, all VMs for in that course for a particular user, all VMs of the same name for the course, or a specific VM for a specific user in that course.

The way this is used is, on a quarterly basis, a course-specific script would be placed in a cron job that executes every 15 minutes (or other time amount). For example, /etc/cron.d/free_vm_resources_tinfo457:

# Check every 15 minutes
*/15 * * * * root /root/scripts/free_vm_resources tinfo457

This can be done easily by this command (which doesn't check or free anything, just creates the crontab):

/root/scripts/free_vm_resources -s tinfo457

and to remove it:

/root/scripts/free_vm_resources -r tinfo457

Except for the -s or -r options, the script will check to see if that course's VM resources can be freed. If so, it could free them, depending on what the opt-out information says to do.

free_vm_resources accepts another option, -c. That means that we only want to check the conditions, to see if they would apply and cause the action to be done; a "dry-run", but one that provides details in the log file (which is /root/free_vm_resources.course.log, where "course" is the course number).

When free_vm_resources does perform an action, besides logging information to its log file, it also adds an entry to $HOME/ssh_vbm.log, with the user @_free_vm_resources and some statistical info. The hope is that some time in the future, that ssh_vbm.log file can be processed for that information and when the VMs were started, to form the basis for mapping when the host is generally available. If that information was presented to the users, they would know when they have a good chance of getting good performance for their VMs vs. during a busy time.

Barring any provided opt-out information, the default condition to check is for VM guest OS's input activity to be idle for more than 60 minutes. If it is, the default action is to save the state of that VM.

Opting Out

We know that for non-interactive VMs, we should not always save their state to free up resources, as that could affect their functionality. In addition, if an instructor needs the students to use their VMs during class time (which is often two hours, much longer than the default input idle activity limit), we may want to not free up the VM resources during that time. Or they may be other unanticipated scenarios where freeing up VM resources should not be done.

We don't want the default to be "do nothing" or "ignore", because then we are back to the current state of affairs. The default should be to save the state of the VM based on some input idle activity criteria, and the user or SET Lab staff can override that by opting out.

This could be implemented by a simple file existence test -- if an opt_out file exists in a VM-specific directory ($HOME/guestlog/vm_name), then that means the VM should be allowed to continue running. However, that isn't very flexible -- it doesn't allow for other conditions or actions that would free VM resources.

For the user documentation (vs. this design and implementation document), see Opting Out of Freeing a VM's Host Resources. That document provides the syntax that the user uses to create an opt_out specification file for a VM. The actual specification is a slightly-encoded version of the nicer user syntax, in the form:

condition:action

"Slightly-encoded" means that the original user syntax is converted to the above form, where spaces between terms, operators or actions are removed, any "op" value is converted to either ">" or "<", any "term2" value is converted to minutes, and the "then" keyword is replaced by a colon.

SET lab staff can create or remove an opt-out specification, either "globally" (for all VMs of a user/team account) or per-VM. Global specifications are placed in the $HOME/guestlog/opt_out file. The specification format is the same as listed above, but if the global file is present, its specification will override any VM-specific opt_out file that the student/team may have requested.

The /root/scripts/opt_out_fvr script provides or removes either the global or the per-VM specification. For example, if we don't want the student's/team's VMs' resources to be freed, this specification would do that:

null:ignore

and the way to do it is via this script (for example course "tinfo457" and user "srondeau"):

/root/scripts/opt_out_fvr -o -s 'null:ignore' tinfo457 srondeau

"srondeau" could be omitted from the above command to disallow freeing any VM resources for any student/team in the "tinfo457" course.

If we want to remove the global specification (using the above example):

/root/scripts/opt_out_fvr -o -r tinfo457 srondeau

and that will basically undo the global override specification, allowing any user-created or default per-VM specifications to be in effect again.

If we want to overwrite or create a per-VM specification, the -o option is NOT used, so the specification gets applied to each VM in the course, each VM owned by a student or team in the course, a specific VM for that student/team, or, by using the period to represent "any", each VM matching the provided VM name in the course. For example, let's gracefully shutdown all tinfo457 fedora VMs that are idle for more than 2 hours. The "2 hours" must be manually converted to minutes, since the specification is always provided in minutes:

/root/scripts/opt_out_fvr -s 'input_idle>120:shutdown' tinfo457 . fedora

Note that this is a per-VM specification, NOT a global one, so it will wipe out any existing opt-out specification that a student/team may have made for the "fedora" VM. This is the only way to set a different opt-out specification for each VM when there are multiple VMs per student/team. That could be useful in a course that has multiple VMs, one of which is the target of a penetration test — that VM should not be freed, For example, course "tinfo442" needs a "kali" VM and a "centos" VM, where the centos VM is the target (needs an unconditional "ignore" action) and the kali VM is the penetration tester VM (which is subject to the default opt-out condition/action):

/root/scripts/opt_out_fvr -s 'true:ignore' tinfo442 . centos

or, if sending that command from one host to a remote host (e.g., cn8), one needs the proper quoting to be used so the resulting command works on the remote host, as in:

ssh cn8 '/root/scripts/opt_out_fvr -s '\''true:ignore'\'' tinfo442 . centos'

A student/team cannot remove an opt-out setting they made, but they can revert it back to the default using the following (for the win10 VM example):

opt_out win10 default then default

Of course, given any choice for a default input idle time (e.g., 60 minutes), someone will want to change it. Before we even started making input activity detection and freeing VM resources the standard, one instructor asked for a longer input idle time. Consequently, a per-course default was created; if an opt_out.default file is found in the course directory (e.g., for tinfo457, it's /classroom/home/tinfo457), then its specification is what is used for the default, not one that is hard-coded in the free_vm_resources script.

Note that a default class opt-out specification is NOT the same as an override specification. Students/teams can still override the class default via a VM-specific opt-out specification, and SET Lab staff can override the class default or any VM specification, at a user level, with an opt-out specification in the user's home directory in the guestlog folder.

To support a class or course default, opt_out_fvr was enhanced to create (via the -d option) or remove (via the -r option) the default specification in the proper location.

How to Overcommit

Given that we are attempting to free VM resources, if the estimate of 12% of the time the host is utilized is correct, then we could commit nearly four times more VMs per host than we currently do! If students would use any of the available hours of the day to run their VMs, and perhaps could be informed of when their host is likely to have a low load on it, then that degree of overcommitment could work.

Realistically, and without any statistics, putting four times more than the current VM load on the host seems to be far too much; perhaps two times more than current loading, or double the load, would work okay.

Consequently, overcommitting in the presence of actively freeing VM resources might mean that one doubles the amount of cores that the host has to determine how many VMs to allocate on it.

Accessing VMs

One creates a VM such that someone can remotely access it. This is the initial concern of a user of the VCL, and it is documented in the SET Lab web pages, under How to Use the VCL: Quickstart Guide.

The description of how VM access is set up is scattered across the Compute Node Phase I implementation documentation. Here, we attempt to pull together and document all that is necessary to do so, including some topics that may not have had a home elsewhere and were therefore undocumented.

The main concept of the VCL and any cloud provider is that one can access a VM from anywhere on the internet. A VM is defined with its virtual hardware capabilities, and the VM creator could leave it at that and let the user install or run a guest operating system inside the VM, given a Live CD/DVD image or an installation image. Or the VM creator could pre-install the operating system and configure it for the needs of the end user, team, project, course or research, as has been done for over a decade with the VCL.

The VCL Information File

The essence of a VM is what is captured in the .vcl file. That file is created every time the compute node or "host" script create_vb_vm_for_user is run. It is also done when the create_batch script is run, since it calls create_vb_vm_for_user for each course/class/project and user defined. That .vcl file is created in the host home directory of the user or team account.

Multiple VMs per user are common in many courses. Often, the VMs will be created such that a virtual network can be easily established. Virtual networks are either predefined for course needs or defined by the user once they have access to the VMs. This need for simple networking is facilitated by placing all of the VMs used in a particular course/class/project for a specific user or team on the same host. Consequently, the VM information of each VM belonging to a user/team on a host is appended to that user's or team's .vcl file when the VM is created.

Normally, only one .vcl file is needed per user, but sometimes a user takes more than one course that uses the VCL. The VM creation process does not take this into account — there is no current (February 2021) catalog of courses that require the VCL. Instead, each user's set of VMs for a course are created independently of that user's VM needs for other courses. There is no current (February 2021) information in the .vcl file that indicates which course uses what VMs, and there probably should be in the future. For now, that means that the VM creator must be aware that a student might be involved in more than one course using the VCL, and then append the host-dependent .vcl file information at gather_vb_vm_info time or manually.

Here is an example of a complex VCL information file:

20201222 13:08:59 vcl user testuser password 'ppppwwwww'
20201222 13:08:59 --------- Host and Guest Login Information ----------
20201222 13:08:59 vcl host cn8-vcl12 '140.142.71.32' (host ip)
20201222 13:08:59 --------- Information for ubuntu ----------
20201222 13:08:59 +-------- VRDP and LOGIN Information ----------
20201222 13:08:59 | vcl vm ubuntu (vrdp) '140.142.71.32:50838' -- use testuser password above
20201222 13:08:59 | vcl vm ubuntu (login) itadmin password 'xxxxxxxx'
20201222 13:08:59 +-------- PORT Information ----------
20201222 13:08:59 | vcl ubuntu cn8-vcl12 host port '140.142.71.32:52633' (ssh) for guest NIC 2 port 22/TCP
20201222 13:08:59 +------- End of PORT Information ----------
20201222 13:08:59 --------- End of Information for ubuntu ----------
20201222 13:18:42 --------- Information for fedora ----------
20201222 13:18:42 +-------- VRDP and LOGIN Information ----------
20201222 13:18:42 | vcl vm fedora (vrdp) '140.142.71.32:50159' -- use testuser password above
20201222 13:18:42 | vcl vm fedora (login) root password 'xxxxxxxx'
20201222 13:18:42 +-------- PORT Information ----------
20201222 13:18:42 | vcl fedora cn8-vcl12 host port '140.142.71.32:52973' (ssh) for guest NIC 2 port 22/TCP
20201222 13:18:42 +------- End of PORT Information ----------
20201222 13:18:42 --------- End of Information for fedora ----------
20201222 13:28:58 --------- Information for win10 ----------
20201222 13:28:58 +-------- VRDP and LOGIN Information ----------
20201222 13:28:58 | vcl vm win10 (vrdp) '140.142.71.32:50583' -- use testuser password above
20201222 13:28:58 | vcl vm win10 (login) administrator password 'xxxxxxxx'
20201222 13:28:58 +-------- PORT Information ----------
20201222 13:28:58 | vcl win10 cn8-vcl12 host port '140.142.71.32:54864' (ms-wbt-server) for guest NIC 2 port 3389/TCP
20201222 13:28:58 +------- End of PORT Information ----------
20201222 13:28:58 --------- End of Information for win10 ----------
20201222 16:06:49 vcl user testuser password 'wwwwpppp'
20201222 16:06:49 --------- Host and Guest Login Information ----------
20201222 16:06:49 vcl host cn5-vcl12 '140.142.71.64' (host ip)
20201222 16:06:49 --------- Information for centos8 ----------
20201222 16:06:49 +-------- VRDP and LOGIN Information ----------
20201222 16:06:49 | vcl vm centos8 (vrdp) '140.142.71.64:52902' -- use testuser password above
20201222 16:06:49 | vcl vm centos8 (login) itadmin password 'xxxxxxxx'
20201222 16:06:49 +-------- PORT Information ----------
20201222 16:06:49 | vcl centos8 cn5-vcl12 host port '140.142.71.64:53472' (ssh) for guest NIC 1 port 22/TCP
20201222 16:06:49 +------- End of PORT Information ----------
20201222 16:06:49 --------- End of Information for centos8 ----------

It contains VMs on two hosts, cn8-vcl12, on which testuser has four VMs, and cn5-vcl12, on which testuser has one VM.

Before we descend into the quagmire of complex VCL information files, let's step back to what the information is in the .vcl file. The order of VCL information is as follows:

  1. VCL user information, consisting of a user name and password
  2. Host and guest login information:
    1. host name and IP address
    2. one or more collections of information per VM
  3. optional additional VCL user information, followed by host and guest login information

The first line of the VCL information file is the vcl user line, as identified by that string. It is the host user name and password, which are the credentials that allow access to the host, whose IP address is provided later in the file.

The credentials together with the host IP address provides ssh access to the host, and commands issued via ssh provide the means of managing and accessing the VM. This is done for a Windows or MacOS user by manage_vc, but must be done manually by a Linux user.

Host-based ssh commands use the information immediately following the Host and Guest Login Information line. The next line contains the host name and the host IP address, but only the host IP address is needed. The host name is not fully-qualified and is for reference only, so instructors and SET Lab staff can easily determine what host is being used vs. looking up the IP address.

Following the host information is a collection of information for each virtual machine. The name of the VM is found after Information for string, and the end of that VM's information is denoted by the line containing End of Information for that VM name. In other words, all of the information between those two lines pertain to the named VM.

The VM-specific information collection varies per VM definition, but can include:

The .vcl file is placed in the user or team account's home directory on cssgate, since cssgate is the gateway to SET Lab information and everyone has a login account there. When there is more than one set of VMs for a user, there will be more than one vcl user line in the user's cssgate .vcl file. That line terminates the information associated with the previous host and starts information for the next host. In essence, one is appending the contents of the second (or n-th) .vcl file to the previous contents, and is what an option of gather_vb_vm_info can do from a VCL host.

Using the VCL Information

Background

While the How to Use the VCL: Quickstart Guide is the user's guide to the VCL, its audience is different from this document's audience — which provides more details and inner workings.

The .vcl file is the only source of VM information from the user's perspective, and hence is the only thing needed to inform the user as to how to access his/her VMs. In the first year of the VCL's existence, it was the only way to gain access, and it was painful for predominantly-Windows users to use, in part due to the cryptic but hard-to-guess password of the "vcl user" information, but also for the configuration of the Microsoft remote desktop client (the mstsc.exe command).

If the VCL was to survive, the students would need something easier to use. In addition, if the VCL would become popular, it would be very difficult and time-consuming for SET Lab staff to manage hundreds of student VMs across several courses.

For example, if the hosts experienced a power failure outside of normal office hours, any running VMs would also fail, and it would take manual intervention to start them again. To handle power outages, the hosts are set up to restart and thus boot the host operating system after power is restored, but that doesn't start the VMs after the host OS is up. At the time, VirtualBox didn't have the capability of starting a VM, and neither did SET Lab staff via scripts and cron jobs.

We also were aware of courses that may require the students to be in full control of their VMs, much like removable hard drives provided for coursework in previous years. Full control is important for installing an OS or configuring it, installing and configuring apps, and also extends to changing VM definitions (e.g., virtual RAM size and networking methods). If one can control the machine, the operating system, and the network, one can do essentially anything related to computing.

Hence, a decision was made to let the users control or "manage" their own set of VMs, and manage_vc was born.

Manually Managing VMs Using SSH

A user wishing to directly manage their VM sends ssh commands and any arguments to the host using the "vcl user" credentials and the host IP address. If the command requires a VM name, that name is pulled from the VCL information file.

For example, using the sample VCL information file , if a user wants start the "fedora" VM associated with the "testuser" account:

ssh testuser@140.142.71.31 start fedora

and manually enter the password in the VCL information file, or

plink -pw ppppwwww testuser@140.142.71.31 start fedora

Issuing ssh commands to the host is the only means a user has to manage his/her VMs (it underlies what manage_vc does), but "connecting to" or interacting with a VM is the most common activity that a user does. Connecting to/interacting with a VM using VCL information is described later.

All host-based ssh commands are run through the /usr/local/bin/ssh_vbm script, which allows only certain VirtualBox subcommands or custom commands — those written by SET Lab staff — to run. Any commands issued by the user are "sanitized" to prevent users from trying to subvert the system.

Connecting to VMs: Overview

Depending on the connection/interaction method chosen, the user will see either a virtual desktop, a command console or a terminal session of the VM. The user can also send keystrokes and mouse movements/clicks to the VM.

Connections can be done from a user's client computer via:

The user's client program must support the protocol of the service desired.

The manage_vc Facility

As of October 2022, the manage_vc facility resides in srondeau's UWTACOMA home directory, in H:\scripts\python\manage_vc, as manage_vc.py. This replaces the older AutoIT Script file called H:\scripts\autoit\manage_vc\anage_vc.au3, which only ran under Windows. Since the VCL was created in 2009, the student population shifted from using primarily Windows personal laptops or home computers to using a mixture of Windows and Mac OS computers, with rare cases of Linux. The long-desired cross-platform version of manage_vc with one source code file was finally implemented in Summer 2022, once it was realized that one could use Python's tkinter package to allow cross-platform windowing.

Retrieving the VCL File

manage_vc has to get the VCL information file from cssgate.insttech.washington.edu so it can read its contents locally. In other words, it has to remotely copy it from cssgate to the local computer.

For Windows, the pscp command from the PuTTY package is used, since Windows versions prior to Windows 10 in 2019 did not have any natively-supported scp/ssh commands. More recent versions of pscp have required an additional option to be provided: -P 22 — to specify the destination port.

Unfortunately, pscp does not have an option to ignore host key checking such as ssh's option -o StrictHostKeyChecking=no, so some special code has to answer a normally-interactive question about accepting the host key.

Gaining remote access to cssgate requires login credentials, and one must use the UW-ubiquitous "UW Net IDs" (or "uwnetid") because that cssgate requires. Login accounts are created when we first detect that there is a new student enrolled in one of the SET courses; first they are created as an INSTTECH domain user account on a Windows server, and then on cssgate. Both Windows and cssgate use Kerberos credentials (UW Net ID with the user's MyUW/Canvas password) to login.

A home/laptop user most likely will not have defined a local login account using the UW Net ID, so the user will need to specify his/her UW Net ID to manage_vc. manage_vc prompts for the user name, which is normally just the UW Net ID. As team accounts were allowed to be used for shared VMs, support for team accounts was added, allowing a "user name" of this form: uwnetid teamacct to be specified. That indicates to manage_vc to use the uwnetid as the credential source, but instead of the VCL information on the cssgate uwnetid account, use the team account's .vcl file.

To support SET Lab staff, if uwnetid is "root", then cssgate's root account will be used to access any user/team account's home directory. That allows the transfer of any .vcl file to whoever knows the cssgate root account password. For example, root srondeau would get UW Net ID srondeau's .vcl file.

Overriding Some manage_vc Information

Eventually, support was included for an INI-style file (manage_vc.ini) to define some user-specific information. manage_vc reads and uses the information in that file to override some information, such as the paths to pscp/plink executables, the server for the VCL information file, the user name, and the name of the VCL information file to retrieve. This allows users to assert more control over what manage_vc does, and is especially useful for defining a UW Net ID for the "user" line.

manage_vc also needs to issue commands to the host, so the Windows PuTTY plink command is used. plink also evolved over the years to require an additional option: -no-antispoof — to remove additional prompts for anti-spoofing mechanisms PuTTY's author deemed necessary.

Unlike scp/ssh, pscp and plink allow passing in the password to the remote login account, which removes the need to generate public keys. For copying the .vcl file, we prompt for the password, after which the file is stored on the local computer, usually in the user's profile.

For scp/ssh, one must use the sshpass command to send passwords to ssh or scp. This is commonly used in the Mac version. The Windows version includes a default plink/pscp, so sshpass is not strictly necessary, although it would be a problem (bug) if the user directed us to use ssh/scp via the manage_vc.ini file.

Parsing the VCL File Information to Populate manage_vc's GUI

Once the .vcl file is local, the first task of manage_vc is to parse it from its somewhat human-readable format into something manage_vc can use.

Perhaps the .vcl file should have been converted to Yet Another Markup Language (YAML) or some other Extensible Markup Language (XML) format, but it wasn't (maybe YAML wasn't written/popular or easily parseable by AutoIT). There is a host-based Perl script called dump_vcl_file which should do that (might not be up to date), but it was written long after manage_vc was first released.

  1. The "vcl user" information is parsed and saved, as is the host IP address. That information is used "under the covers" to issue VM management commands to the host via buttons mentioned later.
  2. Each VM's information is parsed, and a tab panel is created for the VM, with the name of the VM on the tab.
  3. Other VM information is parsed and placed in the tab panel.
  4. Buttons to start and power-off the VM are added.

    Limitations on the button size were circumvented by using tool tips to show the full VM name.

  5. The "Connect" button's functionality is defined.

    Its purpose is to connect to the VM's display via RDP. Its functionality was changed in 2020 to allow for better handling of guest OS RDP information, as "guest RDP" differs from "virtual RDP" information. In addition, there is often a need to edit guest RDP information, since the user may add guest OS user accounts, change passwords, etc. As much as can be automated is for the user, parsing NAT port mapping information if present and using it to define the RDP client connection.

    Virtual RDP uses only information which is present in the VCL information file, and therefore doesn't need and should not nbe editable, thereby lessening the chance that other users can easily connect to a particular user's VM.

  6. Buttons to reveal and paste the guest login account's password are added, and a label is created to display what that login account name is; usually its either "administrator", "root" or "itadmin".
  7. Other useful specific buttons and/or dropdown boxes are added.

    This includes the do-anything "Do cmd:" facility, which uses default commands present in the drop-down box to the right of the "Do cmd:" button, or commands that a user types in, to execute on the host. This is the manage_vc equivalent of being able to manually issue ssh commands.

Sending Commands via manage_vc

manage_vc manages the VMs via sending ssh/plink commands to the host, using the VCL user information for login credentials and the host IP address. On the host end, ssh_vbm receives those commands and attempts to execute them.

On Windows, to perform the "Connect" button functionality, a Microsoft RDP file is created using some information from the VCL information file, the VCL user password is encrypted, and then that RDP file defines the connection information that Microsoft's mstsc command uses to gain access to the VM's virtual display, thus allowing the user to interact with the VM. The Microsoft RDP client works for connecting to the display any VM, as the default virtual RDP service is independent of the guest OS. Other RDP clients that support more recent versions of the RDP protocol may also work.

Because there is no known way for a Mac client to pass Windows credentials like cryptRDP5 does, there is no way to set up the credentials in advance such that a simple click of the manage_vc "Connect" button will open the remote desktop connection. Instead, there is a complicated set of instructions that are told to the user via an information dialog box about how to import most of the RDP info (except credentials) into Microsoft Remote Desktop Client. Entering credentials are an additional, one-time-per-VM step, ultimately via the Mac's KeyChain Access. This is the only place it is not user friendly and differs significantly from the Windows version.

Packaging manage_vc

manage_vc is packaged via the make_zip.cmd Windows batch file, which is in the same folder where the master copy of manage_vc.py resides.

A zip file is created for distribution — one for Windows called manage_vc_win.zip and one for Mac OS called manage_vc_mac.zip. These files are placed on the csslab web site that hosts the UsingVCLQS.html file, which references the .zip files and explains how to install and use VMs in the VCL.

The .zip files are created by running (in the folder above):

make_zip win
make_zip mac

If you want to also update the web site and have it mapped as the L: drive, as srondeau does:

make_zip win update
make_zip mac update

Note that the Windows part of make_zip uses pyInstaller to create an executable called manage_vc.exe with a python interpreter and all required files, resulting in a 10MB+ .zip file. Once unzipped, the executable is ready to use.

The Mac part of make_zip simply bundles the essential files and an install script into a small .zip file. The Mac version's mvc_install script helps the user install Python and its packages, gets Microsoft Remote Desktop Client from the Apple Store, installs developer tools to allow compilation of the sshpass command (critical to supplying passwords to ssh), and creates a script called manage_vc and puts it in /usr/local/bin.

To package it for Windows, run make_zip win, which will:

  1. compile the script into a .exe file and adds an icon
  2. put the executable in a subfolder for easier archiving and unarchiving

    That subfolder (called manage_vc) includes the executable manage_vc.exe, the default manage_vc.ini file, the cryptRDP5.exe for encrypting Windows passwords for mstsc's use, and the latest versions of the plink and pscp executables (currently, in October 2022, version 0.74).

  3. archive the manage_vc folder using 7zip

The resulting manage_vc_win.zip file is placed on the SET Lab web pages, as follows:

net use L: \\itfiles4.d.insttech.washington.edu\lab$ /user:insttech\css_setup *
copy /y manage_vc.zip L:\www\Support\HowtoUse

which replaces the current copy there. The quickstart web page is set up to not be cached, so the latest version of the .zip file is always available. Alternatively, you can use:

make_zip win update

For the Mac version of the .zip file, see above.

There may be more than one .zip file whose name starts with "manage_vc_win" or "manage_vc_mac" in that directory. If there is a major change to manage_vc, especially with respect to the VCL information file format, we may keep the older version around in case someone still needs to access their old VCL information. These additional .zip files contain a version number in the file name, while the most recent one does not.

User Requirements

Network Requirements

If a VM's host is on a UW-private (viz., 10.0.0.0/8) or semi-private (viz., 172.28.0.0/16 or 172.22.71.0/24) IP subnet, users who are not on a UW campus will not be able to directly access the VMs. Instead, the user must have installed, started and authenticated via Husky OnNet VPN, after which they should be able to access the VMs.

If not using the Husky OnNet VPN, a user's device may be blocked from accessing either cssgate or the VM's host. That can occur due to host-based IP firewalls which have an extensive list of subnets which are blocked, based on at least one IP address in that subnet which attacked cssgate in the past. The user must then contact SET Lab staff and supply the IP v4 address shown by navigating to https://whatsmyip.org, and reporting that to SET Lab staff, who will attempt to unblock the IP address and/or subnet.

Note that the IP address reported by the operating system that the user is running is not adequate, since their device is often behind a home router that issues dynamic IP addresses on its own private IP subnet. That private address is NOT what should be investigated for blocking. Rather, what is desired is the IP address provided by the internet service provider (ISP), which is exposed by browsing to the web page mentioned above.

User Platform Requirements

End users can use either Windows, Mac OS or Linux to access the VCL. As noted, Windows and Mac OS users can use manage_vc, which either bundles Python and required packages with it (for Windows version), or on Mac OS, uses an install script to direct the user to install required software.

Mac and Linux users may require a Microsoft Virtual Desktop License (VDA) if they were assigned a Windows VM. It is the responsibility of SET Lab staff to have enough licenses available for all non-Windows users.

The number of non-Windows users could be determined prior to use by a survey, but getting anyone to take a survey is difficult and typically provides only a 10% response rate. There is code in ssh_vbm that can be enabled to disallow starting a Windows VM unless the user has taken the survey, but periodic distribution of the survey results to all hosts has not been finalized to date (February 2021), so more work needs to be done to enforce this.

Alternatively, one can guess at how many non-Windows users there are that will need access to a Windows VM. If a course doesn't use any Windows VMs, then this is a moot point. For those courses/projects that do require a Windows VM, we can use statistics from previous quarters to guesstimate the percentage of non-Windows users there are, or we can simply choose a number well over the best guess, and then validate as the quarter progresses.

The Microsoft Virtual Desktop License (VDA) ticket describes the general technique, and a script on each host called /root/scripts/count_access_to_windows_vms will detect any occurrence of Windows and non-Windows accesses to a Windows VM. If it finds any, it counts it (but only one if many occurrences are found for the same VM). If the script is run on all hosts during the quarter, one can see if there are enough VDA licenses to cover the non-Windows users.

Such VDA licenses last only for the number of months purchased, and are tied to the UW-Microsoft Enrollment for Education Solutions (EES) agreement, which expires on 1 June. The need for licenses will vary per quarter, so an effective strategy for purchasing the right number of licenses will save money.

As mentioned, a VDA license is the responsibility of SET Lab staff, not of the end user. A Mac/Linux user must install a remote desktop connection client and know how to configure it, and must become familiar with it to interact with the VMs and scp/ssh commands to manage them. There are detailed instructions in the quickstart web page about this.

Interacting with VMs

When a user wants to see the display of the remote VM and interact with it, these are the methods available for a VM which is running (or started), in order from easiest to hardest to use:

The choice can be up to the SET Lab staff VM creator or the user can configure the OS to provide services such as RDP or ssh. The user is responsible for client software that can access either the host or the guest OS services desired.

The speed and ease of interaction is based on how the VM is defined, the guest operating system installed in the VM, the resources available on the host, the speed and reliability of the network connection, and many other factors. SET Lab staff configure VMs based on course needs, and try to make using the VCL and interacting with the VMs a pleasant experience. Factors improving the experience involve: