SECTION 1: Cloud Administration

1. Login to your account with provided username & password

          ssh tiw35@

2. Unzip and setup your cloud environment variables

          source novarc

Within novarc, you can see EC2_ACCESS_KEY and EC2_SECRET_KEY which are credentials used to access cloud account. Also, you can find address of Cloud API service (EC2_URL and NOVA_URL).

3. Getting Cloud info. (we will use euca2ools as Cloud administration tool)


This command lists all of available images. aki-*, ari-* and ami-* represents kernel, ramdisk and disk image respectively. If your credentials are setup correctly, you should be able to access cloud API service and the above command should return a list of pre-uploaded images.


This command lists all of running instances on the cloud the belong to you. You can only see instances within your own project.

To have a global view of the cloud, access This webpage shows general cloud information. You need to access this website using Firefox installed on

4. Generate your key

          euca-add-keypair mykey > mykey.priv

          chmod 600 mykey.priv

Upon starting instance, a public key will be injected into the instance to enable remote login. mykey.priv is a private key used to login your cloud instances in future.

euca-describe-keypairs will list all of your keys

5. Enabling network access of instances

          euca-authorize -P icmp -t -1:-1 default

          euca-authorize -P tcp -p 22 default

Configuring network access rule for instance. Allowing icmp (ping) and tcp 22(ssh) traffic on instance.

euca-describe-group will list your group rules

6. Starting your own instances

          euca-run-instances ami-00000003 -k mykey -t siis.small --kernel aki-00000001 --ramdisk ari-00000002

You can use euca-describe-instances to check the status of your instance. The instance status will change from pending(cloud is loading images) to running(instance is running).

7. Checking your instance

          euca-get-console-output i-00000037

You can use euca-get-console-output to check the console output of your instance. i-00000037 is the instance ID which you can see from euca-describe-instances command. If anything goes wrong, for example, the kernel halts, you may see from the console output.

8. Accessing your instance


You will retrieve a public IP from IP pool. In this example, is allocated

          euca-associate-address -i i-00000037

Associate public IP with your instance

Your instance will be started within a private network which is not accessible from outside. In order to access your instance, you will need to first retrieve a public IP and assign it to your instance. Then your instance will become accessible.

9. Login your instance

          ssh -i mykey.priv ubuntu@

You will need to use the corresponding key that you start up the instance with to login to the instance. The username of the instance is ubuntu.

10. Shutdown your instance

          euca-terminate-instances i-00000037

SECTION 2. Auditing the Cloud

1. Start up an instance with -v option

          euca-run-instances ami-00000010 -k mykey -t siis.small --kernel aki-00000006 --ramdisk ari-00000007 -v

The -v option starts up the instance with Integrity Verification Proxy (IVP). In this case, we will use customized kernel(aki-00000006), ramdisk(ari-00000007) and disk_image(ami-00000010) to enable auditing purpose. This may take few minutes due to the measurement interfaces. Then repeat step 8 in Section 1 to assign the instance a public IP address. (Note: in the following experiments, you will login as "root" instead of "ubuntu".)

2. Registering criteria

Copy a sample criteria to your home directory

          cp /etc/nova/cfg/client.crt ./

Register the criteria, specifying the instance public IP address and the service port. (In this example, public IP of instance is

          euca-register-criteria -s -d -p 80 -c ./client.crt

Once the registration is done, a service port number is returned (for example, 9000). This port will be under control of IVP such that once the criteria is violated, connection to this port will be terminated.

Access the service using this new port.


3. Static measurement module

Static measurement modules measure load time properties of an instance. Customer can thus verify if their instances are loaded as expected.


Default criteria only enabled one static measurement module [Hash]. This module measures the kernel/ramdisk/disk_image of an instance. Customer can verify that the images loaded to the cloud are not modified by either cloud insiders or a network attacker.

Violation of [Hash] criteria indicates a modification of images and will return an error message upon criteria registration. If customer registers a criteria and fails, it is an indication that the instance is not loaded as he expected. Then such instance should not be used to host any sensitive services.

For this module, customer specifies the digest (SHA1) of kernel, ramdisk and disk image.


[Vminfo] module measures information related to the instance such as CPU, memory and kernel parameters.

For this module, customer specifies instance configurations including number of CPUs, size of memory (in KB) and kernel boot up parameter. If any of the instance configurations differ from customer's specification, the registration of critiria will fail.


[Policy] module analyzes SELinux policy of an instance. It is able to answer specific questions such as whether there is unconfined user types through analyzing the SELinux policy retrieved from the instance.

For this module, we demonstrated two sample questions. The first one "EXISTS_UNCONFINED" analyzes whether there are unconfined user in SELinux policy. The second one "EXISTS_UNPRIV_USER" analyzes if there are unprivileged users. Customer can specify their expected anwser after the questions. If the policy of instance doesn't meet customer's expectation, the criteria registration will fail.

4. Dynamic measurement module

Dynamic measurement modules measure runtime properties of an instance. Customer can thus verify that their instances are running correctly and as expected. During runtime, if customer's criteria is violated, connection to the service port will be disabled.


[Timing] module is a dummy test module. We firstly use this to show the ability for the Cloud Verifier framework to measure runtime properties of an instance. [Timing] module measures a specific instance kernel variable "printk_ratelimit_state.interval". Every change to this kernel variable will be captured by the framework. We use [Timing] module to demonstrate the ability for the framework to monitor sensitive kernel data structures of instance.

Enable [Timing] module by un-comment (deleting the ";" symbol) the corresponding lines in client.crt file. Register criteria and if registration succeeds, a new service port will be returned. (In this example, 9001 is returned.)

          euca-register-criteria -s -d -p 80 -c ./client.crt

Access the service using new service port, 9001 in this example.


Now, we are going to Trigger runtime events of instance to violate the criteria. (Login the instance as root)

          ssh -i mykey.priv root@

If you get the warning "REMOTE HOST IDENTIFICATION HAS CHANGED", it is because you are using the same public IP for different VMs. Delete file ~/.ssh/known_hosts and continue.

Within the instance, modify the printk_ratelimit value, which will change the value of instance kernel variable "printk_ratelimit_state.interval". (Original value is 5)

          echo 4 > /proc/sys/kernel/printk_ratelimit

This change will be captured by the framework. Since customer's criteria specifies no change in this variable, violation will cause the connection to the service port be cut.

Try again accessing the service


You will get "failed: Connection refused."


[Prima] module makes use of Integrity Measurement Architecture (IMA) within the instance kernel to monitor executable load.

Enable [Prima] module by un-comment the corresponding lines in client.crt file. Under [Prima], you will specify a white list of executable (their digest) that are allowed to run on your instance.

Current IMA measurement list can be found within the instance at /sys/kernel/security/ima/ascii_runtime_measurements. You can check out the digest of all the executable run after the instance boots up.

Test Case 1

In this case, you will select few web server (apache) modules to see how this affects the measurement list and criteria registration.

First, register criteria. In this test, we will focus on [Prima] module, so comment out [Timing] module. For [Prima] module, specify the "default" as the white list. The "default" white list contains measurements of base system and default modules loaded by apache web server.

          Under [Prima] section within criteria, specify "trusted" to be "default"

          euca-register-criteria -s -d -p 80 -c ./client.crt

Access the service using new service port, 9001 in this example.


To see what the default modules are, login to your VM

          ssh -i mykey.priv root@

List current loaded modules

          apachectl -t -D DUMP_MODULES

Now, try to load new module for the web server

          a2enmod Module_Name (you can select from ssl/rewrite/php5/cgi/dav_svn)

Restart web service

          service apache2 restart

Outside the instance, try again accessing the service


You will get "failed: Connection refused."

Above example simulates a situation that web server doesn't comply with customer or client's requirements (it may have loaded a vulnerable module). Now we modify the criteria file to let customer/client to change their requirements. Specify a new whitelist by adding "default" with whitelists of extra modules. There is a sample commented out in the file. You can modify accordingly.

Register criteria again

          euca-register-criteria -s -d -p 80 -c ./client.crt

Try different combinations of whitelists and load different modules. However, remember the IMA is irreversible. Once you load a module, you cannot delete it from the measurement list (undo it as if it has not happened). This is a basic rule of trusted computing.

All whitelists being used can be found under /etc/nova/cfg/imasets/.

SECTION 3. Accessing a Cloud Hosted Service

This section simulates how a client can use Cloud Verifier framework to securely access sensitive services such as on-line banking.

First, login with X11 forwarding enabled

          ssh tiw35@ -X

Running the browser

          cd browser


1. Verifying the Cloud Verifier

We created a Firefox extension to ease up client side operations. The status panel at the upper left corner of the window shows the most recent time of a successful verification of Cloud Verifier. At the beginning, it shows the Cloud Verifier was never verified. Click Verify button to start verifying Cloud Verifier. When status at the upper left corner turns into green, the verification succeeded. The Cloud Verifier will be verified every 10 seconds. Verification details can be found at shell output. Everytime client visits the sensitive service hosted by Cloud, he or she should check the status panel to make sure the Cloud Verifier is verified.

2. Accessing the Service

We used the our workshop homepage as an example. Enter the the URL and PORT number for the previously started instance into the browser:

You should be able to access the service normally.

3. On criteria violation

Once the criteria of customer is violated, (i.e. IMA detects an unauthorized code loading such as a virus), client should be informed to not using such compromised service anymore. To simulate this scenario, follow the instructions below.

Test Case 2

Above Test Case 1 shows how customer/client can audit the configuration of a cloud hosted web server. This example will show how customer/client can detect unauthorized executable loading (e.g. malware) and performs remediation.

Register criteria with Prima enabled. (use "all" whitelist under [Prima] section)

          euca-register-criteria -s -d -p 80 -c ./client.crt

If the criteria is successfully registered, you can proceed. (Otherwise, some of your activities within VM has fallen out of the whitelist. In this case, shutdown your VM (Step 10 in Section 1) and start a new one (Step 1 in Section 2). You need also register the criteria with Prima enabled.)

Trigger runtime events of instance to violate the criteria. (by running an executable that falls out of the white list)

          ssh -i mykey.priv root@

Within the instance, create a simple script as follows and name it "":

          # !/bin/bash
          echo "Simple Script"

make it executable

          chmod +x

run it


Check /sys/kernel/security/ima/ascii_runtime_measurements again, you will see your script is extended to the list.

Outside the instance, try again accessing the service

Your access should be denied due to the violation of criteria.

4. On Cloud Verifier anomaly

Cloud Verifier framework relies on Cloud Verifier (nova-verify service) to function correctly. Thus once Cloud Verifier malfunctions, it can no longer vouch for the correctness of the measurements of instance. Clients should not trust the service any more.

We simulate this scenario through booting the Cloud Verifier into a malicious state. (Changing some files on Cloud Verifier)

In this case, the Cloud Verifier Status at the upper left corner will turn into yellow again to show that the verification of Cloud Verifier failed. Client should not trust the website anymore.Meanwhile, you can check out at the console to see detailed information about verification.


1. "Cloud Verifier: Verifiable Auditing Service for IaaS Clouds" by Joshua Schiffman, Yuqiong Sun, Hayawardh Vijayakumar, and Trent Jaeger. To appear in IEEE 2013 First International Workshop on Cloud Security Auditing, 2013.

2. "Verifying System Integrity by Proxy" by Joshua Schiffman, Hayawardh Vijayakumar, and Trent Jaeger. In 5th International Conference on Trust and Trustworthy Computing, 2012, pp. 179-201.

3. "Network-based Root of Trust for Installation" by Joshua Schiffman, Thomas Moyer, Trent Jaeger, and Patrick McDaniel. IEEE Security & Privacy, Jan/Feb 2011.

4. "Seeding Clouds with Trust Anchors" by Joshua Schiffman, Thomas Moyer, Hayawardh Vijayakumar, Trent Jaeger, and Patrick McDaniel. In CCSW '10: Proceedings of the 2010 ACM Workshop on Cloud Computing Security, 2010.