SGX-LKL provides remote attestation capabilities that allow a remote client to ensure that an enclave has been deployed correctly and is running on a trustworthy Intel SGX hardware platform. In addition, SGX-LKL makes it possible to provide application configurations and secrets remotely.

On launch SGX-LKL automatically creates an enclave quote and starts a Protobuf-based RPC server that allows a remote client to retrieve the quote. The quote contains both the enclave report and attached report data. The enclave report contains the enclave measurement/identity (MRENCLAVE) and the signing identity (MRSIGNER) that are used to verify that the enclave contained the expected code/data pages at launch. The report data contains two SGX-LKL specific fields. First, it contains a nonce that can be specified when an SGX-LKL enclave is created. It can be used to verify that the received quote corresponds to the same enclave. Secondly, it contains the enclave's public Wireguard key used for establishing a secure channel/VPN with the enclave.

In order to verify that the quote corresponds to an actual Intel SGX enclave running on a system with up-to-date microcode/BIOS/platform software, SGX-LKL provides support for verification via the Intel Attestation Service (IAS). It supports 1) generating the IAS attestation report server-side when running sgx-lkl-run and retrieving it remotely for verification, or 2) generating a quote for a specified SPID and retrieving it remotely in order to invoke the IAS manually/on the client side.

To be able to use the IAS, you will have to register with Intel first. You can do this here. You will be assigned a Service Provider ID (SPID) as well as a subscription key. As part of the registration you also choose whether you require the verification of unlinkable or linkable quotes (see Signature Policy). The SPID, the subscription key, and the IAS signing CA certificate can then be used together to request an IAS attestation report and verify it.

SGX-LKL itself is able to request and parse an IAS attestation report. For this, the following sgx-lkl-run parameters exist.

SGXLKL_IAS_SPID: Specifies the Service Provider ID (SPID) required for communication with the Intel Attestation Service (IAS).
SGXLKL_IAS_QUOTE_TYPE: Specifies the quote type: '0' for unlinkable quotes (default), '1' for linkable quotes.
SGXLKL_IAS_SUBSCRIPTION_KEY: IAS subscription key.
SGXLKL_IAS_SERVER: IAS server to use (Default: test-as.sgx.trustedservices.intel.com:443).

The following shows a basic example of how to invoke sgx-lkl-run and the response returned by the IAS:

$sgx-lkl/apps/helloworld$ SGXLKL_VERBOSE=1 SGXLKL_TAP=sgxlkl_tap0 \
  SGXLKL_IAS_SPID=B2608D42F0EC26C73DE50ABF32E02417 \
  SGXLKL_IAS_SUBSCRIPTION_KEY=a6129f9e2987426120b5e54d4acfa2eb \
  ../../build/sgx-lkl-run ./sgxlkl-disk.img /app/helloworld
[...]
[    SGX-LKL   ] Received quote from launch enclave:
[    SGX-LKL   ]  MRENCLAVE: 3a24dfcbc0ce4c674bc808be599c4d58c2dd026311aabefb4dfe9eb5ecf5aaaf
[    SGX-LKL   ]  MRSIGNER:  6f0ee844222c89e67876b714fec7a662e6d9dc2fe06a3d3b36b2ac140959e992
[    SGX-LKL   ] Sending IAS request...
[    SGX-LKL   ] Intel Attestation Service Response:
[    SGX-LKL   ]  Quote status: GROUP_OUT_OF_DATE
[    SGX-LKL   ]  EPID group flags: 0x4
[    SGX-LKL   ]  TCB evaluation flags: 0x0f
[    SGX-LKL   ]  PSE evaluation flags: 0x00
[    SGX-LKL   ]  One or more advisories apply to the attested platform. Refer to https://security-center.intel.com for more information:
[    SGX-LKL   ]   INTEL-SA-00203,INTEL-SA-00106,INTEL-SA-00115,INTEL-SA-00135,INTEL-SA-00088,INTEL-SA-00161
[...]
Hello World!

In this example, the IAS returned a quote status GROUP_OUT_OF_DATE indicating that "EPID signature of the ISV enclave QUOTE has been verified correctly, but the TCB level of SGX platform is outdated" (see IAS API documentation V3).

Note that this output of sgx-lkl-run is generated by untrusted host code. It is useful for debugging, but you might not want to trust it when running on an untrusted system. It also doesn't involve verification of the enclave report itself, i.e. the enclave measurement.

As mentioned above, an SGX-LKL enclave incorporates a Protobuf-based attestation endpoint. It is possible to remotely retrieve and verify the IAS report using the sgx-lkl-ctl tool. In the following example, we launch an SGX-LKL enclave on one server and retrieve and verify the quote on another trusted machine.

On the server:

$sgx-lkl/apps/helloworld$ SGXLKL_VERBOSE=1 SGXLKL_TAP=sgxlkl_tap0 SGXLKL_REMOTE_CONFIG=1 \
  SGXLKL_REPORT_NONCE="$(od -An -vtu8 -N8 < /dev/urandom | tr -d ' ')" \
  SGXLKL_IAS_SPID=B2608D42F0EC26C73DE50ABF32E02417 \
  ../../build/sgx-lkl-run ./sgxlkl-disk.img /app/helloworld
[...]
[    SGX-LKL   ] Enclave report nonce: 7989292929894539561
[    SGX-LKL   ] Received quote from launch enclave:
[    SGX-LKL   ]  MRENCLAVE: 3a24dfcbc0ce4c674bc808be599c4d58c2dd026311aabefb4dfe9eb5ecf5aaaf
[    SGX-LKL   ]  MRSIGNER:  6f0ee844222c89e67876b714fec7a662e6d9dc2fe06a3d3b36b2ac140959e992
[    SGX-LKL   ] No IAS key file provided (via SGXLKL_IAS_KEY_FILE). Skipping IAS attestation...
[    SGX-LKL   ] Starting attestation server, listening on 10.0.1.1:56000...
[    SGX-LKL   ] Starting remote control server, listening on 10.0.2.1:56001...
[    SGX-LKL   ] Waiting for application run request...

SGXLKL_REPORT_NONCE can be used to specify a random nonce that will be attached to the enclave report and can be used to guarantee the freshness of the enclave report during verification. SGXLKL_REMOTE_CONFIG=1 is used to specify that the application configuration will be provided remotely by a trusted client. This will be discussed in more detail later on.

At this point, the enclave is running and we can request the quote remotely. Note that by default SGX-LKL listens on port 56000 for attestation requests as can be seen in the verbose output above. In this case, 10.0.1.1 is the IP of the in-enclave TAP network interface. In order to access it remotely, you might have to add a corresponding forwarding rule (see Networking support). In the following example, 192.168.10.1 is the public IP of the machine the enclave runs on.

On the client:

$ sgx-lkl-ctl attest \
  --server=192.168.10.1:56000 \
  --ias-spid=B2608D42F0EC26C73DE50ABF32E02417 \
  --ias-skey=a6129f9e2987426120b5e54d4acfa2eb \
  --ias-sign-ca-cert=./ias-sign-ca-cert.pem \
  --mrenclave=3a24dfcbc0ce4c674bc808be599c4d58c2dd026311aabefb4dfe9eb5ecf5aaaf \
  --mrsigner=6f0ee844222c89e67876b714fec7a662e6d9dc2fe06a3d3b36b2ac140959e992 \
  --nonce=1975754296801406209

[    SGX-LKL   ] Sending IAS request...
[    SGX-LKL   ] Intel Attestation Service Response:
[    SGX-LKL   ]  Quote status: OK
[    SGX-LKL   ] Quote measurements:
[    SGX-LKL   ]  MRENCLAVE: 3a24dfcbc0ce4c674bc808be599c4d58c2dd026311aabefb4dfe9eb5ecf5aaaf
[    SGX-LKL   ]  MRSIGNER:  6f0ee844222c89e67876b714fec7a662e6d9dc2fe06a3d3b36b2ac140959e992
[    SGX-LKL   ] Verification of quote and attestation report successful.
[    SGX-LKL   ] Enclave report data:
[    SGX-LKL   ]   Nonce: 1975754296801406209
[    SGX-LKL   ]   Public wireguard key: sGdsmrc5JBguBURpVVeMw33MMCnzcyYDH9IkgoEnyk4=

In order to request an IAS attestation report, sgx-lkl-ctl requires you to specify the SPID and your subscription key. It also expects a signing CA certificate that is used to verify the report against a signing CA certificate chain (from here) to ensure the report is genuine. In addition you can specify the expected MRENCLAVE and MRSIGNER values as well as an expected nonce. --strict can be used to make sgx-lkl-ctl attest fail for any enclave quote status other than OK. Without, GROUP_OUT_OF_DATE and CONFIGURATION_NEEDED are considered to be valid quote status results. sgx-lkl-ctl attest will exit with exit code 0 if the attestation has been successful, and with exit code 1 otherwise.

The following sequence diagram shows the basic operations involved in this attestation process. The process of launching an application once the attestation has been successful is described below.

As mentioned above, the attestation response also includes the public Wireguard key of the enclave, allowing a client to establish a secure connection. This is not only used to communicate securely with an application, but also to provide enclave secrets such as disk encryption keys and the application configuration itself. Similar to the attestation endpoint described above, SGX-LKL exposes a remote control endpoint on the Wireguard interface. For testing, it can be made accessible on the public interface by specifying SGXLKL_REMOTE_CMD_ETH0=1. This is NOT available when SGX-LKL is built in release mode in which case the remote control endpoint is always accessible over Wireguard only.

In release mode, an application configuration has to be provided remotely (as it includes sensitive disk encryption keys, application arguments, and environment variables that need to be protected). When not built in release mode, remote configuration can be enabled by specifying SGXLKL_REMOTE_CONFIG=1. If not specified, the application configuration has to be provided to sgx-lkl-run directly. The application configuration includes:

• Disk encryption keys and root hashes
• Additional Wireguard peers
• The path to the executable to run
• Application command line arguments
• Environment variables exposed to the application

For simplicity, in the following example we set SGXLKL_REMOTE_CMD_ETH0=1 to expose the remote control interface on the public interface of SGX-LKL.

On server:

$sgx-lkl/apps/helloworld$ SGXLKL_VERBOSE=1 SGXLKL_TAP=sgxlkl_tap0 SGXLKL_REMOTE_CONFIG=1 SGXLKL_REMOTE_CMD_ETH0=1 ../../build/sgx-lkl-run ./sgxlkl-disk.img 
[...]
[    SGX-LKL   ] Starting attestation server, listening on 10.0.1.1:56000...
[    SGX-LKL   ] Starting remote control server, listening on 10.0.1.1:56001...
[    SGX-LKL   ] Waiting for application run request...

At this point, SGX-LKL is ready to receive a request to run an application remotely. This request can be sent via sgx-lkl-ctl run. The application configuration is provided in a JSON-formatted file and is specified via --app.

On client

$sgx-lkl$ cat helloworld.conf
{
    "run": "/app/helloworld",
    "disk_config": [
        {
            "disk": "/"
        }
    ]
}

$sgx-lkl$ ./build/sgx-lkl-ctl run --server=192.168.10.1:56001 --app=./helloworld.conf
Connecting to 192.168.10.1:56001... done.
[    SGX-LKL   ] Request successful.

On server:

[...]
[    SGX-LKL   ] Waiting for application run request...
[    SGX-LKL   ] Calling application main
Hello World!

The following shows a more complex example of an application configuration including command line arguments, environment variables, a disk encryption key and root hash, and Wireguard peers. Note that that disk configurations are mapped to a disk image file via their mount point. / is a special case and refers to the root file systems.

{
  "run": "/usr/bin/redis-server",
  "args": [
    "--bind",
    "10.0.1.1"
  ],  
  "environment" : { 
    "MYVAR": "UNUSED",
    "MYVAR2": "UNUSED"
  },  
  "disk_config": [
    {   
      "disk": "/",
      "key": "9efac0f1082e345059f3d2bf0c5299731574f4357e7bb67ed4eaac7bfcdc0964cf77d39525f6c0cd8909fbfa5696a91512f1be3daa969b859c76611fe77d3bf1",
      "roothash": "507243152f04f7897e035ccafd6cc88e107cee33345011e989eca87c79d1ca59",
      "roothash_offset": 106954752,
      "readonly": false
    }   
  ],  
  "network_config": {
    "peers": [
      {   
        "key": "EemumcAZTe+ojwy64zE/IIxaq21I6F34lE1NSmkhGFk=",
        "endpoint": "192.168.10.2:56002",
        "allowedips": "10.0.2.2/32"
      }   
    ]   
  }
}

Note that the disk encryption key has to be provided as hex-encoded string and that the application configuration does not support passphrases. In this example, the disk encryption key, root hash, and root hash offset have been generated/determined by the sgx-lkl-disk tool:

$ ./tools/sgx-lkl-disk create --alpine="redis" --encrypt --key-file --keyfile-size=64 --verity --size=100M disk.img
$ xxd -p -c 512 disk.img.key
9efac0f1082e345059f3d2bf0c5299731574f4357e7bb67ed4eaac7bfcdc0964cf77d39525f6c0cd8909fbfa5696a91512f1be3daa969b859c76611fe77d3bf1
$ cat disk.img.roothash 
507243152f04f7897e035ccafd6cc88e107cee33345011e989eca87c79d1ca59
$ cat disk.img.hashoffset 
106954752

The remote control endpoint also allows a client to remotely add new Wireguard peers, e.g. additional enclaves or other trusted nodes that should be part of the same VPN, individually. This can be done via sgx-lkl-ctl addpeer as shown in the following example.

$sgx-lkl$ ./build/sgx-lkl-ctl addpeer --server=192.168.10.1:56001 --key=EemumcAZTe+ojwy64zE/IIxaq21I6F34lE1NSmkhGFk= --endpoint=192.168.10.2:56002 --allowedips=10.0.2.2/32

In the following, we look at a full attestation and remote launch example involving a trusted client and an untrusted server that hosts an SGX-LKL enclave. First, we create an encrypted and integrity-protected disk image containing the application, in this case redis, on the trusted client. This disk image is then transferred to the server (via scp or some other means). Then, sgx-lkl-run is launched on the server side which will create the enclave and a corresponding enclave quote. Then, the client retrieves and verifies the quote, establishes a secure channel with the enclave and instructs the SGX-LKL instance to run the application. In this example, the public client IP will be 192.168.10.2, and the public server IP will be 192.168.10.1. The client VPN (Wireguard) IP will be 10.0.2.2, and the VPN enclave IP will be 10.0.2.1.

SGX-LKL relies on Wireguard for secure communication. We therefore need to have installed Wireguard on the client. For this, follow the official instructions here. Next, we generate a new Wireguard key for the client and add a Wireguard network interface. We call the new interface wgsgx0 and assign the IP 10.0.2.2. As port for incoming Wireguard traffic we will use 56002 (which is also the default port used by SGX-LKL).

# Generate key
wg genkey | tee -a wgclient.priv | wg pubkey > wgclient.pub
# Add interface
sudo ip link add dev wgsgx0 type wireguard
# Set client IP, we will use 10.0.2.2
sudo ip address add dev wgsgx0 10.0.2.2/24
# Set private key and port
sudo wg set wgsgx0 listen-port 56002 private-key wgclient.priv
# Activate device
sudo ip link set up dev wgsgx0
# Add iptables rule to enable forwarding from and to the new interface
sudo iptables -A FORWARD -i wgsgx0 -j ACCEPT

On the server side we have to enable forwarding to allow traffic to and from the enclave. By default, SGX-LKL requires two port forwarding rules, one for the attestation service (TCP port 56000), and one for Wireguard (UDP port 56002). In the following, we assume a TAP device sgxlkl_tap0 has been set up as described in the networking setup instructions here.

# Enable packet forwarding
sudo sysctl -w net.ipv4.ip_forward=1
# Forward traffic to enclave attestation endpoint
sudo iptables -t nat -I PREROUTING -p tcp -d 192.168.10.1 --dport 56000 -j DNAT --to-destination 10.0.1.1:56000
# Forward traffic to enclave Wireguard endpoint
sudo iptables -t nat -I PREROUTING -p udp -d 192.168.10.1 --dport 56002 -j DNAT --to-destination 10.0.1.1:56002
# Forward outgoing traffic from enclave
sudo iptables -I FORWARD -m state -s 10.0.1.0/24 --state NEW,RELATED,ESTABLISHED -j ACCEPT

In this example, our goal is to setup redis to be run inside an SGX-LKL enclave. We can create an encrypted/integrity-protected disk image with a root file system that has redis-server installed using sgx-lkl-disk on the client:

$ sgx-lkl $ ./tools/sgx-lkl-disk create --alpine="redis" --encrypt --key-file --integrity --size=100M disk.img

At this point, the previously generated disk image should have been transferred to the server, e.g. via scp. We now run sgx-lkl-run to create an SGX-LKL enclave. We need to provide a number of arguments/environment variables. Besides the path the disk image file, we provide

• a randomly chosen 64-bit nonce enabling us to guarantee the freshness of the quote we will later verify on the client,
• an IAS SPID that is used by the quoting enclave to generate a quote that can later be verified by the Intel IAS, and a
• Wireguard peer configuration that corresponds to the client.

$ sgx-lkl $
  SGXLKL_VERBOSE=1 \
  SGXLKL_TAP=sgxlkl_tap0 \
  SGXLKL_REMOTE_CONFIG=1 \
  SGXLKL_REPORT_NONCE=10867864710722948371 \
  SGXLKL_IAS_SPID=B2608D42F0EC26C73DE50ABF32E02417 \
  SGXLKL_WG_PEERS='LW5fn7DG+0y9mefoLvHDGp8AqkHdDDYD4RQWJjstoAU=:10.0.2.2/32:192.168.10.2:56002' ./build/sgx-lkl-run ./disk.img

When run with SGXLKL_VERBOSE=1, you will see output like the following:

[    SGX-LKL   ] wg0 has public key 2bC2HX3NCn2aICXladL5Rcy7PJmLcvffW1ro7pYjw2o=
[    SGX-LKL   ] Enclave report nonce: 10867864710722948371
[    SGX-LKL   ] Received quote from launch enclave:
[    SGX-LKL   ]  MRENCLAVE: f2f35339626187c3fb6eed48daba0aa13b000dddd2d286562e88b89a7fc26ca1
[    SGX-LKL   ]  MRSIGNER:  6f0ee844222c89e67876b714fec7a662e6d9dc2fe06a3d3b36b2ac140959e992
[    SGX-LKL   ] No IAS key file provided (via SGXLKL_IAS_KEY_FILE). Skipping IAS attestation...
[    SGX-LKL   ] Starting attestation server, listening on 10.0.1.1:56000...
[    SGX-LKL   ] Starting remote control server, listening on 10.0.2.1:56001...

At this point, the enclave is ready to be remotely attested to. It also accepts run and addpeer requests on the Wireguard interface (10.0.2.1:56001). Note that when built in release mode, SGX-LKL will accept exactly one Wireguard peer via sgx-lkl-run (e.g. via SGXLKL_WG_PEERS). This guarantees that if the trusted client is able to talk to the enclave over the VPN, it is guaranteed that no other parties are part of the VPN.

At this point, we can attest to the enclave, ensuring that it contains a known trusted version of libsgxlk.so and that it is a genuine hardware-protected Intel SGX enclave.

$ sgx-lkl-ctl attest \
  --server=192.168.10.1:56000 \
  --ias-spid=B2608D42F0EC26C73DE50ABF32E02417 \
  --ias-skey=a6129f9e2987426120b5e54d4acfa2eb \
  --ias-sign-ca-cert=./ias-sign-ca-cert.pem \
  --mrenclave=f2f35339626187c3fb6eed48daba0aa13b000dddd2d286562e88b89a7fc26ca1 \
  --mrsigner=6f0ee844222c89e67876b714fec7a662e6d9dc2fe06a3d3b36b2ac140959e992 \
  --nonce=10867864710722948371

[    SGX-LKL   ] Sending IAS request...
[    SGX-LKL   ] Intel Attestation Service Response:
[    SGX-LKL   ]  Quote status: OK
[    SGX-LKL   ] Quote measurements:
[    SGX-LKL   ]  MRENCLAVE: f2f35339626187c3fb6eed48daba0aa13b000dddd2d286562e88b89a7fc26ca1
[    SGX-LKL   ]  MRSIGNER:  6f0ee844222c89e67876b714fec7a662e6d9dc2fe06a3d3b36b2ac140959e992
[    SGX-LKL   ] Verification of quote and attestation report successful.
[    SGX-LKL   ] Enclave report data:
[    SGX-LKL   ]   Nonce: 10867864710722948371
[    SGX-LKL   ]   Public wireguard key: 2bC2HX3NCn2aICXladL5Rcy7PJmLcvffW1ro7pYjw2o=

At this point, we can assume that the enclave is trustworthy and we are willing to establish a secure channel/VPN between the client and enclave.

In order to be able to send the application configuration securely, we need to the enclave as a Wireguard peer to the previously created Wireguard interface wgsgx0.

sudo wg set wgsgx0 peer 2bC2HX3NCn2aICXladL5Rcy7PJmLcvffW1ro7pYjw2o= allowed-ips 10.0.2.1/32 endpoint 192.168.10.1:56002.

Now, we can send a request to the enclave to launch redis through Wireguard. As part of the request, we also provide the disk encryption key, required by the SGX-LKL enclave to decrypt the disk containing the redis-server executable and verifying its integrity. We use a basic application configuration file for this. We specify the disk encryption key and instruct redis to bind to the in-enclave Wireguard interface which has the IP 10.0.2.1.

Hex representation of the disk encryption key

$ sgx-lkl $ xxd -p -c 512 disk.img.key
9efac0f1082e345059f3d2bf0c5299731574f4357e7bb67ed4eaac7bfcdc0964cf77d39525f6c0cd8909fbfa5696a91512f1be3daa969b859c76611fe77d3bf1

sgxlkl.redis.conf

{
  "run": "/usr/bin/redis-server",
  "args": [
    "--bind",
    "10.0.2.1"
  ],  
  "disk_config": [
    {   
      "disk": "/",
      "key": "9efac0f1082e345059f3d2bf0c5299731574f4357e7bb67ed4eaac7bfcdc0964cf77d39525f6c0cd8909fbfa5696a91512f1be3daa969b859c76611fe77d3bf1"
    }   
  ]   
}

We can now send the request via sgx-lkl-ctl.

$ sgx-lkl $ ./build/sgx-lkl-ctl --server=10.0.2.1:56001 run --app=sgxlkl.app.conf
Connecting to 10.0.2.1:56001... done.
[    SGX-LKL   ] Request successful.

At this point, we should be done. redis should be running and be ready to accept requests.

$ redis-cli -h 10.0.2.1 -p 6379 ping
PONG