DTrace Network Providers

The following is a design proposal for a collection of DTrace Networking Providers. These aim to provide networking observability and troubleshooting information for Solaris users. A prototype TCP provider was demonstrated at CEC 2006.

I originally posted this at http://www.opensolaris.org/os/community/dtrace/NetworkProvider/ . This project was later completed and merged into the Solaris kernel.

This document will list the probes that may be made available, their arguments, and examples of their proposed usage. Feedback is welcome, please post to dtrace-discuss. NOTE: this is a work in progress, and this network provider is not yet available! The purpose of this site is to flesh out ideas for a future network provider.

# dtrace -n 'tcp:::receive /args[2]->tcp_dport == 80/ {
        @pkts[args[1]->ip_daddr] = count();
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C

  192.168.1.8                                                       9
  fe80::214:4fff:fe3b:76c8                                         12
  192.168.1.51                                                     32
  10.1.70.16                                                       83
  192.168.7.3                                                     121
  192.168.101.101                                                 192

# dtrace -n 'tcp:::accept-established { @[args[2]->tcp_dport] = count(); }'
dtrace: description 'tcp:::accept-established' matched 1 probe
^C

       79                2
       22               14
       80              327

Author: Brendan Gregg and others, 16-Oct-2006

Contents

• Aims
• Providers
• Modules
• Probes
• Arguments
• Example Usage
• General
• Security Monitoring
• Performance Monitoring
• Solved Problems

Aims

The main aims of the network providers are:

• Support for common network protocols: TCP, UDP, IP.
• Everyday observability: connections/packets/bytes by address/port.
• Performance observability: throughputs, network latencies.
• Socket level tracing: associate events to PIDs.
• Further troubleshooting support: TCP flag analysis, header info.

Future enhancements and additions to the providers include:

• Other protocol support: ICMP, SCTP, ...
• Kernel code path latency probes.

Providers

The Network provider is a collection of several DTrace providers, such as ip and tcp (there is no monolithic net provider). They are:

Other providers such as icmp and igmp may also be added (and many more).

Modules

The module and function names are from the kernel code, and are usually ignored unless deeper code-based analysis is needed.

Probes

tcp:::accept-established
tcp:::accept-refused
tcp:::connect-request
tcp:::connect-established
tcp:::connect-refused
tcp:::state-bound
tcp:::state-close-wait
tcp:::state-closed
tcp:::state-closing
tcp:::state-established
tcp:::state-fin-wait1
tcp:::state-fin-wait2
tcp:::state-idle
tcp:::state-last-ack
tcp:::state-listen
tcp:::state-syn-received
tcp:::state-syn-sent
tcp:::state-time-wait
tcp:::drop-saturation
tcp:::data-send
tcp:::data-resend
tcp:::data-receive
tcp:::ack-send
tcp:::ack-receive
tcp:::fuse-send
tcp:::fuse-receive
tcp:::send
tcp:::receive

udp:::send
udp:::receive
sctp:::send
sctp:::receive
ip:::send
ip:::receive
arp:::send
arp:::receive
icmp:::send
icmp:::receive
gld:::send
gld:::receive

Send and receive probes fire whenever that layer sends or receives. They can be used for observability and measuring network latency, but their placement is not sufficient for kernel code path latency measurements.

TCP fuse probes fire for localhost traffic (TCP fusion), IP probes do not.

drop-saturation probes fire when packets are dropped due to load; other varients may include drop-error.

TCP Probes

To start with, send and receive fire for all traffic that is TCP; this includes the TCP handshake, TCP ACKs, and also for invalid packets such as those with corrupt flags. Since the TCP flags are available in the probe arguments, any particular types of TCP traffic can be matched using DTrace predicates.

Additional probes have been provided for convienence in addition to send and receive. This includes the data-* probes for matching on TCP packets containing data, connect-* probes for client connections, accept-* probes for servers accepting connections, state-* for RFC 793 state changes, and so on.

Outbound TCP Connections

For outbound connections, connect-request fires before the first SYN, connect-established fires after the SYN-ACK as been received, and the final ACK is being sent.

In summary:

An outbound connection to an open port will fire a connect-request followed by a connect-established. An outbound connection to a closed port will fire a connect-refused.

A possible connect-closed probe (normal close) has been dropped from this design due to difficulties encountered in the tcp code; the state-closed probe may serve a similar purpose.

Inbound TCP Connections

For inbound connections, accept-established fires after the final ACK was received in the three way handshake in response to an inbound SYN. accept-refused fires if a connection was refused, such as a SYN to a closed port.

In summary:

An inbound connection to an open port will fire an accept-established, and an inbound connection to a closed port will fire an accept-refused.

A possible accept-closed probe (normal close) has been dropped from the design due to difficulties encountered in the tcp code; the state-closed probe may serve a similar purpose.

Note: TCP events such as the three way handshake can also be traced using tcp:::receive and tcp:::send, and examination of the TCP flags (args[2]->tcp_flags).

Arguments

The following information is made available for the probes, where possbile and appropriate:

For IPv4 traffic, the ip probes provide header details in args[2] and args[3] is NULL; for IPv6 traffic, the header details are in args[3] and args[2] is NULL. For both traffic, summary header details are in args[1] - ipinfo_t.

Argument Details

The arguments contain the following members:

The info structures are new, and will be provided by DTrace. They have been based on the RFCs so that their contents are widely understood.

Example Usage

The following are demonstrations how the net provider could be used to answer various common observability questions.

1. New inbound TCP connections by port
2. New inbound TCP connections by source IP address
3. Inbound TCP packets by local port
4. Inbound TCP packets by remote address
5. Inbound TCP packets by remote address, for a specific port, eg: port 80 (HTTP)
6. Sent TCP bytes by local port
7. TCP bytes by address and port
8. TCP bytes by address and port, per second
9. Detect TCP connect() scan by IP address
10. Detect TCP SYN stealth scan by IP address
11. Detect TCP null scan by IP address
12. Detect TCP FIN stealth scan by IP address
13. Detect TCP Xmas scan by IP address
14. Network Utilization by TCP Port
15. Connect Latency
16. 1st Byte Latency
17. Throughput Latency
18. Connection Lifespan
19. TCP Saturation Drops by IP address

Examples: General

New inbound TCP connections by port

# dtrace -n 'tcp:::accept-established { @[args[2]->tcp_dport] = count(); }'
dtrace: description 'tcp:::accept-established' matched 1 probe
^C

       79                1
       22                2
      515                5
       80               94

This one-liner shows new inbound TCP connections by port. The sample output shows 94 new connections occured on port 80, HTTP.

New inbound TCP connections by source IP address

# dtrace -n 'tcp:::accept-established { @[args[1]->ip_saddr] = count(); }'
dtrace: description 'tcp:::accept-established' matched 1 probe
^C

  192.168.1.31                                                      1
  fe80::214:4fff:fe3b:76c8                                          2
  192.168.1.2                                                       7
  192.168.1.51                                                      9
  192.168.1.8                                                      11
  192.168.7.100                                                    17
  192.168.101.101                                                  31

This one-liner shows new inbound connections by source ip address, which is the string form of either an IPv4 or IPv6 address. Here the host 192.168.101.101 has made 31 successful connections to our server while DTrace was tracing.

Inbound TCP packets by local port

# dtrace -n 'tcp:::receive { @pkts[args[2]->tcp_dport] = count(); }'
dtrace: description 'tcp:::receive' matched 1 probe
^C

       79                2
      515                9
       22               32
     2049              121
       80             8462

This one-liner shows inbound TCP packets by port. The sample output shows that 8462 packets arrived for port 80, HTTP.

Inbound TCP packets by remote address

# dtrace -n 'tcp:::receive { @pkts[args[1]->ip_daddr] = count(); }'
dtrace: description 'tcp:::receive' matched 1 probe
^C

  192.168.1.8                                                       9
  fe80::214:4fff:fe3b:76c8                                         12
  192.168.1.51                                                     32
  192.168.1.8                                                      54
  192.168.7.3                                                     121
  192.168.101.101                                                 192

This one-liner shows which hosts TCP packets have been received from.

Inbound TCP packets by remote address, for a specific port, eg: port 80 (HTTP)

# dtrace -n 'tcp:::receive /args[2]->tcp_dport == 80/ {
        @pkts[args[1]->ip_daddr] = count();
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C

  192.168.1.8                                                       9
  fe80::214:4fff:fe3b:76c8                                         12
  192.168.1.51                                                     32
  192.168.1.8                                                      54
  192.168.7.3                                                     121
  192.168.101.101                                                 192

This short script shows which hosts are sending packets to TCP port 80, the web server. This script can easily be tweaked to show which hosts are sending packets to any of the local services, such as DNS, NFS, etc. RPC services can be resolved by comparing output with an "rpcinfo -p".

Sent TCP bytes by local port

# dtrace -n 'tcp:::send { 
        @bytes[args[2]->tcp_sport] = sum(args[1]->ip_plength - args[2]->tcp_offset);
}'
dtrace: description 'tcp:::sent' matched 1 probe
^C

      515               16
       79              204
       22              427
     2049             8264
       80           108268

The DTrace oneliner subtracts the TCP header length from the IP payload length, which provides is with the TCP payload length. The output shows that 108,268 bytes (106 Kbytes) were sent from port 80, HTTP, during the time DTrace was tracing.

TCP bytes by address and port

# dtrace -n '
tcp:::receive {
        @bytes[args[1]->ip_saddr, args[2]->tcp_dport] = 
            sum(args[1]->ip_plength - args[2]->tcp_offset);
}
tcp:::send {
        @bytes[args[1]->ip_daddr, args[2]->tcp_sport] =
            sum(args[1]->ip_plength - args[2]->tcp_offset);
}'
dtrace: description 'tcp:::receive' matched 2 probes
^C

  192.168.1.8                                           22                    68
  fe80::214:4fff:fe3b:76c8                              22                   582
  192.168.1.6                                           80                  1216
  192.168.101.101                                      512                  2082
  192.168.1.1                                           22                  2187
  192.168.1.51                                          80                  3763
  192.168.3.5                                          109                  4080
  192.168.1.8                                           80                  8039
  192.168.7.3                                           80                 10921
  192.168.1.1                                         2049                 54080
  192.168.101.101                                       80                287623

This provides a report of IP address, TCP port and bytes transferred. This provides an immediate view of which hosts are communicating with our web server, and which ports they are using. During the above sample, host 192.168.101.101 transferred 287623 bytes with our port 80, HTTP.

TCP bytes by address and port, per second

# dtrace -qn '
tcp:::receive {
        @bytes[args[1]->ip_saddr, args[2]->tcp_dport] =
            sum(args[1]->ip_plength - args[2]->tcp_offset);
}
tcp:::send {
        @bytes[args[1]->ip_daddr, args[2]->tcp_sport] =
            sum(args[1]->ip_plength - args[2]->tcp_offset);
}
profile:::tick-1sec {
        printf("\n   %-32s %16s\n", "HOST", "BYTES/s");
        printa("   %-32s %@16d\n", @bytes);
        trunc(@bytes);
}
'
dtrace: description 'tcp:::receive' matched 3 probes

   HOST                                     BYTES/s
   192.168.1.8                                   22
   fe80::214:4fff:fe3b:76c8                      54
   192.168.3.5                                  516
   192.168.2.11                                8142

   HOST                                     BYTES/s
   fe80::214:4fff:fe3b:76c8                      54
   192.168.2.11                                 148
   192.168.3.5                                  686
   192.168.101.101                            60872

   HOST                                     BYTES/s
   192.168.1.8                                    6
   192.168.3.5                                 8044
   192.168.101.101                           826306
...

This DTrace script provides a rolling summary of which hosts have transferred how many bytes by TCP, per second. This makes for a useful tool to immediately identify which host is responsible for consuming network bandwidth. Various enhancements to this script would include adding a field for the destination port, and printing in Kbytes instead of bytes.

Examples: Security Monitoring

While other options such as installing a NIDS (such as snort) would usually offer a better long term traffic monitoring solution, DTrace can be handy for short term traffic analysis.

Detect TCP connect() scan by IP address

# dtrace -n 'tcp:::accept-refused { @num[args[1]->ip_saddr] = count(); }'
dtrace: description 'tcp:::accept-refused' matched 1 probe
^C

  192.168.1.1                                                         1
  220.241.6.37                                                      307

The accept-refused probe fires when a connection is attempted to a closed port. Host 220.241.6.37 attempted 307 times to connect to closed ports, which is likely to be malicious activity such as port scanning. Host 192.168.1.1 had only 1 refused connection, which may be normal activity depending on what their OS is doing.

Detect TCP SYN stealth scan by IP address

# dtrace -n '
tcp:::receive
/syn[args[0]->tcps_cid] && (args[2]->tcp_flags & TH_RST)/
{
        @num["TCP_stealth_scan", args[1]->ip_saddr] = sum(1);
}
tcp:::receive
{
        syn[args[0]->tcps_cid] = 0;
}
tcp:::receive
/args[2]->tcp_flags & TH_SYN && ! (args[2]->tcp_flags & TH_ACK)/
{
        syn[args[0]->tcps_cid]++;
}
'
dtrace: description 'tcp:::receive' matched 2 probes
^C

  192.168.3.5                                                         1
  220.241.6.37                                                        4

This script (or one similar to this) detects half connections - SYNs that are followed by a RST, a sign of a TCP SYN stealth scan. In the above output, host 220.241.6.37 has made 4 half connections, and is most probably scanning our host (or this is a spoofed IP address). The host 192.168.3.5 made 1 half connection, which is not suspicious as this may have completed had we traced for longer. If you are being stealth scanned, you will see many more accept-refused events that appear like a connect scan, and only a few half connections for the ports you have open.

Detect TCP null scan by IP address

# dtrace -n '
tcp:::receive
/args[2]->tcp_flags == 0/
{
        @num[args[1]->ip_saddr] = sum(1);
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C

  220.241.6.37                                                      197

The net provider can be easily used to analyse esoteric TCP traffic, such as null scans. Here null scan packets by host are reported, identifying 220.241.6.37 as sending 197 TCP null packets.

Detect TCP FIN stealth scan by IP address

# dtrace -n '
tcp:::receive
/args[0]->tcp_state != TCPS_ESTABLISHED && args[2]->tcp_flags & TH_FIN/
{
        @num[args[1]->ip_saddr] = sum(1);
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C

  220.241.6.37                                                       45

Here stealth FIN scans are identified by matching on packets to non-established connections (using the TCP state from args[0]), and also matching on the TCP FIN flag. Our evil host sent 45 of these packets. There may certainly be other ways available to identify these scans.

Detect TCP Xmas scan by IP address

# dtrace -n '
tcp:::receive
/args[2]->tcp_flags == (TH_URG + TH_PUSH + TH_FIN)/
{
        @num[args[1]->ip_saddr] = sum(1);
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C

  220.241.6.37                                                      167

There are variations on the Xmas scan, and many different ways to write predicates to match the TCP flags. Here we found 167 likely Xmas packets.

It would be straightforward to write a DTrace script to monitor for many types of malicious TCP traffic, and produce a report of host and traffic type; and as such can serve as a temporary Host based NIDS (Network Intrusion Detection System). Producing such reports without the DTrace net provider usually involves installing additional software, which either sets the network interface to promiscuous mode and analysip_saddr)] = sum(1); }'s every packet, such as snort; or processes the output of network capture files, such as snoop's RFC1761 or libpcap.

Examples: Performance Monitoring

Network Utilization by TCP Port

# dtrace -n '
tcp:::receive {
        /* assume 1 packet with standard ether + IP + TCP header */
        @util[args[2]->tcp_dport] = sum(args[1]->ip_plength + 14);
}
tcp:::send {
        /* assume 1 packet with standard ether + IP + TCP header */
        @util[args[2]->tcp_sport] = sum(args[1]->ip_plength + 14);
}
profile:::tick-1sec {
        /* many assumptions, including 1 NIC @ 100 Mbps */
        normalize(@util, 10000000)
        printf("   %12s %12s\n", "PORT", "%UTIL");
        printa("   %12d %@12d\n", @util);
        trunc(@util);
}'
dtrace: description 'tcp:::receive' matched 3 probes
^C

           PORT       %UTIL
             22           0
           2049           2
             80          12

           PORT       %UTIL
           2049           0
             22           1
             80          23
...

This is a short and crude script based on numerous assumptions, however it will serve to provide an approximation for a single 100 Mbps NIC's utilization by port. While we prefer things to be 100% accurate, this is far better than nothing at all - so long as we bear in mind that the values are an approximation.

Connect Latency

# dtrace -n '
tcp:::connect-request
{
        start[args[0]->tcps_cid] = timestamp;
}

tcp:::connect-established
/start[args[0]->tcps_cid]/
{
        @latency["Connect Latency (ns)", args[1]->ip_daddr] =
            quantize(timestamp - start[args[0]->tcps_cid]);
        start[args[0]->tcps_cid] = 0;
}'
dtrace: description 'tcp:::connect-request' matched 2 probes
^C

  Connect Latency (ns)                                 192.168.101.101

           value  ------------- Distribution ------------- count    
           32768 |                                         0        
           65536 |@@@@@@@                                  2        
          131072 |@@@@@@@@@@@@@@@                          4        
          262144 |@@@@                                     1        
          524288 |@@@@@@@@@@@                              3        
         1048576 |@@@@                                     1        
         2097152 |                                         0        

  Connect Latency (ns)                                  10.1.1.7

           value  ------------- Distribution ------------- count    
         8388608 |                                         0        
        16777216 |@@@@@                                    3        
        33554432 |@@@@@@@@@@@@@@@@                         9        
        67108864 |@@@@@@@@@@@@@@@                          8        
       134217728 |@@@@                                     2        
       268435456 |                                         0        

This script measures outbound connection latency by IP address, and presents this as a distribution plot by nanosecond and count of occurances. We define this to be the time from the SYN to the SYN-ACK (matched by the first receive on the same conn_s *), so most of the TCP handshake costs. The output shows that one IP address is slower to connect than the other. This metric includes network latency and destination kernel overheads.

1st byte Latency

# dtrace -n '
tcp:::connect-established
{
        start[args[0]->tcps_cid] = timestamp;
}

tcp:::receive
/start[args[0]->tcps_cid] && (args[1]->ip_plength - args[2]->tcp_offset) > 0/
{
        @latency["1st Byte Latency (ns)", args[1]->ip_saddr] =
            quantize(timestamp - start[args[0]->tcps_cid]);
        start[args[0]->tcps_cid] = 0;
}'
dtrace: description 'tcp:::send' matched 2 probes
^C

  1st Byte Latency (ns)                                192.168.101.101

           value  ------------- Distribution ------------- count    
           32768 |                                         0        
           65536 |@@@@@@                                   2        
          131072 |@@@@@                                    1       
          262144 |@@@@@@@@@@@@@@@                          4       
          524288 |@@@@@@@@@@@                              3        
         1048576 |@@@@                                     1        
         2097152 |                                         0        

This measures a 1st byte latency on an outbound connection, defined as the time from the TCP handshake completing, to the time that the 1st application byte is returned. This metric includes network latency, destination kernel scheduling latency, and application latency.

Throughput Latency

Connection Lifespan

The following script measures the lifespan of a connection in milliseconds.

# cat tcplifespan1.d
#!/usr/sbin/dtrace -qs

dtrace:::BEGIN
{
        printf("Tracing... Hit Ctrl-C to end.\n");
}

tcp:::accept-established
{
        raddr[args[0]->tcps_cid] = args[1]->ip_saddr;
        lport[args[0]->tcps_cid] = args[2]->tcp_dport;
        stime[args[0]->tcps_cid] = timestamp;
}

tcp:::state-closed,
tcp:::state-time-wait
/stime[args[0]->tcps_cid]/
{
        @lifespan[raddr[args[0]->tcps_cid], lport[args[0]->tcps_cid]] =
           quantize((timestamp - stime[args[0]->tcps_cid]) / 1000000);
        raddr[args[0]->tcps_cid] = 0;
        lport[args[0]->tcps_cid] = 0;
        stime[args[0]->tcps_cid] = 0;
}

dtrace:::END
{
        printf("Inbound Connection Lifespan Report (ms),\n\n");
        printa("   %-32I %21P %@d\n", @lifespan);
}

And now running this script:

# ./tcplifespan1.d
^C
Inbound Connection Lifespan Report (ms),

   deimos                                            http
           value  ------------- Distribution ------------- count
               1 |                                         0
               2 |@@@@@@@@@@@@@@@@@@@@@@@                  7
               4 |@@@@@@@@@@                               3
               8 |@@@                                      1
              16 |                                         0
              32 |@@@                                      1
              64 |                                         0

   deimos                                          finger
           value  ------------- Distribution ------------- count
               8 |                                         0
              16 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@         4
              32 |@@@@@@@@                                 1
              64 |                                         0

   deimos                                             ssh
           value  ------------- Distribution ------------- count
            2048 |                                         0
            4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
            8192 |                                         0

tcplifespan1.d measures the lifespan of an inbound TCP connection and prints a distribution plot. In the above output, we captured a dozen HTTP connections, five finger connections and an SSH connection. One HTTP connection was much slower than the others as it was in the 32 to 63 ms bucket, most of the rest were from 2 to 3 ms. The SSH connection was the slowest (naturally, as this was an interactive session), which was in the 4 to 8 second bucket.

TCP Saturation Drops by IP address

# dtrace -n 'tcp:::drop-saturation {
        @drops[args[1]->ip_saddr, args[1]->ip_daddr] = count();
}'
dtrace: description 'tcp:::drop-saturation' matched 1 probe
^C
  10.1.1.17                         192.168.1.1                             1
  192.168.1.1                       192.168.6.2                             5
  192.168.6.2                       192.168.1.1                             9

This measures TCP drops caused by saturation, by IP address. Nine packets from 192.168.6.2 to 192.168.1.1 were dropped.

Solved Problems

Some of the (numerous) network obserability problems that this provide will solve include:

• Who is connecting to my web server? (TCP accepts by IP address)
• Which services are busy? (bytes by destination port)
• Which hosts are responsible for network load? (bytes by IP address)
• Are hackers/crackers port scanning my server? (TCP flag matching by IP address)
• Why are outbound connections slow? (Connect latency, 1st byte latency, throughput latency)

Comments?