Quick Start Guide

INSTALLING OR UPGRADING AN FNA CACHE

March 2021
Copyright & Trademarks
© 2021 Facebook, Inc. All rights reserved.

Quick Start Guide: Installing or Upgrading an FNA Cache | 2

Table of Contents
Introduction ................................................................................................................................................... 4

Hardware Inventory ....................................................................................................................................... 5

Pre-Installation Checklist ............................................................................................................................... 7

Physical Installation ....................................................................................................................................... 8

1. INSTALL THE SERVERS AND THE SWITCH. ................................................................................................. 8

2. CONNECT THE NETWORK CABLES FROM THE SERVERS TO THE SWITCH ......................................................10

3. INSTALL OPTICS INTO PORTS AND CONNECT THE FNA SWITCH TO YOUR ROUTER ........................................12

4. INSTALL QSFP-SFP ADAPTERS AND SFP+ OPTICAL TRANSCEIVERS ............................................................12

5. POWER ON THE FNA ........................................................................................................................13

6. VERIFY SERVER OPERATION................................................................................................................14

7. CHECK OPTICAL LIGHT LEVELS ............................................................................................................14

Network Configuration ................................................................................................................................15

8. CONFIGURE POINT-TO-POINT LINK..........................................................................................................15

9. CONFIGURE BORDER GATEWAY PROTOCOL (BGP) ....................................................................................15

10. STEP-BY-STEP REPRESENTATION OF IP/BGP CONNECTIVITY AND TRAFFIC FLOW ..............................................17

Appendix 1: HPE Proliant DL380 Generation 10 Overview .........................................................................19

Appendix 2: Server <-> Switch Full Port Map (For Future Upgrades) .........................................................19

Quick Start Guide: Installing or Upgrading an FNA Cache | 3

Introduction
Thank you for choosing to install the Facebook Network Appliance (FNA)!

Facebook Network Appliance is Facebook’s content caching program. FNA provides Internet Service
Providers (ISPs) with an efficient means of delivering static Facebook content from within their network.
Upon deployment, an ISP will offload a significant amount of Facebook content from its backbone
network and vastly improve the Facebook user experience.

An FNA cluster consists of a Top-of-Rack (ToR) switch and from two to twenty servers. The illustrations
in this guide will show a typical 4-server cache deployment, so please consider these illustrations as an
example and follow the instructions taking into account the number of servers actually deployed.

This document provides hardware specifications and instructions on installing your FNA and configuring
it, whether it is a new cluster installation or an upgrade kit for an existing cluster

Quick Start Guide: Installing or Upgrading an FNA Cache | 4

Hardware Inventory
Ensure you have received all equipment in accordance with the checklist below. The table below is a
reference for a 4-server deployment of a new cache. Number of items will vary depending on
deployment size.
An upgrade kit will be almost identical, but I will not include a switch. This is because an upgrade kit
adds servers and uplinks to an existing cluster, that already has a switch.

If you are missing any items, please contact Facebook via Network Partner Portal (NPP).

FNA Provided Hardware – New Cluster Kit

Dimensions
QTY Item
per Unit

(1) Arista DCS-7060CX-32S-R-DC Switch 1 RU

(4) Hewlett Packard ProLiant Server DL380p G10 Server 2 RU

(4) Post rail kit N/A

(8) AC Power Cord (only if AC power required) 2 Meters

(4) Network Cable (to connect servers to the switch) 3 Meters

(1) HP ProLiant DL380 G10 server user manual1 N/A

(1) Switch user manual N/A

(4) For 10G uplink, SFP+ optical transceiver, LR or SR2 N/A

(1 per SFP+) For 10G uplink, QSFP to SFP+ adapter2 N/A

(1-4) For 100G uplink, QSFP transceiver, LR4 or SR42 N/A

1
Model numbers vary per cluster.

2
Uplink capacity is selected for every order, either with 10G or 100G ports. Optical transceivers will
be sent according to the port type selected. Facebook provides transceivers only for the FNA, while
optics for the ISP devices must be provided by the ISP. For 10g uplink, by default FB will send as
many SFP+ as servers in the order. For 100g uplink, FB will send as many as selected in the order
form.

Quick Start Guide: Installing or Upgrading an FNA Cache | 5

Table 1: FNA Cluster Provided Hardware

The following table shows additional equipment required for the FNA installation that is not provided by
Facebook.

Equipment Required but Not Included

QTY Item

(1 per uplink port) Fiber patch cables from Arista DCS-7060CX-32S-R-DC Switch to ISP router1

As needed QSFP / SFP from router side and rack2

1
Fiber connectivity on the FNA side is the Lucent Connector (LC) type. Fiber connectivity on the
router side can be LC-LC, LC-SC, (Lucent Connector – Subscriber Connector), etc.

2
Optics for the ISP devices must be provided by the ISP. The FNA accepts 10G SR and LR, and 100G
SR4 and LR4 optical transceivers. Facebook provides the most optimal optical transceiver type,
corresponding to the type you currently use to connect your routers. FNA does not support 40GE
interfaces for the uplink.

Table 2: Equipment Required but Not Included

Quick Start Guide: Installing or Upgrading an FNA Cache | 6

Pre-Installation Checklist

Use the pre-installation checklist provided below to confirm that your facility has the capacity and
resources required to install an FNA cluster:

Pre-installation Checklist

Item

Sufficient power for each cluster (1kW for a 4 servers cluster)

Power connections are divided evenly between two different power sources

n available input power connections1

Sufficient rack space is available2

n x 10 Gigabit (10Gb or 100gb) available ports on ISP switch or router3

n available SFP+ (or compatible) optics of same type – Long Range (LR) or Short
Range (SR)3

n available fiber patch cables (multi-mode or single mode)3

1
Where n = 2 times the number of devices (e.g., Cluster 1 is (5) devices, so (10) input power
connections are required).

2
For initial deployment (Cluster 1), 9RU (nine rack unit) is required. For additional servers, consider
2RU additional per server.

3
Where n = 1 for each server deployed.

Table 3: Pre-installation Checklist

Quick Start Guide: Installing or Upgrading an FNA Cache | 7

Physical Installation
1. Install the servers and the switch.

a. Install the servers and connect power cables.

i. Install the servers into a compatible rack. Installing servers in a 2-post rack requires
appropriate 4-post to 2-post conversion hardware. Please leave enough rack space at
the top part of the rack to install the switch later. Also, please install the server labeled
as fna001 at the highest position of the rack. Install the rest of the racks accordingly, you
can use Figure 1.1 for guidance. Use manufacturer’s instructions for the server model
included in your order if further guidance is needed.

ii. Connect C13-C14 power cables from each server to the Power Distribution Units (PDUs).
Figure 1.1 illustrates AC power supplies. For DC power installations, please consult an
electrician.

Quick Start Guide: Installing or Upgrading an FNA Cache | 8

Rear of Rack

Fna001

Fna002

Fna003

Fna004

Figure 1.1: Install the servers and connect AC Power

b. Install the Switch.

i. Install the switch on top of the rack with the airflow as port side exhaust (switch ports
must face the rear of the rack as shown in Figure 1.2). Use manufacturer’s instructions
for the switch model included in your order if further guidance is needed.

Rear of Rack

Quick Start Guide: Installing or Upgrading an FNA Cache | 9

Figure 1.2: Install switch on top of the rack

IMPORTANT: We won’t connect the switch to power at this stage, as the switch will immediately turn on
as soon it receives power, and we don’t want that for now. We will do this in a later stage.

2. Connect the Network Cables from the Servers to the Switch

a. Each server’s NIC and network cable connector have a green sticker to show the correct port
connection and orientation (as shown in Figure 2.1).

Quick Start Guide: Installing or Upgrading an FNA Cache | 10

Figure 2.1: Server port connexion and orientation

b. Connect one end of the network cable (green sticker facing up) to the server’s NIC port 2 (begin
with the server at the highest position in the rack labelled fna001) and connect the other end to
the switch’s port 32. Repeat for every server as illustrated in Figure 2.2. Table 4 provides a port

map for a 4-servers cache deployment, plus an example of a 4 to 6-servers upgrade.

Figure 2.2: Switch <> Server connectivity

Quick Start Guide: Installing or Upgrading an FNA Cache | 11

Server NIC's port FNA Switch port
Hostname Number number

fna001 2 32
fna002 2 31
4-Servers Cache
fna003 2 30
fna004 2 29

Server NIC's port FNA Switch port

Hostname Number number

fna001 2 32
fna002 2 31
fna003 2 30
6-Servers Cache
fna004 2 29
fna005 2 28
fna006 2 27

Table 4: Server <> Switch connectivity port map (4-servers and 4 to 6-servers cache upgrade)

Note: Go to Appendix 2 at the end of this document to get a complete port map for future upgrades.

3. Install Optics into ports and connect the FNA Switch to your Router

4. Install QSFP-SFP adapters and SFP+ optical transceivers

into ports 1, 2, 3 and 4 in FNA switch as shown in Figure 4.1.

Please note: Full switch port assignment is provided in Figure 4.1 for clusters over four servers.

Quick Start Guide: Installing or Upgrading an FNA Cache | 12

Figure 4.1: Switch port assignment
Ports 1-4

5. Power on the FNA

a. Press the power buttons on all four servers as shown in Figure 5.1.

b. Connect two power cables to the network switch as shown in Figure 5.1 (the switch powers on
when connected to power).

c. The switch’s system status LED displays as blinking green while powering up, then steady green
for normal and amber indicates a fan disconnect or malfunction. Port LEDs display green for up,
yellow for software disabled and flashing yellow for failed diagnostics (refer to Figure 5.2).

5.b

5.a

Quick Start Guide: Installing or Upgrading an FNA Cache | 13

Figure 5.1 Figure 5.2

6. Verify Server Operation

a. Verify server operation by checking the status LEDs as illustrated in Figure 6.1.

b. A server is powered on when the status LED is green. If the status LED is amber, then you need
to press the power button until the LED turns green. For Gen10 servers, wait a few seconds until
the status LED turns green.

Figure 6.1

7. Check Optical Light Levels

a. Check the optical light levels to ensure the cluster has adequate signal stretch. Light levels need
to be between -2 dB and -7 dB.

Quick Start Guide: Installing or Upgrading an FNA Cache | 14

Network Configuration
8. Configure Point-to-Point Link
a. The point-to-point connection is configured on the Link-Aggregation Control Protocol (LACP)
interface. Make sure the following Parameters are configured as follows:

i. Set a static route to the allocated subnets via the link-aggregation interface.

ii. Set the appropriate maximum transmission unit (MTU) for your router software:

• For CISCO IOS XR router software, set the MTU to 1514. CISCO IOS SR software
includes both the L2 and L3 overhead in the interface MTU command.

• For all other router software, set the MTU to 1500.

The FNA cluster must have internet connectivity BEFORE BGP peering can be established. The
BGP peering session does not need to be routed before the FNA has internet connectivity.

The FNA kit is preconfigured with the IP addresses that were specified in the order.

If IP addresses need to be changed at any time, please let us know by posting in the kit request
ticket in the Network Partner Portal.

Our team monitors the updates in the ticket portal, where you can also request assistance. You
may also contact the FNA Operations team and ask for support by emailing [email protected]

9. Configure Border Gateway Protocol (BGP)

iii. Use the IPv4 and IPv6 addresses originally indicated in the FNA order form.

iv. The IP prefixes assigned to the FNA should be announced to all your Facebook peering
connections and to your transit provider(s) to ensure they are reachable through the
internet.

v. (Optional) Enable Graceful Restart (GR). Enabling GR is not required, but it is

Quick Start Guide: Installing or Upgrading an FNA Cache | 15

recommended.

vi. Enable External BGP (EBGP) Multihop.

vii. Use the peering Autonomous System Number (ASN): 63293

viii. Configure your router’s BGP settings.

ix. After the BGP settings are saved, the FNA cluster will begin downloading application
data. Within two business days, BGP peering will be established:

1. The connection will be coming from the 16th IP address in each allocated subnet
(for example, 192.168.1.16 for IPv4, and 2001:db8::10 for IPv6).

2. FNA will not advertise any routes/prefixes to the ISP.

x. BGP advertisements to the FNA:

1. All your customer prefixes and every servable public IP prefix should be
announced to the FNA for network traffic optimization

2. Do not advertise private IP addresses to the FNA, as this is not supported

3. The smallest IP prefix sizes accepted are /30 for IPv4 and /64 for IPv6

xi. We hold a limit on the number of IPv4 / IPv6 advertisements allowed to your FNA:

• For IPv4 we allow a maximum of 90k

• For IPv6 we allow a maximum of 25k

xii. As this may be indicative of prefix leakage, we will inform you via a Direct Support ticket
in the Network Partner Portal when you have reached 90% of these thresholds. Should
you cross either of these thresholds, traffic will be drained from your cluster. To prevent
this, please ensure your BGP advertisements are below the defined thresholds. Should
you have a reason to cross this, please contact [email protected]

xiii. When installation is complete, please share an update in the kit request ticket in the
Network Partner Portal. Our team monitors the updates in the portal ticket, where you
can also request assistance. You may also contact the FNA Operations team and ask for
support by emailing [email protected]

Quick Start Guide: Installing or Upgrading an FNA Cache | 16

10. Step-by-step representation of IP/BGP connectivity and traffic flow

STEP 1

Figure 10.1: Physical connection and LACP uplink

STEP 2

Figure 10.2: Configuring BGP

Quick Start Guide: Installing or Upgrading an FNA Cache | 17

STEP 3

Figure 10.3: Adding prefixes, FNA starts serving traffic

STEP 4

Figure 10.4: Full diagram with example IP addressing

Quick Start Guide: Installing or Upgrading an FNA Cache | 18

Appendix 1: HPE ProLiant DL380 Generation 10 Overview

Figure A1.1

Figure A1.2

You can find the technical specifications for the FNA devices in the Technical Overview document
available in the Network Partner Portal documentation.

Appendix 2: Server <-> Switch full port map (for future

upgrades)

Quick Start Guide: Installing or Upgrading an FNA Cache | 19

Server Hostname NIC's port Number FNA Switch port number

fna001 2 32

fna002 2 31

fna003 2 30

fna004 2 29

fna005 2 28

fna006 2 27

fna007 2 26

fna008 2 25

fna009 2 24

fna010 2 23

fna011 2 22

fna012 2 21

fna013 2 20

fna014 2 19

fna015 2 18

fna016 2 17

fna017 2 16

fna018 2 15

fna019 2 14

fna020 2 13

Table 5: Server <-> Switch full port map

Quick Start Guide: Installing or Upgrading an FNA Cache | 20

Quick Start Guide: Installing or Upgrading an FNA Cache | 21