pmacct [IP traffic accounting : BGP : BMP : RPKI : IGP : Streaming Telemetry]

pmacct is Copyright (C) 2003-2024 by Paolo Lucente

The keys used are:

!: fixed/modified feature, -: deleted feature, +: new feature

1.7.9 -- 01-08-2024

+ pmtelemetryd: gRPC data collection functionalities were added to

the daemon. https://github.com/scuzzilla/mdt-dialout-collector is

the library linked to add the functions required to collect data

via gRPC dialout. Thanks to Salvatore Cuzzilla ( @scuzzilla ) for

this contribution, along with the ninja support of Marc Sune

( @msune ).

+ BMP daemon: HA support introduced: in order to add redundancy at

data collection, let multiple collectors to establish indentical

BMP session, while only one is sending data to the backend. Thanks

to Zhuoyao Lin ( @Zephyre777 ) and Leonardo Rodoni ( @rodonile )

for this contribution.

+ nfacctd: developed a more robust and streamlined NetFlow v9/

IPFIX template handling framework based on libcdada structures,

hence dropping the previous ad-hoc home-grown memory structures.

+ nfacctd: introduced support for sampling rate calculation based

on IANA entities IE309 (samplingSize), IE310 (samplingPopulation),

which are being exported in IPFIX sampling options for a random-n-

out-of-N sampler. Also, support for 16-bit and 32-bit selectorID

is added as well. Thanks to Leonardo Rodoni ( @rodonile ) for this

contribution.

+ nfacctd: extended the support of IE130 (exporterIPv4Address) and

IE131 (exporterIPv6Address) for when this info is contained in

data packets. Thanks to Leonardo Rodoni ( @rodonile ) for this

contribution.

+ nfacctd: added nfacctd_ignore_exporter_address config knob to

not honour IE130 (exporterIPv4Address), IE131 (exporterIPv6Address)

and use the socket address instead.

+ nfacctd: improved support for Route Distinguisher (RD): added the

case where the RD information has finer-grained scope in Option

packets; also RD in IPFIX Options can now be used for correlating

data against BGP/BMP. Finally priority of supplied RD info is

streamlined as: RD in flow_to_rd.map > RD in IPFIX/NFv9 data

packet > RD in IPFIX/NFv9 option packets. Thanks to Leonardo

Rodoni ( @rodonile ) for these contributions.

+ nfacctd: added IEs 44 (sourceIPv4Prefix), 170 (sourceIPv6Prefix),

45 (destinationIPv4Prefix), and 169 (destinationIPv6Prefix) to

flow heuristics.

+ nfacctd: as part of src_mac, dst_mac primitives, added support

for IEs 365 (staMacAddress) and 366 (staIPv4Address).

+ nfacctd: introduced support for IE497 (srhSegmentIPv6ListSection),

The SRv6 Segment List as defined in Section 2 of [RFC8754] as a

series of octets in IPFIX. Also added Path Delay measurements

PathDelay{Mean,Min,Max,Sum}DeltaUsecs.

+ nfacctd: tunnel primitives (tunnel_src_host, tunnel_dst_host,

tunnel_proto, tunnel_tos, tunnel_src_port, tunnel_dst_port and

tunnel_tcpflags) were linked to SRv6 code. Also, in this context,

support for repeating IEs has been added. Thanks to Uwe Storbeck

( @ustorbeck ) for this contribution.

+ nfacctd: extended IE89 (forwardingStatus) support to be 2 or 4

bytes long. Also introduced support for IE31 (flowLabelIPv6).

Thanks to Uwe Storbeck ( @ustorbeck ) for these contributions.

+ nfacctd: a new primitive 'nvgre' is defined representing the

Tenant Network Identifier (TNI) for NVGRE L2 tunnels, which is

encoded in IANA IE351 (layer2SegmentId) with MSB 0x02. Thanks to

Leonardo Rodoni ( @rodonile ) for this contribution.

+ sfacctd: added support for accounting for ARP packets via a new

aggregate_unknown_etype config knob.

+ nfacctd, sfacctd, pmacctd: added new 'in_cvlan' and 'out_cvlan'

primitives. It identifies the VLAN to which the frame belongs to

when it is transmitted in the customer network.

+ BGP daemon: added support for "Only to Customer" OTC attribute,

RFC9234.

+ BGP daemon: extend MP nexthop length to support 48 bytes to fit

the BGP VPNv6 nexthop length. Thanks to @FRIDM636 for this

contribution.

+ BGP daemon: as specified by RFC8950, routers cannot advertise

IPv4 or VPNV4 routes with an IPv6 next hop without an extra

capability advertisement (Extended Next Hop Encoding). This has

been added so that the daemon can reply with this capability to

the remote peer. Thanks to Leonardo Rodoni ( @rodonile ) for

this contribution.

+ BGP, BMP daemons: Two new configuration keys are introduced:

bgp_comms_encode_as_array and as_path_encode_as_array to allow

to specify that BGP communities and AS_PATH are encoded as an

array in JSON or AVRO encodings.

+ BMP daemon: a new bmp_dump_exclude_stats config knob has been

added: if enabled, BMP Stats messages are not going to be cached

and thus not being included in the regular dump.

+ All daemons: added a -T / dry_run config knob. With the 'config'

argument only configuration is validated; with the 'setup' one

the daemon and its plugins are also instantiated and validated.

+ Kafka plugin: allowing to configure sub-minute historical time

bins (ie. kafka_history, etc.).

+ pre_tag_map: introduced pre_tag_map_dont_recirculate config knob

to explicitely disable v4/v6 recirculation of entries without an

'ip' key specified. Also if 'ip' key is not defined, still define

the address family (AF) so to be memory-savvy and avoid creating

duplicate entries in case of v4/v6 recirculation.

+ nDPI support: updated API calls to compile against nDPI 4.6. Also

dropped support for previous versions of the library.

! fix, nfacctd: treat the result of (SysUptime - fstime) as signed

to avoid a underflow in the case where fstime > SysUptime in

NetFlow v9. Thanks to Jim Westfall ( @jwestfall69 ) for this

contribution.

! fix, nfacctd: addressed a memory leak in handling NetFlow/IPFIX

templates. Also performed code re-factoring to better encapsulate

the template functionality, add a template header file, separate

the module interface from local functions and reduce the scope of

local functions. Thanks to Uwe Storbeck ( @ustorbeck ) for these

contributions.

! fix, nfacctd: parsing of IPFIX/NetFlow data when a template does

contain multiple padding octet fields IE 210 (paddingOctets) with

different length. Thanks to Uwe Storbeck ( @ustorbeck ) for this

contribution.

! fix, nfacctd: 4 bytes long IE95 (applicationID) is now supported;

as part of this work the setup of nDPI and NBAR classifiers have

been harmonized.

! fix, sfacctd: mispelled daemon type in sampling_direction handler

was causing counters not to print.

! fix, pmacctd: restored packet data pointer for correct collection

of ICMPv6 data.

! fix, uacctd: the daemon was crashing upon receipt an ICMP Echo

Request packet (as a result, for example, of a ping). Thanks to

Alexei A Smekalkine ( @ikle ) for this contribution.

! fix, pmtelemetryd: when no backend dump method is configured (and

telemetry_dump_time_slots is not set) pmtelemetryd was crashing

with SIGFPE. Also fixed the dump interval calculation when time

slots are used. Thanks to Uwe Storbeck ( @ustorbeck ) for these

contributions.

! fix, pmtelemetryd: restored ability of the daemon re-loading maps

via SIGUSR2 signal. Thanks to Salvatore Cuzzilla ( @scuzzilla )

for this contribution.

! fix, BGP, BMP daemons: the hash function used to distribute per-

peer information attached to RIB entries was enriched with the

addition of Route Distinguisher (RD) data in order to minimize

collisions. It is in fact crucial to have an efficient hash-table

to perform data correlation between BGP/BMP & IPFIX. Thanks to

Salvatore Cuzzilla ( @scuzzilla ), Leonardo Rodoni ( @rodonile )

for this contribution.

! fix, BMP daemon: finer grained control on support of ADD-PATH

capability as part of the Peer Up message in order to support

the case of remote peer receive-only scenario.

! fix, SQL plugins: 'in_vlan' primitive is not muxed anymore on

'vlan' keyword at configure time. Also, 'out_vlan' primitive is

now properly handled if sql_optimize_clauses left false.

! fix, MySQL plugin: if timestamps_utc is set to true, enforce UTC

for the current session.

! fix, tee plugin: missing variable definition when BSD definition

is on was preventing the code to compile correctly.

! fix, pre_tag_map: check added to avoid daemon hanging if 'next'

label was mentioned as part of the last map entry.

! fix, Redis: avoid to create a new file descriptor for every

reconnect. Use redisReconnect instead of redisConnect. Thanks to

Uwe Storbeck ( @ustorbeck ) for this contribution.

! fix, util.c: weekly time roundoff has been made consistent by

correctly including the first day of the week.

! fix, util.c: when pidfile is specified, use mkdir_multilevel() to

build dir structure if needed.

! fix, plugins_hook.c: when processing a pcap_savefile, perform an

inter-buffer sleep of 1ms only if using home-grown buffering as

ZeroMQ (plugin_pipe_zmq) would instead do fine absorbing the data

burst.

! fix, rpki_msg.c: missing json_decref() in rpki_roas_file_load()

was leaking memory on map reload. Also, solved a SEGV observed at

times when reloading rpki_roas_file. Finally free'd the output of

aspath_make_str_count() in rpki_roas_file_load().

- pmtelemetryd: removed legacy Python decoders and associated utils,

also removed support for Kafka and ZeroMQ telemetry collection.

- pre_tag_map: obsoleted 'fwdstatus' key.

- GeoIP support: removed support for Maxmind API v1.

1.7.8 -- 31-12-2022

+ Introduced support for eBPF for all daemons: if SO_REUSEPORT is

supported by the OS and eBPF support is compiled in, this allows

to load a custom load-balancer. To load-share, daemons have to

be part of the same cluster_name and each be configured with a

distinct cluster_id.

+ Introduced support for listening on VRF interfaces on Linux for

all daemons. The feature can be enabled via nfacctd_interface,

bgp_daemon_interface and equivalent knobs. Many thanks to

Marcel Menzel ( @WRMSRwasTaken ) for this contribution.

+ pre_tag_map: introduced limited tagging / labelling support for

BGP (pmbgpd), BMP (pmbmpd), Streaming Telemetry (pmtelemetryd)

daemons. ip, set_tag, set_label keys being currently supported.

+ pre_tag_map: defined a new pre_tag_label_encode_as_map config

knob to encode the output 'label' value as a map for JSON and

Apache Avro encodings, ie. in JSON "label": { "key1": "value1",

"key2": "value2" }. For keys and values to be correctly mapped,

the '%' delimiter is used when composing a pre_tag_map, ie.

"set_label=key1%value1,key2%value2 ip=0.0.0.0/0". Thanks to

Salvatore Cuzzilla ( @scuzzilla ) for this contribution.

+ pre_tag_map: introduced support for IP prefixes for src_net

and dst_net keys for indexed maps (maps_index set to true).

Indexing being an hash map, this feature currently tests data

against all defined IP prefix lenghts in the map for a match

(first defined matching prefix wins).

+ pre_tag_map: introduced two new 'is_nsel', 'is_nel' keys to

check for the presence of firewallEvent field (233) and

natEvent field (230) in NetFlow/IPFIX respectively in order

to infer whether data is NSEL / NEL. If set to 'true' this

does match NSEL / NEL data, if set to 'false' it does match

non NSEL / NEL data respectively.

+ Introduced a new mpls_label_stack primitive, encoded as a

string and includes a comma-separated list of integers (label

values). Thanks to Salvatore Cuzzilla ( @scuzzilla ) for this

contribution.

+ Introduced a new fw_event primitive, to support NetFlow v9/

IPFIX firewallEvent 233 Information Element.

+ Introduced a new tunnel_tcp_flags primitive for pmacctd and

sfacctd to record TCP flags for the inner layer of a tunneled

technology (ie. VXLAN). Also tunnel_dst_port decoding was

fixed for sfacctd.

+ Introduced support for in/out VLAN support for sfacctd. To be

savy, 'in_vlan' and 'vlan' were muxed onto the same primitive

depending on the daemon being used. Thanks to Jim Westfall

( @jwestfall69 ) for this contribution.

+ Introduced a new mpls_label_stack_encode_as_array config knob

to encode the MPLS label stack as an array for JSON and Apache

Avro encodings, ie. in JSON "mpls_label_stack": [ "0-label0",

"1-label1", "2-label2", "3-label3", "4-label4", "5-label5" ]

and in Avro "name": "mpls_label_stack", "type": { "type":

"array", "items": { "type": "string" } }. Thanks to Salvatore

Cuzzilla ( @scuzzilla ) for this contribution.

+ Introduced a new tcpflags_encode_as_array config knob to encode

TCP flags as an array for JSON and Apache Avro, ie. in JSON

"tcp_flags": [ "URG", "ACK", "PSH", "RST", "SYN", "FIN" ] and

in Avro "name": "tcp_flags", "type": { "type": "array",

"items": { "type": "string" } }. Thanks to Salvatore Cuzzilla

( @scuzzilla ) for this contribution.

+ Introduced a new fwd_status_encode_as_string config knob to

encode the 'fwd_status' primitive in human-readable format

like described by RFC-7270 Section 4.12 when JSON or Avro

formats are selected for output. Thanks to Salvatore Cuzzilla

( @scuzzilla ) for this contribution.

+ Introduced a new protos_file to define a list of (known/

interesting/meaningful) IP protocols. Both protocol names, ie.

"tcp", and protocol numbers, ie. 1 (for icmp), are accepted.

IANA reserved protocol value 255 is used to bucket as 'others'

those IP protocols not matching the ones defined in the list.

+ Introduced a new tos_file to define a list of (meaningful) IP

ToS values; if tos_encode_as_dscp is set to true then DSCP

values are expected as part of the file. The directive uses

value 255 to bucket as 'others' those ToS/DSCP values not

matching the ones defined in the list.

+ A new tos_encode_as_dscp config knob makes pmacct to honour

only the 6 bits used by DSCP and report only on those.

+ BGP, BMP, Streaming Telemetry daemons: introduced a new

dump_time_slots config knob to spread the load deriving by

dumps over the configured refresh time interval. The interval

is divided into time slots and nodes are assigned to such

slots. The slot for each node is determined using its IP

address. Thanks to Raphael Barazzutti ( @rbarazzutti ) for

this contribution.

+ BGP, BMP daemons: End-of-RIB messages are now being exposed

in the output feed in order to facilitate tracking their

arrival (or not!).

+ pmtelemetryd: aligned daemon to the latest Unyte UDP-Notif API

(0.6.1) and related standardization draft-ietf-netconf-udp-notif

+ RPKI daemon: added case for input "asn" value being integer (ie.

"asn" : 2914) on top of the string case (ie. "asn" : "AS2914").

+ Kafka, amqp plugins: introduced a new writer_id_string config

knob to allow to customize the the "writer_id" field value. A

few variables are supported along with static text definitions.

+ Added a new aggregate_unknown_etype config knob to account also

frames with EtherTypes for which there is no decoding support

and allow to aggregate them by the available Ethernet L2 fields

(ie. 'src_mac', 'dst_mac', 'vlan', 'cos', 'etype'). Thanks to

@singularsyntax for this contribution.

+ Added a new bgp_daemon_add_path_ignore config knob to ignore

(do not advertise back) the ADD-PATH capability advertised by

remote BGP peers.

+ nfacctd, sfacctd: extended the possibility to run daemons from

a user with non root privileges to these daemons.

+ nfacctd: if Information Element 90 (MPLS VPN RD) is present in

NetFlow v9/IPFIX, make it available for BGP/BMP correlation.

+ pmacctd, sfacctd: introduced basic support for QinQ, 802.1AD.

+ [print|kafka|amqp]_preprocess: added suppport for 'maxp',

'maxb' and 'maxf' keys when preprocessing aggregates of non-

SQL plugins. Thanks to Andrew R. Lake ( @arlake228 ) for this

contribution.

+ nDPI: newer versions of the library (ie. >= 4.0) bring changes

to the API. pmacct is now aligned to compile against these. At

the same time support for nDPI 3.x was dropped.

! fix, plugin_common.[ch]: when stitching feature was enabled,

ie. nfacctd_stitching, timestamp_min was never reset. Also both

timestamp_min and timestamp_max were clamped to sec granularity.

! fix, BGP, BMP daemons: added a tmp_bgp_daemon_origin_type_int to

print out BGP "origin" field as int (legacy behaviour) instead

of string (current behaviour). In a future major release the

legacy behaviour will be dropped.

! fix, BGP, BMP daemons: MPLS labels are now encoded in both JSON

and Apache Avro as 'mpls_label' instead of 'label'. This is to

align behaviour with pre_tag_map where 'label' has a different

semantic.

! fix, BGP, BMP daemons: resolved memory leak when encoding log

messaging (logmsg) in Avro format with Schema Registry support.

! fix, BGP daemon: improved handling of ADD-PATH capability,

making it per-AF (as it is supposed to be) and not global.

! fix, BMP daemon: now checking that ADD-PATH capability is

enabled at both ends of the monitored session (check both BGP

OPEN in a Peer Up message) in order to infer that the capability

exchange was successful. Also some heuristics were added to

conciliate BGP Open vs BGP Update 4-bytes ASN reality.

! fix, nfacctd: improved parsing of NetFlow v9 Options data

particularly when multiple IEs are packed as part of a flowset.

! fix, nfacctd: corrected parsing of Information Element 351

(layer2SegmentId).

! fix, pmacctd: improved processing of pcap_interfaces_map for

cases where the same interface is present multiple times (maybe

with different directions). Also, if the map is empty then bail

out at startup.

! fix, pmacctd: SEGV when ICMP/ICMPv6 traffic was processed and

'flows' primitive was enabled.

! fix, pmacctd: sampling_rate primitive value was not reported

correctly when 'sampling_rate' config directive was specified.

! fix, pmbgpd, pmpmbd, pmtelemtryd: changed SIGCHLD handler to

prevent zombification of last spawned data dump writer.

! fix, Kafka plugin: moved the schema registration from the dump

writer to the plugin process in order to register the schemas

only once at plugin startup and not on every start of a writer

process. Thanks to Uwe Storbeck ( @ustorbeck ) for this

contribution.

! fix, Kafka plugin: a check for kafka_partition was missing,

leading the plugin to always use the default partitioner

instead of sending data to the configured fixed partition.

Thanks to Martin Pels ( @rodecker ) for this contribution.

! fix, nfprobe plugin: BGP data enrichment was not working due to

a mistakenly moved pointer.

! fix, sfprobe plugin: AS-PATH was being populated even when null;

added a check to see if the destination AS is not zero in order

to put the destination AS into the AS-PATH for sFlow packets.

Thanks to Marcel Menzel ( @WRMSRwasTaken ) for this contribution.

! fix, networks_file: remove_dupes() was making partial commits

of valid rows hence creating data inconsistencies.

! fix, pre_tag_map: resolved a potential string overflow that was

being triggered in pretag_append_label() when data would be

assigned more than one single label. Also now allow ',' chars

in set_label.

! fix, maps_index: uninitialized var could cause SEGV in case no

results are found in the map index. Also introduced support for

catch-all rules, ie. "set_label=unknown".

! fix, maps_index: optimized the case of no 'ip' key specified

(for nfacctd and sfacctd): when indexing is enabled, prevent

recirculation from happening, ie. test v4 first then v6, since

the 'ip' key is not going to be part of the hash serializer.

! fix, pretag.c: allow to allocate maps greater than 2GB in size.

Also several optimizations were carried out yelding to a better

memory utilization for allocated maps along with improved times

to resolve JEQs.

! fix, pre_tag_label_filter: optimized and improved runtime

evaluation part of this feature, avoiding a costly strdup() and

returning immediately on certain basic mismatch conditions.

! fix, kafka_common.[ch]: a new p_kafka_produce_data_and_free()

is invoked to optimize memory allocations and releases.

! fix, plugin_cmn_avro.c: when a schema registry is being defined,

ie. kafka_avro_schema_registry, the logic to generate the schema

name has been changed: use topic plus record name as the schema

name, use underscore as separator within the record name, stop

adding a "-value" suffix. Thanks to Uwe Storbeck ( @ustorbeck )

for this contribution.

! fix, util.c: roundoff_time() to reason always with the locally

configured time, like for the rest of functional (as in non-data)

timestamps, ie. refresh time, deadline, etc.

! fix, log.c: when log messages are longer than message buffer,

the message gets cut off. As the trailing newline also gets cut

off the message will be concatenated with the following message

which makes the log hard to read. Thanks to Uwe Storbeck

( @ustorbeck ) for this contribution.

- Completed the retirement of legacy packet classification based

on home-grown code (Shared Objects) and the L7 layer project.

- Removed the mpls_stck_depth primitive due to the introduction

of the mpls_label_stack primitive.

1.7.7 -- 07-11-2021

+ BGP, BMP, Streaming Telemetry daemons: introduced parallelization

of dump events via a configurable amount of workers where the unit

of parallelization is the exporter (BGP, BMP, telemetry exporter),

ie. in a scenario where there are 4 workers and 4 exporters each

worker is assigned one exporter data to dump.

+ pmtelemetryd: added support for draft-ietf-netconf-udp-notif:

a UDP-based notification mechanism to collect data from networking

devices. A shim header is proposed to facilitate the data streaming

directly from the publishing process on network processor of line

cards to receivers. The objective is a lightweight approach to

enable higher frequency and less performance impact on publisher

and receiver process compared to already established notification

mechanisms. Many thanks to Alex Huang Feng ( @ahuangfeng ) and the

whole Unyte team.

+ BGP, BMP, Streaming Telemetry daemons: now correctly honouring the

supplied Kafka partition key for BGP, BMP and Telemetry msg logs

and dump events.

+ BGP, BMP daemons: a new "rd_origin" field is added to output log/

dump to specify the source of Route Distinguisher information (ie.

flow vs BGP vs BMP).

+ pre_tag_map: added ability to tag new NetFlow/IPFIX and sFlow

sample_type types: "flow-ipv4", "flow-ipv6", "flow-mpls-ipv4" and

"flow-mpls-ipv6". Also added a new "is_bi_flow" true/false key to

tag (or exclude) NSEL bidirectional flows. Added as well a new

"is_multicast" true/false config key to tag (or exclude) IPv4/IPv6

multicast destinations.

+ maps_index: enables indexing of maps to increase lookup speeds on

large maps and/or sustained lookup rates. The feature has been

remplemented using stream-lined structures from libcdada. This is

a major work that helps preventing the unpredictable behaviours

caused by the homegrown map indexing mechanism. Many thanks to

Marc Sune ( @msune ).

+ maps_index: support for indexing src_net and dst_net keywords has

been added.

+ Added <daemon_name>_ipv6_only config directives to optionally

enable the IPV6_V6ONLY socket option. Also changed the wrong

setsockopt() IPV6_BINDV6ONLY id to IPV6_V6ONLY.

+ Added log function to libserdes to debug transactions with the

Schema Registry when kafka_avro_schema_registry is set.

+ nDPI: newer versions of the library (ie. >= 3.5) bring changes

to the API. pmacct is now aligned to compile against these.

+ pmacctd: added pcap_arista_trailer_offset config directive since

Arista has changed the structure of the trailer format in recent

releases of EOS. Thanks to Jeremiah Millay ( @floatingstatic )

for his patch.

+ More improvements carried out on the Continuous Integration

(CI) side by migrating from Travis CI to GitHub Actions. Huge

thanks to Marc Sune ( @msune ) to make all of this possible.

+ More improvements also carried out in the space of the Docker

images being created: optimized image size and a better layered

pipeline. Thanks to Marc Sune ( @msune ) and Daniel Caballero

( @dcaba ) to make all of this possible.

+ libcdada shipped with pmacct was upgraded to version 0.3.5. Many

thanks Marc Sune ( @msune ) for his work with libcdada.

! build system: several improvements carried out in this area,

ie. improved MySQL checks, introduced pcap-config tool for

libpcap, compiling on BSD/old compilers, etc. Monumental thanks

to Marc Sune ( @msune ) for his continued help.

! fix, nfacctd: improved euristics to support the case of flows

with both IPv4 and IPv6 source / destination addresses (either

or populated). Also improved euristics to distinguish event data

vs traffic data in NetFlow v9/IPFIX from Cisco 9300/9500, ASA

firewalls and Cisco 4500X.

! fix, nfacctd: improved support for initiatorOctets (IE #231) and

responderOctets (IE #232). Thanks to Esben Laursen ( @hyberdk )

for reporting the issue.

! fix, nfacctd: in NF_mpls_vpn_id_handler() double ntohl() calls

were applied for the case of 'vrfid'-encoded mpls_vpn_rd field.

! fix, sfacctd: wrong ethertype set for VLAN-tagged, MPLS-labelled

IPv6 traffic. Impacting BGP resolution among others. Thanks to

Jeremiah Millay ( @floatingstatic ) for his help resolving the

problem.

! fix, BGP, BMP daemons: parsing improvements: added a check for

BGP Open message and BGP Open Options lengths. Strengthened

parsing of Peer Up, Route Monitoring and Peer Down v4 messages.

! fix, BGP, BMP daemon: when using Avro encoding and Avro Schema

Registry, attempt to reconnect if serdes schemas are voided.

Also now checking for serdes schema definitions before doing a

serdes_schema_serialize_avro() to avoid triggering a SEGV.

Finally improved serdes logging.

! fix, BGP, Streaming Telemetry daemons: in daemon logs, summary

counters for amount of tables / entries dumped were wrong.

! fix, BGP daemon: distinguish among null and zero value AIGP

and Prefix SID attributes. Same applies for Local Preference

and MED attributes.

! fix, BMP daemon: resolved a memory leak in bgp_peers_free().

Thanks to Peter Pothier ( @pothier-peter ) for his patch. Also

resolved a leak caused by an invalid BGP message contained in a

BMP Route Message v4.

! fix, BMP daemon: correctly setting peer_ip and peer_tcp_port

JSON fields for Term messages. Also the correct bmp_router

value when bmp_daemon_parse_proxy_header feature is enabled.

! fix, BMP daemon: several encoding issues when using Apache Avro

ie. u_int64_t now correctly encoded with avro_value_set_long(),

certain u_int32_t fields switched to avro_value_set_long() due

to lack of unsignedness in Avro encoding, improved various

aspectes of Avro-JSON format output, etc.

! fix, pmtelemetryd: wrong parsing of pm_tfind() output was

leading to mistaken data attribution of UDP-based peers (always

first peer to connect was being picked).

! fix, pmtelemetryd: when set, the pidfile config directive was

not being correctly honoured.

! fix, RPKI: the RTR PDU element for maxLength is uint8, therefore

it might have been possible to transmit incorrect RTR data.

Thanks to Job Snijders ( @job ) for his patch.

! fix, SQL plugins: amended the text composition of SQL queries

that are involving latitude and longitude keys.

! fix, MySQL plugin: check for 'unix:' prefix string only when a

sql_host configuration directive is specified.

! fix, nfprobe: modernized Application Information export. Until

the previous release pmacct was adhering to aging NBAR model

whereas now NBAR2 has been implemented. Thanks to Rob Cowart

( @robcowart ) for helping out resolving this issue.

! fix, tee plugin: restored usefulness of tee_source_ip which was

broken in 1.7.6. Thanks to Jeremiah Millay ( @floatingstatic )

for reporting the issue.

! fix, maps_index: indexing of mpls_pw_id was broken. Also now,

when the feature is enabled, actual data is being referenced in

the index structure instead of creating a copy of it; thanks to

Sander van Delden ( @SanderDelden ) for reporting the memory

leak that was resulting from the copy.

! fix, kafka_common.c: solved memory leak in p_kafka_set_topic()

when Kafka session was getting in down state. Many thanks to

Peter Pothier ( @pothier-peter ) for nailing the issue.

! fix, net_aggr.[ch]: when a networks_file is specified in the

config, gracefully handle max memory structure depth; added

also de-duplication of entries.

! fix, pmacct-defines.h: if PCAP_NETMASK_UNKNOWN is not defined,

ie. in libpcap < 1.1.0, let's define it.

! fix, SO_REUSEPORT feature was being restricted to Linux only in

previous releases: now it has been unlocked to all other OS that

do support the feature.

! fix, split SO_REUSEPORT and SO_REUSEADDR setsockopt() calls.

Thanks to @eduarrrd for reporting and resolving the issue.

! fix, several code warnings catched gcc9 and clang.

- Obsoleted sql_history_since_epoch, pre_tag_map_entries and

refresh_maps configuration directives.

1.7.6 -- 07-02-2021

+ Added dependency to libcdada in an effort to streamline basic

data structures needed for everyday coding. All new structures

will make use of libcdada, old ones will be ported over time.

Libcdada offers basic data structures in C: ie. list, set, map/

hash table, queue and is a libstdc++ wrapper. Many thanks to

Marc Sune ( @msune ) for his work with libcdada and his enormous

help facilitating the integration.

+ BGP daemon: added suppport for Accumulated IGP Metric Attribute

(AIGP) and Label-Index TLV of Prefix-SID Attribute.

+ BGP daemon: added SO_KEEPALIVE TCP socket option (ie. to keep the

sessions alive via a firewall / NAT kind of device). Thanks to

Jared Mauch ( @jaredmauch ) for his patch.

+ BGP daemon: if comparing source TCP ports among BGP peers is

being enabled (config directive tmp_bgp_lookup_compare_ports),

print also BGP Router-ID as distinguisher as part of log/dump

output.

+ BMP daemon: added support for HAProxy Proxy Protocol Header in

the first BMP message in order to determine the original sender

IP address and port. The new bmp_daemon_parse_proxy_header config

directive enables the feature. Contribution is by Peter Pothier

( @pothier-peter ).

+ BMP daemon: improved support and brought implementation on par

with the latest drafting efforts at IETF wrt draft-cppy-grow-bmp-

path-marking-tlv, draft-xu-grow-bmp-route-policy-attr-trace,

draft-ietf-grow-bmp-tlv and draft-lucente-grow-bmp-tlv-ebit.

+ BMP daemon: added 'bgp_agent_map' equivalent feature for BMP.

+ nfacctd, nfprobe plugin: added support for collection and export

of NetFlow/IPFIX data over Datagram Transport Layer Security (in

short DTLS). The feature depends on the GnuTLS library.

+ nfacctd: added support for deprecated NetFlow v9 IE #104

(layer2packetSectionData) as it is implemented for NetFlow-lite

on Cisco devices. Reused code from IPFIX IE #315.

+ nfacctd: added support for MPLS VPN RD IE #90. This comes in two

flavours both found across vendor implementations: 1) IE present

in flow data and 2) IE present in Options data as a lookup from

IE #234 (ingressVRFID) and #235 (egressVRFID).

+ nfacctd: added a new timestamp_export aggregation primitive to

record the timestamp being carried in the header of NetFlow/IPFIX

messates (that is, the time at which the export was performed).

+ nfprobe plugin: added support for ICMP/ICMPv6 information as part

of the NetFlow/IPFIX export. The piece of info is encoded in the

destination port field as per the current common understandings

across vendors. As a result of that, the 'dst_port' primitive is

to be part of the aggregation method in order to leverage this

feature.

+ MySQL plugin: introduced support to connect to a MySQL server

via UNIX sockets.

+ tee plugin: added crc32 hash algorithm as a new balancing option

for nodes in the receiving pool. It hashes original exporter IP

address against a crc32 function. Thanks to @edge-intelligence

for the contribution.

+ Massive improvements carried out on the Continuous Integration

(CI) side, ie. to ensure better quality of the code, and on the

containerization side by offering official stable / bleeding edge

Docker images. Huge thanks to Marc Sune ( @msune ) to make all of

this possible.

! fix, BGP daemon: re-worked internal structuring of 'modern' BGP

attributes: for the sake of large-scale space optimization

certain attributes are confined in a separate (less used)

bgp_info_extra structure.

! fix, BGP daemon: improved support for BGP ADD-PATH, ie. made it

per Address-Family rather than global. Also comparisons upon

doing route looup were improved and normalized.

! fix, BGP daemon: use split buffers for recv and send functions

of the BGP x-connects feature. Also improved validation when

processing a bgp_daemon_xconnect_map.

! fix, BGP daemon: when using BGP x-connects, close unused file

descriptors in bgp_peer_xconnect_init() in order to avoid

quickly reaching the maximum amount of allowed open descriptors

in case of BGP flaps.

! fix, BGP daemon: trigger a log message for a missing entry while

processing bgp_daemon_xconnect_map in bgp_peer_xconnect_init().

! fix, BGP daemon: enabled log notifications (that is, log anti-

spam measure) upon reaching limit of allowed BGP peers.

! fix, BGP daemon: ecommunity_ecom2str(), first thing make sure

that the destination size is enough! Missing this did cause some

SEGVs due to heap corruption. Thanks to Chris Danis ( @cdanis )

for his help resolving the issue.

! fix, BGP daemon: solved a memory leak in aspath_make_str_count()

by returning result from aspath_make_empty(), if any. Thanks very

much to Peter Pothier ( @pothier-peter ) for his contribution.

! fix, BMP daemon: several encoding issues when using Apache Avro

ie. missing conditional branching, wrong field names, etc. Thanks

also to Raphael Barazzutti ( @rbarazzutti ) for several of those

fixes.

! fix, BMP daemon: throw an error for any issues (error or zero

length) related to the BGP Update PDU parsing; also added marker

and length checks for BGP Open PDU in Peer Up messages.

! fix, BMP daemon: both timestamp of the BMP event and its

arrival at the collector are now recorded and printed out

separately; before they were wrongly muxed on one single field

making it uncertain for the user what was the time reference.

! fix, BMP daemon: correctly print Peer Distinguisher for Route

Monitoring messages. Also improved BMP lookup comparisons in

order to factor in Peer Distinguisher if any.

! fix, BMP daemon: print 'is_in' boolean for Adj-Rib-In data

instead of having it implicit. Also print 'is_post' for Post-

Policy Adj-Rib-In data.

! fix, BMP daemon: upon receipt of a Termination message, do

proactively close the TCP session.

! fix, nDPI: newer versions of the library (ie. >= 3.2) require

calling ndpi_finalize_initialization() somewhere after the

detection module init finished. Contribution is from Toni Uhlig

( @lnslbrty ).

! fix, pmacctd: link checks were being mistakenly skipped when

reading from a pcap_savefile. Also now if a selected aggregation

primitive is unsuitable for a given Layer2, it is simply cleared

(with an info message issued) instead of making the daemon bail

out.

| fix, print plugin: bail the plugin out if its output was set to

stdout while the daemon was started as daemonized.

! fix, PostgreSQL plugin: in PG_compose_conn_string() allow any

intersection of host, port and cafile options.

! fix, nfprobe plugin: changed default export version from NetFlow

v5 to IPFIX.

! fix, sfprobe plugin: FreeBSD was complaining of errno 22 (Invalid

argument) upon sendto().

! fix, tee plugin: replication of IPv6 packets has been now tested

working. Previously the output message size was obviously encoded

wrongly and the checksum (mandatory piece of info to fill in IPv6,

contrary to IPv4 where it is optional) was not being computed.

! fix, kafka_common.c: improved p_kafka_check_outq_len() error log

message to report the amount of elements have been successfully

processed in order to better assess impact and dynamics of the

problem when inspecting logs.

! fix, net_aggr.c: if networks_file_filter is set to true, don't

add a default route to the table.

! fix, cfg.c: throw error if config file is not a regular file.

! fix, compiling against gcc10: renamed some variables and unified

declaration of others in order to be more friendly to the new

version of gcc. Also fixed several code warnings catched gcc8.

- Removed the IP prefix label feature that was enabled via the

--enable-plabel configure script switch.

1.7.5 -- 17-06-2020

+ pmacct & Redis: pmacct daemons can now connect to a Redis cache.

The main use-case currenly covered is: registering every stable

daemon component in a table so to have, when running a cluster

comprising several daemons / components, an olistic view of what

is currently running and where; shall a component stop running

or crash it will disappear from the inventory.

+ BMP daemon: as part of the IETF 107 vHackaton, preliminar support

for draft-xu-grow-bmp-route-policy-attr-trace and draft-lucente-

grow-bmp-tlv-ebit was introduced. Also added support for Peer

Distinguisher field in the BMP Per-Peer Header.

+ BMP daemon: added support for reading from savefiles in libpcap

format (pcap_savefile, pcap_savefile_delay, pcap_savefile_replay,

pcap_filter) as an alternative to the use of bmp_play.py.

+ BMP daemon: re-worked, improved and generalized support for TLVs

at the end of BMP messages. In this context, unknown Stats data

is handled as a generic TLV.

+ BMP daemon: added SO_KEEPALIVE TCP socket option (ie. to keep the

sessions alive via a firewall / NAT kind of device). Thanks to

Jared Mauch ( @jaredmauch ) for his patch.

+ nfacctd, nfprobe plugin: added usec timestamp resolution to IPFIX

collector and export via IEs #154, #155. For export, this can be

configured via the new nfprobe_tstamp_usec knob.

+ nfacctd: new nfacctd_templates_receiver and nfacctd_templates_port

config directives allow respectively to specify a destination

where to copy NetFlow v9/IPFIX templates to and a port where to

listen for templates from. If nfacctd_templates_receiver points to

a replicator and the replicator exports to nfacctd_templates_port

of a set of collectors then, for example, it gets possible to share

templates among collectors in a cluster for the purpose of seamless

scale-out.

+ pmtelemetryd: in addition to existing TCP, UDP and ZeroMQ inputs,

the daemon can now read Streaming Telemetry data in JSON format

from a Kafka broker (telemetry_daemon_kafka_* config knobs).

+ pmgrpcd.py: Use of multiple processes for the Kafka Avro exporter

to leverage the potential of multi-core/processors architectures.

Code is from Raphael P. Barazzutti ( @rbarazzutti ).

+ pmgrpcd.py: added -F / --no-flatten command-line option to disable

object flattening (default true for backward compatibility); also

export to a Kafka broker for (flattened) JSON objects was added (in

addition to existing export to ZeroMQ).

+ nDPI: introduced support for nDPI 3.2 and dropped support for all

earlier versions of the library due to changes to the API.

+ Docker: embraced the technology for CI purposes; added a docker/

directory in the file distribution where Dockerfile and scripts to

build pmacct and dependencies are shared. Thanks to Claudio Ortega

( @claudio-ortega ) for contributing his excellent work in the area.

! fix, pmacctd: pcap_setdirection() enabled and moved to the right

place in code. Libpcap tested for function presence. Thanks to

Mikhail Sennikovsky for his patch.

! fix, pmacctd: SEGV has been detected if passing messages with an

unsupported link layer.

! fix, uacctd: handle non-ethernet packets correctly. Use mac_len = 0

for non-ethernet packets in which case a zeroed ethernet header is

used. Thanks to @aleksandrgilfanov for his patch.

! fix, BGP daemon: improved handling of withdrawals for label-unicast

and mpls-vpn NLRIs.

! fix, BGP daemon: improved decoding of MPLS labels by passing it via

a "0x%02x%02x%01x" formatter. In the past some labels may have been

printed out incorrectly.

! fix, BGP daemon: decoding origin field correctly. Thanks to Peter

Pothier ( @pothier-peter ) for his patch.

! BGP Looking Glass: LG code was moved from pmbgpd to pmacct libbgp

so to be re-used in other components (ie. BGP thread of a flow

daemon, BMP daemon, etc.).

! fix, BMP daemon: timestamps for Route Monitoring message were set

to the current time rather than timestamp in the Per-Peer Header.

Thanks to Peter Pothier ( @pothier-peter ) for reporting the issue.

! fix, BMP daemon: V flag test to allow IPv6 prefixes/peers was not

correct (details in PR #362 on GitHub). Thanks @bcavns01 for his

patch.

! fix, BGP, BMP daemons: modified approach to message segmentation

by ensuring reading exactly one message at a time (instead of the

previous unaligned approach that was aiming to fill the available

read buffer up) and waiting for all segments be available before

processing a message (instead of the previous greedy approach that

was trying to parse also partial messages).

! fix, RPKI daemon: in rpki_prefix_lookup_node_match_cmp() prevent

invalid results to over-write a previously computed valid one.

! fix, pmtelemetryd: recv() does include a MSG_WAITALL option to make

sure all data is available before processing a given message; now

n alarm() syscall is introduced so to prevent stalls due to bogus /

incomplete data.

! fix, tee plugin: Tee_init_socks() now does not overwrite previously

computed address length anymore: issues were repoted with IPv6 IPv4-

mapped addresses.

! fix, nfprobe plugin: take into account ingress/egress interfaces,

if available, when comparing flows. Thanks to Mikhail Sennikovsky

for his patch.

! fix, pretag.c: incorrect string termiantion in pretag_copy_label()

was making the daemon SEGV upon certain conditions when defining

set_label statements in a pre_tag_map.

! fix, pretag_handlers.c: pretag_copy_label() instead of memcpy() in

pretag_label_handler() to not borrow reference to label value and

consequently lead to SEGV.

! fix, zmq_common.c: missing variable init in p_zmq_zap_handler() was

giving troubles with gcc7 compiler optimizations. Thanks to Yuri

Lachin ( @yuyutime ) for his support resolving this bug.

! fix, sql_common.c: print custom primitives with hex semantics among

quotes. Hexs require a char type defined in the SQL table schema.

! fix, addr.c: passing right aligned argument to ip6_addr_cmp() in

host_addr_mask_cmp(). This was found to impact selective replication

of IPv6 flows basing on src_net and/or dst_net primitives in the

pre_tag_map.

! fix, several code warnings catched by gcc7 and gcc8. Also several

functions were renamed to avoid namespace conflicts with linked

libraries.

- Obsoleted --enable-64bit knob which was already defaulting to true

for some releases already.

- Obsoleted savefile_wait config knob (pcap_savefile_wait must be

used instead).

1.7.4p1 -- 09-02-2020

! fix, pre_tag_map: a memory leak in pretag_entry_process() has been

introduced in 1.7.4. Thanks to Fabien Vincent and Olivier Benghozi

for their support resolving the issue.

1.7.4 -- 31-12-2019

+ Released pmgrpcd.py v3: a Streaming Telemetry collector and decoder

for multi-vendor environments written in Python3. It supports gRPC

transport along with Protobuf encoding as input and can output to

Kafka with Avro encoding. Output to files and JSON encoding is

currently supported sending data via ZMQ to pmtelemetryd first. It

was tested working with data input from Cisco and Huawei routers

and v3 replaces v2. Thanks to the Streaming Telemetry core team:

Matthias Arnold ( @tbearma1 ), Camilo Cardona ( @jccardonar ),

Thomas Graf ( @graf3 , @graf3net ), Paolo Lucente ( @paololucente ).

+ Introduced support for the 'vxlan' VXLAN/VNI primitive in all traffic

daemons (NetFlow/IPFIX, sFlow and libpcap/ULOG). Existing inner tunnel

primitives (ie. tunnel_src_host, tunnel_dst_host, tunnel_proto, etc.)

have been wired to the VXLAN decoding and new ones (tunnel_src_mac,

tunnel_dst_mac, tunnel_src_port, tunnel_dst_port) were defined.

+ BMP daemon: added support for Peer Up message namespace for TLVs

(draft-ietf-grow-bmp-peer-up) and also support for Route Monitor

and Peer Down TLVs (draft-ietf-grow-bmp-tlv).

+ BGP, BMP daemons: in addition to existing JSON export, data can now

be exported in Apache Avro format. There is also support for the

Confluent Schema Registry.

+ Introduced support for JSON-encoded Apache Avro encoding. While the

binary-encoded Apache Avro is always recommended for any production

scenarios (also to optionallly leverage Confluent Schema Registry

support), JSON-encoded is powerful for testing and troubleshooting

scenarios.

+ sfprobe plugin: added support for IPv6 transport for sFlow export.

sfprobe_agentip is an IP address put in the header of the sFlow

packet. If underlying transport is IPv6, this must be configured to

an IPv6 address.

+ zmq_common.[ch]: Improved modularity of the ZMQ internal API and

decoupled bind/connect from push/pull and pub/sub; also improved

support for inproc sockets. All to increase the amount of use-cases

covered by the API.

+ bgp_peer_src_as_map: added 'filter' key to cover pmacctd/uacctd use

cases.

+ nfprobe, sfprobe plugins: introduced [sn]fprobe_index_override to

override ifindexes dynamically determined (ie. by NFLOG) with values

computed by [sn]fprobe_ifindex.

+ MySQL, PostgreSQL plugins: added support for SSL/TLS connections by

specifying a CA certificate (sql_conn_ca_file).

+ Kafka, AMQP plugins: amqp_markers and kafka_markers have now been

properly re-implemented when output encoding is Avro using an own

Avro schema (instead of squatting pieces of JSON in the data stream

for the very purpose).

+ print plugin: introduced print_write_empty_file config knob (true,

false) to create an empty output file when there are no cache entries

to purge. Such behaviour was present in versions up to 0.14 and may

be preferred by some to the new >= 1.5 versions behaviour. Thanks to

Lee Yongjae ( @setup74 ) for the contribution.

! fix, signals.c: signals handling has been restructured in order to

block certain signals during critical sections of data processing.

Thanks to Vaibhav Phatarpekar ( @vphatarp ) for the contribution.

! fix, signals.c: slimmed reload() signal handler code and moved it to

a synchronous section. The handler is to reset logging output to

files or syslog. Thanks to Jared Mauch ( @jaredmauch ) for his

support resolving this.

! fix, pmbgpd, pmbmpd and pmtelemetryd daemons: added extra signals

handling (SIGINT, SIGTERM, SIGCHLD) consistently to traffic daemons.

! fix, BGP daemon: withdrawals of label-unicast (support introduced in

1.7.3) and mpls-vpn NLRIs did fail to parse in release 1.7.3 and

were silently discarded.

! fix, nfacctd: wired (BGP, BMP, ISIS, etc) lookups to NetFlow (Secure)

Event Logging (NEL/NSEL).

! fix, pmtelemetryd: re-implemented a decoder for so-called Cisco v1

Streaming Telemetry proprietary header over UDP/TCP streams.

! fix, pmtelemetryd: improved sanitization of input JSON objects by

also checking for isspace() other than isprint() for pretty-printed

objects.

! maps_index: optimized lookups, improved debugging output upon loading

indexes.

! fix, tee plugin: overwriting computed IP address length with socket

container length was found to prevent output data on some BSDs.

! fix, kafka_common.c: if taking the p_kafka_close() route, ensure to

return and not perform any further polling in order to avoid SEGVs.

! fix, BMP daemon: incorrect decoding of type was preventing correct

logging of Init and Term messages extra info. Also in Term messages

TLV data was incorrectly consumed twice triggering length check

warnings.

! fix, BMP daemon: added checks for successful BGP PDU parsing in both

Peer Up (BGP OPEN) and Route Monitor (BGP UPDATE) messages.

! fix, BMP daemon: improved length checks and making sure that strings

potentially non null-terminated are now terminated. Also TLV-related

code has been refactored.

! fix, pmbgp.py: the example client for BGP Looking Glass was migrated

to Python3: thanks to @brusilov for the contribution.

! fix, nfacctd: if src_port or dst_port primitives are selected, enable

IP fragment handling. Needed to process L4 of IPFIX IE #351.

! fix, nfv9_template.c: correct handling of variable-length IPFIX

fields. Thanks to Nimrod Mesika ( @nimrody ) for the contribution.

! fix, PostgreSQL plugin: ABSTIME was replaced with to_timestamp() in

queries as support for ABSTIME was dropped as of PostgreSQL 12. Many

thanks to Manuel Mendez ( @mmlb ) for the contribution.

! fix, PostgreSQL plugin: SEGVs were observed when the queue of pending

queries was non-empty (ie. nfacctd_time_new set to false, default);

thanks to Guo-Wei Su ( @nansenat16 ) for the contribution.

! fix, cfg_handlers: [sn]facctd_disable_checks, nfacctd_disable_opt_

scope_check could not be properly set to false.

! fix, sql_common.c: src_host_coords and dst_host_coords primitives

have been correctly spaced in SQL queries. Also float values are now

quoted. Finally, sampling_direction primitive is encoded correctly.

! fix, kafka plugin: if kafka_avro_schema_registry is in use, subject

name is aligned to Kafka topic name (if topic is not dynamic).

! fix, pretag.c: when using 'label', store the label string in the

heap (instead of the stack). Thanks to Raphael P. Barazzutti

( @rbarazzutti ) for the contribution.

! fix, pretag.c: JEQ labels are now correctly free() during init upon

map reload.

! fix, zmq_common.c: missing variable init in p_zmq_zap_handler() was

causing plugin_pipe_zmq operations to fail on certain compilers (ie.

gcc7). Thanks to Yuri Lachin ( @yuyutime ) for his support.

! fix, cfg_handlers.c: reviewed handling of parsed 'zero' value for

several config directives.

! fix, countless code warnings when enabling -Wall (--enable-debug);

also included -Wall in Continuous Integration tests. Restructured

globals, header inclusions, function prototypes definition, etc.

Many thanks to Marc Sune ( @msune ) for all his efforts.

! fix, configure.ac: evaluation of --enable-debug pushed to the end of

the script so to not interfere with tests (ie. alignment, endianess,

etc.).

- BMP daemon: retired support for draft-hsmit-bmp-extensible-routemon-

-msgs.

- AMQP plugin: obsoleted amqp_avro_schema feature (which includes

amqp_avro_schema_routing_key and amqp_avro_schema_refresh_time keys

config keys). Avro schemas can now only be written to files.

1.7.3 -- 16-05-2019

+ Introduced the RPKI daemon to build a ROA database and check prefixes

validation status and coverages. Resource Public Key Infrastructure

(RPKI) is a specialized public key infrastructure (PKI) framework

designed to secure the Internet routing. RPKI uses certificates to

allow Local Internet Registries (LIRs) to list the Internet number

resources they hold. These attestations are called Route Origination

Authorizations (ROAs). ROA information can be acquired in one of the

two following ways: 1) importing it using the rpki_roas_file config

directive from a file in the RIPE Validator format or 2) connecting

to a RPKI RTR Cache for live ROA updates; the cache IP address/port

being defined by the rpki_rtr_cache config directive (and a few more

optional rpki_rtr_* directives are available and can be reviwed in

the CONFIG-KEYS doc). The ROA fields will be populated with one of

these five values: 'u' Unknown, 'v' Valid, 'i' Invalid no overlaps,

'V' Invalid with a covering Valid prefix, 'U' Invalid with a covering

Unknown prefix. Thanks to Job Snijders ( @job ) for his support and

vision.

+ Introducing pmgrpcd.py, written in Python, a daemon to handle gRPC-

based Streaming Telemetry sessions and unmarshall GPB data. Code

was mostly courtesy by Matthias Arnold ( @tbearma1 ). This is in

addition (or feeding into) pmtelemetryd, written in C, a daemon to

handle TCP/UDP-based Streaming Telemetry sessions with JSON-encoded

data. Thanks to Matthias Arnold ( @tbearma1 ) and Thomas Graf for

their support and contributing code.

+ pmacctd, uacctd: added support for CFP (Cisco FabricPath) and Cisco

Virtual Network Tag protocols. Both patches were courtesy by Stephen

Clark ( @sclark46 ).

+ print plugin: added 'custom' to print_output. This is to cover two

main use-cases: 1) use JSON or Avro encodings but fix the format of

the messages in a custom way and 2) use a different encoding than

JSON or Avro. See also example in examples/custom and new directives

print_output_custom_lib and print_output_custom_cfg_file. The patch

was courtesy by Edge Intelligence ( @edge-intelligence ).

+ Introducing mpls_pw_id aggregation primitive and mpls_pw_id key in

pre_tag_map to filter on signalled L2 MPLS VPN Pseudowire IDs.

+ BGP daemon: added bgp_disable_router_id knob to enable/disable BGP

Router-ID check, both at BGP OPEN time and BGP lookup. Useful, for

example, in scenarios with split BGP v4/v6 AFs over v4/v6 transports.

+ BGP, BMP daemons: translate origin attribute numeric value into IGP

(i), EGP (e) and Incomplete (u) strings.

+ plugins: added new plugin_exit_any feature to make the daemon bail

out if any (not all, which is the default behaviour) of the plugins

exits.

+ maps_index: improved selection of buckets for index hash structure

by picking the closest prime number to the double of the entries of

the map to be indexed in order to achieve better elements dispersion

and hence better performances.

+ nfacctd: added support for IPFIX templateId-scoped (IE 145) sampling

information.

+ pmacctd, uacctd, sfacctd, nfacctd: added a -M command-line option to

set *_markers (ie. print_markers) to true and fixed -A command-line

option to set print_output_file_append to align to true/false.

! fix, BGP, BMP, Streaming Telemetry daemons: improved sequencing of

dump events by assigning a single sequence number per event (ie. for

streaming pipeline scenarios in order to reduce correlation with

dump_init/dump_close messages). Also amount of record dumped was

added to the close message.

! fix, BGP, BMP, Streaming Telemetry daemons: removed hierarchical

json_decref() since json_object_get() borrows reference. This was

occasionaly leading to SEGVs.

! fix, uacctd: dynamically allocate jumbo_container buffer size as

packets larger than 10KB, previous static allocation, would lead to

crashes.

! fix, nfacctd: wired (BGP, BMP, ISIS, etc.) lookups to the NEL/NSEL

use-case.

! fix, nfacctd: search for IE 408 (dataLinkFrameType) was leading to

SEGVs. Also improved handling of variable-length IPFIX templates.

! fix, BMP daemon: solved an occasional truncation of the last message

in a packet.

! fix, BGP daemon: when processing bgp_daemon_md5_file, ipv4 addresses

were incorrectly translated to ipv4-mapped ipv6 ones as a result of

which TCP-MD5 hashes were not correctly bound to sockets.

! fix, BGP daemon: improved label-unicast and mpls-vpn SAFIs handling

(some bogus messages, multiple labels, etc.).

! fix, BGP daemon: introduced PREFIX_STRLEN to make enough room for

prefix2str() calls (before unsufficient INET6_ADDRSTRLEN was used).

! fix, BMP daemon: improved handling of ADD-PATH capability.

! fix, plugins: an incorrect evaluation in P_cache_attach_new_node did

make possible to buffer overrun in plugins cache allocation. This was

found related to a "[..]: Assertion `!cache_ptr->stitch' failed."

daemon bail-out message.

! fix, plugins: if pidfile directive was enabled, exit_gracefully() was

mistakenly deleting the plugin pidfile when called by a child process

(ie. writer, dumper, etc.).

! fix, plugins: when taking exit_gracefully(), if the process is marked

as 'is_forked', just exit and don't perform extra ops in exit_all()

or exit_plugin().

! fix, plugins: re-evaluate dynamic tables/files name if *_refresh_time

is different than *_history period.

! fix, SQL plugins: a missing 'AND' was making SQL statements related

to src_host_coords and dst_host_coords fail.

! fix, GeoIPv2: if no match is returned by libmaxminddb, return O1 code

(Other Country) instead of a null value.

! fix, flow_to_rd_map: mpls_vpn_id was not working when maps_index was

enabled. Also partly re-written mpls_vpn_id handler.

! fix, nfprobe plugin: serialize_bin() function introduced for correct

serialization of custom primitives defined with 'raw' semantics.

! fix, PostgreSQL plugin: testing for presence of PQlibVersion() in

libpq to prevent compiling issues (ie. on CentOS 6).

! fix, MySQL plugin: including mysql_version.h to compile successfully

against newer MariaDB releases.

! fix, nDPI classification: send log message if 'class' primitive is

selected but nDPI is not compiled in; also updated code to follow

API changes in versions >= 2.6 of the library. Dropped support for

versions < 2.4.

! fix, sfprobe plugin: added (and documented) conditional for optional

export of classification info.

! fix, aggregate_primitives: field_type is now also allowed for pmacctd

and uaccd daemons so that it can be used for NetFlow v9/IPFIX export

(nfprobe plugin) purposes.

! fix, pre_tag_map: if no 'ip' keyword is specified, an entry of the

map gets recirculated in order to be set for both v4 and v6 maps. If