Config: Flow Archive Control Options

Comprehensive user manual for CySight Ai-Driven Network and Endpoint Detection and Response (NDR, EDR) Forensics and Application Performance Monitoring (APM)

Config: Flow Archive Control Options

CySight Next Generation NetFlow Analyzer

CySight is a scalable Next Generation NetFlow Analyzer providing access to multiple value propositions from the scalable, granular and flexible manner in which it collects and displays flow analytics.

CySight can scale by license key and/or by using many different flow tuning methods to mould flows to suit the specific or hybrid Performance analytic, Anomaly Detection, Peering, Forensics or Billing needs and to function within hardware and operating system resources provided.

Requirements for flow analytics tend to fall into four main areas with various degrees of emphasis and CySight uniquely provides access to these benefits from the smallest to the largest network environments. CySight was architected out-of-the-box to deliver;
  • Scalable Performance Analytics (Bandwidth Utilization and Visibility),
  • Granular Forensics (High visibility usually for deep Root Cause Analysis, Security and Compliance),
  • Intelligent Behavior Anomaly Detection (Discovery of traffic issues caused by attacks, abuse of bandwidth, policy such as p2p, onion routing, email spamming, nefarious traffic and informative; e.g. video streaming and voip analytics) and
  • Accountable Billing/Trending (ROI by usage billing or trending of resource usage e.g. Month by month growth of interface/service/resource usage etc).

Customers can generally expect CySight to handle between 9M to 20M/minute on a single server running in top 5000 flows per minute per device archival mode and between 1M to 5M per minute running in Full archival mode. Actual performance depends on the type of flows, the flow variance and environment and system hardware provided and operating system and configurations.

Customers can also benefit from sending multiple flows to different CySight instances with different granularity of data.

CySight offers a number of options to allow data to be recorded at a full or lesser granularity balanced to the needs and system resources provided for collection.

Also please read Scalable Netflow - 3 key questions to ask to appreciate why CySight scales well beyond any other flow based network monitoring system in the market today whilst retaining the deepest granularity available and providing the most flexible toolset to access the information to address multiple requirements.

CySight Flow Rates and Flow Management

License control

The primary and simplest method of granularity control is by using the allowed configurations and defaults provided by the license key purchased or provided for evaluation.

CySight licensing is offered as a Device-based license or an Interface based license. Device-based licensing allows the monitoring of unlimited interfaces on a specified exporter device. Interface based licensing allows the monitoring of a specified number of Interfaces that can be situated on multiple devices.

For Device based licensing there are also three sub-base license modes; Performance, Intelligence, and Compliance.
  • The Performance license is fixed to allow archival of up to the top 500 flows per device per minute
  • The Intelligence license is fixed to allow archival of up to 5000 flows per device per minute.
  • The Compliance license is fixed to allow archival of up to Unlimited flows per device per minute.
The Interface based license is based on the maximum number of flows per device which is set in the license key. The maximum number flows per device per minute (TopN) can be between 100 and 50000.
  • In a perfect world where all interfaces were made equal a TopN key of 5000 would equate to 500 flows per interface per minute for 10 interfaces and a TopN key of 50000 would equate to 5000 flows per interface per minute for 10 interfaces. In real life, Network traffic is by its nature bursty and therefore the Interface license will archive more flows for a busier interface than a smaller one because the flows per Interface per minute are allocated on a first-come, first serve basis up to the maximum of the TopN.
  • If there are 9 or fewer Interfaces on a Device then the actual available TopN will be the ((TopN / 10) * Number of Interfaces). Eg if TopN provided is 5000 and Number of interfaces is 9 then the actual Topn=((5000/ 10)* 9)=4500.
Whatever the archive management model utilized CYSIGHT WILL ALWAYS COLLECT ALL THE FLOWS irrespective of the archive settings selected.
The Default Archived Flows can be changed from the default Top 5000 to the minimum or maximum flows archived per minute allowed by the license to suit the depth of granularity required.


Default Archived Flows per Minute Setting

Prior to sending any new flows to CySight the Default Archived Flows can be configured to tune all flows seen for the first time. If flows are already being sent to CySight then the Default Archived Flows need to be configured on a per device basis (discussed below).
The Compliance license mode is set by default to archive the Top 5000 on installation and it is up to the Administrator to select and confirm that the allocated hardware and operating system resources are sufficient to archive and analyze Unlimited flows per minute.
Sorted by

By Default CySight sorts flows 3 ways to find the top offenders in each category, the top Byte consumers are always available and Top packet generators and Top flow generators are calculated in order to identify DDoS, p2p and other nefarious traffic. The higher the granularity archived in the Default Archived Flows the more forensic details will be retained and the more accurately a network can be base-lined to detect traffic anomalies along with the ability to follow every conversation over time if desired.

Aggregated Flow Policy

The Aggregated Flow policy enables a quick way to choose which fields in the aggregated flows will be maintained.

By Default Application is the only field set to aggregate. Application refers to the Applications as defined in the Application / NBAR / NBAR2 Mapping. This is because applications can be very spammy in some environments. If your resources are sufficient then unselect it and click "Confirm" to activate the new policy. Keep an eye on your Collection Status as well as your disk space when deselecting options as they will force more information to be archived when running in Performance (TopN) mode.

ASN can also become quite noisy if origin_as has been enabled on the flow exporter. We recommend in huge environments to either change the exporter to peer_as or select ASN in the Aggregated Flow Policy to manage.

The following fields are allowed to be aggregated"
  • ToS
  • Business Group
  • ASN
  • Application
  • VLAN
  • Next Hop
  • BGP Next Hop
Top Archived Flows

The default flow aggregation can be overridden for each individual Device. This can be set to anywhere from Unlimited (subject to licensing) and below for each device.


Flow Retention Rate

Although CySight has superior collection scalability there are still time where the flows are excessive for the hardware provided. In this case you can set the flow retention rate to archive every Nth flow and have the remaining flows be governed by the Aggregated Flow Policy.

For example you may have selected Unlimited archival mode because you want the highest granularity to be retained. For your process you consider a Sampling of the data would be more important than the top of the data because your needs are more security orientated than bps orientated. However, you soon recognize that the size of the flows is so large that you need another means to manage the flow in this case Flow Retention Rate is for you.