Parquet Schema

Rockfish exports flow data in Apache Parquet format with IPFIX-compliant field naming. The schema varies by license tier.

Schema by Tier

Community Schema (44 Fields)

Basic flow capture with core network fields.

Basic Schema (54 Fields)

Community schema + nDPI application detection + GeoIP (country, city, ASN).

GeoIP fields:

nDPI fields:

Professional Schema (60 Fields)

Basic schema + GeoIP AS organization names and nDPI fingerprinting.

Additional GeoIP fields (AS organization):

nDPI fingerprint fields:

Enterprise Schema (63+ Fields)

Professional schema + anomaly detection and ML predictions.

Anomaly detection fields:

File Naming

S3 Path Structure

With Hive partitioning enabled:

s3://<bucket>/<prefix>/v1/year=YYYY/month=MM/day=DD/*.parquet
s3://<bucket>/<prefix>/v2/year=YYYY/month=MM/day=DD/*.parquet

Field Descriptions

Flow Identification

• flowid: Unique UUID for deduplication and correlation
• obname: Observation domain name (sensor identifier)

Timing

• stime/etime: Timestamps with microsecond precision, UTC
• dur: Duration in milliseconds
• rtt: Estimated TCP round-trip time

Network Addresses

• saddr/daddr: IPv4 or IPv6 addresses as strings
• sport/dport: Port numbers (0 for non-TCP/UDP)
• smac/dmac: MAC addresses in standard notation

Traffic Volumes

• spkts/dpkts: Packet counts per direction
• sbytes/dbytes: Byte counts per direction
• pcr: Producer-consumer ratio: (sent-recv)/(sent+recv)

TCP Flags

• iflags: Initial TCP flags (SYN, ACK, etc.)
• uflags: Union of all flags seen in flow

Payload Analysis

• sentropy/dentropy: Shannon entropy (0-255)

  • 230: Likely encrypted/compressed

  • ~140: English text
  • Low: Sparse or zero-padded

Flow Termination

• reason: Why the flow ended
  • idle: Idle timeout
  • active: Active timeout
  • eof: End of capture
  • end: FIN exchange
  • rst: TCP reset

GeoIP (Professional+)

• scountry/dcountry: ISO 3166-1 alpha-2 codes
• sasn/dasn: Autonomous System Numbers
• sasnorg/dasnorg: AS organization names

nDPI Detection (Basic+)

• ndpi_appid: Application identifier (e.g., “TLS.YouTube”)
• ndpi_category: Category (e.g., “Streaming”)
• ndpi_risk_score: Cumulative risk score
• ndpi_risk_severity: 0=none, 1=low, 2=medium, 3=high

nDPI Fingerprints (Professional+)

• ndpi_ja4: JA4 TLS client fingerprint
• ndpi_ja3s: JA3 TLS server fingerprint
• ndpi_tcp_fp: TCP fingerprint with OS detection hint (format: “fingerprint/os”)
• ndpi_fp: nDPI composite fingerprint for device correlation

Anomaly Detection (Enterprise)

• anomaly_score: 0.0-1.0 indicating how unusual the flow is
• anomaly_severity: Classification based on score percentile
• anomaly_factors: Fields contributing most to the score

Parquet File Metadata

Each file includes custom metadata:

Example Queries

DuckDB - Read from S3

SELECT * FROM read_parquet(
    's3://bucket/v2/year=2025/month=01/day=28/*.parquet',
    hive_partitioning=true
);

Count by Protocol

SELECT proto, COUNT(*) as count
FROM read_parquet('flows/*.parquet')
GROUP BY proto
ORDER BY count DESC;

Filter by Country (Professional+)

SELECT saddr, daddr, scountry, dcountry, ndpi_appid
FROM read_parquet('flows/*.parquet')
WHERE scountry = 'US' AND dcountry != 'US';

High-Risk Flows (Basic+)

SELECT stime, saddr, daddr, ndpi_appid, ndpi_risk_score
FROM read_parquet('flows/*.parquet')
WHERE ndpi_risk_score > 100
ORDER BY ndpi_risk_score DESC;

Anomalous Flows (Enterprise)

SELECT stime, saddr, daddr, anomaly_score, anomaly_severity
FROM read_parquet('flows/*.parquet')
WHERE anomaly_severity IN ('HIGH', 'CRITICAL')
ORDER BY anomaly_score DESC
LIMIT 100;