There’s a recurring belief in infrastructure teams that goes like this:

“If we just log everything, we’ll understand everything.”

It’s comforting. Like thinking that keeping all receipts automatically makes you good at accounting.

Then production breaks and suddenly you’re not doing observability anymore; you’re reconstructing reality from fragments and hoping the timestamps were honest.

What we’re building here is a pipeline based on Zeek, OpenTelemetry Collector, and NetBox: not to collect more data, but to make the data slightly less insulting to interpret.

1. Problem statement#

A lot of teams are using the wrong type of network logs.

They rely on:

  • DNS server logs
  • firewall logs
  • router logs

Which feels safe, but isn’t really the truth.

Because:

  • DNS logs only show traffic that actually hit your resolver; if a client bypasses DNS or overrides it, you see nothing
  • firewall logs are policy artifacts, not behavior
  • router logs are infrastructure noise, not intent

So you end up with a partial story and a lot of assumptions filling the gaps.

A network tap changes that. It shows actual traffic, not “approved visibility paths”.

Here’s the core problem with device and server logs: every one of them is a witness, and witnesses only report what passed through them. A DNS server log can only tell you about queries that reached the DNS server. The moment a client hardcodes 8.8.8.8, ships its own resolver, or speaks DoH (DNS over HTTP) to something off-box, your DNS log goes quiet, and silence reads identically to “nothing happened”. You’re not observing the network; you’re observing one component’s opinion of the network, after that component already decided what was worth writing down.

A tap doesn’t have an opinion. It sits on the wire and copies every packet that crosses it, before any application gets to decide whether the event is interesting, allowed, or embarrassing. No log rotation policy, no sampling, no “we didn’t think to log that”. If it happened on the link, the tap saw it.

  flowchart LR
    C[Client]

    subgraph LOGS["Device / server logs"]
        D[DNS server]
        F[Firewall]
        R[Router]
        D -->|logs only queries it answered| LD[(DNS log)]
        F -->|logs only policy hits| LF[(Firewall log)]
        R -->|logs only what it routes| LR[(Router log)]
    end

    C -->|normal DNS query| D
    C -.->|DNS hardcoded 8.8.8.8 / DoH / direct IP| X((bypasses classic DNS Server logs))

    subgraph TAP["Network tap"]
        T{{SPAN / TAP}}
        T -->|copies every packet| Z[(Full traffic record)]
    end

    C ==>|all traffic crosses the wire| T
    X ==> T

    style X fill:#3a1a1a,stroke:#a33,color:#fbb
    style Z fill:#16301a,stroke:#3a3,color:#bfb

The dashed path is the one that ruins your incident review: the client that decided your DNS server was optional. The device logs never see it. The tap sees it because it has no choice.

💡 A tap is not x-ray vision. Encrypted traffic (TLS, DoH, DoT) still shows up as a connection: who talked to whom, when, and how much. You just don’t get the contents. Which is fine, because the client that hardcoded a sketchy DoH resolver is suspicious whether or not you can read the query. Behavior leaks even when payloads don’t.

But taps alone aren’t enough either: they lack infrastructure context.

SOC teams end up mapping IPs manually. “Who is IP A and why is it talking to IP B” becomes the daily ritual.

This is where enrichment becomes the missing layer.

2. Zeek: turning packets into structured truth#

Zeek is the tap we’re using here. It’s an open-source network analysis framework; you may remember it as Bro, the name it carried for about two decades before a 2018 rebrand that everyone pretends they immediately got used to. It doesn’t replace the physical SPAN port or optical tap; it’s the software that sits behind it and turns the copied packets into something readable.

Zeek sits in that sweet spot between “packet capture tool” and “security system that accidentally became an operating system for network visibility”.

Zeek listens on a SPAN port via a Linux interface and generates structured logs from raw traffic.

It takes raw packets and emits structured logs:

  • DNS queries
  • HTTP requests
  • TLS metadata
  • SSH sessions
  • connection flows

So instead of: “packet observed” you get something like this:

{
  "rejected": false,
  "opcode_name": "query",
  "proto": "udp",
  "ts": 1781867577.003075,
  "zeek.uid": "CI0B4p2F8LTZAVfdia",
  "log.file.name": "dns.log",
  "zeek.source_ip": "159.223.222.191",
  "zeek.destination_ip": "8.8.8.8",
  "zeek.source_port": 56293,
  "zeek.destination_port": 53,
  "zeek.dns_query": "google.com",
  "zeek.dns_response_code": "NOERROR",
  "zeek.dns_answers": {
    "values": [
      {
        "stringValue": "2a00:1450:400e:806::200e"
      }
    ]
  },
  "TTLs": {
    "values": [
      {
        "doubleValue": 110
      }
    ]
  }
}

That’s already a massive upgrade.

You’ve gone from noise to structured sentences.

But sentences are not context yet.

And context is where incidents actually live.

3. OpenTelemetry: the boring middleware that prevents future regret#

OpenTelemetry (OTEL) is an open-source, vendor-neutral standard for collecting and shipping telemetry: logs, metrics, and traces. The Collector is its workhorse: a single binary that ingests data in one format, reshapes it, and exports it somewhere else, all driven by config rather than code. Being vendor-neutral matters here, because it means nothing in this pipeline is married to a particular backend; swap OpenSearch for something else later and the rest of the setup doesn’t notice.

OpenTelemetry Collector sits in the middle and does what it does best: moving data without developing opinions about it.

In this setup it:

  • ingests Zeek JSON logs
  • normalizes fields
  • enriches with NetBox data using a custom processor (if available)
  • forwards events to OpenSearch

Think of it as a very disciplined courier: it doesn’t care what the message says, only that it arrives intact.

The real value is separation:

  • Zeek = truth extraction
  • OTEL = transport + shaping
  • storage = query layer

If you ever merge these responsibilities, you don’t get an architecture; you get a YAML-shaped cry for help.

🔧 Example: OTEL Processor (Go-style enrichment logic)#

OTEL processors aren’t a fixed menu; you can write your own. A custom processor is just code that sees every event passing through the Collector and can mutate it, and nothing stops it from reaching out to external data sources to do so: a CMDB, a cloud API, a threat-intel feed, whatever gives the event more meaning. Here I’ve wired it to exactly one source, NetBox, because that’s where this infrastructure’s truth lives. The same hook could enrich from a dozen sources; I just didn’t need to.

This is the “glue” that makes Zeek logs SOC-useful. It’s intentionally simple. No magic. No ML. And definitely no LLMs. Just IP lookups and JSON mutation.

type NetBoxEntry struct {
    Name         string
    Type         string
    Role         string
    Platform     string
    Site         string
    Cluster      string
    Manufacturer string
}

var netboxCache map[string]NetBoxEntry

func enrichZeekLog(log map[string]any) map[string]any {
    srcIP := log["zeek.source_ip"].(string)
    dstIP := log["zeek.destination_ip"].(string)

    if src, ok := netboxCache[srcIP]; ok {
        log["netbox.source.name"] = src.Name
        log["netbox.source.type"] = src.Type
        log["netbox.source.role"] = src.Role
        log["netbox.source.platform"] = src.Platform
        log["netbox.source.site"] = src.Site
        log["netbox.source.cluster"] = src.Cluster
    }

    if dst, ok := netboxCache[dstIP]; ok {
        log["netbox.destination.name"] = dst.Name
        log["netbox.destination.type"] = dst.Type
        log["netbox.destination.role"] = dst.Role
        log["netbox.destination.platform"] = dst.Platform
        log["netbox.destination.site"] = dst.Site
        log["netbox.destination.manufacturer"] = dst.Manufacturer
    }

    return log
}

This is basically:

“turn IP soup into infrastructure-aware events before anyone tries to debug it manually”

4. NetBox enrichment in working: where IPs stop being numbers#

For those unaware, NetBox is an open-source IPAM/DCIM system used to document infrastructure and address space.

NetBox + OTEL is where logs stop being abstract and start becoming infrastructure-aware.

Without enrichment:

{
  "zeek.source_ip": "159.223.222.191",
  "zeek.destination_ip": "8.8.8.8",
  "zeek.dns_query": "google.com"
}

With enrichment:

{
  "zeek.source_ip": "159.223.222.191",
  "netbox.source.name": "virtual-machine-01",
  "netbox.source.type": "vm",
  "netbox.source.role": "Cloud Server",
  "netbox.source.platform": "Ubuntu 22.04",
  "netbox.source.site": "AMS3",
  "netbox.source.cluster": "ams3-cluster",

  "zeek.destination_ip": "8.8.8.8",
  "netbox.destination.name": "google-dns-8888",
  "netbox.destination.type": "device",
  "netbox.destination.role": "DNS Resolver",
  "netbox.destination.manufacturer": "Google",
  "netbox.destination.platform": "Google DNS"
}

Same event.

Completely different interpretation.

One is data. The other is a story.

💡 Enrichment is only as honest as NetBox. If your IPAM is stale, you don’t get truth; you get confident fiction. DHCP churn and IP reuse mean today’s virtual-machine-01 was three VMs ago. Garbage in, authoritative-looking garbage out. Keep the source of truth actually true, or the enrichment layer just lies faster.

5. Why raw connection logging is the real foundation#

This is where Zeek earns its keep.

Because connection-level telemetry shows behavior, not intention.

Example conn.log event:

{
  "log.file.name": "conn.log",
  "zeek.source_ip": "159.223.222.191",
  "zeek.destination_ip": "159.223.208.1",
  "zeek.transport": "icmp",
  "zeek.connection_state": "OTH",
  "zeek.destination_port": 0
}

Add enrichment here from NetBox, connecting the IPs to actual infrastructure roles:

{
  "zeek.source_ip": "159.223.222.191",
  "netbox.source.name": "virtual-machine-01",
  "netbox.source.type": "vm",
  "zeek.destination_ip": "159.223.208.1",
  "netbox.destination.name": "edge-router-ams3",
  "netbox.destination.type": "device",
  "netbox.destination.role": "Cisco ISR 4331",
  "zeek.transport": "icmp"
}

And when you zoom out across flows:

vm → google-dns-8888     53/udp
vm → cloudflare-dns-1111 443/tcp
vm → edge-router-ams3    icmp
vm → external services   443/tcp burst

Now you’re no longer looking at logs.

You’re looking at behavior under pressure.

Which is usually where the truth leaks out.

6. NetBox enrichment turns behavior into infrastructure graphs#

Once enrichment kicks in, the same flows become meaningful relationships:

{
  "zeek.source_ip": "159.223.222.191",
  "netbox.source.name": "virtual-machine-01",
  "netbox.source.type": "vm",
  "netbox.source.site": "AMS3",

  "zeek.destination_ip": "8.8.8.8",
  "netbox.destination.name": "google-dns-8888",
  "netbox.destination.type": "device",
  "netbox.destination.role": "DNS Resolver"
}

Now you can actually answer questions like:

  • which VMs depend on external DNS?
  • which systems talk to edge infrastructure unexpectedly?
  • what changed before traffic anomalies?

And more importantly: who is going to explain this in the incident review?

7. Even simple aggregation becomes powerful#

Once everything is structured and enriched, even “dumb” summaries become meaningful. No more IP A → IP B on port X and a guessing game; you’re reading named infrastructure talking to named infrastructure:

33 vm:virtual-machine-01 → device:google-dns-8888 [Google] 53/udp
18 vm:virtual-machine-01 → device:cloudflare-dns-1111 [Cloudflare] 443/tcp
10 vm:virtual-machine-01 → device:cloudflare-dns-1111 [Cloudflare] 80/tcp
6  vm:virtual-machine-01 → device:edge-router-ams3 [Cisco] 22/tcp
6  vm:virtual-machine-01 → device:edge-router-ams3 [Cisco] 179/tcp

This is the point where logs stop being forensic artifacts and start becoming infrastructure telemetry.

Not because the format changed.

But because the meaning did.

8. Architecture reality: keep it physically boring#

This works best as:

  • dedicated machine (bare metal preferred)
  • single NIC on SPAN/tap
  • Zeek for packet interpretation
  • OpenTelemetry Collector for transport
  • OpenSearch for querying
  • small Go enrichment service caching NetBox state

No distributed orchestration.

No “AI-driven anomaly reasoning layer”.

No hero architecture diagrams.

Just:

packets → structure → enrichment → searchable reality

Or, if you insist on a diagram, a deliberately boring one; every box does exactly one job and nothing else:

  flowchart LR
    NET[Network traffic] -->|SPAN / TAP| Z[Zeek]
    Z -->|structured JSON logs| O[OTEL Collector]
    NB[(NetBox<br/>IPAM / DCIM)] -.->|IP → infra context| O
    O -->|enriched events| OS[(OpenSearch)]
    OS --> Q[Queries · dashboards · the 3 a.m. incident review]

    style NET fill:#1a2330,stroke:#37c,color:#cde
    style OS fill:#16301a,stroke:#3a3,color:#bfb

💡 A tap can lie by omission too. Point Zeek at a saturated 10G+ link with no tuning and it quietly drops packets, and a dropped packet is just a tidier version of the gap you were trying to escape in the first place. Size the box, pin the CPU, watch your capture stats. If you must sample, sample on purpose and write it down, so future-you doesn’t mistake “we didn’t capture it” for “it didn’t happen”.

And that’s already enough complexity for most systems.

9. Closing thought: logs don’t create understanding#

Zeek gives you visibility. NetBox gives you meaning. OpenTelemetry keeps it moving without falling apart.

But understanding still comes from correlation:

  • what talked
  • what it is supposed to be
  • and what it is actually doing instead

Everything else is just structured noise.

Beautiful noise, sometimes.

But still noise.

And noise, as always, is very confident right up until the moment it isn’t.