Using Tcpdump to analyze HTTP won't always work. Unless I'm mistaken, Tcpdump doesn't decode packet fragments into a stream, so if the request is fragmented, your grep won't work. If there are multiple HTTP requests in one TCP stream, it's less likely they are aligned completely in one packet, also fragmenting the request. And if it's a request to Google it could be in QUIC rather than TCP. You may wonder what on your network is broken because you can't see the requests, but it's just the network fragmenting packets.

Tshark (nee Wireshark) I believe does decode packet fragments into streams, and allows you to use more advanced Wireshark protocol filters. My suggestion is to start a pcap capture, and at the same time replay it using Tshark with the filters you want. This way you can re-analyze the same live traffic both now and later.

I use tcpdump continuously to analyse multiple RTP feeds coming in to our edge network. Found all sorts of issues with it, including packet reordering in Cisco and mikrotik routers (shouldn't be a problem with rtp - but tell that to ateme decoders), actual packet loss on juniper srx, and of course minor outages on international and even local links on the order of 100-200ms of loss.

Packet stream is dumped into some perl which keeps across the sequence number looking for out of sequence packets, summarise output every second into an elastic stack. Also have it comparing multiple streams for dual streaming and comparing skew.

Ironically I rarely use tcpdump to look at tcp packets - about the only interesting thing to me there is the SYN packets to see if the packet is getting through the firewall!

Any links to something similar, sounds really interesting.

BTW I really recommend unbuffer from expect, if you are using tcpdump and pipes.

I think a similar post on leveraging Wireshark would be neat. The deepest I tend to need to go is searching packets for a known substring and then following the TCP stream to see where something unexpected happened. Add to this the analyzing of the frames on this stream and you can diagnose tricky timeout issues, bad protocol usage, and much more.

Yeah, my workflow is generally to use tshark (wireshark's command-line tool) for server-side capture, then scp file to local and use wireshark gui to poke around. I find tshark's interface to be simpler than tcpdump. tcpdump feels like bringing a gun to a knife fight when all you want to do is look at application layer payloads/streams (http/etc). But it's good to know more about tcpdump than I did before, because when things get complicated it's good to have more options.

You can also pipe tshark capture to the Wireshark GUI on your machine through ssh in realtime, e.g. `ssh host.example.com 'tshark -i eth0 -w - -F pcapng -f "not tcp port 22"' | wireshark -k -i -`

that is sweet. thanks

The website references its Wireshark article under "Wrapping Up": https://hackertarget.com/wireshark-tutorial-and-cheat-sheet/

I recently presented my tool HTTPanalyzer at RIPE76. It's a C tool I developed in 2016 available at Github for processing HTTP traffic up to 10Gbps from PCAP files or straight from the interface. I recommend the branch "revisited", much better coded in my opinion. Of course, it's limited to the first packet of the request and first packet of the response. It's aimed to aggregated statistics like response codes, user agents, response time (immediate one, not full load of the resource) etc. More info here: https://carlosvega.github.io/httpDissector/

Only HTTP traffic, you can process decrypted HTTPS traffic (like some devices do, i.e. IXIA network devices) which is transformed into HTTP traffic. Regarding HTTPS or HTTP2 etc. The current approach is to correlate the application information from log events against traffic measurements.

It's pretty sad, that the best way of analyzing HTTP traffic from the command line appears to be comparing payload bytes:

  ~$ sudo tcpdump -s 0 -A -vv 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420'

There should be a better way to do that. Ideally I would want a tool shows Request Method, Headers, Query String, POST Payload of requests as they come in (and let's me filter on those). It should support HTTP2 and know how to stitch together the payload from multiple packets.

HTTPS is a difficult topic. I think it's fair for a tool like this, not to mess with encrpytion. Usually you don't want to had a precious private key file to a command line tool for debugging. A better way seems to be to terminate SSL on a separate host, and analyze the un-encrypted traffic.

Lots of tools can do this. I use mitmproxy personally. You can easily customize it by using it as a python library.

Looks like this requires me to redirect all traffic through a proxy server. This sounds very different from what I want to do.

I don't want to change the data path in any way. I just want to listen in.

If you've already terminated SSL, you can do this easily by using iptables nflog rule to duplicate traffic and send it to mitmproxy. Or you can setup a transparent proxy (but sounds like that violates your requirement). Either way, if SSL is not terminated you're not going to be able to do much.

My tips:

1. Use `-i any` -- just in case the traffic you want to look at is not on the interface you think it is. Also doesn't put the interface(s) into promiscuous mode, which can be preferable.

2. Read the `pcap-filter` man page.

Good stuff, thanks for sharing. I always forget you can do indexing into the packet itself as in tcp[((tcp[12:1] & 0xf0) >> 2):4].

Also sometimes I reach for ngrep if it is installed:

https://github.com/jpr5/ngrep/blob/master/EXAMPLES.md

This is one of my favorite oneliners:

  sudo stdbuf -oL -eL /usr/sbin/tcpdump -A -s 10240 \
   "tcp port 4080 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)" | \
    grep -a --line-buffered ".+(GET |HTTP\/|POST )|^[A-Za-z0-9-]+: " | \
    perl -nle 'BEGIN{$|=1} { s/.*?(GET |HTTP\/[0-9.]* |POST )/\n$1/g; print }'

fyi you should be able to use tcpdump -l instead of calling stdbuf.

Shameless plug, but if you’re interested in this sort of thing, I published a proof of concept docker container that makes packet analysis with scapy very easy. [0] The trick is to leverage iptables nfqueue to filter packets you’re interested in, and have scapy listen to the queue.

[0] https://github.com/milesrichardson/docker-nfqueue-scapy

Nitpick: The HTTP examples will only catch v1 traffic, not HTTP2

The "-nn" explanation seems wrong. Neither libpcap's tcpdump nor BSD's need more than one "-n" to not translate port. Even in their examples, we can spot that using a single "-n" do not translate port names.

Additionally, it's always better to always use "-p" if you don't want to accidentally attract additional traffic to the host (additional VLAN on a 802.1q interface, or additional multicast traffic).

I have always had to use -nn to not translate ports. If what you say is true, it must be recent.

I've always used -nn (or -n -n), it's really annoying - usually I want to resolve IPs, but not ports.

Really annoying, in the common case you don't want /etc/services names since they're always wrong for the ephemeral >1024 client side port. Also, not to be confused with `lsof -n -P`!

NetEng here, I'm usually troubleshooting lower layer issues (Ethernet, MPLS, IP) than HTTP however, tcpdump is one of my go-to tools every time.

If anyone is interested I've made some notes here: https://null.53bits.co.uk/index.php?page=tcpdump-notes

For example, imagine spanning/mirroring a 10G backbone link, how many people are pinging 8.8.8.8 all the time. I can ping with a specific DSCP value set to isolate my pings from anyone elses, looking into the reported issue of 8.8.8.8 latency, then apply a filter to tcpdump on my mirrored port that matches ICMP traffic to a specific IP, with a specific DSCP value, inside specific L3 VPN (specific MPLS labels) etc.

sudo tcpdump -nlASX -s 65535 -vvv -i eth3 '(mpls 52634 and (ip and (ip[1] & 0xfc) >> 2 > 0x01) and host 11.22.33.44 and icmp)'

I love tcpdump's filtering capabilities.

You build your own huge website and don't have an About page? How could someone interested in what you're doing contact you?

I mostly make notes for myself, so no need for an about page, I'm the target audience. Someone else might find the info useful which is why it's publically visible, but I'm mainly writing for myself, so no contact or comments. If you want to contact me, jwbensley /at/ gmail /dot/ com. Or Google this username, I just did, you'll find me on GitHub, LinkedIn, StackExchnage and many other places.

I am always amazed at how many people have heard of tcpdump but how few seem to have heard of tcpflow

https://linux.die.net/man/1/tcpflow

If you are looking to do TCP level protocol analysis it is one of the simplest and easiest linux command line tools out there.

It's biggest strength is that it can take a payload split over multiple packets and spit it out as a file with the whole payload

tshark is supposed to be a tcpdump alternative but it seems tcpdump is still dominant on the command line these days.

tshark always hogs my RAM and eventually crashes on > 4GB pcaps. I'd love to have a solution for this.

Try dumpcap [0], also part of the Wireshark suite. It's the back-end engine used by the Wireshark GUI as well as tshark. tshark tracks state for streams the same way the GUI will and eats your RAM, whereas dumpcap is a dumb siphon (with filtering).

0: https://www.wireshark.org/docs/man-pages/dumpcap.html

I can't count the number of times tcpdump has made debugging a problem a zillion times faster. It's my go to tool for any network related issue.

I tend to end up using wireshark for analysis of more complicated SIP issues but I always start with tcpdump.

tcpdump + CloudShark is my go-to. Usually using cURL with their API or SCP via Couchdrop.

how practical are examples ignoring https?... or am I mis-understanding target use case?

Reading through a long man page in terminal to get to el' result and then find and use cmd extensions. vs 2s google "how to inurl: tcpdump http packet"

What is the „correct“ way? (real question, I am curious)

You can search man pages by "/" then your search term, then press "n" to cycle forward, "N" to cycle back through occurrences.

Might I also suggest apropose.

> https://en.m.wikipedia.org/wiki/Apropos_(Unix)

Also, of course see:

> http://www.man7.org/linux/man-pages/man1/apropos.1.html

grep is pretty handy, -C 5 or so will get you a nice amount of context. Maybe GibBreakPls has better tricks.