Re: Figuring out odd web traffic from clients

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


sounds like a timeout value discrepancy issue somewhere either between ur backend and ur firewall or on an application level

hit me up offline if u nee any help, i can take a look 

cheers

Sent from my iPhone

On 2012-03-16, at 1:22 PM, Miles Stevenson <miles@xxxxxxxxxxxxxx> wrote:

> I'm trying to figure out a strange traffic pattern that I've seen on
> our firewalls for a few years now. It doesn't affect our production
> services, but I never figured out exactly what is going on. Recently,
> I switched out our OpenBSD firewalls to Ubuntu firewalls, and I see
> the same traffic pattern, so I know that it wasn't some oddity with
> OpenBSD, carp, or pf. Maybe someone here has an idea?
> 
> My setup is a single Ubuntu server firewall acting as a gateway
> between the internet, a DMZ network which hosts web servers, and an
> internal/private network which hosts DB servers. We are a LAMP shop.
> So, web connections are coming in from the Internet to our Apache
> servers on the DMZ.
> 
> My firewall configuration is using fairly straight forward
> state-tracking rules with no rate limiting or burst control or
> anything like that. Here is a sample:
> 
> #Allow HTTP traffic to all DMZ IPs
> -A FORWARD -p tcp -o $dmz_if -s 0/0 --sport 1024: -d $dmz_net --dport
> 80 -m state --state NEW -j LOGNEW
> -A FORWARD -p tcp -o $dmz_if -s 0/0 --sport 1024: -d $dmz_net --dport
> 80 -m state --state NEW,ESTABLISHED -j ACCEPT
> -A FORWARD -p tcp -i $dmz_if -s $dmz_net --sport 80 -d 0/0 --dport
> 1024: -m state --state ESTABLISHED,RELATED -j ACCEPT
> 
> 
> So, nothing fancy here, other than the fact that I try to be as
> descriptive as possible with my rules, and I like to log the SYN
> packets for new connections simply so I have those logs for auditing
> and troubleshooting. Of course, I REJECT anything else by default.
> 
> The strange thing, is that during peak traffic hours, I see a lot of
> traffic getting rejected by the firewall that is destined to port 80
> on the web servers. These indicate to me that the packets are not part
> of an existing session in the state table. Here are some sample logs:
> 
> [REJECT] IN=eth2 OUT= SRC=<client> DST=<server> LEN=40 TOS=0x00
> PREC=0x00 TTL=118 ID=8748 PROTO=TCP SPT=63533 DPT=80 WINDOW=65535
> RES=0x00 ACK FIN URGP=0
> 
> [REJECT] IN=eth2 OUT= SRC=<client> DST=<server> LEN=40 TOS=0x00
> PREC=0x00 TTL=118 ID=11934 PROTO=TCP SPT=63532 DPT=80 WINDOW=65535
> RES=0x00 ACK FIN URGP=0
> 
> 
> [REJECT] IN=eth2 OUT= SRC=<client2> DST=<server> LEN=40 TOS=0x00
> PREC=0x00 TTL=51 ID=16661 DF PROTO=TCP SPT=59503 DPT=80 WINDOW=0
> RES=0x00 ACK RST URGP=0
> 
> [REJECT] IN=eth2 OUT= SRC=<client2> DST=<server> LEN=40 TOS=0x00
> PREC=0x00 TTL=51 ID=16662 DF PROTO=TCP SPT=59505 DPT=80 WINDOW=0
> RES=0x00 ACK RST URGP=0
> 
> 
> Notice that some clients are attempting to send FIN, while others
> attempt to send RST. I get clumps of 4-5 of these every 20 seconds or
> so, usually from different client IP's when the web servers are busy.
> Also, our datacenter is on the east coast USA, and most of our clients
> are connecting from Europe, but we have small pockets all over the
> world. Just in case you were wondering about the low TTL values.
> 
> Apache logs show valid conversations with these same client IP's, and
> the timestamps on those are about 1-2 minutes before these packets
> described above get rejected. All of our servers are NTP synchronized,
> which helps creating a timeline across servers.
> 
> To try and figure out more of what is going on, I set up the following test:
> 
> -Set up tcpdump to capture all the web traffic on the external
> interface of the firewall.
> -Set up tcpdump to capture all the web traffic on the web server interface.
> -Compared the data between the timestamps of when things get blocked
> on the firewall, web log data from the same client IP, and what I see
> in both tcpdump data files.
> 
> What I'm seeing is kind of strange. I'll paraphrase instead of posting
> lots of logs to keep things short.
> 
> 1. The client starts up a legit TCP 80 session with the web server.
> Those get through the firewall normally and I see the full
> conversation with tcpdump.
> 2. There are usually some duplicate ACK's, retransmissions, or a few
> out of order packets, since most of our clients are several hops away
> in the UK, but TCP always handles the problem.
> 3. At the end of the web session, the web server closes the session
> cleanly using FIN.
> 4. The client ACK's the FIN, with the next sequence number.
> 5. For some reason, the client sends a RST, which seems unnecessary to
> me, since the connection is already closed with FIN.
> 6. All packets related to this socket stop.
> 7. Steps 1-6 happen at the same time for a few different sockets
> simultaneously, since the client tends to open 2-4 HTTP sessions at
> once.
> 8. A few minutes later, after all those sessions have closed, the
> client for some reason sends either a FIN, or a RST for a socket that
> never existed. This gets blocked by iptables because it does not
> belong to any open session in the state table. They are using a SRC
> PORT and sequence number that has never existed in any of the
> conversations. None of my tcpdump data captured from both the firewall
> or the web server shows a SYN ever coming in with that SRC PORT.
> 
> This seems like some weird bug or problem with the client browser. But
> we get this constantly from different clients all over the world, and
> this has been going on for a few years at least for us. The user agent
> string for the connection I just looked at was:
> 
> User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SHTTP/1.1 200 OK
> 
> Which is a fairly wide-spread browser and version. Maybe this is some
> kind of odd bug in IE or something? Does anyone have an
> experience/advice with this? This really doesn't seem to cause a
> problem at all, so I'm considering completely filtering this out from
> getting logged and completely ignoring it. But I'd really like to know
> what is going on if possible.
> 
> The only other thing I can think of, is that maybe when the client
> first connects to our web server, some of the SYN requests from the
> client never make to the server but some do. The SYN's that never make
> it to us simply expire from the clients' perspective, and then they
> eventually end up sending a RST or FIN for a socket that our firewall
> never received. As far as the reason why some of the SYN's would make
> it to us and some wouldn't, I guess that could be any number of things
> from router problems to firewalls on the client side.
> 
> Thanks a bunch for any advice.
> 
> -Miles
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Linux Netfilter Development]     [Linux Kernel Networking Development]     [Linux Networking Development]     [Linux Kernel Development]     [Linux Resources]     [LARTC]     [Bugtraq]     [Consulting]     [Free Internet Dating]     [Yosemite Forum]     [Photo]

Add to Google Powered by Linux