dslreports logo
site
 
    All Forums Hot Topics Gallery
spc

spacer




how-to block ads


Search Topic:
uniqs
2390
share rss forum feed

HELLFIRE
Premium
join:2009-11-25
kudos:18

Burned by IP INSPECT -- My Own Personal Journey

Figured I'd mention this story as an abject lesson in the old saying "you never truly appreciate things
till you experience it yourself." Basically, had a CBAC config with http/s inspection turned on. Crazy
thing is NOTHING behind CBAC seemed to complain with the CBAC config (which made tracking it down
that much harder) EXCEPT Jdownloader updates. No matter what I did, any computer behind my CBAC
config would not allow Jdownloader to update the program. Put said computer behind a Cisco router
with reflexive ACLs, a Cisco ASA in transparent mode, or a basic home router, Jdownloader would
update no problems.

Finally happy, and somewhat embarassed, to report that I figured out that the lines "IP INSPECT
HTTP/S" was to blame all along... seemed that CBAC thought neither the outbound or inbound packets
were part of a valid connection (even though wireshark showed a proper syn/synack/ack setup) and
sent RST packet to both endpoints. Turned off HTTP/S inspection in the config, and Jdownloader is
updating properly and cleanly, and no resets in the wireshark captures.

Application intelligence, inspection, et al is SUCH a bloody oxymoron.

Which leads me to my next question, anyone else know of any other specific INSPECT engines in
CBAC or ZBFW that seem to take such "liberties" with interpreting the RFC, and that I should never
turn on for the sake of one's network sanity?

Regards

nosx

join:2004-12-27
00000
kudos:5
Its the networks job to deliver packets, not mangle them. Keep the intelligence (and security) at the host or as close to the host as possible. Network security is the oxymoron, it provides a false sense of protection that turns our infrastructure into complex time sucking tar pits that looks like swiss cheese to savvy attackers.

Are you really more protected running all those application inspection features? More secure than just outbound NAT with state tracking? And if so... how? I can see how inspection could play a role with inbound requests, but it is somewhat less valuable for outbound traffic.

That said, one mans interpretation of of a standard may not match anothers, the Jdownloader may violate the protocol in some way, or Cisco may be interpreting too strictly.

Have you tried inspect on different & newer IOS? What did TAC say? etc.

aryoba
Premium,MVM
join:2002-08-22
kudos:4
said by nosx:

Network security is the oxymoron, it provides a false sense of protection that turns our infrastructure into complex time sucking tar pits that looks like swiss cheese to savvy attackers.

Tell that to network security auditors and they will stare at you funny

aryoba
Premium,MVM
join:2002-08-22
kudos:4

2 edits
reply to HELLFIRE
I have been testing reliability of HTTP/HTTPS IP Inspection features on IOS and the results are never good. The ASA's IP Inspection features seemed to be more reliable since perhaps the behaviors inherited the PIX fixup commands among other things. If I had to use a box with enhanced routing and firewall features, I would rather use Juniper SRX than Cisco router with IP Security featured IOS. For basic firewall (without HTTP/HTTPS inspection), IOS IP Inspection is good enough

nosx

join:2004-12-27
00000
kudos:5
reply to aryoba
I tell the paranoid QSA's what I think of their ideas every year lol
"But somebody could scale the telephone pole and strip the cable and tap it and steal data!" - Real QSA.
Risk management means average loss per event times annual rate of occurance = potential loss. If potential loss is less than the cost of some action actually capable of remediating their dilusional attack vector, there is no justification to do it.

aryoba
Premium,MVM
join:2002-08-22
kudos:4
said by nosx:

I tell the paranoid QSA's what I think of their ideas every year lol

Once I had an honest response from one of those network security auditor. He did admit that a lot of the technical requirements to consider a network as a secure network are simply fabrication that bears no real meaning. Unless you are backed by expert lawyers and some government lobbyists, there is nothing you can do or say to change the game rule

aryoba
Premium,MVM
join:2002-08-22
kudos:4
reply to nosx
said by nosx:

"But somebody could scale the telephone pole and strip the cable and tap it and steal data!" - Real QSA.

This reminds me of a story that a lot of government entities encrypt their data over point-to-point dedicated private links as a requirement in order to avoid the situation where the ISP or telco stealing their data. There are however no such requirements coming from some federal entities such as Federal Reserve and financial exchanges. I guess some rules and/or mindsets are not applicable to all government entities

cramer
Premium
join:2007-04-10
Raleigh, NC
kudos:9
Cost and speed trump security.

And for the record, the only .gov systems I've ever known to use encryption are systems carrying sensitive information. ("top secret", "classified", etc. i.e. not for the public to see. I couldn't believe the shear volume of crap they stamp sensitive -- 'tho partly because they don't want to take any time evaluating it.)

HELLFIRE
Premium
join:2009-11-25
kudos:18
reply to HELLFIRE
@nosx
said by nosx:

Its the networks job to deliver packets, not mangle them.

Agreed, which makes me question one unnamed vendor's comment once upon a time of the ASA TCP-MAP
as "mangling packets" even less... though I don't have any operational experience with TCP-MAPs.

said by nosx:

the Jdownloader may violate the protocol in some way, or Cisco may be interpreting too strictly.

The fact that JDownloader was the ONLY one that didn't play nicely, makes me suspect the former... but how do you
prove that to the developers? Perfect Catch-22 situation.

said by nosx:

Have you tried inspect on different & newer IOS? What did TAC say? etc.

Just to give some context, this was on the home 1811 on 12.4T code. AFAIK, Cisco said that CBAC wouldn't
have new features added to it after 12.4T, and I haven't found any indications this is necessarily a bug, per se.
I get some time, I'll review the 15.x release notes for anything related to CBAC / HTTP/S inspection if anything
pops out or not.

@aryoba
said by aryoba:

I have been testing reliability of HTTP/HTTPS IP Inspection features on IOS and the results are never good.

Do you mind sharing some specific examples (non-NDA protected ones, of course) of what you experienced?
Like I said, this was a home environment rather than prod (thanks be for THAT!), so I just wanted to hear about
other experiences from the field for personal reference and learning.

said by aryoba:

This reminds me of a story that a lot of government entities encrypt their data over point-to-point dedicated private links as a requirement in order to avoid the situation where the ISP or telco stealing their data.

Financial institutions are nuts about this... and the funny thing is, especially for the client I deal with most, it's
not that hard to leave the encryption up to the ENDPOINTS to do (AES isn't that computationally expensive
on modern server hardware), yet I keep coming across multiple MACD requests to the carriers to set up
a P2P encrypted VPN requests as if it were the "in" thing to do these days...

@cramer
said by cramer:

Cost and speed trump security.

You're preaching to the choir on that one, brother! Amen!

Again, if anyones got more stories or experiences about IP INSPECT, please share.

Regards

nosx

join:2004-12-27
00000
kudos:5
Set up a packet capture and produce a PCAP of the behavior between the router and host running jDownloader. If there is any funny business with HTTP, many tools can decode those sessions and something obvious might jump out. Watch for the exchange right before the inspect engine kills the traffic and injects spoofed reset packets in both directions.

Getting developers on-board is always tricky, depending on how interested they are in fixing whatever is going wrong. There is also the possibility that the developers and Cisco are BOTH innocent, and the underlying library that the developers used is at fault. Lets face it, not many people are gung ho to write their own underlying application dependancies, we all use existing librarys that get the job done to save time, cost, and sanity.

aryoba
Premium,MVM
join:2002-08-22
kudos:4
reply to HELLFIRE
said by HELLFIRE:

said by aryoba:

I have been testing reliability of HTTP/HTTPS IP Inspection features on IOS and the results are never good.

Do you mind sharing some specific examples (non-NDA protected ones, of course) of what you experienced?
Like I said, this was a home environment rather than prod (thanks be for THAT!), so I just wanted to hear about
other experiences from the field for personal reference and learning.

Basically you can test HTTP/HTTPS inspection against any application such as Youtube for video streaming or news feed. I find the best application to troubleshoot and/or test is live streaming or constant traffic flowing since you can actually monitor any slowness or problems in real time. Start from some configuration template and tune your way in during test to get the best result. Time outs or keepalives, buffer sizes, are some factors you can tune. If you are unsure which factors you can tune, reading Cisco online documentations on the IOS command you are testing while doing the test I always find helpful.

aryoba
Premium,MVM
join:2002-08-22
kudos:4
reply to nosx
said by nosx:

Getting developers on-board is always tricky, depending on how interested they are in fixing whatever is going wrong. There is also the possibility that the developers and Cisco are BOTH innocent, and the underlying library that the developers used is at fault. Lets face it, not many people are gung ho to write their own underlying application dependancies, we all use existing librarys that get the job done to save time, cost, and sanity.

This is one of the reason why I prefer to work for small companies with tight relationship across highly technical people who actually know what they are doing; networks, servers, developers. The goal is about get things done quickly and efficiently without politics and finger pointing

aryoba
Premium,MVM
join:2002-08-22
kudos:4
reply to cramer
said by cramer:

Cost and speed trump security.

Basically network security is about where and how technical understanding your company lawyers are. In one of my previous company, we got Infrastructure Security VP that had JD and MBA degrees in addition to network engineering and support background that enabled him to see eye to eye with anybody; management and technical people; which helped tremendously in implementing policies and procedures. So no fancy nor frivolous stuff, just necessary things to keep the cost minimal yet we still passed the network security audit and compliance

HELLFIRE
Premium
join:2009-11-25
kudos:18
reply to HELLFIRE
@nosx
I've got the packet captures, and like I said, they definately show a syn/synack/ack setup, then a RST packet
almost immediately after. "ip inspect log-drop-packet" didn't offer much more than indicating dropping
a packet with 0x5010 flags (SYN) from said PC updating JDownloader, which didn't tell a whole lot.

I don't do a whole lot of programming, and I'd THINK Java had a standard TCPIP IO library SOMEwhere, unless
like you said, the devs of JDownloader wrote their own. Anywho, the cause has been determined, how much
time and effort I want to hunting this down is in my court now.

@aryoba
Thanks for the advice, will keep it in mind. Like I said in the beginning, nearly ALL HTTP/S traffic I ever ran
with CBAC on didn't have an issue; I don't think I use all possible HTTP/S connections out there, but all the typical
stuff these days -- std HTTP/S pages, logins, youtube, HTTP downloads, etc. ALL played nicely EXCEPT JDownloader
updates which used an HTTPGET BUT "ip inspect http" SOMEhow thought it was not a valid setup / connection
and killed the session.

said by aryoba:

This is one of the reason why I prefer to work for small companies with tight relationship across highly technical people who actually know what they are doing; networks, servers, developers. The goal is about get things done quickly and efficiently without politics and finger pointing

Always the goal / dream, never the reality....

Regards

aryoba
Premium,MVM
join:2002-08-22
kudos:4
said by HELLFIRE:

@aryoba
Thanks for the advice, will keep it in mind. Like I said in the beginning, nearly ALL HTTP/S traffic I ever ran
with CBAC on didn't have an issue; I don't think I use all possible HTTP/S connections out there, but all the typical
stuff these days -- std HTTP/S pages, logins, youtube, HTTP downloads, etc. ALL played nicely EXCEPT JDownloader
updates which used an HTTPGET BUT "ip inspect http" SOMEhow thought it was not a valid setup / connection
and killed the session.

This is not the first time some "legitimate" HTTP/HTTPS application get zapped. It is possible that the application does not employ standard RFC somewhere hence the zapper sees them as illegitimate traffic to zap. It is then about tune the zapping level in addition to be using the trusted/proven zapping method and/or appliance.

aryoba
Premium,MVM
join:2002-08-22
kudos:4
reply to HELLFIRE
said by HELLFIRE:

said by aryoba:

This is one of the reason why I prefer to work for small companies with tight relationship across highly technical people who actually know what they are doing; networks, servers, developers. The goal is about get things done quickly and efficiently without politics and finger pointing

Always the goal / dream, never the reality....

In today's struggling economy, it has been more difficult to find such company. From my experience, you simply have to be willing to learn anything, connecting with the right social network, and keep looking for companies that will see you as asset and not cost. Good luck in searching

nosx

join:2004-12-27
00000
kudos:5
See im the exact opposite, I wont ever work for small businesses again. The large multinational 2b-20b range is a great place. they have enough incentive to prioritize stability for existing revenue over the quest for new revenue and invest heavily in both the talent and infrastructure that make it a good place to lead. The smaller guys were always full of nepatism cronyism and the chronic underfunding of critical infrastructure. Id rather not work somewhere that decides to be single threaded and then call me at 3am because something broke.

aryoba
Premium,MVM
join:2002-08-22
kudos:4

1 edit
said by nosx:

See im the exact opposite, I wont ever work for small businesses again. The large multinational 2b-20b range is a great place. they have enough incentive to prioritize stability for existing revenue over the quest for new revenue and invest heavily in both the talent and infrastructure that make it a good place to lead. The smaller guys were always full of nepatism cronyism and the chronic underfunding of critical infrastructure. Id rather not work somewhere that decides to be single threaded and then call me at 3am because something broke.

From my experience, basically the key is to work for a company that sees you as regarded asset instead of costs regardless of the company size. Since we are in IT business (specifically network), the company then has to be technology-heavy-invested one that sees having bleeding edge or most advanced network as critical requirement to business prospective. You will have higher visibility to management (possibly higher level of regarded assets) as you work closer to serving the management directly (report directly to CTO or CEO certainly have such high visibility compared to report to some manager).

Realistically speaking, there are only handful of companies that offer network engineering work opportunities to report directly to CTO or CEO. The list is getting shorter when you add requirement such as big bonuses or compensations