In this third post we are going to see what we need to look at when hunting and detecting adversaries in the perimeter. We are also going to look at some of the firewall technologies and their log formats in order to detect anomalies in the inbound and outbound traffic in your network.
Most traffic in modern organizations goes through different choke points in order to inbound or outbound your organization. This choke points are your perimeter firewalls where your organization is monitoring the network traffic and normalizing this traffic so it adheres to the company security policy. Many of the techniques you use to hunt in the perimeter can be used in the internal firewalls to add segmentation and security levels to different parts of your network, however I am only going to focus on the detection in the external firewalls.
If you have not figured this out yet, the perimeter is probably one of the most difficult places to hunt as literally all type of traffic goes through the external firewalls. With such a big amount of traffic and protocols, it is very easy to get hopeless when looking at huge amounts of data in this part of your network and that’s why it is important to understand your company security policy and environment.
Assuming you company firewalls were setup according to your security company policy in mind, you should be able to understand some basics such as:
- Firewall security requirements for your organization
- Permitted communications
- Enforcement points
- Allowed transaction flows in your environment
- Identify connection with your business partners and guess access networks
- Identify how resources, applications and services need to be protected by your firewall
- Identify traffic baseline for your firewall
This list only covers a few of the basics, the more you understand the purpose of your firewall in regards to what your company policy enforces, the faster you will be able to spot anomalies in your firewalls.
Now that we have a good understanding of our perimeter firewalls, let’s start applying some techniques to spot potential adversaries in your perimeter. There are many techniques out there, I am only going to cover some of them but definitely you can implement any approach that leads you to a good place.
1. Denied inbound traffic
Many people argue here and they end up fooling themselves. If the firewall is stopping traffic why would I look into it? is it being stopped and that’s it. It is a valid opinion but from the security point of view has little benefit to offer, why is that traffic being stopped at your doors? is it a misconfiguration or is it a potential adversary probing or scanning your firewall? Where is that IP coming from? Is it a company owned IP or an IP in the addressing space of an ISP? Is the IP blacklisted or has a history of known bad reputation?What traffic and ports are being blocked? Are there any services running behind the firewalls on these ports? Are they vulnerable? Is always the same port being blocked or there are evidence of the same IP or group of IP’s probing other ports? Maybe a vertical scan? Or horizontal? Could it be residual traffic from a malware infection? Is there evidence of connections to these blocked destinations?
The amount of questions you can think of will lead your investigation, and it is very important to extract as much information from these sort of events as you can. Once you have all the information you may want to pass this to other teams to collaborate in your investigation. This information can be used to establish correlation with internal information systems, it can be used to start an open source intelligence investigation to discover additional intelligence on these events or it can be used as threat intelligence.
Looking at this type of activity can only bring benefit to your organization and it can be used as an early warning for your teams of what it is coming towards you in an early future.
2. Denied outbound traffic
You did not find any important anomalous inbound traffic being denied in your firewall? Give yourself one more chance. At some point your attacker went in, he also needs to go out and guess what…he has to use your choke points to do so. To have denied outbound traffic is concerning, ruling out misconfiguration issues in your infrastructure you need to wonder what is happening inside your network that the traffic is being denied in the firewall? Do you have a considerable amount of IP’s being denied to an external destination? How does that traffic look like? What’s the IP or set of IP’s? What’s the protocol used? How often is happening? To which entity does that IP or set of IP’s belong to? What sort of traffic it is? Could it be data exfiltration – insider threat? Could it be an external attacker or malware trying to leave your organization?
All these questions and more should trigger an internal investigation to understand the behaviour inside your network. If the traffic is being denied it is because your security policy contemplated that scenario before or because that communication it is not supposed to happen.
3. Permitted Inbound and Outbound traffic
This one is a difficult one as the amount of traffic to analyse make this a daunting task. One of the fastest ways to approach the problem is to use the rinse and repeat technique. Look at the traffic logged, define what is normal (whitelist), remove it from your log investigation and start the process again. At the end of the exercise you end up with some new information about your network or you end up with a set of logs where your investigation should begin.
In my previous post I mentioned that it would be desirable to have a scripting language. Well, here is one the use cases for that scripting language, hunting through a log file with 5,000 to 10,000 lines will take a long time without a scripting language. If you do hunting for entertainment though you can get along without it however the amount of time increases considerably and if you have to respond to an incident time is critical.
What other indicators can we monitor in the firewalls?
The important indicators were already discussed above however there are still other indicators that can be used to spot anomalies in your perimeter traffic such as, user activity, bandwith usage, protocol usage and amount of bytes transferred. I will not get into details as they are self-explanatory however I will do a brief stop on protocol usage but I have to clarify first that this case is only applicable in cases in which you are doing mainly packet capture as you may be able to observe the real traffic traversing the wire.
4. Port Independent protocol identification
PIPI technique allows the identification of the protocol used regardless of the port in use. PIPI makes possible to discover covert channels, back doors and in general any policy -violating channels.
The technique is easy to implement if you have an understanding and you can recognize different application layer protocols. You also have some tools out there that will allow you to implement the technique without a deep understanding of the protocols in used.
Being able to identify application layer protocols without relying on the TCP or UDP port number is crucial when analyzing malicious traffic, such as malware, Command & Control (C2) communication, covert backdoors and channels since such communication often uses services on standard ports to blend the malicious traffic with the legitimate traffic.
- Some botnets C2 protocols communicate port TCP 443, but using a proprietary protocol rather than HTTP over SSL.
- Backdoors on hacked endpoints and networks typically run on standard services such as SSH on port 22 in order to blend with normal traffic and hide
- Some backdoors use port knocking ro run a proprietary C2 protocol on an standard port such as port 80
What does all of this mean?
By analyzing network traffic for port-protocol anomalies, like an outgoing TCP connection to TCP port 443 that is not SSL you can effectively detect intrusions.
Remember this is only useful when having packet capture as it is the only way to analyze the traffic that goes through the wire.
Now that we have an understanding of where to start hunting in our perimeter firewalls let’s look at the log format that many firewalls out there in the market are throwing at us.
Needless to say that if you do not master the logging format of your perimeter firewalls before going out there on a hunting trip, you are wasting your time.
The logs we are covering are for the following firewalls, Cisco, Palo Alto and Juniper. This last section is not very long is just a matter of understanding what type of firewall is each of them and how the logs look like before starting a hunting trip. Why specifically these Vendors? They are Enterprise Grade Firewalls.
If Cisco is a good at something in their firewalls – it is detecting potential network attacks or anomalies in the network layer. These firewalls are not the most advanced in the market however they do well their job. The log format is pretty easy to understand.
Cisco ASA logs have different levels,
These levels are very useful when you are searching in your log files. As you can see the different levels have different criticality and they inform of different conditions in the traffic routed through the firewall.
Below you can find a subset of the most common logs used to detect incidents in the network layer.
If you look within your Cisco logs and you search for the Mnemonic field you can start easily seeing incidents being flagged up in your firewalls. The section above does not cover all of them but the most important ones.
Here are some additional resources for you to study in detail:
Palo Alto firewalls
These firewalls are top of the shelf, this brand introduced in the market the so called NGF (Next Generation Firewalls). These firewalls are capable of inspecting traffic in the upper layer protocols. They do packet filtering, they are protocol aware and they inspect the packet payload to detect signatures of attacks. To put it simple they are hybrids between packet filtering firewalls and Intrusion Detection Systems.
The firewall can function as packet filtering device or IDS, you can see the difference in the way the log is tagged. Traffic log when working in layer 4 and Threat when working in layer 7 as IDS. Beyond this what is left it is for you to study in detail these logs. Please see below some of the documentation.
Juniper is not a firewall but a whole network operating system. You can find out more about them by doing a little research. Juniper’s origins are in the ISP’s and core network markets however they have also evolved to cover the edge routers market. The complexity of their hardware can be seen in their system log reference for security devices below.
When we look a Juniper firewalls we find the SRX’s series firewall which are the evolution of their legacy ScreenOS solutions.
These are some of the firewalls you can find in the market with Enterprise Grade Capabilities. I suggest you research the documentation of your current firewalls and start understanding the log format and capabilities in order to protect your network. Of course this is not an easy task and the size and complexity of your networks also influence your results however some technologies such as signature, visualization and automation can assist the hunter in his hunting trips to pinpoint the area to cover.
I haven’t mentioned it before, but it makes sense you document all actions taken during your hunt trip (hunting logbook). This will help to feedback management and your team and it will avoid getting into loopholes in your next hunting trip.