… or accessed from any unauthorized party. In some cases it’s not even necessary to hack a system to gain access to it. For example it may be enough to leak a configuration file to allow unauthorized system access. Fortunately, many cases of a suspected hack turn out to be false alarm.
This guide will give you a guideline at hand on how to proceed in such cases.
First we need to rule out false-positive alerts. We do this by:
Ensure that you Really got Hacked
Often, an administrator has the suspicion that a device got hacked when he finds logs or other suspicious traces that he is not very sure that they are harmless. In this case, we recommend to check your setup for traces of a hack or an attack. Possible sources of information are:
- Results of the comparison of configuration files from different dates. (Aka Change-Log analyze)
- Crashlog entries (Crashed daemons, conserve mode events, session clashed, error/drop counter, …)
- Log messages (Event logs, traffic logs, security logs…)
Just for completeness: The possibly best way to know that you are under attack or your system has been hacked is a notification from you SIEM or SOAR solution. In most cases, when a SIEM or SOAR solution recognizes such an event, it just happened some seconds ago. This is the most convenient way to get alert of a system breach because you can respond immediately and therefore the attacker has less time to steal data and harm your infrastructure.
When you are sure that no false positive event happened and your system really got attacked and compromised, you want to involve Fortinet into the process at this point:
Involve Fortinet into the Process
Please note, that – if you have a distributor that maintains an own support department – a good first step would be to involve the distributor to coordinate the next steps.
There is a standard procedure how to proceed when a FortiGate is compromised: Open a support ticket on the Fortinet support portal support.fortinet.com as timely as possible. The engineer working on your ticket will proceed to investigate the case and then decide if other Fortinet ressources have to be involved. In some cases, the Fortinet Product Security Incident Response Team (PSIRT) will be involved. The Fortinet PSIRT also has wide possibilities to make a thorough forensic analysis of your appliance.
Since in most cases, the priority of such cases need to be very high, we recommend to open a ticket as “P3”. P3 is the highest priority one can open a ticket in over the support center. Then, call in at the Fortinet support department and ask for an escalatation of the ticket to P2. Do not use the chat, call by phone.
Please be aware and mind:
- When opening the ticket, invest the time and upload all required information and also all relevant logs and observations. It is very crucial to provide as much information as possible to the support engineer.
- Note in the ticket the impact to your business operations. If not in minimum a part of the network is offline or another important reason for a priorization is available, the ticket will not be escalated.
- Reaching “P1” in a case is an extremely rare case. You can request an escalation, but in most cases P2 is more than high enough.
- Customers are not entitled to escalate a ticket to P1 or P2 on a product that only has a FortiCare Essential support contract.
The documentation of the Fortinet technical support procedures is located under https://support.fortinet.com/Information/DocumentList.aspx in the “FortiCompanion to Technical Support” guide.
Also, depending from your situation, it’s recommended to …
Involve the Official Federal Department, the National Cyber Security Centre (NCSC) into the Investigation
The National Cyber Security Centre is the federal government’s competence centre for cybersecurity and thus the first point of contact for businesses, public services, educational institutions and the population when it comes to cyber issues. It is responsible for the coordinated implementation of the national cyber strategy.
The NCSC has a report form to contact them: https://www.report.ncsc.admin.ch/de
Depending on the situation it is also appropriate to inform the local law enforcement over the emergency number 117.
If you can confirm, that you FortiGate has been hacked, you may want to know HOW the attacker has entered your system:
Enumerate how your FortiGate got Hacked
A first point of contact to find out how your FortiGate got hacked are the FortiOS release notes of the version your are using. To be more specific: The “known issues” sections of the release notes (https://docs.fortinet.com).
Also, you can find relevant information to severe bugs under https://www.fortiguard.com/psirt.
Log files are a very important source of information to investigate attacks. Note, however, that log files can be manipulated and an attacker may have already removed the telltale traces.
If customer personal data has been breached,
Involve the Federal Data Protection and Information Commissioner
The FDPIC provides the data controllers with an online form with which their reports can be submitted in a digital and secure manner. After submitting the report, the data controller can download a confirmation with the submitted data.
Only personal data breaches that result in the unintentional or wrongful loss, deletion, destruction or alteration of personal data, or made accessible or disclosed to unauthorized persons, and that are likely to result in a high risk to the personality or fundamental rights of the data subjects must be reported.
If it is necessary for the protection of the data subjects, the data controllers must inform them of the personal data breach. Further information on notifications is available on the online form:
Form reserved exclusively for notifications by data controllers (DataBreach)
As soon as you know how the attacker has entered your system, you want to save any evidence for investigations:
Save Traces and Forensic Data
If needed, you may consider involving external specialists or government agencies that can support you in this step. In Switzerland, the National Cyber Security Centre is the central federal competence center that is responsible for such cases and in case of Fortinet products, Fortinet itself has a PSIRT unit to support you here. Those units were already mentioned above and they may also be a good supporting resource at this point.
Important information sources to save for troubleshooting are:
- Logfiles
- Configuration files
- Crashlogs / Debuglogs
- Recorded traffic (sniffer)
- Monitoring data
- Dashboard data
- CLI debug data
- Alert emails
It is very important to save any relevant information before destroying traces. In some cases, companies keep an attacker accessing and leave them in the knowledge that they are still undiscovered to get more information about the intention of an attacker.
After saving any relevant data to bring the attacker behind bars, you now can close the entry point which the attacker used to break in:
Close Access to Vulnerabilities that have been exploited
Do not forget to think a step ahead: If the attacker was able to download your FortiGate configuration file, you may also want to prevent the attacker from accessing a SSL VPN portal and so on.
Often a software upgrade will be available to install in which the bugs have been fixed.
In other cases a bug fix requires more time to implement and a software update can not be made available momentarily. In this cases, the manufacturer will inform how to close the vulnerable part with a workaround like local in policies, ips signatures, geo blocking and so on.
If it’s not absolutely sure, that the FortiGate system has not been tampered, we recommend to format the FortiGate internal storage and install the FortiOS over TFTP afterwards. Do not restore a configuration backup afterwards without checking the whole configuration thoroughly. If you have a clean configuration backup at hand, you can restore a “safe” configuration version.
If you are not sure if the threat is persistent in the system and survives a configuration restore or a factory reset, the safe version is to format the FortiGate, reinstall the FortiOS over TFTP and rebuild the configuration by hand.
Actions to take on the FortiGate
Among all the other adjustments, that may be useful depending from your specific case, you have to check if the following actions have to be taken:
Change all usernames
Change all passwords
Change all ssh keys
Change all certificates
Delete unknown users / administrators
Check automation stitches
Check WebAdmin and API access
Overthink your logging and alerting settings and adjust them accordingly
After you have excluded the attacker from your infrastructure, you now need to monitor the situation to prevent that the attacker has a backdoor in place which can be used to gain access to the system again:
Monitor the Situation
It’s very recommendable to configure extended logging. This measure may be configured only for a limited time or may be enabled for future use. Log messages are a very valuable information source to determine what happened at what time and what was the impact.
As an additional measure, you may want to enable alerting for suspicious actions.
After you have implemented a secure solution to lock the attacker out and you have a monitoring in place, you can proceed to remediate the problem and implement a solution to prevent future similar attacks.
Implement a Solution to Prevent Future issues, also Known as: Harden your Infrastructure
Minimize exposure to the internet (local in policies, geo blocking, admin access, and so on).
Keep your network equipment devices up to date.
Use technologies to authenticate access to resources like zero trust network access.
Reduce the attack surface of your infrastructure devices.
Actions to take on the rest of your IT Infrastructure
If an LDAP server was in use on the FortiGate, check all locations where the LDAP service user has access to. Also, external authentication systems are a good point for an attacker for information gathering attacks. Since the FortiGate is a very central part of your network, an attacker may also have gained VPN access. Therefore it’s recommended to check every system that is reachable from the hacked FortiGate for indicators of compromise. This also includes client systems.
A very important last step is to comply with your local law:
Evaluate Legal Steps and Insurance Cases
There are some legal related actions you should consider:
- You may consider to contact a lawyer for advice on how to proceed.
- Inform local authorities and file charges at the local law enforcement agency.
- Inform the national cyber security centre.
- Inform the federal data protection and information commissioner.
- Inform your insurance to file a case.
- Inform affected customers about data loss, service interruptions or stolen confidential information.
There are some important details worth to know on the FortiOS and in it’s configuration:
- If you do not set the private-data-encryption, passwords and keys (except administrator passwords) are stored using a reversible algorithm. That means, that it is possible to translate an encrypted string (marked with “ENC” before the encrypted part) back to a cleartext string.
The configuration option “private-data-encryption” is documented in the FortiOS Hardening guide. - The FortiGate has a BIOS-level signature and file integrity checking implemented since FortiOS 7.4.0. The BIOS-level signature and integrity checking is enforced on FortiOS GA firmware image, AV engine file, and IPS engine file. It checks, if those files are dually-signed by the Fortinet CA and a third-party CA. The BIOS verifies that each file matches their secure hash as indicated by their certificates. Users are warned when there is a failed integrity check, and the system may be prevented from booting depending on the severity and the BIOS security level.
The BIOS-level signature and file integrity checking is documented in the release notes. - The FortiGate has a Real-time file system integrity check in addition to the BIOS-level signature check.
When the FortiGate boots, the system performs a BIOS level integrity check on important internal files, the AV engine file, and the IPS engine file. These files are signed by the process described in Enhance BIOS-level signature and file integrity checking, and the BIOS verifies their signature against their certificates.
Once these files are verified to be authentic, the BIOS can boot the root filesystem and other executables and libraries. Once loaded, real-time protection begins. The important executables and binaries are protected from write access and any modifications. It also blocks the kernel from loading any modules. Any unauthorized loading of modules is blocked. If violations are found, logs are triggered.
A hash of all executable binaries and libraries is taken and stored in memory. If there is a hash mismatch when attempting to run a binary, that binary is blocked from running, and the system is rebooted.
The system also runs a periodic check to verify the integrity of important binaries and AV and IPS engines.
This is also documented in the release notes as well as in the admin guide.
Due to the sensitivity of the content of this article, the following notice applies: This article is provided “as is” and makes no claim to completeness or accuracy. This article is a rough guide and not legal advice. If you have any valuable additions or your own experience, please feel free to share your thoughts in our comments section.
