FortiGate Memory and CPU Troubleshooting

From time to time we face performance problems on FortiGate units in our daily support life. Most often the impacts of performance problems on the FortiGate are not typical. Or let’s say “not as an admin that is not familiar with FortiGates would expect”. The expectations vary from high delay on network traffic up to unresponsiveness of the system or even a system crash. But what we experience in reality differs very much from this expectations: We see skipped UTM inspections, slow webadmin access, notification emails about conserve mode and, in very rare cases, even the the blocking of new sessions. Slowliness and delayed packets are very rarely seen, even in performance bottleneck situations. We see those symptoms much more often when there are layer 1 or layer 2 problems on the network which lead to crc errors and/or packet loss. This behaviour in overload situations makes the FortiGate a very tough solution even when the system is badly overloaded.

Table of Contents

    Memory load too high? – conserve mode!

    If the memory usage on a FortiGate is very high, the FortiGate goes into the so called “conserve mode”. The conserve mode protects memory ressources with different measures to prevent daemons (services) from crashing and the system from becoming instable.

    In FortiOS 7.2.x, the memory usage limit to enter the conserve mode is 88% by default. At 95% memory usage, the FortiGate will drop new sessions. As soon as the memory load is under 82% again, the FortiGate will automatically exit conserve mode again.

    config system global
        set memory-use-threshold-extreme 95
        set memory-use-threshold-green 82
        set memory-use-threshold-red 88
    end

    During conserve mode no more configuration changes can be made. Also, all quarantine actions (and also sandboxing) are skipped.

    There is a detailed KB article that describes what conserve mode is.

    You can find out if your FortiGate is running in conserve mode really quick either by the red notice on the WebAdmin portal or with the CLI command “diag hardware sysinfo conserve”:

    Conserve mode message in the FortiGate WebAdmin GUI.
    Conserve mode message in the FortiGate WebAdmin GUI.

    You can also find the relevant information by using the CLI command “diagnose hardware sysinfo conserve”:

    gw01 (global) # diagnose hardware sysinfo conserve
    memory conserve mode:                        on
    total RAM:                                         3040 MB
    memory used:                                       2706 MB   89% of total RAM
    memory freeable:                                    334 MB   11% of total RAM
    memory used + freeable threshold extreme:          2887 MB   95% of total RAM
    memory used threshold red:                         2675 MB   88% of total RAM
    memory used threshold green:                       2492 MB   82% of total RAM
    
    gw01 (global) # diagnose hardware sysinfo conserve
    memory conserve mode:                        off
    total RAM:                                         1918 MB
    memory used:                                       1306 MB   68% of total RAM
    memory freeable:                                    199 MB   10% of total RAM
    memory used + freeable threshold extreme:          1860 MB   97% of total RAM
    memory used threshold red:                         1687 MB   88% of total RAM
    memory used threshold green:                       1572 MB   82% of total RAM

    If you are interested to know more details about each line of information out of this output, you may take a look at this website that described memory handling on linux operating systems.

    The tension intensifies…

    What most administrators are not aware about is, that the FortiGate also has another mechanism to prevent instability caused by high memory load: Memory tension drops. This mechanism has nothing to do with the conserve mode. As soon as the kernel is not able to allocate anymore memory pages, it removes the oldest sessions in the session table.

    There is a counter telling us, if any sessions have been dropped by the “memory tension mechanism”:

    gw01 (global) # diag sys session stat 
    misc info:       session_count=75 setup_rate=3 exp_count=0 clash=0
            memory_tension_drop=0 ephemeral=0/126976 removeable=0 extreme_low_mem=0
            npu_session_count=21
            nturbo_session_count=21
    delete=10, flush=13, dev_down=274/41 ses_walkers=0
    TCP sessions:
             26 in ESTABLISHED state
             1 in TIME_WAIT state
    firewall error stat:
    error1=00000000
    error2=00000000
    error3=00000000
    error4=00000000
    tt=00000000
    cont=00000000
    ips_recv=000947dd
    policy_deny=001703c4
    av_recv=00000000
    fqdn_count=00000012
    fqdn6_count=00000000
    global: ses_limit=0 ses6_limit=0 rt_limit=0 rt6_limit=0

    Is it a CPU or Memory problem?

    When you have checked the system for conserve mode events and memory tension drops and you have not noticed anything special, you want to proceed troubleshooting to find the source of the problem.

    On the FortiGate, you can find out what is the cause of the performance issue really fast with the CLI command “get system performance status”:

    gw01 (global) # get system performance status
    CPU states: 20% user 1% system 0% nice 79% idle 0% iowait 0% irq 0% softirq
    CPU0 states: 11% user 1% system 0% nice 88% idle 0% iowait 0% irq 0% softirq
    CPU1 states: 8% user 2% system 0% nice 90% idle 0% iowait 0% irq 0% softirq
    CPU2 states: 15% user 0% system 0% nice 85% idle 0% iowait 0% irq 0% softirq
    CPU3 states: 23% user 3% system 0% nice 74% idle 0% iowait 0% irq 0% softirq
    CPU4 states: 33% user 4% system 0% nice 63% idle 0% iowait 0% irq 0% softirq
    CPU5 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
    CPU6 states: 67% user 1% system 0% nice 32% idle 0% iowait 0% irq 0% softirq
    CPU7 states: 4% user 0% system 0% nice 96% idle 0% iowait 0% irq 0% softirq
    Memory: 1964036k total, 1346896k used (68.6%), 369604k free (18.8%), 247536k freeable (12.6%)
    Average network usage: 181 / 39 kbps in 1 minute, 216 / 63 kbps in 10 minutes, 229 / 71 kbps in 30 minutes
    Maximal network usage: 413 / 340 kbps in 1 minute, 1456 / 1321 kbps in 10 minutes, 2505 / 2351 kbps in 30 minutes
    Average sessions: 45 sessions in 1 minute, 45 sessions in 10 minutes, 44 sessions in 30 minutes
    Maximal sessions: 80 sessions in 1 minute, 80 sessions in 10 minutes, 80 sessions in 30 minutes
    Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
    Maximal session setup rate: 9 sessions per second in last 1 minute, 9 sessions per second in last 10 minutes, 9 sessions per second in last 30 minutes
    Average NPU sessions: 7 sessions in last 1 minute, 8 sessions in last 10 minutes, 8 sessions in last 30 minutes
    Maximal NPU sessions: 7 sessions in last 1 minute, 11 sessions in last 10 minutes, 11 sessions in last 30 minutes
    Average nTurbo sessions: 7 sessions in last 1 minute, 8 sessions in last 10 minutes, 8 sessions in last 30 minutes
    Maximal nTurbo sessions: 7 sessions in last 1 minute, 11 sessions in last 10 minutes, 11 sessions in last 30 minutes
    Virus caught: 0 total in 1 minute
    IPS attacks blocked: 0 total in 1 minute
    Uptime: 41 days,  5 hours,  18 minutes

    In this CLI output, we can see very clear if the cpu or the memory is the limiting factor on our appliance.

    Let’s first focus on memory troubleshooting and discuss the cpu load troubleshooting as a second topic.


    Memory Troubleshooting

    FortiGate memory is splitted into different parts. Cached, slab, kernel, shared and user space memory are the important and good to know parts. Let’s see what those different parts are used for on the FortiGate and how to determine the current load in the respective part of the memory:

    • Kernel memory
      • Contains the memory used by the operating system kernel and modules as well as drivers.
    • Shared memory
      • Memory ressources are allocated for a specific process. A process is able to access only it’s own memory ressources. Shared memory (SHM) contains information that needs to be available for multiple processes.
      • As example, databases like the IPS database are stored in the shared memory so that multiple processes can access the information.
      • You can view the amount of available shared memory with the following CLI command:
    gw01 (global) # diag hardware sysinfo shm
    SHM FS total:      1379164160       1315 MB
    SHM FS free:       1376931840       1313 MB
    SHM FS avail:      1376931840       1313 MB
    SHM FS alloc:         2232320          2 MB
    • User space memory
      • User Space memory contains the memory used by active processes. The processes are running under a user account (one or more process per account, depending from implementation).
      • The CLI command “diagnose sys top” lists all running processes. In this list, that is being refreshed every second, you can sort the output by typing C, for CPU or M, for memory. The last column of the “diag sys top” output shows the ID of the CPU core on which this process is running. This columns was not shown before FortiOS 7.2.
    gw01 (global) # diagnose sys top
    
    Run Time:  42 days, 2 hours and 17 minutes
    0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 1918T, 395F
              httpsd     1043      S       2.4     1.1    2
                node     9733      S       0.9     3.8    0
           ikecryptd      223      S       0.4     0.4    1
           ipsengine      351      S <     0.0     5.0    7
           ipsengine      353      S <     0.0     4.8    6
           ipsengine      352      S <     0.0     4.7    5
             cmdbsvr      148      S       0.0     3.0    7
    [...]
    • Cached memory (System I/O Cache)
      • Cached memory is memory that the FortiGate uses for disk caching. The FortiGate writes data from this part of the ram to the disk. Cached memory does not count as “used” memory, since it will be freed when applications require it. Therefore it is no reason to be worried about if the cached memory cosumes much space.
      • Harddisk logging, Explicit Proxy Utilization or WAN optimization can lead to more memory consumption in the cached section since those applications do use the harddisk.
      • If a harddisk is failing in a FortiGate, there are several indicators for this. Apart from file system errors in the crash log and miglogd processes with status “Z” (zombie) in the “diag sys top” CLI utility, an increased cached memory can also indicate a problem with the hard disk. If the disk has failed completely, it is just “gone” and will not cause any troubles anymore (beside that nothing is being logged anymore).
      • There are two types of cached memory: active and inactive. Active pages were recently accessed from the kernel, while inactive pages were inactive for some time. If there is a memory shortage, the kernel may reclaim those inactive pages to store more relevant information.
      • You can query how much cached memory is used by using the following CLI command:
    gw01 (global) # diag hardware sysinfo memory 
    MemTotal:        1964020 kB
    MemFree:          441880 kB
    Buffers:            9208 kB
    Cached:           479412 kB
    SwapCached:            0 kB
    Active:           801652 kB
    Inactive:         150680 kB
    Active(anon):     658728 kB
    Inactive(anon):    49572 kB
    Active(file):     142924 kB
    Inactive(file):   101108 kB
    Unevictable:      111664 kB
    Mlocked:              44 kB
    SwapTotal:             0 kB
    SwapFree:              0 kB
    Dirty:                 0 kB
    Writeback:             0 kB
    AnonPages:        575424 kB
    Mapped:           179212 kB
    Shmem:            132968 kB
    Slab:             217484 kB
    SReclaimable:      12096 kB
    SUnreclaim:       205388 kB
    KernelStack:        3776 kB
    PageTables:        33548 kB
    NFS_Unstable:          0 kB
    Bounce:                0 kB
    WritebackTmp:          0 kB
    CommitLimit:      982008 kB
    Committed_AS:   12881840 kB
    VmallocTotal:   260046784 kB
    VmallocUsed:       93784 kB
    VmallocChunk:   259828816 kB
    • Slab Memory
      • Slab is a memory allocation mechnism, that allocates memory ressources for often used elements that need the same amount of space for every similar element. It is like a pre-defined table to which you can add or remove rows dynamically. Those elements are as example sessions, nat table entries, routes and so on.
      • You can diagnose the use of slab memory with the CLI utility “diagnose hardware sysinfo slab”:
    gw01 (global) # diagnose hardware sysinfo slab
    slabinfo - version: 2.1
    # name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
    packet_de_duplication6      0      0    192   20    1 : tunables  252  126    8 : slabdata      0      0      0
    packet_de_duplication      0      0    128   30    1 : tunables  252  126    8 : slabdata      0      0      0
    ip6_nat_record         0      0    128   30    1 : tunables  252  126    8 : slabdata      0      0      0
    tcp6_session           0      0   1472    5    2 : tunables   60   30    8 : slabdata      0      0      0
    ip6_session            0      0   1344    3    1 : tunables   60   30    8 : slabdata      0      0      0
    ip_nat_record         16     59     64   59    1 : tunables  252  126    8 : slabdata      1      1      0
    sctp_session           0      0   1536    5    2 : tunables   60   30    8 : slabdata      0      0      0
    tcp_session           60     60   1536    5    2 : tunables   60   30    8 : slabdata     12     12      0
    ip_session            50     60   1344    3    1 : tunables   60   30    8 : slabdata     20     20      0
    fib6_nodes            28    118     64   59    1 : tunables  252  126    8 : slabdata      2      2      0
    ip6_dst_cache         87    170    384   10    1 : tunables  124   62    8 : slabdata     17     17      0
    [...]

    You can see on this output, how much memory is being consumed by sessions (splitted into IPv6 and IPv4 sessions).

    You can multiply the column “num_objs” and “objsize” together, to get the space in bytes used by one type of slab.


    What are performance intensive workloads on the FortiGate?

    You can do some things to optimize the performance of your FortiGate by disabling or reducing performance intensive tasks. Some of the most performance consuming tasks of the FortiGate are:

    • WAN Optimization (Is a performance intensive task. But we see in our work, that this feature is not really in use anymore, since WAN bandwidth is not a bottleneck to worry about.)
    • Threat feeds (Fetching data can utilize a lot of memory when big files need to be read into the FortiGate.)
    • Logging to RAM and Disk as well as report generation (Safe logs, view logs and generate log reports directly on the FortiAnalyzer to optimize the FortiGate performance.)
    • Proxy based UTM inspection (Flow based inspection is the default and Fortinet recommends to use flow based inspection wherever possible. Use proxy based only where absolutely needed.)

    Furthermore there are some other measures you can take to optimize system performance:

    • Tune the number of process workers (wad, ipsengine, sslvpn, scanunit, etc.). TAC support (support.fortinet.com) is available to optimize the count of worker processes in your specific setup. Every worker needs its own part of memory. Having too many or too few worker units may cause high CPU and memory load, packet drops and even unevenly distributed load on the CPU cores. In general, a lower number of workers will slow down traffic (because load balancing of traffic between two workers is less efficient than on four workers) handling and maybe (if count is too low) cause packet drops, but it will prevent the FortiGate from going into conserve mode.
    • Extend the allocated memory size on VM based systems. Modern FortiOS licenses do not limit memory on VMs anymore. Therefor a conserve mode situation on a VM can be resolved pretty quick by allocating more memory to the FortiGate VM in many cases.
    • Take into consideration, that hardware offloading on a physical FortiGate appliance is a ultimatively powerfull performance booster. Therefore, we recommend to offload and accelerate as much of the workload as possible. Since this is a very broad topic, we have created another blog post regarding hardware acceleration troubleshooting for you.

    CPU Troubleshooting

    If you can see with the CLI utility “get system performance status”, that the CPU load is too high, you may want to know which process is the cause of the high load. On the FortiGate we have the well known tool named “top” available for this troubleshooting step. You can open top with the command “diagnose sys top” on the FortiGate CLI.

    The CLI command “diagnose sys top” lists all running processes. In this list, that is being refreshed every second, you can sort the output by typing C, for CPU or M, for memory.

    gw01 (global) # diagnose sys top
    
    Run Time:  37 days, 4 hours and 29 minutes
    0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 1917T, 456F
                node    25757      S       0.9     2.7    2
           forticron      191      S       0.4     1.4    7
           ipsengine      416      S <     0.0     5.3    7
           ipsengine      414      S <     0.0     5.3    5
           ipsengine      415      S <     0.0     5.2    6
             cmdbsvr      144      S       0.0     2.8    0
           ipshelper      208      S <     0.0     2.5    2
                 wad      261      S       0.0     2.2    2
             fgtlogd      220      S       0.0     2.0    5
             miglogd      201      S       0.0     2.0    1
           scanunitd      214      S <     0.0     1.8    2
              cw_acd      250      S       0.0     1.7    3
           extenderd      271      S       0.0     1.6    1
             sslvpnd      209      S       0.0     1.4    3
                csfd    32583      S       0.0     1.4    3
             miglogd      356      S       0.0     1.4    2
              httpsd      182      S       0.0     1.2    4
             sslvpnd      281      S       0.0     1.2    6
               fgfmd      249      S       0.0     1.2    4
                 cid      263      S       0.0     1.2    3

    Process states

    The FortiGate knows the following process states:

    Killing processes

    Since it is very prone to problems if you just “kill” a task on the FortiGate, we do not recommend to wildly kill any task in the hope to solve a problem. But if you are in a non productive environment you may be interested about the behavior of the FortiGate if a specific task is being killed. In this case, Fortinet has created a manual on how to kill a process on the FortiGate. If there are multiple processes with the same name running, you may want to take a look into this KB article where “killall” is documented.

    If you kill a process, you need to specify a termination signal. The following termination signals are available:

    Signal numberDescription
    4Illegal instruction
    6Abort command from FortiOS
    7Bus error
    9Unconditional kill
    11Invalid memory reference
    14Alarm clock
    15Graceful kill

    You can find the process ID by using one of the following commands:

    gw01 (global) # diagnose sys process pidof httpsd
    173
    5720
    5811
    5816
    5818
    5915
    
    gw01 (global) # fnsysctl ps
    PID       UID     GID     STATE   CMD
    1         0       0       S       /bin/initXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 
    2         0       0       S       [kthreadd]
    3         0       0       S       [ksoftirqd/0]
    4         0       0       S       [kworker/0:0]
    5         0       0       S       [kworker/u:0]
    6         0       0       S       [migration/0]                     
    5940      0       0       S       /bin/httpsd                              
    5944      0       0       S       /bin/httpsd       

    Now you know which process ID’s are running for the “https” process. You may now decide to kill only one of them by using the “kill” command as following:

    gw01 (global) # diagnose sys kill <termination-signal> <process-id>

    But if you decide to kill all the running processes of “https”, you can use the following command:

    gw01 (global) # fnsysctl killall <process-name>

    Please note: We mentioned it before, but we really do not recommend to kill any processes on the FortiGate since is is not necessary in most cases and may also cause system instability and unpredictable behaviour.


    Processes on the FortiGate

    The FortiGate is running a huge amount of different processes on it’s system. Every process has it’s own tasks to fulfill. Since Fortinet does not provide a full list of all available processes, we try to have a hopefully complete list of all processes here.

    If you find another process which is not yet documented here, please comment the reported process as a comment below. If you already know what the process is for, please also mention this so we can supplement this information in the table below.

    ProcessFull nameDescription and additional information
    pyfcgidCore processMain/Core running process of FGT
    wadWebproxy DaemonDoes inspection of all “proxy based” inspection for HTTP, HTTPS and FTP as well as explicit proxy operations. WAN optimization (caching).
    wad_diskdWAN acceleration disk daemonWAN acceleration disk daemon
    wad_usrinfohistory
    httpsdHTTPS Webserver daemonWebserver for WebAdmin GUI
    dhcpdDHCP server ####falsch!!!DHCP server process
    dhcprdDHCP relayDHCP relay process
    dhcpcdDHCP clientDHCP client process
    dhcp6sIPv6 DHCP serverDHCP server process for IPv6
    dhcp6rIPv6 DHCP relayDHCP relay process for IPv6
    dhcp6cIPv6 DHCP clientDHCP client process for IPv6
    cmdbsvrDatabase serverConfiguration and management database server
    (According to KB 195863, also updates. But there is an update process also.)
    forticronCRL update process
    fgtlogd
    miglogdLogging daemonThis process is handling all logging operations on the Fortigate. Since it also writes the logs to the disk, it sometimes causes problems when the disk is too slow in writing logs or if the disk is failing.
    extenderdFortiExtender daemonManagement and control of FortiExtender devices.
    sslvpndSSL-VPN Server daemonServes the SSL VPN portal for web- and tunnel based SSL VPN access.
    nodenode.js processIn FortiOS 6.4, the node process is used for
    Report management (which includes Security Fabric (csf) / Fortiview / Security Rating). WebSockets. Maintaining the CLI console widget when accessing the Fortigate via HTTP/HTTPS. In FortiOS 7.0 onwards, the node process is also responsible for: processing all incoming HTTP/HTTPS to serve static files (prior to 7.0 the process HTTPSD served static files).
    On 7.0. the 3 main node.js scripts on a FortiGate are for: Report runner (Security Rating). CLI console. SSL VLN QR code generation.
    updatedUpdate daemonChecks for Updates of the FortiGate licensing status, the FortiOS and the FortiGuard signature databases.
    Troubleshooting: diagnose debug application updated -1
    Force Update: execute update-now
    ipsengineIPS EngineThe IPS engine is responsible for all flow based inspection on the FortiGate. This process does the packet inspection. Count of simultaneous running engines id depending from the model and configuration.
    ipsatest(Suspicion: “diag test application ipsmonitor” process)
    ipsmonitorIPS monitoringWatchdog and diagnostics process for the IPS engine
    ipshelperIPS helper and configurationConfiguration management inside the IPS engine. Looks for CMDB changes affecting the IPS engine. Compiles the IPS rule DB and generates the direct filter approach.
    scanunitdFilescanner daemonFile scanning daemon (Antivirus scan)
    npdNP daemonNetwork Processor daemon (hardware acceleration)
    samldSAML daemonSAML authentication daemon
    Troubleshooting: diagnose debug application samld -1
    cw_acdWireless daemonThe process cw_acd handles communication between Fortigate and APs.
    cw_acd_helper
    eap_proxyEAP Proxy daemonExtensible Authentication Proxy process
    newcliShell access daemonCLI commands execution – Used over ssh, gui-cli and telnet
    hasyncHA sync daemonHigh availability synchronization process (configuration, files, sessions, ppp & dhcp status, …)
    hatalkHA talk daemonHigh availability heartbeat process & primary / secondary decision process
    fssoFSSO communication daemonFortinet single-sign-on communication daemon
    fsso_ldapFSSO LDAP lookup daemonFortinet single sign on LDAP query daemon
    ikedIPSec IKE daemonInternet Key Exchange daemon for IPSec key exchange
    ikecryptdIPSec IKE cryptography daemonInternet Key Exchange daemon for IPSec cryptography tasks
    proxyworkerProxy-based IMAP, POP and SMTP daemonDaemon for proxy based IMAP, SMTP and POP inspection
    l2tpdL2TP daemonLayer 2 tunneling protocol daemon
    l2tpcdL2TP client daemonLayer 2 tunneling protocol client daemon
    pptpdPPTP daemonPoint to point tunneling protocol daemon
    pptpcdPPTP client daemonPoint to point tunneling protocol client daemon
    forticlddFortiCloud daemonFortiCloud connection daemon
    alertmailAlert Email daemonAlert Email generating daemon
    urlfilterWebfilter daemonFortiGuard webfilter daemon
    zebos_launcherZEBOS Launcher daemon
    snmpdSNMP daemonSimple network management protocol (SNMP) server process
    ospfdOSPF router serverOSPF router server daemon
    ospf6dIPv6 OSPF router serverIPv6 OSPF router server daemon
    bgpdBGP router serverBGP router server daemon
    pppoedPPPoE daemonEncapsulating and decapsulating traffic in PPP over Ethernet (DSL)
    pppdPPP daemonEncapsulating and decapsulating traffic in PPP
    ipldbdIP load balancing daemonIP load balancing daemon
    acdaggregate controller daemon
    vpdvpn policy daemon – handle vpn traffic to know to which policy the traffic corresponds
    rlogdreliable syslog daemon
    acs-sdn-change
    acs-sdn-status
    acs-sdn-update
    alarmd
    authdUser authentication daemon
    autod
    awsd
    azd
    chat
    chlbd
    cid
    clearpass
    cli_grepGrep toolThe grep binary to use on the FGT Admin CLI
    confsyncd
    confsynchbd
    csfdSecurity fabric daemonSecurity Fabric daemon
    cu_acd
    ddnscdDynDNS client daemonDynamic DNS client daemon
    dnp
    dnsproxyDNS Proxy daemonDNS proxy (forward, recursive and dns database queries). Also, information about the applied DNS filter profile is debugged by using diag deb appl dnsproxy.
    dpdk_early_init
    dsd
    e2fsck
    eap_proxyExtensible Authentication Protocol proxyExtensible Authentication Protocol proxy process
    eap_supp
    fas
    fclicense
    fcnacd
    fnbamdFortinet non blocking authentication daemonFortinet non blocking authentication daemon
    fctrlproxyd
    flpold
    fds_notify
    fgfmdFortiManager daemonDaemon that handles the communication with the FortiManager
    fips_self_test
    flcfgdFortiLink daemonFortiLink controlled by FortiGate daemon (The FortiGate part of the FortiLink controller)
    foauthd
    forticron
    fortilinkd
    fsd
    ftm2
    garpd
    gcpd
    getty
    grep
    hamonitord
    harelay
    hotplug
    httpclid
    httpsnifferd
    iflpd
    imi
    inat
    iotd
    ipamd
    ipamsd
    iperfiperf processThe command “diag traffictest” uses iperf as testing tool.
    ipmc_sensord
    ipmitool
    isisdISIS daemonISIS routing service daemon
    killallKill all processUsed to kill all processes running under the same name.
    kmiglogd
    kubed
    lldprx
    lldptx
    lnkmtd
    lpmd
    lspci
    ltedUSB LTE daemonUSB LTE modem daemon
    memuploadd
    merged_daemons
    mke2fs
    mlcli
    mld
    modemdModem daemonModem daemon
    netxd
    npd
    nqd
    nsm
    ntpd
    ocid
    openstackd
    ovrd
    pdmd
    pim6dIPv6 multicast routing daemonPIM-SM/SSM multicast routing daemon for IPv6
    pimdIPv4 multicast routing daemonPIM-SM/SSM multicast routing daemon for IPv4
    proxyd
    ptpd
    quard
    radius-das
    radiusd
    radvd
    reportd
    resize2fs
    ripdRIP routing server daemon for IPv4RIP routing server daemon for IPv4
    ripngdRIP routing server daemon for IPv6RIP routing server daemon for IPv6
    rm
    rtmon
    scp
    sipSIP ALGSIP application layer gateway process
    sdncd
    sdnd
    sepmd
    setpci
    sflowd
    sfupgraded
    sg_raw
    shShow commandShows the FortiGate configuration in the CLI
    sla_probe
    slbc_garpsd
    smartctl
    smbcd
    smit
    snifferd
    speedtestd
    sshSSH clientSSH client process
    ssh-keygenSSH keygenSSH keygen process
    sshdSSH serverSSH server daemon
    stpdSTP daemonSpanning tree protocol daemon
    sysctl
    syslogdSyslog clientSyslog client service daemon
    telnetdTelnet serverTelnet server daemon
    tftpTFTP clientTFTP client process
    tune2fs
    tvc
    uploadd
    usb_modeswitch
    usbmuxd
    vned
    voipd
    vwl
    wf_monitor
    wiredapd
    wlacWireless controller serverWireless controller server
    wlac_hlp
    wmwd
    wpad_ac
    wpad_client
    This list shows the processes that can run on a FortiGate appliance and describes what they do.

    Information sources

    You want to know more regarding FortiGate troubleshooting? We offer the Fortinet Network Security Support Engineer (FCSS) training in our Fortinet Authorized Training Center in Wettingen.

    All CLI commands are listed in our FortiOS Cheat Sheet:

    https://blog.boll.ch/category/cheatsheet/

    Memory Troubleshooting information:

    https://community.fortinet.com/t5/FortiGate/Technical-Tip-Memory-Debugs/ta-p/202800

    List of processes on the FortiOS:

    https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-list-processes-in-FortiOS/ta-p/195863

    https://community.fortinet.com/t5/FortiGate/Technical-Tip-Short-list-of-processes/ta-p/190775

    WAD process structure:

    https://community.fortinet.com/t5/FortiGate/Technical-note-Overview-of-WAD-process-structure/ta-p/197183

    IPS Engine structure:

    https://community.fortinet.com/t5/FortiGate/Technical-Tip-Introduction-of-IPS-process/ta-p/217550

    Conserve Mode troubleshooting:

    https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-How-to-do-initial-troubleshooting-of-high/ta-p/194874

    Process troubleshooting information:

    https://docs.fortinet.com/document/fortigate/6.2.13/cookbook/76710/running-processes

    Kill processes on the FortiOS:

    https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-restart-kill-all-processes-with-the/ta-p/194434

    Loading

    2 thoughts on “FortiGate Memory and CPU Troubleshooting

      • vla Post authorReply

        Dear Peter
        Thank you for your comment.
        We are glad that you like our post 🙂
        Kind regards from the
        Boll Tech Team

    Leave a Reply

    Your email address will not be published. Required fields are marked *