Skip to content

Attack Detection and Blocking Troubleshooting

Attacks are not displayed

If attacks are not displayed the Cloud and the gauge-export_drops_flag metric gets the 1 value, this means the uploading of attacks from the node to the Cloud has stopped. Follow the steps to check and and fix this:

  1. Make sure that the filtering node mode is different from off. The node does not process incoming traffic in the off mode.

    The off mode is a common reason for the wallarm-status metrics not to increase.

  2. If the node is NGINX-based, restart NGINX to be sure that settings have been applied:

    sudo systemctl restart nginx

    sudo service nginx restart

    sudo systemctl restart nginx

    sudo systemctl restart nginx

    sudo systemctl restart nginx

  3. Check whether the script for uploading data to the analytical cluster has been run:

    ps aux | grep wcli

  4. If the process is not running, start the supervisord service.

  5. Make sure in the /opt/wallarm/etc/supervisord.conf or /opt/wallarm/etc/supervisord.conf.postanalytics the wcli script is configured properly (mode is should be NOT specified - by default it is all or should be -mode post_analytic).

  6. Check logs:

    grep reqexp /opt/wallarm/var/log/wallarm/wcli-out.log

  7. Check whether the wallarm and tarantool services are running:

    ps aux | grep wallarm-tarantool

  8. If it is not, run it:

    systemctl start wallarm

  9. If this did not help, check IP:port on which tarantool is running in the /opt/wallarm/env.list file, section #tarantool, HOST and PORT parameters. Set:

    1. HOST=127.0.0.1
    2. PORT=3313

    If the postanalytics module is installed at a separate server, check and configure the same data in /opt/wallarm/etc/wallarm/node.yaml.

  10. Restart the wallarm service:

    systemctl restart wallarm

  11. If this did not help, check request timeouts in wcli-out.log:

    grep reqexp /opt/wallarm/var/log/wallarm/wcli-out.log

  12. If request timeout errors are presented, check the availability of Wallarm's API, grant access to it, it it is not granted yet.

  13. If some other errors are presented in wcli-out.log, contact the Wallarm support team.

Some attacks are not displayed

If some attacks are not displayed the Cloud and the gauge-export_drops_flag metric gets the 1 value, this means that requests with attacks have been deleted from the postanalytics module but not sent to the Cloud. Usually this happens due to the insufficient memory allocated form the postanalytics module. Follow the steps to check and and fix this:

  1. Check logs:

    grep reqexp /opt/wallarm/var/log/wallarm/wcli-out.log

  2. If there are errors in the log, act as described here.

  3. If this did not help, check the timeframe_size metric value:

    /opt/wallarm/usr/bin/collectdctl -s /opt/wallarm/var/run/wallarm-collectd-unixsock getval "$(/opt/wallarm/usr/bin/collectdctl -s /opt/wallarm/var/run/wallarm-collectd-unixsock listval | awk -F "/" '{ print $1 }' | head -n1)"/wallarm-tarantool/gauge-timeframe_size

    If the value is < 300, in /opt/wallarm/env.list, increase the value of SLAB_ALLOC_ARENA and then restart the wallarm service.

  4. If this did not help, check the export_delay metric value:

    /opt/wallarm/usr/bin/collectdctl -s /opt/wallarm/var/run/wallarm-collectd-unixsock getval "$(/opt/wallarm/usr/bin/collectdctl -s /opt/wallarm/var/run/wallarm-collectd-unixsock listval | awk -F "/" '{ print $1 }' | head -n1)"/wallarm-tarantool/gauge-export_delay

    If the value is > 10, then check the stability of connection to Wallarm API.

  5. Restart wallarm and wcli:

    systemctl restart wallarm

Filtering node RPS and APS values are not exported to Cloud

If filtering node information about RPS (requests per second) and APS (attacks per second) are not exported to Wallarm cloud, the possible reason is SELinux.

SELinux is installed and enabled by default on RedHat‑based Linux distributions (e.g., CentOS or Amazon Linux 2.0.2021x and lower). SELinux can also be installed on other Linux distributions, such as Debian or Ubuntu.

Check SELinux presence and status by executing the following command:

sestatus

If the SELinux mechanism is enabled on a host with a filtering node, during node installation or upgrade, the all-in-one installer performs its automatic configuration for the node not to interfere with it.

If after automatic configuration you still experience the problems that can be caused by SeLinux, do the following:

  1. Temporarily disable SELinux by executing the setenforce 0 command.

    SELinux will be disabled until the next reboot.

  2. Check whether the problem(s) disappeared.

  3. Contact Wallarm's technical support for help.

    SELinux permanent disabling not recommended

    It is not recommended to disable SELinux permanently due to the security issues.

Filtering node does not block attacks when operating in blocking mode (wallarm_mode block)

Using the wallarm_mode directive is only one of several methods of traffic filtration mode configuration. Some of these configuration methods have a higher priority than the wallarm_mode directive value.

If you have configured blocking mode via wallarm_mode block but Wallarm filtering node does not block attacks, please ensure that filtration mode is not overridden using other configuration methods:

More details on filtration mode configuration methods →

User gets blocking page after legitimate request

If your user reports a legitimate request being blocked despite the Wallarm measures, you can review and evaluate their requests as this articles explains.

To resolve the issue of a legitimate request being blocked by Wallarm, follow these steps:

  1. Request the user to provide as text (not screenshot) the information related to the blocked request, which is one of the following:

    • Information provided by the Wallarm blocking page if it is configured (may include user’s IP address, request UUID and other pre-configured elements).

      Wallarm blocking page

      Blocking page usage

      If you do not use the default or customized Wallarm blocking page, it is highly recommended to configure it to get the appropriate info from user. Remember that even a sample page collects and allows easy copying of meaningful information related to the blocked request. Additionally, you can customize or fully rebuild such page to return users the informative blocking message.

    • Copy of user's client request and response. Browser page source code or terminal client textual input and output suits well.

  2. In Wallarm Console → Attacks or Incidents section, search for the event related to the blocked request. For example, search by request ID:

    attacks incidents request_id:<requestId>

  3. Examine the event to determine if it indicates a wrong or legitimate blocking.

  4. If it is a wrong blocking, solve the issue by applying one or a combination of measures:

  5. If the information initially provided by the user is incomplete or you are not sure about measures that can be safely applied, share the details with Wallarm support for further assistance and investigation.

Errors in postanalytics module

If the tnt_errors metric gets +1 at each processed request, this means some problems in Wallarm's postanalytics module functioning occur. Follow the steps to check and and fix this:

  1. Check if wallarm-tarantool service is running:

    systemctl status wallarm

  2. If it is not running, run it:

    start wallarm

  3. Additionally, check the tarantool process managed by supervisord:

    ps aux | grep wallarm-tarantool

  4. Restart tarantool:

    systemctl restart wallarm

  5. If this did not solve the problem, proceed to the next steps.

  6. Check /opt/wallarm/var/log/wallarm/tarantool-out.log, if there are errors like "Index 'primary' already exists" or "can't initialize storage", delete all files from /opt/wallarm/var/lib/wallarm-tarantool and restart Wallarm:

    systemctl restart wallarm

  7. If this did not solve the problem, collect the debug information:

    sudo /opt/wallarm/collect-info.sh

  8. Create ticket in Wallarm support system, providing the collected information.

Filtering node not processing requests

If the segfaults metric gets value different from 0 and increases, there are problems with filtering node processing requests. Follow the steps to check and and fix this:

  1. Check whether other filtering nodes (if any) work properly.

  2. Restart nginx:

    sudo systemctl restart nginx

    sudo service nginx restart

    sudo systemctl restart nginx

    sudo systemctl restart nginx

    sudo systemctl restart nginx

  3. If this did not help, for the machine with problematic node, start collecting core files:

    1. Create folder to store core files mkdir /tmp/cores/.

    2. Make sure NGINX can save files to the folder:

      sudo chown root:root /tmp/cores
sudo chmod 1777 /tmp/cores

    3. To /etc/nginx/nginx.conf, add the strings:

      ...
working_directory /tmp/cores/;
worker_rlimit_core 500M;

    4. Restart NGINX.

  4. If segfaults continues to increase, collect core files, output of dpkg -l and files by the /var/log/nginx/*error*.log mask.

  5. If problem affects significantly the application normal functioning, set filtering node to the off mode temporarily.

  6. Send the collected data to Wallarm support team.

  7. Get a fix data from Wallarm support team.

  8. Check NGINX configuration correctness with nginx -t.

  9. Re-enable Wallarm node by setting mode different from off.

  10. Restart NGINX.

Requests are not proxied to applications

If one of the following happens:

  • Requests reach filtering node but do not reach target applications

  • Filtering node does not process requests

This may mean problems with proxying requests from node to applications. Follow the steps to check and and fix this:

  1. Check NGINX status and restart it if necessary:

    sudo systemctl restart nginx

    sudo service nginx restart

    sudo systemctl restart nginx

    sudo systemctl restart nginx

    sudo systemctl restart nginx

  2. If this did not help, check NGINX logs for incoming requests, if they are absent:

    1. Check connection between the load balancer and the node.
    2. Check the load balancer configuration.

  3. If requests reach filtering node, but do not reach the target application, check connection between node and application.

  4. If you have timeout errors, increase NGINX's proxy_read_timeout and proxy_connect_timeout in /etc/nginx/nginx.conf.