Common issues

Installation

Issue - scontrol \ sacct command not found

OKA cannot find SLURM binaries.

Reason

The SLURM commands are not available in the environment of OKA.

Solution

Modify ${OKA_INSTALL_DIR}/conf/environment.sh to source the missing environment variables. The needed variables are often defined in /etc/profile.d directory. The environment.sh file will be used to set up the OKA environment.

#!/bin/bash
# Environment settings
# Add here all environment variables needed for OKA to work properly (e.g. to access JobScheduler bins).
# You can also directly source a bash script.
export PATH="$PATH:/opt/local/bin"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/opt/local/lib"
export MANPATH="${MANPATH}:/opt/local/share/man"
export TERM=xterm

Issue - ModuleNotFoundError: No module named '_curses'

Installation crashes due to a missing python module.

Reason

The Python3.x found by OKA has not been compiled properly and a library is missing.

Solution

• Install the missing ncurses-devel package.

• Recompile Python and retry (see Python requirement guideline for an example).

• If you are thinking of using a different version and/or to have a new path for the installation of Python, remember to clean the existing {OKA_INSTALL_DIR} created by previous installation attempt. At least, there is a need to remove the file Pipfile and the hidden folder .venv in order to avoid potential error of selection between the multiple Python version.

Update

Issue - Disk usage exceeded flood-stage watermark

Elasticsearch does not have sufficient disk space to work.

Reason

There is not enough space for Elasticsearch to store its data or to migrate its database.

Solution

• Free some disk space: at least for the time of the update to allow the database migration. A rule of thumb is to have at least twice the size of the largest Index in Elasticsearch as free space. To list the available Indexes you can run the following command: curl 'host:port/_cat/indices?v' (with host and port the hostname/IP and port to contact the Elasticsearch server).

• Make sure that the free disk space will stay below the flood_stage threshold in Elasticsearch configuration. You can check these values by running the following command

  curl -X GET "host:port/_cluster/settings?include_defaults=true&pretty"
  

  Extract of output

  {
  "defaults" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "disk" : {
            "threshold_enabled" : "true",
            "watermark" : {
              "flood_stage" : "95%",
              "high" : "90%",
              "low" : "85%",
              "enable_for_single_data_node" : "false"
            }
  }}}}}}
  

• Unblock Elasticsearch indices that shifted to read-only states:

  curl -XPUT -H "Content-Type: application/json" http://host:port/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
  

  Warning

  This step can take some time to execute, depending on the size of your indexes in Elasticsearch.

• Clean the Elasticsearch indices: curl -X POST "host:port/_forcemerge?pretty".

• Turn off the verification of the available space left for the duration of the update:

  curl -XPUT -H "Content-Type: application/json" http://host:port/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
  

• Restart the update.

Tensorflow

Issue - tensorflow.python.framework.errors_impl.NotFoundError

tensorflow.python.framework.errors_impl.NotFoundError:
`/usr/local/lib64/python3.9/site-packages/tensorflow/core/kernels/libtfkernel_sobol_op.so:
undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb`

Reason

Pip must have all access to install tensorflow. Must use sudo to probably access non-python packages.

Solution

pip3 uninstall tensorflow
[sudo yum groupinstall "Development Tools"]
sudo pip3 install tensorflow==2.4.1

NGINX

Issue - 502 bad gateway

OKA cannot be reached by Nginx.

Reason

Several reasons can be the cause of this problem:

1. Nginx might not be configured properly.

2. On RHEL environment, you probably need to check SELinux configuration.

Solution

1. There is an Nginx configuration file for OKA located in ${OKA_INSTALL_DIR}/conf. Copy this file into Nginx configuration directory:

   sudo cp ${OKA_INSTALL_DIR}/conf/oka.nginx.conf /etc/nginx/conf.d/
   sudo systemctl restart nginx
   

2. Disable SELinux by either:

   • running setenforce 0 as root to disable it temporarily

   • editing /etc/selinux/config and set SELINUX=disabled, then reboot the system

   Check the documentation: https://www.redhat.com/en/topics/linux/what-is-selinux

---

Issue - The layout of OKA interface is missing and no images are displayed

Nginx does not serve OKA files properly.

Reason

The Nginx user might not have the rights to access the static files of OKA.

Solution

The user running Nginx must have read and execute access on the whole folder tree to ${OKA_INSTALL_DIR}/data. For example, if OKA is installed in /opt, the Nginx user must have read and execute rights on /opt folder to be able to navigate to ${OKA_INSTALL_DIR}/data directory.

• Either change the Nginx user to a user with the correct access rights.

• Or change the directories rights to give read and execute rights to the Nginx user. The user running Nginx is system specific (it might be nginx, www-data…).

---

Issue - 414 URI Too Long

Nginx does not allow large sized requests.

Reason

The Nginx configuration is set to allow 8k maximum for a request when the current request is probably of a bigger size and is therefore not allowed to be processed.

Solution

You can update you current Nginx configuration to allow for larger requests by adding or updating the following option:

• large_client_header_buffers 4 16k;

---

Issue - 403 CSRF verification failed

Connection to OKA is not possible due to Django rejecting CSRF token validation.

Reason

Django will not allow connection if the origin is not trusted.

Solution

Update your OKA configuration to trust your current instance domain either in http or https depending on your Nginx configuration.

• Edit/add the CSRF_TRUSTED_ORIGINS option in ${OKA_INSTALL_DIR}/conf/oka.conf.

• Then restart OKA: sudo systemctl restart OKA.

Gunicorn

Issue - 400 bad request

Gunicorn does not allow your host reach OKA.

Reason

Configuration of the ALLOWED_HOST parameter is probably wrong in oka.conf

Solution

Update ALLOWED_HOST parameter in file ${OKA_INSTALL_DIR}/conf/oka.conf to give access yo specific host/domain names Django can serve (https://docs.djangoproject.com/en/3.1/ref/settings/#allowed-hosts), or set it to '*' to allow all hosts.

---

Issue - 110: Connection timed out (Nginx/Gunicorn)

Long requests fail.

Reason

Configuration of the TIMEOUT parameter is probably too short in oka.nginx.conf and/or gunicorn.conf to increase the reading time out of the Gunicorn.

Solution

Update TIMEOUT parameter in file ${OKA_INSTALL_DIR}/conf/gunicorn.conf

and/or

Update or add proxy_connect_timeout and proxy_read_timeout parameters in ${OKA_INSTALL_DIR}/conf/oka.nginx.conf to increase or defines a timeout for reading a response from the proxied server (Gunicorn). https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout

Elasticsearch

Issue - java.lang.OutOfMemoryError: Java heap space

Not enough memory.

Reason

Your JVM heap size is probably too low.

Solution

Edit the JVM configuration (/etc/elasticsearch/jvm.options) and set the total heap space either to:

• Half of the node’s total memory. For example, if the total memory is 32GB, set -Xms16g and -Xmx16g.

• Or comment those 2 lines to let Elasticsearch manage the heap space (see set-jvm-heap-size).

---

Issue - java.lang.IllegalStateException: failed to obtain node locks / java.io.IOException: No locks available

Problem while accessing Elasticsearch files.

Reason

Elasticsearch is incompatible with NFSv3.

Solution

If you must stick with NFS, use NFSv4 instead of NFSv3.

Warning

Elasticsearch had been know to work on NFSv4, but this is not an officially supported deployment.

---

Issue - Limit of total fields [1000] has been exceeded

Problem while training Predict-IT.

Reason

Elasticsearch limits to 1000 the number of fields in the index mapping.

Solution

You must increase the number of fields allowed for the index:

curl -s -XPUT http://host:port/index-name/_settings  -H 'Content-Type: application/json' -d '{"index.mapping.total_fields.limit": 2000}'

---

Issue - Out of memory

Not enough memory.

Reason

When training many predictors at the same time in Predict-IT or doing heavy analysis, it can happen that Elasticsearch runs out of memory and crashes. One way to prevent a downtime of OKA is to setup Elasticsearch services to automatically restart Elasticsearch in case of failure.

Solution

Automatically restart Elasticsearch in case of failure.

You need to add the following configurations in the Elasticsearch Systemd service file (e.g., /lib/systemd/system/elasticsearch.service):

[Unit]
...
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
...
Restart=on-failure
RestartSec=5

Then reload Systemd: sudo systemctl daemon-reload.

You can test the auto-restart by killing the Elasticsearch process: kill -9 <ES_PID>, and checking that the service restarts: systemctl status elasticsearch.

PostgreSQL

Issue - Cannot connect to PostgreSQL database

postgresql logs:
  FATAL: could not access private key file “/etc/ssl/private/ssl-cert-snakeoil.key”: Permission denied

systemctl status okaserver.service
  okaServer.sh[1234]: django.db.utils.OperationalError: connection to server at "localhost" (::1), port 5432 failed: Connection refused
  okaServer.sh[1234]: connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused

Reason

The access rights on the folder containing the key file might have changed.

Solution

Update the rights on the /etc/ssl/private/ folder as follow:

chmod 710 /etc/ssl/private/

ls -ld /etc/ssl/private/
drwx--x--- 2 root ssl-cert 4096 Jan 20 00:00 /etc/ssl/private

Rosetta

Issue - Rosetta does not take into account the changes.

Problem while modifying/translating some strings.

Reason

It is using the dictionary saved in cache.

Solution

Restart OKA server to update Rosetta’s dictionary.

Log Ingestion

Issue - Expected X fields in line Y, saw Z. Error could possibly be due to quotes being ignored when a multi-char delimiter is used.

Problem while ingesting Slurm logs.

Reason

Parsing logs extracted from Slurm accounting is done using a separator character to first identify all available fields in the data file. By default, the separator used by Slurm is set to | so when calling sacct to extract the logs you will receive something like this:

field0|field1|field2

Because users can still use the | character in their submit command lines or job names etc. you might find yourself in cases where an extra separator exists within the data. You can therefore end up with some cases like those:

field0|fie|ld1|field2
or
field0|fie_|_ld1|field2

Solution

The Slurm parser will handle most cases automatically however, we still cannot guaranty that all of them will be detected. To avoid this type of problems we recommend to either:

• Add this option to your sacct call to specify a custom separator: --delimiter=@|@

• Directly use our extraction script (see Retrieve job scheduler logs)

• Let OKA handle the extraction by connecting it to the scheduler. For this you need to change the Command type in the configuration for your cluster in DATA MANAGER > Conf job schedulers to be LOCAL, FORWARDED_PWD or FORWARDED_KEYFILE (see Other variables for more info regarding those parameters)

---

Issue - Data are displayed on the wrong timezone.

Problem while visualizing Slurm logs ingested directly through an sacct call.

Reason

Due to the way sacct works, the datetime data retrieved when calling the scheduler are set in the timezone of the current environment from where the call originated. When creating a cluster (see Clusters management), the default timezone is set to UTC in its configuration.

Solution

In order to fix this, you can switch to the appropriate timezone in the cluster configuration (see Cluster configurations) accessible through the admin panel. You will then have to clean your existing data affected by the previous upload on the wrong timezone and let OKA call sacct again with the new configuration.