FAQ

OKA lifecycle management

Question - Why is OKA update taking so long?

OKA update can take until several hours or days.

Answer

During the update procedure, migrations are applied to update your Elasticsearch database.

• Each minor version update can comes with its own migration.

• In the case where you are upgrading several versions, the update procedure will chain the successive migrations.

• If you have multiple environments, migrations will be applied to each environment.

• Depending on the migration, an index can be temporarily duplicated to update its documents. This will lead to a brief increase of your database storage and can be time consuming

  • A rule of thumb is to have at least twice the size of the largest Index in Elasticsearch as free disk space,

  • A single migration can take approximately 10 minutes per million documents (jobs) for one index in one environment.

---

Question - How to set up a Disaster Recovery Plan (DRP)?

In order to quickly resume OKA after an unplanned incident, I want to create a DRP.

Answer

In order to recover OKA after an incident, you must backup OKA databases on regular bases. You can manually backup the databases following the Upgrade procedure. Or you can setup an automatic procedure to backup both databases:

1. Automation is available for Elasticsearch using the Snapshot Lifecycle Management API (SLM). Using SLM, you can set up snapshot lifecycle policies to automate the timing, frequency, and retention of snapshots. For more details, see SLM.

2. To backup PostgreSQL database, we recommend to create a cron job that runs the backup pg_dump command available in the Upgrade procedure.

Make sure to backup both databases on the same bases.

---

Question - How to automatically restart OKA in case of failure?

If a problem occurs on the VM where OKA runs, it can happen that OKA crashes. In order to reduce downtime, how can I automatically restart OKA?

Answer

One way to prevent a downtime of OKA is to setup OKA services to automatically restart in case of failure.

OKA services are pre-configured for automatic restart, you just need to update one parameter in the service files (/etc/systemd/system/okascheduler.service and /etc/systemd/system/okaserver.service): Restart=on-failure (the default value is no). Then reload Systemd: sudo systemctl daemon-reload. You can test the auto-restart by killing the OKA processes (server or scheduler): kill -9 <OKA_PIDs>, and checking that the services restarts: systemctl status 'oka*'.

---

Question - How to backup and restore when upgrading the Elasticserch version?

For example, when updating to a OKA 2.X that requires Elasticsearch >=7.14, you might need to migrate and use data if for example the previous Elasticsearch version does not allow for a straight forward update to 7.14. This might be the case if you had an OSS 7.10 version.

Answer

For a smooth upgrade to a new Elasticsearch version, use snapshot and restore. Create a snapshot of your data, then restore it in the upgraded system. Check the Elasticsearch backup and restore documentation for more details Elasticsearch.

Question - How to replicate my production environment into a preproduction environment?

For example, if I want to test OKA update first on the preproduction environment before updating the production environment.

Answer

To create an iso-prod preproduction environment you must follow these steps:

1. Make sure that production and preproduction environments have the same requirements installed (see Requirements).

2. In the production environment, backup both OKA databases following the Elasticsearch and PostgreSQL procedures.

3. In the preproduction environment:

   1. Uninstall OKA if it was already installed following the Uninstalling OKA procedure.

   2. Run the OKA installer (see Installation) corresponding to the version installed in production. Do not start OKA services.

   3. Copy all the directories present in ${OKA_INSTALL_DIR}/data from production to preproduction, apart from the staticfiles directory.

   4. If this is the first time a snapshot repository is declared for Elasticsearch in preproduction, follow the Elasticsearch procedure to configure a snapshot repository but do not create the snapshot directory.

      Note

      A shared volume can be used as snapshot repository for both production and preproduction environments. This way, the preproduction environment will have direct access to backup the snapshots saved by the production environment. If you have shared repositories, proceed directly to step 9.

   5. If a snapshot repository was already declared for Elasticsearch in preproduction, you must delete it to be able to replace it by the one from production:

      1. Delete Elasticsearch snapshot repository curl -XDELETE "host:port/_snapshot/my_backup".

      2. Delete the backup directory sudo rm -rf /opt/esdata/backup.

   6. Copy the Elasticsearch backup directory (snapshot repository) created by the production environment during the backup procedure at the same location in the preproduction environment.

   7. Give ownership to Elasticsearch on the backup directory sudo chown elasticsearch: /opt/esdata/backup.

   8. Create the new snapshot directory for Elasticsearch following the Elasticsearch procedure (last step of the snapshot directory configuration).

   9. Restore both PostgreSQL and Elasticsearch databases following the Elasticsearch and PostgreSQL procedures.

   10. Give the rights to oka user on restored PostgreSQL tables:

       sudo -i -u postgres psql oka_db -c "GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO oka;"
       sudo -i -u postgres psql oka_db -c "GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO oka;"
       

   11. Start OKA services: sudo systemctl start oka.

   12. Connect to the admin panel, go to DATA MANAGER/Conf apis. For each output_db, update the Url parameter to point to the preproduction Elasticsearch. Click on SAVE.

Elasticsearch

Question - How can I move Elasticsearch data directory?

For example, if I want to move my Elasticsearch data directory from /var/lib to /foo/data?

Answer

1. Stop Elasticsearch service: systemctl stop elasticsearch

2. Check the permissions of /var/lib repository. The new repository should have the same permissions to grant Elasticsearch access.

3. Move the data directory to the chosen location: mv /var/lib/elasticsearch /foo/data

4. Make sure that the new directory has the same permissions as of /var/lib/elasticsearch

5. In elasticsearch.yml, modify the path.data option to /​foo/data/elasticsearch (instead of /var/lib/elasticsearch)

6. Restart Elasticsearch service systemctl start elasticsearch

---

Question - What is the maximum length allowed for a string field?

What should be the maximum length of any field contained in the scheduler logs? For example, what can be the maximum length of a submission line? (e.g. SubmitLine field in SLURM.)

Answer

Elasticsearch Lucene’s term byte-length limit is 32766. Lucene counts in bytes therefore, if you use UTF-8 text with many non-ASCII characters, you may want to set the limit to 32766 / 4 = 8191 since UTF-8 characters may occupy at most 4 bytes. For this reason, OKA will automatically truncate a string field containing more than 8191 characters upon parsing the logs.

---

Question - How can we configure a secure connection?

How can we configure OKA to handle user/password connection? How can we configure OKA to handle ssl connection?

Answer

• user/password:

  • Those parameters can be specified before the URL like this: http://user:password@{ELASTICSEARCH_URL}

  • This can be set upon installation when asked for the Elasticsearch URL.

  • It can also be set after installation by updating the ELASTICSEARCH_HOSTS value in the configuration file ${OKA_INSTALL_DIR}/conf/oka.conf. However, this update will only apply to clusters that would be created after the modification.

  • To update the value for existing clusters, there is a need to update directly the impacted ConfAPI objects output_db and their Url field through the admin panel (see Conf apis).

• ssl: You will need to configure your Nginx to provide access to it.URL as you would for the user/password in order to specify https instead of http.

Integration with an AWS cluster

Efficiently running an HPC infrastructure is complex, and often lacks the proper tools to track down and get insights on how the users are behaving and how the cluster is responding to the demand. This is even more complicated when using Cloud clusters. Due to their transient and dynamic nature, information about instances types, location and costs are important assets to monitor, especially when you pay for what you use. The task of managing and presenting these metrics becomes increasingly difficult when the scale of the infrastructure grows or undergoes changes over time, which is a common situation in Cloud environments.

“Standard” metrics and information provided by the job scheduler might not be sufficient to efficiently manage the Cloud clusters. For example, tracking the cost of running jobs becomes even more important in order to monitor and manage your budget, and redistribute the costs to your users/departments. Due to the wide variety of compute instance types in the cloud, it can also be interesting to track on which instance types the jobs have run, in order to check their performances and associated costs, and further improve the placement and instance selection of the jobs.

OKA offers many benefits to monitor your jobs, and deep-dive into how your clusters behave and how they are being used by your end-users. Accessing information about the Cloud in OKA is straightforward once your environment is properly configured. In this article, we present a simple integration that can be made in a Slurm cluster on AWS to retrieve the type of instances jobs runs on, their pricing information (on-demand/spot, per hour price…), the AWS region, and virtually any information about the AWS environment you are using. The scripts provided below are given as examples, and can easily be adapted to retrieve more detailed information, or be adapted to work with other job schedulers (e.g., LSF, PBS…).

There are many ways to create a cluster on AWS, the details are out of the scope of this article, but you can use for example AWS ParallelCluster or CCME (note: the solution presented here is extracted from CCME where it is available out of the box).

The principle depicted here is very simple, and relies on two components:

1. A Slurm epilog script that will gather information about the AWS environment on which the job runs, and store this information as comma separated values (CSV) in the Comment field of the job. The gathered information are:

   • instance type

   • instance id of the “main” job node

   • availability zone

   • region

   • instance price

   • cost type: ondemand or spot

   • tenancy: shared, reserved…

2. An OKA Data Enhancer that will parse the values of the Comment field, and store them as additional information with each job.

Note

The solution depicted here can also easily be adapted to other Cloud providers. For example, you could follow the indications presented in the Azure integration with Slurm presented here in the “Granular Cost Control” section.

Slurm epilog script

This Slurm epilog script retrieves information about the instance type and its pricing when the job ends, and stores the information in the Comment field of the job in sacct. The user provided comments are kept, and the information are added at the end after a semicolon. The format of the Comment field is the following:

:PricingInfo=${instance type};${instance id};${availability zone};${region};${instance price};${cost type};${tenancy}

The following packages are required and should be available on all the nodes of the cluster:

• jq

• awscli

Also note that this solution requires that Slurm has been configured to keep accounting information about the jobs. See Slurm documentation to configure accounting manually, or if you are using AWS ParallelCluster you can follow this guide.

As the epilog script contacts AWS APIs to gather the information, it needs to run on instances having (at least) the following policy attached to the role of the instance:

{
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "VisualEditor0",
        "Effect": "Allow",
        "Action": [
            "ec2:DescribeSpotPriceHistory",
            "ec2:DescribeInstances",
            "ec2:DescribeRegions",
            "pricing:GetProducts"
        ],
        "Resource": "*"
    }
]
}

Script

#!/bin/bash
declare logFile="/var/log/ccme.slurmepilog.log"
touch "${logFile}"; chmod -v 600 "${logFile}"
exec  > >(awk '{printf "[%s] %s\n", strftime("%FT%T%z"), $0; fflush()}' >>"${logFile}" || true)
exec 2>&1;
# Uncomment the following line for debug traces
#set -x

slurm_env="/opt/slurm/etc/slurm.sh"
if [[ -f "${slurm_env}" ]]; then
    # shellcheck source=/dev/null
    . "${slurm_env}"
fi

declare -A regions
regions["af-south-1"]="Africa (Cape Town)"
regions["ap-east-1"]="Asia Pacific (Hong Kong)"
regions["ap-east-1"]="Asia Pacific (Hong Kong)"
regions["ap-northeast-1"]="Asia Pacific (Tokyo)"
regions["ap-northeast-2"]="Asia Pacific (Seoul)"
regions["ap-south-1"]="Asia Pacific (Mumbai)"
regions["ap-south-2"]="Asia Pacific (Hyderabad)"
regions["ap-southeast-1"]="Asia Pacific (Singapore)"
regions["ap-southeast-2"]="Asia Pacific (Sydney)"
regions["ap-southeast-3"]="Asia Pacific (Jakarta)"
regions["ap-southeast-4"]="Asia Pacific (Melbourne)"
regions["ca-central-1"]="Canada (Central)"
regions["eu-central-1"]="EU (Frankfurt)"
regions["eu-central-2"]="Europe (Zurich)"
regions["eu-north-1"]="EU (Stockholm)"
regions["eu-south-1"]="Europe (Milan)"
regions["eu-south-2"]="Europe (Spain)"
regions["eu-west-1"]="EU (Ireland)"
regions["eu-west-2"]="EU (London)"
regions["eu-west-3"]="EU (Paris)"
regions["me-central-1"]="Middle East (UAE)"
regions["me-south-1"]="Middle East (Bahrain)"
regions["sa-east-1"]="South America (Sao Paulo)"
regions["us-east-1"]="US East (N. Virginia)"
regions["us-east-2"]="US East (Ohio)"
regions["us-west-1"]="US West (N. California)"
regions["us-west-2"]="US West (Oregon)"

# Gather Information about job environment on AWS
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
insttype=$(curl -s -H "X-aws-ec2-metadata-token: ${TOKEN}" -v http://169.254.169.254/latest/meta-data/instance-type)
instid=$(curl -s  -H "X-aws-ec2-metadata-token: ${TOKEN}" -v http://169.254.169.254/latest/meta-data/instance-id)
az=$(curl -s -H "X-aws-ec2-metadata-token: ${TOKEN}" -v http://169.254.169.254/latest/meta-data/placement/availability-zone)
region=$(curl -s -H "X-aws-ec2-metadata-token: ${TOKEN}" -v http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region || true)


# We try to detect if we use spot through AWS APIs
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-purchasing-options.html#check-instance-lifecycle
# Use the following describe-instances command:
# aws ec2 describe-instances --instance-ids i-1234567890abcdef0
# - If the instance is running on a Dedicated Host, the output contains the following information: "Tenancy": "host"
# - If the instance is a Dedicated Instance, the output contains the following information: "Tenancy": "dedicated"
# - If the instance is a Spot Instance, the output contains the following information: "InstanceLifecycle": "spot"
# - If the instance is a Scheduled Instance, the output contains the following information: "InstanceLifecycle": "scheduled"
# - Otherwise, the output does not contain InstanceLifecycle.
lifecycle=$(aws --region="${region}" ec2 describe-instances --instance-ids "${instid}" | jq -r ".Reservations[0].Instances[0].InstanceLifecycle" || true)
tenancy=$(aws --region="${region}" ec2 describe-instances --instance-ids "${instid}" | jq -r ".Reservations[0].Instances[0].Placement.Tenancy" || true)

costtype="ondemand"  # Default value
if [[ "${lifecycle}" == "spot" ]]; then
    costtype="spot"
elif [[ "${lifecycle}" == "" ]]; then
    costtype="ondemand"
fi

# Get instance price
if [[ "${costtype}" == "ondemand" ]]; then
    if [[ "${tenancy}" == "default" ]]; then
        tenancy="shared"
    fi

    filters=(
    "Type=TERM_MATCH,Field=instanceType,Value=${insttype}"
    "Type=TERM_MATCH,Field=location,Value=${regions[${region}]}"
    "Type=TERM_MATCH,Field=operatingSystem,Value=Linux"
    "Type=TERM_MATCH,Field=preInstalledSw,Value=NA"
    "Type=TERM_MATCH,Field=capacitystatus,Value=Used"
    "Type=TERM_MATCH,Field=tenancy,Value=${tenancy}"
    )
    # Warning: if tenancy==host (reserved), then the price will be $0.00
    instprice=$(aws --region=us-east-1 pricing get-products --service-code AmazonEC2 --filter "${filters[@]}" | jq -rc '.PriceList[0]' | jq -r '[.terms.OnDemand[].priceDimensions[].pricePerUnit.USD][0]' || true)
elif [[ "${costtype}" == "spot" ]]; then
    instprice=$(aws --region "${region}" ec2 describe-spot-price-history --availability-zone "${az}" --instance-types "${insttype}" --start-time "$(date '+%Y-%m-%dT%H:%M:%S')" --product-descriptions "Linux/UNIX" | jq -r '.SpotPriceHistory[0].SpotPrice' || true)
else
    # Currently we do not manage other types of pricing
    instprice=0
fi

comment=$(scontrol show job "${SLURM_JOBID}" | grep 'Comment=' | awk -F'Comment=' '{print $2}' || true)
comment+=":PricingInfo=${insttype};${instid};${az};${region};${instprice};${costtype};${tenancy}"

echo "Setting comment for job ${SLURM_JOBID}: ${comment}"

sacctmgr -i -Q modify job where JobID="${SLURM_JOBID}" set Comment="${comment}"

Installation

• Copy the epilog script to a folder accessible on all nodes, e.g., /shared_nfs/slurm/slurm-epilog.sh, and give it execution rights: chmod +x /shared_nfs/slurm/slurm-epilog.sh

• Edit /etc/slurm/slurm.conf (on all nodes), and set the Epilog option to /shared_nfs/slurm/slurm-epilog.sh

• Reconfigure Slurm daemons: scontrol reconfigure, or restart them: systemctl restart slurmd

Then submit a job. Once finished, check that in the output of sacct you have in the Comment field the expected output: sacct --format "jobid,comment".

OKA Data Enhancer

A Data Enhancer needs to be created and configured in OKA in order to parse the additional data gathered by the Slurm Epilog script. We propose here an example Data Enhancer that you can adapt to your needs (included in comment is the generation of “fake” data if you wish to test it first):

import logging
import numpy as np

logger = logging.getLogger("oka_main")

class EnhancerAWSSLURMFeature():
    VERSION = "1.0.0"

    def parse_comment(comment):
        # :pricinginfo=${instance type};${instance id};${availability zone};${region};${price};${spot|ondemand};{tenancy}
        # e.g., :pricinginfo=c5n.18xlarge;i-0cd3c13fa4599d4d5;eu-west-1b;eu-west-1;1.241400;spot;shared
        scsv = comment.split(':pricinginfo=')[1].split(';')
        insttype = scsv[0]
        instid = scsv[1]
        az = scsv[2]
        region = scsv[3]
        instprice = float(scsv[4]) if scsv[4].lower() not in ["na", "nan"] else 0.0
        costtype = scsv[5]
        tenancy = scsv[6]
        return insttype, instid, az, region, instprice, costtype, tenancy


    def run(self, data, **kwargs):
        try:
            # Uncomment the following lines for testing with fake data
            # from random import random, randint
            # azs = [["a", "b", "c"][randint(0,2)] for y in range(len(data))]
            # prices = [random() for y in range(len(data))]
            # inst = [["c5n.18xlarge", "c5n.9xlarge", "c5n.4xlarge", "c5.4xlarge", "g4dn.8xlarge"][randint(0,4)] for y in range(len(data))]
            # data.loc[:, "Comment"] = ["comment:pricinginfo={};id12;eu-west-1{};eu-west-1;{};spot;shared".format(inst[x], azs[x], prices[x]) for x in range(len(data))]

            # Fill missing values with a set of default values
            data["Comment"].fillna(":pricinginfo=NA;NA;NA;NA;0;ondemand;shared", inplace=True)

            data.loc[:, "instance_type"], data.loc[:, "instance_id"], data.loc[:, "availability_zone"], data.loc[:, "region"], instprice, data.loc[:, "price_type"], data.loc[:, "tenancy"] = zip(*data["Comment"].apply(EnhancerAWSSLURMFeature.parse_comment))

            duration = (data["End"].astype(np.datetime64) - data["Start"].astype(np.datetime64)).astype('timedelta64[s]')/3600.0
            data.loc[:, "Cost"] = instprice * duration * data["Allocated_Nodes"]

        except Exception as e:
            logger.error(f"Cost information not available: {e}")

Installation

Please refer to Data enhancers section for explanations on how to install and configure this Data Enhancer in the ingestion pipeline.

Accessing AWS information in OKA

The information ingested through the Data Enhancer are then available in OKA in multiple plugins and through the filters. We present below a few examples of where the information can be accessed and used to analyze your workloads

AWS costs in KPI.

Filters allow to select workloads based on the information gathered from AWS.

Detailed information in Consumers.

Cost per job status in State to detect waste.

Nginx

Question - How can I configure Nginx to access Flower monitoring?

If you started Flower using our okaCeleryFlower.sh script and are not using the auto generated oka.nginx.conf flower will not be accessible for you straight away.

Answer

Edit your Nginx configuration and add the following code to it:

location /flower/ {
    proxy_pass ${FLOWER_SERVER_URL};
}

By default, ${FLOWER_SERVER_URL} will be http://127.0.0.1:8001 if you started the process using our bash script.