Linux host system configuration

Docker

Installing Docker

Docker installation instructions vary slightly by distribution. Please follow the links below to docker.com to find the instructions specific to your distribution:

After installing Docker, because Malcolm should be run as a non-root user, add your user to the docker group with something like:

$ sudo usermod -aG docker yourusername

Following this, either reboot or log out, then log back in.

Docker starts automatically on DEB-based distributions. On RPM-based distributions, users must start Docker manually or enable it using the appropriate systemctl or service command(s).

You can test Docker by running docker info, or (assuming you have internet access), docker run --rm hello-world.

Installing docker compose

Please follow this link on docker.com for instructions on installing the Docker Compose plugin.

Operating system configuration

The host system (i.e., the one running Docker) must be configured for the best possible OpenSearch performance. Here are a few suggestions for Linux hosts (these may vary from distribution to distribution):

Append the following lines to /etc/sysctl.conf:

# the maximum number of open file handles
fs.file-max=2097152

# increase maximums for inotify watches
fs.inotify.max_user_watches=131072
fs.inotify.max_queued_events=131072
fs.inotify.max_user_instances=512

# the maximum number of memory map areas a process may have
vm.max_map_count=262144

# decrease "swappiness" (swapping out runtime memory vs. dropping pages)
vm.swappiness=1

# the maximum number of incoming connections
net.core.somaxconn=65535

# the % of system memory fillable with "dirty" pages before flushing
vm.dirty_background_ratio=40

# maximum % of dirty system memory before committing everything
vm.dirty_ratio=80

In addition, the some suggest lowering the TCP retransmission timeout to 5. However, if your host communicates with other systems over a low-quality network, this low of a setting may be detrimental to those communications. To set this value, add the following to /etc/sysctl.conf:

# maximum number of TCP retransmissions
net.ipv4.tcp_retries2=5

Depending on your distribution, create either the file /etc/security/limits.d/limits.conf containing:

# the maximum number of open file handles
* soft nofile 65535
* hard nofile 65535
# do not limit the size of memory that can be locked
* soft memlock unlimited
* hard memlock unlimited

OR the file /etc/systemd/system.conf.d/limits.conf containing:

[Manager]
# the maximum number of open file handles
DefaultLimitNOFILE=65535:65535
# do not limit the size of memory that can be locked
DefaultLimitMEMLOCK=infinity

Change the readahead value for the disk where the OpenSearch data will be stored. There are a few ways to do this. For example, users could add this line to /etc/rc.local (replacing /dev/sda with their disk block descriptor):

# change disk read-adhead value (# of blocks)
blockdev --setra 512 /dev/sda

Change the I/O scheduler to deadline or noop. Again, this can be done in a variety of ways. The simplest is to add elevator=deadline to the arguments in GRUB_CMDLINE_LINUX in /etc/default/grub, then running sudo update-grub.
Enable cgroup accounting for memory and swap space. This can be done by adding cgroup_enable=memory swapaccount=1 cgroup.memory=nokmem to the arguments in GRUB_CMDLINE_LINUX in /etc/default/grub, then running sudo update-grub.
If you are planning on using very large data sets, consider formatting the drive containing the opensearch volume as XFS.

After making allthese changes, do a reboot for good measure!

Podman

See Docker vs. Podman.