A powerful, easily deployable network traffic analysis tool suite for network security monitoring
There are several ways to customize Malcolm’s runtime behavior via local changes to configuration files. Many commonly-tweaked settings are discussed in the project README (see Environment Variable Files and Customizing event severity scoring for some examples).
Some configuration changes can be put in place by modifying local copies of configuration files and then using a bind mount to overlay the modified file onto the running Malcolm container. This is already done for many files and directories used to persist Malcolm configuration and data. For example, the default list of bind mounted files and directories for each Malcolm service is as follows:
$ yq eval '.services = (.services | with_entries(.value = {"volumes": .value.volumes}))' docker-compose.yml
services:
opensearch:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./.opensearch.primary.curlrc
target: /var/local/curlrc/.opensearch.primary.curlrc
read_only: true
- type: bind
bind:
create_host_path: false
source: ./.opensearch.secondary.curlrc
target: /var/local/curlrc/.opensearch.secondary.curlrc
read_only: true
- type: bind
bind:
create_host_path: false
source: ./opensearch
target: /usr/share/opensearch/data
- type: bind
bind:
create_host_path: false
source: ./opensearch-backup
target: /opt/opensearch/backup
- type: bind
bind:
create_host_path: false
source: ./opensearch/opensearch.keystore
target: /usr/share/opensearch/config/persist/opensearch.keystore
dashboards-helper:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./.opensearch.primary.curlrc
target: /var/local/curlrc/.opensearch.primary.curlrc
read_only: true
- type: bind
bind:
create_host_path: false
source: ./.opensearch.secondary.curlrc
target: /var/local/curlrc/.opensearch.secondary.curlrc
read_only: true
dashboards:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./.opensearch.primary.curlrc
target: /var/local/curlrc/.opensearch.primary.curlrc
read_only: true
logstash:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./.opensearch.primary.curlrc
target: /var/local/curlrc/.opensearch.primary.curlrc
read_only: true
- type: bind
bind:
create_host_path: false
source: ./.opensearch.secondary.curlrc
target: /var/local/curlrc/.opensearch.secondary.curlrc
read_only: true
- type: bind
bind:
create_host_path: false
source: ./logstash/maps/malcolm_severity.yaml
target: /etc/malcolm_severity.yaml
read_only: true
- type: bind
bind:
create_host_path: false
source: ./logstash/certs/ca.crt
target: /certs/ca.crt
read_only: true
- type: bind
bind:
create_host_path: false
source: ./logstash/certs/server.crt
target: /certs/server.crt
read_only: true
- type: bind
bind:
create_host_path: false
source: ./logstash/certs/server.key
target: /certs/server.key
read_only: true
filebeat:
volumes:
- nginx-log-path:/nginx:ro
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./.opensearch.primary.curlrc
target: /var/local/curlrc/.opensearch.primary.curlrc
read_only: true
- type: bind
bind:
create_host_path: false
source: ./zeek-logs
target: /zeek
- type: bind
bind:
create_host_path: false
source: ./suricata-logs
target: /suricata
- type: bind
bind:
create_host_path: false
source: ./filebeat/certs/ca.crt
target: /certs/ca.crt
read_only: true
- type: bind
bind:
create_host_path: false
source: ./filebeat/certs/client.crt
target: /certs/client.crt
read_only: true
- type: bind
bind:
create_host_path: false
source: ./filebeat/certs/client.key
target: /certs/client.key
read_only: true
arkime:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./.opensearch.primary.curlrc
target: /var/local/curlrc/.opensearch.primary.curlrc
read_only: true
- type: bind
bind:
create_host_path: false
source: ./arkime/lua
target: /opt/arkime/lua
read_only: true
- type: bind
bind:
create_host_path: false
source: ./arkime/rules
target: /opt/arkime/rules
read_only: true
- type: bind
bind:
create_host_path: false
source: ./pcap
target: /data/pcap
arkime-live:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./.opensearch.primary.curlrc
target: /var/local/curlrc/.opensearch.primary.curlrc
read_only: true
- type: bind
bind:
create_host_path: false
source: ./arkime/lua
target: /opt/arkime/lua
read_only: true
- type: bind
bind:
create_host_path: false
source: ./arkime/rules
target: /opt/arkime/rules
read_only: true
- type: bind
bind:
create_host_path: false
source: ./pcap
target: /data/pcap
zeek:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./pcap
target: /pcap
- type: bind
bind:
create_host_path: false
source: ./zeek-logs/upload
target: /zeek/upload
- type: bind
bind:
create_host_path: false
source: ./zeek-logs/extract_files
target: /zeek/extract_files
- type: bind
bind:
create_host_path: false
source: ./zeek/intel
target: /opt/zeek/share/zeek/site/intel
- type: bind
bind:
create_host_path: false
source: ./zeek/custom
target: /opt/zeek/share/zeek/site/custom
read_only: true
zeek-live:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./zeek-logs/live
target: /zeek/live
- type: bind
bind:
create_host_path: false
source: ./zeek-logs/extract_files
target: /zeek/extract_files
- type: bind
bind:
create_host_path: false
source: ./zeek/intel
target: /opt/zeek/share/zeek/site/intel
- type: bind
bind:
create_host_path: false
source: ./zeek/custom
target: /opt/zeek/share/zeek/site/custom
read_only: true
suricata:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./suricata-logs
target: /var/log/suricata
- type: bind
bind:
create_host_path: false
source: ./pcap
target: /data/pcap
- type: bind
bind:
create_host_path: false
source: ./suricata/rules
target: /opt/suricata/rules
read_only: true
- type: bind
bind:
create_host_path: false
source: ./suricata/include-configs
target: /opt/suricata/include-configs
read_only: true
suricata-live:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./suricata-logs
target: /var/log/suricata
- type: bind
bind:
create_host_path: false
source: ./suricata/rules
target: /opt/suricata/rules
read_only: true
- type: bind
bind:
create_host_path: false
source: ./suricata/include-configs
target: /opt/suricata/include-configs
read_only: true
file-monitor:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./zeek-logs/extract_files
target: /zeek/extract_files
- type: bind
bind:
create_host_path: false
source: ./zeek-logs/current
target: /zeek/logs
- type: bind
bind:
create_host_path: false
source: ./yara/rules
target: /yara-rules/custom
read_only: true
pcap-capture:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./pcap/upload
target: /pcap
pcap-monitor:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./.opensearch.primary.curlrc
target: /var/local/curlrc/.opensearch.primary.curlrc
read_only: true
- type: bind
bind:
create_host_path: false
source: ./zeek-logs
target: /zeek
- type: bind
bind:
create_host_path: false
source: ./pcap
target: /pcap
upload:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./pcap/upload
target: /var/www/upload/server/php/chroot/files
htadmin:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./htadmin/config.ini
target: /var/www/htadmin/config/config.ini
- type: bind
bind:
create_host_path: false
source: ./htadmin/metadata
target: /var/www/htadmin/config/metadata
- type: bind
bind:
create_host_path: false
source: ./nginx/htpasswd
target: /var/www/htadmin/auth/htpasswd
freq:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
netbox:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./netbox/config
target: /etc/netbox/config
read_only: true
- type: bind
bind:
create_host_path: false
source: ./netbox/media
target: /opt/netbox/netbox/media
- type: bind
bind:
create_host_path: false
source: ./netbox/preload
target: /opt/netbox-preload/configmap
read_only: true
- type: bind
bind:
create_host_path: false
source: ./netbox/custom-plugins
target: /opt/netbox-custom-plugins
read_only: true
netbox-postgres:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./netbox/postgres
target: /var/lib/postgresql/data
netbox-redis:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./netbox/redis
target: /data
netbox-redis-cache:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
api:
volumes:
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./.opensearch.primary.curlrc
target: /var/local/curlrc/.opensearch.primary.curlrc
read_only: true
nginx-proxy:
volumes:
- nginx-log-path:/var/log/nginx
- type: bind
bind:
create_host_path: false
source: ./nginx/ca-trust
target: /var/local/ca-trust
read_only: true
- type: bind
bind:
create_host_path: false
source: ./nginx/nginx_ldap.conf
target: /etc/nginx/nginx_ldap.conf
read_only: true
- type: bind
bind:
create_host_path: false
source: ./nginx/htpasswd
target: /etc/nginx/auth/htpasswd
read_only: true
- type: bind
bind:
create_host_path: false
source: ./nginx/certs
target: /etc/nginx/certs
read_only: true
- type: bind
bind:
create_host_path: false
source: ./nginx/certs/dhparam.pem
target: /etc/nginx/dhparam/dhparam.pem
read_only: true
So, for example, if a user wanted to make a change to the nginx-proxy
container’s nginx.conf
file, they could add the following line to the volumes:
section of the nginx-proxy
service in the docker-compose.yml
file:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
The change would take effect after stopping and starting Malcolm.
See the documentation on bind mounts for more information on this technique.
Another method for modifying local copies of Malcolm’s services’ containers is to build custom containers with the modifications baked-in.
For example, imagine a user wanted to create a Malcolm container that includes a new dashboard for OpenSearch Dashboards and a new enrichment filter .conf
file for Logstash. After placing these files under ./dashboards/dashboards
and ./logstash/pipelines/enrichment
, respectively, in the Malcolm working copy, run ./build.sh dashboards-helper logstash
to build just those containers. After the build completes, run docker images
to see the fresh images for ghcr.io/idaholab/malcolm/dashboards-helper
and ghcr.io/idaholab/malcolm/logstash-oss
. Users may need to review the contents of the Dockerfiles to determine the correct service and filesystem location within that service’s image depending on the nature of the task.
Alternately, forks of Malcolm on GitHub contain workflow files that contain instructions for GitHub to build the images and sensor and Malcolm installer ISOs. The resulting images are named according to the pattern ghcr.io/owner/malcolm/image:branch
(e.g., if the GitHub user romeogdetlevjr
has forked Malcolm, the arkime
container built for the main
branch would be named ghcr.io/romeogdetlevjr/malcolm/arkime:main
). To run a local instance of Malcolm using these images instead of the official ones, users would need to edit their docker-compose.yml
file(s) and replace the image:
tags according to this new pattern, or use the bash helper script ./shared/bin/github_image_helper.sh
to pull and re-tag the images.