Skip to content

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 

Repository files navigation

Official configurations for the docker setup of TOLERANT Compliance Suite

The configurations for each released version can be found under tags

Usage with compose

The docker compose commands should be executed in the compose directory.

Starting

docker compose up -d

Stoping

docker compose down

Dataset imports

Set SECRET_ENCRYPTION_KEY to a stable, high-entropy secret before creating SFTP import sources. If no Secret is configured, the service uses its development fallback changeit;do not use that fallback in a deployed environment. Keep the configured key stable across upgrades, because changing it makes existing encrypted SFTP credentials unreadable. The scheduler remains disabled unless DATASET_IMPORT_SCHEDULER_ENABLED=true is set; DATASET_IMPORT_SCHEDULER_DELAY defaults to 1m, DATASET_IMPORT_RUNNING_TIMEOUT defaults to 1h, and DATASET_IMPORT_WORKERS defaults to 2 parallel workers.

LOCAL dataset import sources can read only below the comma-separated roots in DATASET_IMPORT_LOCAL_ALLOWED_ROOTS, which defaults to the read-only ./imports bind mount at /opt/tolerant/data/imports. Their filePattern is relative to the configured source root and supports Ant-style ?, *, and ** wildcards. DATASET_IMPORT_MAX_FILE_SIZE_BYTES limits selected LOCAL and SFTP files and defaults to 100 MiB.

SFTP sources accept either a password or an OpenSSH/PKCS#8 private key with an optional passphrase. These credentials are encrypted in the database with SECRET_ENCRYPTION_KEY. Host verification is enabled by default. Set DATASET_IMPORT_SFTP_KNOWN_HOSTS_LOCATION to a mounted known_hosts file, and configure connection/read limits with DATASET_IMPORT_SFTP_CONNECT_TIMEOUT and DATASET_IMPORT_SFTP_READ_TIMEOUT. DATASET_IMPORT_SFTP_MAX_TRAVERSAL_DEPTH and DATASET_IMPORT_SFTP_MAX_TRAVERSAL_ENTRIES bound remote directory scans. DATASET_IMPORT_SFTP_LOAD_KNOWN_HOSTS=false is intended only for controlled development environments.

Import runs can be triggered with POST /api/v1/dataset/{datasetId}/import-config/{importConfigId}/run. History is available below /api/v1/dataset/{datasetId}/import-config/{importConfigId}/runs. Scheduled successors are calculated after completion, so intervals missed during downtime are not replayed.

Dataset-import completion publishes a notification event transactionally when the corresponding import configuration flag is enabled. The generic notification dispatcher resolves recipients and handles delivery retries. During the notification upgrade, the legacy DATASET_IMPORT_MAIL_ENABLED, subject, and template values DATASET_IMPORT_MAIL_* are imported once for companies that already exist. Newly provisioned companies receive code-owned notification defaults that administrators can subsequently change through the notification-setting API.

Generic notification delivery is configured with NOTIFICATION_MAIL_* variables documented in config.adoc. This section controls dispatcher enablement, polling and retry timing, separate maximum attempt counts for event processing and recipient delivery, and interrupted-processing recovery. NOTIFICATION_MAIL_FROM configures the notification-specific sender. When it is empty, MAIL_FROM is used.

Steps to use your own smtp server

  1. Remove mailserver and self-signed from the compose.yml

  2. Configure the following environment variables for the compliance-service:

    • MAIL_SMTP_ENABLED
    • NOTIFICATION_MAIL_FROM
    • MAIL_SMTP_HOST
    • MAIL_SMTP_PORT
  3. If your smtp requires authentication configure the following environment variables for the compliance-service:

    • MAIL_SMTP_AUTH
    • MAIL_SMTP_USER
    • MAIL_SMTP_PASSWORD

Usage with Helm Chart

This Helm chart deploys the Tolerant Compliance Suite, including a Java-based Backend and a database (PostgreSQL or MariaDB). The helm commands should be executed in the helm directory

1. Prerequisites

  • Kubernetes: 1.21+ (e.g., Docker Desktop, Minikube)
  • Helm: 3.0+
  • Ingress Controller: NGINX Ingress Controller
  • Database: PostgreSQL 14+ or MariaDB 10.11+ when using an externally managed database
  • Hosts File: Map the local domain to your loopback IP:
127.0.0.1 tolerant-compliance-suite.local

2. Setting Up the Ingress Controller

If not already installed, deploy the NGINX Ingress Controller in Docker Desktop to handle external traffic:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/cloud/deploy.yaml

Wait until the controller is ready:

kubectl get pods -n ingress-nginx --watch

3. Installation & Deployment

Run the following commands from the root directory of the chart

Install or Upgrade

helm upgrade --install tolerant-compliance-suite .

Install with MariaDB

PostgreSQL is the default database. To deploy the chart with MariaDB instead, set global.db.type to mariadb during install or upgrade:

helm upgrade --install tolerant-compliance-suite . --set global.db.type=mariadb

The same global.db.name, global.db.user, and global.db.password values are used for the MariaDB pod and for the Compliance Suite services.

Database Migration Notes

The generic search creates full-text indexes on tables that can be large. Rehearse the migration with production-sized data before an upgrade. For MariaDB, verify that innodb_ft_min_token_size is no greater than 3 and that the configured InnoDB stopword lists do not exclude required search terms. Building the FULLTEXT indexes can affect writes and should run in a tested maintenance window or as a separately monitored migration step.

PostgreSQL creates these indexes concurrently. If index creation is interrupted, check pg_index.indisvalid before retrying and drop any invalid index with the intended name using DROP INDEX CONCURRENTLY. Otherwise, CREATE INDEX CONCURRENTLY IF NOT EXISTS can skip the invalid index.

Populate Reference Data

The Compliance Service loads reference data from its persistent input directory:

/opt/tolerant/data/incoming

The loader recognizes these exact file names:

Source Type Required File Name
PEP Info4C_PEP_Desk.txt
Sanction euv_usdpl_v2_utf8.txt

Find the running Compliance Service pod.

PowerShell:

$pod = kubectl get pods `
  -l "app.kubernetes.io/instance=tolerant-compliance-suite,app.kubernetes.io/component=compliance-service" `
  -o jsonpath="{.items[0].metadata.name}"

sh:

pod=$(kubectl get pods \
  -l "app.kubernetes.io/instance=tolerant-compliance-suite,app.kubernetes.io/component=compliance-service" \
  -o jsonpath="{.items[0].metadata.name}")

Copy the PEP file:

kubectl cp ./Info4C_PEP_Desk.txt "${pod}:/opt/tolerant/data/incoming/Info4C_PEP_Desk.txt" -c compliance-service

Copy the Sanction file:

kubectl cp ./euv_usdpl_v2_utf8.txt "${pod}:/opt/tolerant/data/incoming/euv_usdpl_v2_utf8.txt" -c compliance-service

Verify the files:

kubectl exec "$pod" -c compliance-service -- ls -l /opt/tolerant/data/incoming

Trigger the asynchronous reference-data load through the administration API:

Set the API base URL and request a token.

PowerShell:

$base_url = "http://tolerant-compliance-suite.local"
$token = (curl.exe -s -X POST "$base_url/auth/v1/token" `
  -H "Content-Type: application/json" `
  -d '{ "username": "<SUPER_ADMIN_USERNAME>", "password": "<PASSWORD>" }' | ConvertFrom-Json).access_token

sh:

base_url="http://tolerant-compliance-suite.local"
token=$(curl -s -X POST "$base_url/auth/v1/token" \
  -H "Content-Type: application/json" \
  -d '{ "username": "<SUPER_ADMIN_USERNAME>", "password": "<PASSWORD>" }' | jq -r '.access_token')

The sh example uses jq to extract the token from the JSON response.

Trigger the load:

PowerShell:

curl.exe -i -X POST "$base_url/admin/v1/reference-data" `
  -H "Authorization: Bearer $token"

sh:

curl -i -X POST "$base_url/admin/v1/reference-data" \
  -H "Authorization: Bearer $token"

The endpoint returns 202 Accepted when the load request has been scheduled. The service scans the input directory, skips already loaded files with the same checksum, imports new PEP/Sanction records, and refreshes Match afterward. Loading is delayed while match jobs are running.

Check the loading status with:

PowerShell:

curl.exe -s "$base_url/api/v1/reference-data?size=20" `
  -H "Authorization: Bearer $token"

sh:

curl -s "$base_url/api/v1/reference-data?size=20" \
  -H "Authorization: Bearer $token"

Both reference-data entries should eventually report status: done.

Alternatively, restart the Compliance Service after copying the files. The service triggers the same reference-data scan during startup:

Restart the deployment:

kubectl rollout restart deployment tolerant-compliance-suite-service

Wait until the restarted deployment is ready:

kubectl rollout status deployment tolerant-compliance-suite-service

Uninstall

To remove all resources created by the chart:

helm uninstall tolerant-compliance-suite

4. Architecture & Networking

The chart uses Services to ensure stable communication between components.

Component Service Port Target Port Access Type
PostgreSQL/MariaDB 5432/3306 5432/3306 Internal (ClusterIP)
TOLERANT Match 8080 8080 Internal (ClusterIP)
Backend 8080 8080 Internal & via Ingress
Frontend 8080 80 Internal & via Ingress

Init-Container (Database Readiness)

The Backend deployment includes an init container that uses nc (Netcat) to ensure the database is accepting connections before the application starts. This prevents application crashes during the initial startup phase.


5. Persistence & Data Safety

The database uses a PersistentVolumeClaim (PVC) to store data.

  • Resource Policy: The PVC is annotated with "helm.sh/resource-policy": keep.
  • Effect: When you run helm uninstall, the database data will not be deleted.
  • Manual Cleanup: If you wish to wipe the database entirely, you must delete the PVC manually: kubectl delete pvc <pvc-name>

6. Configuration Highlights (values.yaml)

1. Global & General Settings

These values manage shared credentials and base image behaviors across the entire suite.

Parameter Description Default Value
global.db.type Database type (postgres or mariadb) "postgres"
global.db.name The name of the database "compliance-suite"
global.db.user Username for database authentication "compliance-suite"
global.db.password Password for database authentication "compliance-suite"
global.match.licnese Tolerant Match Licnese key ""
image.pullPolicy Policy to pull images (e.g., check for updates) IfNotPresent
imagePullSecrets Credentials for private registries (e.g., Harbor) []

2. Container & Microservice Configuration

Specific settings for the individual components of the suite.

Container Image Source Features RAM (Limit) CPU (Limit)
postgres postgres:16.1 Includes Persistence (10Gi, Keep Policy) 1Gi 500m
mariadb mariadb:11.4 Includes Persistence (10Gi, Keep Policy) 1Gi 500m
match tolerantsoftware/match:12.1 2 Threads, Console Logging 4Gi 2000m
complianceService compliance-service:SNAPSHOT Secure mode enabled, 10Gi Persistence 1Gi 2000m
complianceBatch compliance-batch:SNAPSHOT 4 Threads, 2 Replicas for scaling 6Gi 4000m
complianceFrontend compliance-frontend:SNAPSHOT none 512Mi 500m

2.1 License Configuration

Set the shared license string once and it will be used by both Match and Compliance Batch:

global:
  match:
    license: "YOUR-TOLERANT_MATCH_LICENSE-KEY"

This creates a secret:

  • {{ releaseName }}-match-license with match.lic mounted at /opt/tolerant/config/match.lic

If the value is empty, no license secrets or mounts are created.


3. Networking & Ingress

Settings for internal cluster communication and external web access.

Parameter Description Value
service.type Networking type within the cluster ClusterIP
service.port Internal port used by the Backend 8080
ingress.enabled Enables external access via HTTP/HTTPS true
ingress.hosts.host Domain name for browser access tolerant-compliance-suite.local
ingress.annotations NGINX tuning (SSL redirect & 100MB upload) proxy-body-size: 101m

4. Persistence (Storage)

Configuration for how data is stored on disk to ensure it survives pod restarts. These settings are relevant for containers (postgres or mariadb, complianceService).

Parameter Description Value
persistence.size Total disk space reserved for the service 10Gi
persistence.annotations helm.sh/resource-policy: keep Prevents data loss on uninstall

5. Health & Stability (Probes)

All application containers (match, complianceService, complianceBatch) use these checks to maintain uptime.

Probe Type Path Logic
Liveness /health/liveness Restarts the container if it stops responding (freezes).
Readiness /health/readiness Prevents traffic from hitting the pod until it is fully loaded.
Startup /health/readiness Gives Match enough time for initial startup before liveness checks apply.

7. Troubleshooting

Check Initialization Logs

If the Backend pod stays in the Init state for too long:

kubectl logs -l app=match -c wait-for-postgres

Verify Ingress Path Mapping

Ensure the Ingress has received an IP/Address:

kubectl get ingress

8. Custom configs for match and batch

To use custom configs for match and batch adjust the files in the files directory.

Afterward run the following command from the root directory of the chart:

helm upgrade --install tolerant-compliance-suite .

Dataset imports

Create a Kubernetes Secret containing a stable, high-entropy encryption key before creating SFTP import sources. Set containers.complianceService.datasetImport.secretEncryptionSecretName and, if needed, secretEncryptionSecretKey to reference it. If no Secret is configured, the service uses its development fallback changeit; do not use that fallback in a deployed environment. Keep the configured key stable across upgrades, because changing it makes existing encrypted SFTP credentials unreadable. The key is read with secretKeyRef and is not embedded in the Deployment specification. The same section controls scheduler enablement, polling delay, stale-run timeout, parallel workers (default 2), and SFTP known-host verification.

LOCAL import roots must be visible inside the service container and listed in localAllowedRoots as a comma-separated allowlist. Use the chart's volumes and volumeMounts values to mount them read-only. Import configuration filePattern values are relative to their source root and accept Ant-style ?, *, and ** wildcards. maxFileSizeBytes limits selected LOCAL and SFTP files and defaults to 100 MiB.

SFTP sources support encrypted passwords or encrypted OpenSSH/PKCS#8 private keys with optional passphrases. Set sftpKnownHostsLocation to a mounted known_hosts file. sftpConnectTimeout and sftpReadTimeout bound remote operations; sftpMaxTraversalDepth and sftpMaxTraversalEntries bound directory scans. Host verification should only be disabled in controlled development environments. Manual runs and paged run history are exposed by the dataset import-config API; missed cron intervals are skipped rather than replayed.

Dataset-import completion publishes a notification event transactionally when the corresponding import configuration flag is enabled. The generic notification dispatcher resolves recipients and handles delivery retries. During the notification upgrade, the legacy mailEnabled, subject, and template values below containers.complianceService.datasetImport are imported once for companies that already exist. Newly provisioned companies receive code-owned notification defaults that administrators can subsequently change through the notification-setting API.

Generic notification delivery is configured below containers.complianceService.notification. This section controls dispatcher enablement, polling and retry timing, separate maximum attempt counts for event processing and recipient delivery, and interrupted-processing recovery. mailFrom configures the notification-specific sender. When it is empty, fallbackMailFrom is used through the shared MAIL_FROM fallback.


Compliance Suite Documentation

The Compliance Suite documentation: docs

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages