> This page is for Tools. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.moveworks.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.moveworks.com/_mcp/server. # On-Prem Agent Troubleshooting # Installation Errors These errors occur while running `setup_agent.sh` during initial setup or upgrade. For errors after the agent is running, see [Diagnosing Common Issues](#diagnosing-common-issues) below. ### 🔴 Docker service fails to start ```shell Failed to enable unit: Unit file docker service does not exist [ERROR] Error enabling Docker service to start on boot Failed to start Docker service: Unit docker service not found [ERROR] Error starting Docker service ``` There may be an issue with the Docker installation. To confirm: 1. Run `docker logs --follow` 2. If you see `exec /bin.sh: operation not permitted`, uninstall Docker and reinstall using the [Docker convenience script](https://docs.docker.com/engine/install/ubuntu/#install-using-the-convenience-script) --- ### 🔴 Docker or Podman will not install ``` Error installing Podman. Install Podman and Rerun Error installing Docker, Install Docker and Rerun ``` The script can only auto-install Docker or Podman on select Linux distributions. Manually install Docker or Podman and rerun the script with the same options. --- ### 🔴 Podman repository error (Ubuntu earlier than 22.04) ``` Error adding Podman repository to package sources. Install Podman and Rerun Error adding GPG key for Podman repository. Install Podman and Rerun ``` Ubuntu versions earlier than 22.04 do not support Podman via the official repository. Manually install Docker or Podman and rerun the script. --- ### 🔴 yum-utils missing (RHEL) ``` Error installing yum-utils to install docker. Install docker and Rerun ``` RHEL-based distributions require `yum-utils` to install Docker. Manually install Docker and rerun the script. --- ### 🔴 loginctl not installed (Podman) ``` Error enabling linger Loginctl not installed. Error enabling linger ``` If `loginctl` is not installed, the script continues. This is not an issue for Docker-based containers, but for Podman the agent may stop running when the user session ends. --- ### 🔴 systemd not installed ``` Warning: System does not use systemd. Docker service may not start on boot ``` Without `systemctl`, the Docker or Podman container may not restart automatically on boot. --- ### 🔴 Docker or Podman version too old ``` Podman version $podman_version does not meet the minimum required version $PODMAN_VERSION. Please upgrade Podman Docker version $docker_version does not meet the minimum required version $DOCKER_VERSION. Uninstall Docker and rerun. ``` Upgrade Docker or Podman to meet the minimum version requirement, then rerun the script. --- ### 🔴 systemd directory missing (Podman) ``` Error creating directory /etc/systemd/system. Create the directory and Rerun Error creating directory $HOME/.config/systemd/user. Create the directory and Rerun ``` Manually create the directory and rerun the script with the same options. --- ### 🔴 Podman container naming or systemd unit error ``` Error renaming Podman container CONTAINER_NAME to moveworks_agent_{i} Error generating systemd unit file for Podman container moveworks_agent_ ``` Clean up old containers, networks, and volumes: [`podman system prune`](https://docs.podman.io/en/stable/markdown/podman-system-prune.1.html) --- ### 🔴 systemd daemon reload error (Podman) ``` Error reloading systemd daemon for Podman containers Error enabling service for Podman container moveworks_agent_ Error disabling or stopping service for Podman container Error reloading systemd daemon for Podman containers. ``` Verify that `systemd` is correctly installed and functional. --- ### 🔴 Failed to pull agent image from ECR ``` Error: Failed to pull agent image with version '$AGENT_VERSION' using podman from ECR repository '$ECR_URL'. ``` Verify the version tag is correct at [gallery.ecr.aws/moveworks/agent](https://gallery.ecr.aws/moveworks/agent). If the VM cannot reach ECR due to firewall restrictions, see [Fetching the Agent Image Without ECR](/tool-studio/moveworks-agent/moveworks-agent-installation-guide#fetching-the-agent-image-without-ecr) in the Installation Guide. --- # Diagnosing Common Issues This page covers the most common configuration and runtime issues encountered when running the Moveworks On-Prem Agent. ## Configuring LDAP over SSL (LDAPS, Port 636) ### Prerequisites Determine whether you need a certificate file to connect to the LDAP server over SSL. If so, you will need a base64-encoded `.pem` certificate file (typically the root cert or full cert chain). Wildcard certs are not accepted — see [Microsoft's guidance](https://docs.microsoft.com/en-US/troubleshoot/windows-server/identity/enable-ldap-over-ssl-3rd-certification-authority). The certificate file should look like this and live in the `./certs` directory: ```text -----BEGIN CERTIFICATE----- content of your domain certificate -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- content of any intermediate CA certificate -----END CERTIFICATE----- ``` If the exported certificate is a `.p7b` file, convert it to `.pem`: ```shell openssl pkcs7 -print_certs -in certnew.p7b -out ca_chain.pem ``` ### Verifying the certificate is valid ```shell openssl x509 -in -text -noout ``` The Active Directory fully qualified domain name of the domain controller should appear in either the CN of the Subject field or the DNS entry in the Subject Alternative Name extension. ### Pulling the certificate directly from the server ```shell openssl s_client -connect : -showcerts /dev/null | openssl x509 -outform PEM > cert.pem ``` ### Applying the configuration The recommended approach is to re-run the setup script and select the SSL option to supply the LDAP certificate path: ```shell sudo ./setup_agent.sh --configure ``` To apply manually, edit `/home/moveworks/agent/conf/agent_config.yml` with the agent stopped: 1. Use the LDAP hostname, not IP address 2. Set `port: 636` 3. Set `use_ssl: true` 4. Set the certificate path: `path_to_cert: /home/moveworks/agent/certs/mycert.pem` **Example — with certificate:** ```yaml ldap_config: host: ldaps.company.com port: 636 service_user: COMPANY\svc_moveworks ldap_service_password: encrypted_value: use_ssl: true path_to_cert: /home/moveworks/agent/certs/mycert.pem ``` **Example — with well-known CA (no cert needed):** ```yaml ldap_config: host: ldaps.company.com port: 636 service_user: COMPANY\svc_moveworks ldap_service_password: encrypted_value: use_ssl: true ``` **Example — LDAP without SSL (port 389):** ```yaml ldap_config: host: ldap.company.com port: 389 service_user: COMPANY\svc_moveworks ldap_service_password: encrypted_value: ``` **Example — self-signed certificate (TLS skip verify):** ```yaml ldap_config: host: ldaps.company.com port: 636 service_user: COMPANY\svc_moveworks ldap_service_password: encrypted_value: use_ssl: true path_to_cert: /home/moveworks/agent/certs/mycert.pem tls_skip_verify: true ``` > **Warning** > > `tls_skip_verify: true` disables certificate verification. Do not use this in production. --- ## Validating the LDAP Service Account ### Via AD Explorer [AD Explorer](https://docs.microsoft.com/en-us/sysinternals/downloads/adexplorer) lets you verify service account access to the LDAP server. 1. Open AD Explorer and connect using the host, service account username, and password. You may need to connect to VPN first. 2. Once connected, navigate to groups to validate access as needed. > **Note** > > Passwords containing special characters can cause authentication failures. If other steps check out, try resetting the password to one without special characters. ### Via command line Find the LDAP hostname: ```shell nslookup -q=SRV _ldap._tcp..com ``` Test service account credentials: ```shell # Test authentication ldapwhoami -x -H -w -D "\" # Search for objects ldapsearch -x -H ldaps://:636 -W -D "\" -b "" # Return users (LDAPS example) ldapsearch -x -LLL -H ldaps://ldap.company.com:636 -D "username" -w "password" -b "dc=company,dc=com" -s sub "(objectClass=user)" givenName ``` > **Note** > > On Red Hat OS, install openldap first: `yum install openldap*` --- ## Confirming the NETBIOS Domain for an AD Service Account Some environments require the service account username in the format `NETBIOS_DOMAIN\username` (e.g. `MOVEWORKS\svc_ad_moveworks`). To find the NETBIOS domain: 1. Open **Active Directory Users and Computers** 2. Click the **Find Objects** icon (folder with magnifying glass) 3. Search for the service account and double-click it 4. Select the **Account** tab — the value in **User logon name** is the NETBIOS domain Update the agent config to use: `NETBIOS_DOMAIN\username` --- # Known Issues and Fixes ### 🔴 Moveworks access secret is invalid Your org access secret is incorrect or has expired. Tokens expire if not used within 30 days of generation. Generate a new secret in **Moveworks Setup > Core Platform > On-Premise Agents > Generate Secret**. To validate a token manually: ```shell curl --header "Content-Type: application/json" --request POST \ --data '{"access_secret":"", "access_key": ""}' \ https://agent.moveworks.com/api/v1/auth ``` --- ### 🔴 Permission denied when starting the agent ```shell open /home/moveworks/agent/conf/agent_config.yml: permission denied ``` The user running the script has a group ID that does not match the expected value. Ensure all agent files have read and write permissions for **user ID 17540 / group ID 17540**: ```shell sudo chown -R 17540:17540 . ``` --- ### 🔴 Permission denied when accessing a certificate file When copying a new cert into the certs directory via SCP, the new file may not inherit the correct permissions from the previous cert. Copy permissions from the old cert: ```shell cd /home/moveworks/certs ls -ltr sudo chmod --reference=oldCERT.pem newCERT.pem ``` --- ### 🔴 Agent cannot communicate — proxy authentication required ```text rest_pool: Get "https://...": Proxy Authentication Required ``` Your proxy applies to one system but not another. Add the following to the REST config entry that needs to bypass the proxy: ```yaml do_not_use_rest_proxy: true ``` --- ### 🔴 Agent cannot communicate with an on-prem system — Forbidden ```text rest_pool: Get "https:///...": Forbidden ``` Same cause as above. Add `do_not_use_rest_proxy: true` to the relevant REST config entry. --- ### 🔴 401 Unauthorized when authenticating to Moveworks ```text retry: Received status code 401 while trying to authenticate (moveworks_user=ORGNAME) ``` The access secret is invalid or expired (tokens expire after 30 days without use). Generate a new one in Moveworks Setup and reconfigure the agent: ```shell sudo ./setup_agent.sh --configure ``` **Firewall tests:** ```shell ping google.com curl https://agent.moveworks.com/api/v1/web ``` --- ### 🔴 Agent fails to start — "Operation Not Permitted" The container user lacks executable permissions. Work with your sysadmin to resolve at the VM level. As a workaround, remove `--security-opt=no-new-privileges` from the start script. --- ### 🔴 I/O timeout in agent logs ```text retry: Post "https://agent.moveworks.com/...": dial tcp: ... i/o timeout ``` The agent container cannot make an outbound connection to Moveworks. Common causes: firewall rules, DNS issues, or Docker networking. Try running the agent in host network mode — see the [Docker networking issue](#-docker-networking-issue--no-route-to-host) section below. --- ### 🔴 CA bundle certificate missing ```text context deadline exceeded (Client.Timeout exceeded while awaiting headers) ``` CA bundle certs are not being recognized by the container. 1. Find the CA bundle path: `openssl version -d` (typically `/etc/pki/tls/certs/ca-bundle.crt`) 2. Mount the volume in the start script: `-v /etc/pki/:/etc/pki/` 3. Reference it in `agent_config.yml`: ```yaml moveworks_config: path_to_cert: /etc/pki/tls/certs/ca-bundle.crt ``` --- ### 🔴 Docker networking issue — No Route to Host 1. Restart the Docker daemon: `sudo systemctl restart docker` 2. Use telnet to verify the server can reach the LDAP domain controller 3. If networking is the issue, enable host network mode by adding `--net=host` to the start script --- ### 🔴 Docker container shuts down on server restart ```shell sudo systemctl enable docker ``` --- ### 🔴 Docker log size error ```text unknown log opt 'max-size'/'max-file' for journald log driver ``` Edit the start script and add `--log-driver json-file`. --- ### 🔴 Container in "Restarting" state, logs not generating Likely a permissions issue on the config or certs directory: ```shell chmod 777 /path/to/agent_config.yml chmod 777 /path/to/cert.pem ``` Restart the agent. If the issue persists, run `chmod 777` on all agent-related files. --- ### 🔴 x509: certificate signed by unknown authority — connecting to Moveworks API ```text retry: Post "https://agent.moveworks.com/...": x509: certificate signed by unknown authority ``` The system cert used to connect to the Moveworks API is not from a known CA. Ask your admins to install a cert from a trusted certificate authority. As a last resort, copy the system CA bundle (`/etc/ssl/certs/ca-certificates.crt`) to the agent's `/certs` folder, mount it, and reference it: ```yaml moveworks_config: path_to_cert: /home/moveworks/agent/certs/ca-certificates.crt ``` Test with: ```shell curl -vs --cacert https://agent.moveworks.com/api/v1/web ``` --- ### 🔴 x509: certificate signed by unknown authority — connecting to LDAP ```text retry: DialURL: LDAP Result Code 200 "Network Error": x509: certificate signed by unknown authority ``` The cert used to connect to LDAP does not include the full chain. Ensure the `.pem` file contains all intermediate certificates, then add it to `ldap_config`: ```yaml ldap_config: use_ssl: true path_to_cert: /home/moveworks/agent/certs/cert.pem ``` --- ### 🔴 No such file or directory: TLS cert path The cert exists on the host but is not mounted into the container. Add the volume to the start script: ```shell -v "$(pwd)/certs":/home/moveworks/agent/certs ``` --- ### 🔴 x509: certificate signed by unknown authority — connecting to an on-prem REST service ```text x509: certificate signed by unknown authority ``` Use `ca_cert_path` in the REST config to point to the cert inside the container: ```yaml rest_configs: MY_SERVICE: service: MY_SERVICE ca_cert_path: /home/moveworks/agent/certs/myCert.crt ``` If the cert is expired, you can temporarily disable verification (not recommended for production): ```yaml rest_configs: MY_SERVICE: tls_skip_verify: true ``` --- ### 🔴 Unknown certificate authority — wrong file format If the cert is a `.p7b` file: ```shell openssl pkcs7 -print_certs -in certnew.p7b -out ca_chain.pem ``` If the cert is a `.cer` file: ```shell openssl x509 -inform der -in cert.cer -out cert.pem ``` --- ### 🔴 Certificate relies on legacy Common Name field ```text x509: certificate relies on legacy Common Name field, use SANs ``` The certificate only has a CN with no SAN, which is deprecated. Set `tls_skip_verify: true` in the agent config as a workaround, or request a new cert with a SAN from your CA. --- ### 🔴 404 for config request ```text received non-200 response status [404 Not Found] for config request ``` The `config_url` in `agent_config.yml` is incorrect. Verify it matches your region — for US Commercial: `https://agent.moveworks.com/api/v1/config`. --- ### 🔴 404 for authentication request ```text Received status code 404 while trying to authenticate ``` The `auth_url` is incorrect. Ensure there are no spaces or quotes around the value. For US Commercial: `https://agent.moveworks.com/api/v1/auth`. --- ### 🔴 Unable to read LDAP response packet — connection reset by peer The LDAP server is denying the connection. Likely a bad cert or wrong port. Try port 389 as a workaround to rule out network issues, then work with your team to obtain the correct LDAPS cert. --- ### 🔴 TLS handshake error — first record does not look like a TLS handshake Typically caused by a proxy configured for the wrong protocol (HTTP vs HTTPS). Check whether your `HTTP_PROXY` or `HTTPS_PROXY` setting matches the protocol the agent is using. --- ### 🔴 Container not visible in docker ps / podman ps after start Check `agent_config.yml` for typos. Even a single indentation error will prevent the container from starting. --- ### 🔴 LDAP Result Code 200 — certificate valid for wrong host ```text x509: certificate is valid for NMH-DC01.domain.com ``` The agent is connecting to a load-balanced domain that resolves to multiple servers, but the cert is issued for a specific server. Point directly to one server in `agent_config.yml`: ```yaml ldap_config: host: nmh-dc02.domain.com ``` --- ### 🔴 LDAP Result Code 200 — server misbehaving (DNS) ```text LDAP Result Code 200 "Network Error": dial tcp: lookup ... server misbehaving ``` DNS resolver mismatch between host and container. 1. Compare `cat /etc/resolv.conf` on the host and inside the container 2. Stop and remove all containers: ```shell docker stop $(docker ps -a -q) docker rm $(docker ps -a -q) ``` 3. Restart Docker and start a new agent: ```shell sudo systemctl restart docker sudo ./setup_agent.sh --start ``` Alternatively, enable host networking mode in the start script. --- ### 🔴 Podman — user namespaces not enabled ```text user namespaces are not enabled in /proc/sys/user/max_user_namespaces ``` On CentOS 7, run as root: ```shell echo "user.max_user_namespaces=10000" > /etc/sysctl.d/42-rootless.conf sysctl --system ``` --- ### 🔴 LDAP Result Code 200 — certificate expired ```text x509: certificate has expired or is not yet valid ``` Check if the cert on the VM is expired: ```shell openssl x509 --enddate --noout cert.pem ``` Check if the LDAP host cert is expired: ```shell openssl s_client -connect :636 -showcerts /dev/null | openssl x509 -outform PEM > cert.pem keytool -printcert -v -file cert.pem ``` --- ### 🔴 LDAP connection reset by peer The port is likely closed. If using port 389, switch to port 636 with a cert and configure LDAPS. --- ### 🔴 LDAP Result Code 8 — Strong Auth Required The server requires LDAPS (port 636 with a cert), but the agent is configured for plain LDAP (port 389). Reconfigure using LDAPS. --- ### 🔴 ITSM_DISCONNECTED — consistent timeouts for ticketing * Check for WAFs throttling the connection between the agent and your ITSM * Lower environments often have reduced CPU/storage — verify the timeout is not caused by resource limits on the ITSM instance itself --- ## Common LDAP Error Codes | Error Code | Error | Description | | ---------- | ----------------------------------- | ------------------------------------------------------------------ | | 525 | User not found | Invalid username supplied | | 52e | Invalid credentials | Valid username but wrong password. Also occurs on password expiry. | | 530 | Not permitted at this time | Login restricted to certain hours | | 531 | Not permitted from this workstation | Login restricted to certain machines | | 532 | Password expired | Password is valid but expired | | 533 | Account disabled | Account has been disabled | | 701 | Account expired | Account has expired | | 773 | User must reset password | Password reset required before login | | 775 | Account locked out | Account is locked, regardless of whether password is correct | --- # Debugging Tips ## Check agent logs ```shell # Follow live logs (Docker) docker logs -f # Tail INFO logs from log files tail -f /var/log/moveworks/*.INFO.log # Access the container shell docker exec -it $(docker ps --format '{{.Names}}') /bin/bash ``` ## Useful Docker commands ```shell # Kill all running containers docker kill $(docker ps -q) # Remove all stopped containers docker rm $(docker ps -aq) # Remove a specific agent image docker rmi # Test network connectivity telnet 636 ``` ## Test connectivity ```shell # Test outbound access ping google.com curl https://agent.moveworks.com/api/v1/web # Test LDAP ldapsearch -x -H ldaps://:636 -D "DOMAIN\user" -W # Test with a specific cert curl -vs --cacert https://agent.moveworks.com/api/v1/web ``` --- ## See Also * [Operations and Health](/tool-studio/moveworks-agent/operations-maintenance-guide) — common commands, health monitoring, credential rotation * [Installation Guide](/tool-studio/moveworks-agent/moveworks-agent-installation-guide) — fresh install or reinstall steps