On-prem runtime and performance issues

This guide covers memory-related issues that can occur during the runtime of the on-prem agent (OPA). These include heap space errors, crashes caused by large API responses, and logging errors.

Java heap space errors

The following error log indicates the agent has reached its memory limit:

java.lang.OutOfMemoryError: Java heap space

This error indicates that OPA doesn't have enough memory to process large jobs or payloads. By default, the heap size is set to a fraction of available system memory.

Open the run.sh file and adjust the -Xmx flag to resolve the issue. For example, you can increase it to -Xmx10G to allocate 10 GB of heap memory.

High memory usage from large responses

The agent may crash or slow down when it buffers large API responses in memory. By default, it loads entire responses during execution, which increases memory usage.

OPA version 2.7.1 and later allows you to disable response buffering in the configuration file by adding the following flag to your config.yml:

agent:
  disable_response_buffering: true

Resolve gateway names on servers with OPA installed

You may encounter the following errors when resolving Workato gateway names on servers with OPA installed: - sg3.workato.com

• sg4.workato.com

Complete the following troubleshooting steps to resolve gateway names:

1

Connect to the server where your OPA instance runs.

2

Open your server's console.

3

Run the following commands to verify if the gateway names can be resolved from your Windows, Linux, and MacOS server:

nslookup sg3.workato.com

4

Run the following command on Linux and MacOS servers to verify if the gateway names can be resolved:

dig sg3.workato.com

Your output is similar to the following:

sg3.workato.com	canonical name = public-v10-awsprod-opg3-nlb-d6da47859547995a.elb.us-east-1.amazonaws.com.
Name:	public-v10-awsprod-opg3-nlb-d6da47859547995a.elb.us-east-1.amazonaws.com
Address: 54.224.75.148
Name:	public-v10-awsprod-opg3-nlb-d6da47859547995a.elb.us-east-1.amazonaws.com
Address: 52.206.161.203
Name:	public-v10-awsprod-opg3-nlb-d6da47859547995a.elb.us-east-1.amazonaws.com
Address: 52.204.114.159

This successful output indicates that it is possible to resolve the domain name.

Check gateway connectivity

You may encounter issues with establishing gateway connectivity on servers with OPA installed.

Complete the following troubleshooting steps to check gateway connectivity:

1

Connect to the server where your OPA instance runs.

2

Open your server's console.

3

Run the following commands to verify if the gateway names can be resolved from your Windows, Linux, and MacOS server:

curl -k -vvv -GET --no-alpn --key conf/cert.key --cert conf/cert.pem https://sg3.workato.com/gateway/ping

Your output is similar to the following:

* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* 
(LINES SKIPPED)
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Mon, 25 Jul 2022 21:33:42 GMT
< Content-Length: 81
< 
{"gateway_version":"1.0.1","os_platform":"linux","os_release":"4.19.0-10-amd64"}

Verify the server certificate chain

You may encounter issues verifying the server certificate chain on servers with OPA installed.

Complete the following troubleshooting steps to verify the server certificate chain:```

1

Go to your OPA installation folder. Refer to the Accessing on-prem documentation for more information about installation.

2

Download the Workato root certificate and save it in your OPA installation folder. Refer to the HTTP SSL documentation for instructions.

3

Run the following command for Linux and MacOS:

curl -vvv -GET --no-alpn --cacert root_ca_cert.pem --key conf/cert.key --cert conf/cert.pem https://sg3.workato.com/gateway/ping

The expected output contains the following:

SSL certificate verify ok

This indicates that the server certificate chain was successfully verified.

OPA stopped creating logs

You may encounter an issue where OPA stops generating logs when running on Linux. This issue occurs when there is either no free space remaining on the disk or when the logging level settings are incorrect. Complete the following troubleshooting procedure to resolve this issue:

1

Run the following command to determine if there is free space left on the disk:

df -h
cat /etc/fstab
parted -l

This determines if the issue is related to disk space.

2

Verify that the logging permissions are correct.

1

Run ls -lah /var/log/workato-agent to view log permissions.

2

Run the following command if a user or group does not have permissions set to workato:

chown -R workato:workato /var/log/workato-agent

3

Check that the logging level is set to debug in the config file. For example:

logger: debug