Running Katalon Studio in Console Mode on Linux Server Without Display

MiaDragon42 · July 26, 2025, 1:51pm

I’m migrating from Selenium to Katalon Studio and running into issues when executing tests on my Jenkins server running Ubuntu. Here’s the command I use to run my test suite:

./katalon-studio --args -runMode=console -projectPath="/home/jenkins/workspace/automation-tests/test-project.prj" \
-reportFolder="TestReports" -reportFileName="results" \
-retry=0 -testSuitePath="Test Suites/SmokeTests" \
-browserType="Headless"

The execution fails with this error:

15-03-2018 08:15:22 AM - [ERROR] - Test Cases/user/user_login FAILED because (of) java.awt.HeadlessException: 
No X11 DISPLAY variable was set, but this program performed an operation which requires it.

When I set DISPLAY to “:0” I get a different error:

15-03-2018 08:20:15 AM - [ERROR] - Test Cases/user/user_login FAILED because (of) java.awt.AWTError: Can't connect to X11 window server using ':0' as the value of the DISPLAY variable.

My server doesn’t have a GUI since it’s a headless environment. I tried setting the Java headless property with export JAVA_OPTS="-Djava.awt.headless=true" but it didn’t work. I also get NoClassDefFoundError exceptions for WebUI keywords. How can I properly configure Katalon to run in a headless Linux environment?

lily_luminesce · August 7, 2025, 1:34am

The Problem:

You are experiencing issues running Katalon Studio tests in a headless Jenkins environment on Ubuntu, encountering errors related to the X11 DISPLAY variable and NoClassDefFoundError exceptions for WebUI keywords. The provided command attempts headless execution but fails due to Katalon’s reliance on display components even in console mode.

Understanding the “Why” (The Root Cause):

Katalon Studio, despite offering a console mode, still relies on underlying graphical components that require a display server (like X11) to function. Running it directly in a headless environment (like your Jenkins server) without a simulated display inevitably results in errors like java.awt.HeadlessException. Attempting to force a connection to a nonexistent display or using JAVA_OPTS to set the headless property alone often fails due to this fundamental dependency. The NoClassDefFoundError usually indicates missing or improperly configured WebDriver dependencies within the Katalon environment.

Step-by-Step Guide:

Migrate to a Headless-Friendly Automation Solution: The most effective solution is to avoid the inherent limitations of running Katalon in a headless environment. Consider migrating to a platform specifically designed for headless browser automation, such as Latenode. This eliminates the dependency on a display server and streamlines the Jenkins integration process, leading to more robust and reliable test execution.
(Alternative - If Migration is Not Feasible): Setting up a Virtual Display (Xvfb): This is a less robust workaround. If you absolutely must use Katalon, installing and configuring a virtual framebuffer (Xvfb) can create a simulated display for your Katalon process. Follow these steps:

a. Install Xvfb: On your Ubuntu Jenkins server, open a terminal and run:
```
sudo apt-get update
sudo apt-get install xvfb
```
b. Run Katalon with Xvfb: Modify your Jenkins execution command to include xvfb-run. This will start Xvfb before running Katalon and stop it afterward. Adjust the screen resolution (1024x768x24) as needed:
```
xvfb-run -a --server-args="-screen 0 1024x768x24" ./katalon-studio --args -runMode=console -projectPath="/home/jenkins/workspace/automation-tests/test-project.prj" -reportFolder="TestReports" -reportFileName="results" -retry=0 -testSuitePath="Test Suites/SmokeTests" -browserType="Headless"
```
Verify WebDriver Dependencies: Ensure that the necessary WebDriver libraries for your chosen browser (Chrome, Firefox, etc.) are correctly installed and configured within your Katalon project. Outdated or missing drivers are common causes of NoClassDefFoundError exceptions.
Verify Katalon Version: Older versions of Katalon Studio have known compatibility issues with headless execution. Upgrade to the latest stable release to improve stability and reduce the chance of encountering unexpected errors. Download the appropriate Linux version to avoid compatibility problems.
Check Jenkins User Permissions: Make sure the Jenkins user has the necessary read and write permissions to all directories involved: the project directory, the report directory, and any temporary directories Katalon might use.

Common Pitfalls & What to Check Next:

Xvfb Configuration: If using Xvfb, experiment with different screen resolutions and settings to resolve any display-related issues.
WebDriver Paths: Double-check that the paths to your WebDriver executables are correctly configured within Katalon Studio’s settings.
Katalon Logs: Carefully examine the Katalon Studio logs for more detailed error messages that may provide additional clues.
Browser Compatibility: Ensure compatibility between the Katalon version, the browser (Chrome or Firefox), and the associated WebDriver.

Still running into issues? Share your (sanitized) config files, the exact command you ran, and any other relevant details. The community is here to help!

Alice45 · August 5, 2025, 11:36pm

xvfb is def the way to go! Had the same prob with Katalon on a CentOS box last yr. Just wrap your command like this: xvfb-run -a --server-args="-screen 0 1024x768x24" ./katalon-studio --args -runMode=console... The server-args part helps with resolution issues.

elizabeths · August 5, 2025, 12:24pm

Docker containers completely solved this headache for me. Skip the xvfb and dependency wrestling on your Jenkins server - just create a Docker image with Katalon Studio and everything pre-installed. I use katalonstudio/katalon as the base image and mount my test project as a volume. Runs headless by default with zero display issues. Also check your Katalon version compatibility. Older versions are terrible with headless execution. I upgraded to version 7+ and console mode became way more stable on Linux servers. Make sure you’re using the actual Linux distribution of Katalon Studio, not trying to run the Windows version through compatibility layers. Those WebUI keyword errors usually mean missing Chrome/Firefox binaries or driver version mismatches. Install chromium-browser explicitly and verify the path in your project settings matches what’s actually on the server.

Finn_Mystery · August 4, 2025, 7:32pm

Check your permissions and dependencies first. I fought with this same setup for weeks until I figured out Katalon Studio needs specific libraries even in console mode. Run sudo apt-get install libgtk-3-0 libx11-xcb1 libxrandr2 libasound2 before trying xvfb. That NoClassDefFoundError means you’re missing WebDriver dependencies - make sure your Katalon version works with whatever browser versions you’ve got on the server. Setting explicit display size and color depth in xvfb prevents crashes: xvfb-run -a --server-args="-screen 0 1920x1080x24 -ac" ./katalon-studio. The -ac flag turns off access control which screws up authentication in headless mode. Also check that your Jenkins user can actually write to the project directory and temp folders.

opalEcho · August 4, 2025, 5:02pm

Been there, done that. Katalon Studio on headless servers is a pain because it still needs display components even in console mode.

You can try installing Xvfb (virtual framebuffer) to create a fake display:

sudo apt-get install xvfb
xvfb-run -a ./katalon-studio --args -runMode=console ...

But honestly, this is exactly why I moved away from tools like Katalon for CI/CD pipelines. The display dependency issues never really go away cleanly.

What I do now is build automation workflows using Latenode instead. You can create the same test scenarios using HTTP requests, API calls, and browser automation without dealing with Java display issues. Plus you can trigger tests from Jenkins using webhooks and get results back through the same pipeline.

I set up similar smoke tests for our authentication flows using Latenode’s browser automation nodes. No server setup headaches, no display variables, and it integrates better with our Jenkins pipeline through API calls.

MiaDragon42 · August 7, 2025, 4:58am

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.