Skip to main content

Continuous Testing with GitLab CI

Serenity/JS integrates with industry-standard test runners, build tools, and CI/CD pipelines to help your team continuously deliver high-quality software.

In this article, I'll show you how to configure GitLab CI to run automated acceptance tests built using Serenity/JS and publish test execution reports and living documentation to GitLab Pages, so that you can share them with your team.

You will learn:

  • How to run Serenity/JS test scenarios as part of a GitLab CI pipeline
  • How to produce Serenity BDD reports and living documentation and publish them to GitLab Pages
  • How to make your test suite produce JUnit-standard reports and use them to decorate GitLab Merge Requests

Writing tests​

To run your automated tests on GitLab CI you first need to write them 😊 The easiest way to get started with automating tests using Serenity/JS is to use one of the available Serenity/JS Project Templates as they come with example scenarios and combine some of the most popular configurations of Serenity/JS modules and test automation tools:

If you're new to Serenity/JS, you might want to follow the tutorial to learn how to write your first web scenario!

Running tests on GitLab CI​

GitLab CI uses Docker containers to run your CI/CD jobs. To ensure maximum stability of your test suite, you should run Serenity/JS in a container that offers the latest Long-Term Support version of Node.js, as per the the Serenity/JS installation guide.

The exact Docker image you choose depends on the runtime dependencies required for your tests.

For example:

Below example shows basic configuration of a GitLab CI Pipeline you can place in the .gitlab-ci.yml file in the root directory of your GitLab repository:

.gitlab-ci.yml
stages:
  - test

serenity:
  stage: test
  image: node:lts
  script:
    - npm ci
    - npm test
  artifacts:
    when: always
    paths:
      - target

# other configuration

How a GitLab CI pipeline works:

  • When a code change is pushed to your source control repository, GitLab CI runs your code through the stages of a pipeline you've defined in your pipeline configuration file. The pipeline in our example has one stage so far: test, however most pipelines will have additional stages like build, report, or deploy.
  • Each pipeline stage can have multiple jobs, in this case a job called serenity provisions a Docker container specified as image, checks out your code, and invokes a script responsible for running your automated tests.
  • In our example, script runs:
    • npm ci, which installs the Node.js modules that your automation projects depends on
    • npm test, which invokes the test script defined in your package.json
  • Since all the Serenity/JS Project Templates are configured to run tests via npm test and produce Serenity BDD reports under target/site/serenity, we configure GitLab to archive any artifacts under target. This way you can inspect them in GitLab UI or use them in subsequent stages of your pipeline.

Generating Serenity BDD reports​

Serenity/JS uses Serenity BDD CLI to generate Serenity BDD HTML reports and living documentation. Serenity BDD CLI is a Java program downloaded and managed by Serenity/JS. However, since Serenity BDD CLI is a Java program, it requires a Java Runtime Environment (JRE) which might not be available in your Docker image.

If the Docker image you use doesn't offer a Java Runtime Environment (JRE), you can:

  • Install a JRE as part of your GitLab CI job
  • Create a custom Docker image with both Node.js and a JRE

Installing a Java Runtime Environment​

To install a JRE before running the tests, add apt-get update && apt-get install default-jre -y to your GitLab CI script:

.gitlab-ci.yml
stages:
  - test

serenity:
  stage: test
  image: node:lts
  script:
    - apt-get update && apt-get install default-jre -y
    - npm ci
    - npm test
  artifacts:
    when: always
    paths:
      - target

# other configuration

Building a custom Docker image​

To avoid installing the Java Runtime Environment every time your GitLab CI pipeline runs, you might prefer to build a custom Docker image, publish it to your organisation's artifact registry, and use that image instead.

To add a JRE to your base Docker image of choice, use a Dockerfile like in the example below:

Dockerfile
FROM node:lts

RUN apt-get update \
 && apt-get install default-jre -y \
 && apt-get clean

Next, build and publish your Docker image as per the instructions from your artifact registry vendor:

Publishing Serenity BDD reports to GitLab Pages​

GitLab Pages is a feature of GitLab CI that allows you to publish static HTML websites, such as Serenity BDD HTML reports, and share them with your team.

GitLab Pages rely on convention over configuration, so to use this feature your pipeline needs to meet the below criteria:

  • You must declare a GitLab CI job called pages in your .gitlab-ci.yml
  • There must be an explicit dependency between the serenity job and the pages job so that the artifacts are propagated between the stages of your pipeline
  • Any static content you wish to publish must be moved to a directory called public

Additionally, since GitLab Pages are branch/tag agnostic, you should use rules:if to limit their deployment only to builds executed on the default branch in your repository.

To publish your Serenity BDD reports to GitLab Pages, add a pages job to your .gitlab-ci.yml as per the example below:

.gitlab-ci.yml
stages:
  - test
  - publish

serenity:
  # ...

pages:
  stage: publish
  dependencies:
    - serenity
  script:
    - mv target/site/serenity public
  artifacts:
    paths:
      - public
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Note that GitLab Pages for repositories hosted on gitlab.com are published to gitlab.io. For example, project hosted at gitlab.com/my-group/my-project will have its GitLab Pages published to my-group.gitlab.io/my-project.

Consult GitLab documentation to learn more about the GitLab Pages URL structure.

Generating JUnit reports​

GitLab CI recognises test execution reports that conform to the JUnit XML standard. Serenity/JS integrates with all the native reporters offered by the supported test runners, including the ones producing JUnit reports, such as:

To make GitLab CI recognise a JUnit-standard report produced by your Serenity/JS test suite and surface test results in your Merge Requests, you should:

  • Use a native JUnit reporter appropriate for your test runner
  • Configure artifacts:reports:junit to tell GitLab where to find the report.

For example, if you have configured your JUnit reporter to produce a report at target/junit-results.xml, you can configure your GitLab CI job as follows:

.gitlab-ci.yml
stages:
  - test

serenity:
  stage: test
  # ...
  artifacts:
    when: always
    paths:
      - target
    reports:
      junit: target/junit-results.xml

# other configuration
Pro Tip

Note that in the above example I use artifacts:when: always. This instructs GitLab to upload and analyse the test report even when the test run fails, which is exactly when you need your test reports the most 😊

Next steps​

Congratulations! 🥳 You've just learnt how to run your Serenity/JS test scenarios on GitLab CI! If you enjoyed this tutorial, please leave a 👍 in the reactions section below.

New tutorials and videos are coming soon, follow Serenity/JS on LinkedIn and subscribe to Serenity/JS YouTube channel to get notified when they're available!

LinkedIn Follow YouTube Follow

Help us help you

Serenity/JS is a free open-source framework, so we rely on our wonderful GitHub sponsors to keep the lights on.

If you appreciate all the effort that goes into making sophisticated tools easy to work with, please support our work and become a Serenity/JS GitHub Sponsor today!

GitHub Sponsors