Skip to content

CI machines

Jump to bottom
Sanakan8472 edited this page Dec 11, 2022 · 1 revision

Introduction

Setting up a new Linux Agent

Create VM

  • Create a new VM, reuse the ezEngine resource group.
  • Give the VM a name.
  • Image: Select See all images and select Ubuntu Server 22.04 LTS x64 Gen2.
  • Size: Standard_D4s_v3
  • Auth type: Password. User name and password can be found in the EZ keyvault: AzureVM-User and AzureVM-PW.
  • Allow inbound ports: SSH, http, https
  • OS disk type: Standard SSD, locally redundant.
  • Virtual Network: ezEngine-vnet
  • Auto Shutdown: Disable

Leave everything else as their defaults. Once created, the VM will start automatically and you can SSH into it using the user name and password. The current IP can be seen when trying to connect via SSH on the VM page.

Basic package dependencies

For a new Linux agent we need to install several prerequisites besides just the Azure agent software.

sudo apt-get update
sudo apt-get install -y cmake uuid-dev gcc-12 g++-12 libx11-dev build-essential qt6-base-dev libqt6svg6-dev qt6-base-private-dev libxrandr-dev libxinerama-dev libxcursor-dev libxi-dev xvfb lightdm libtinfo5 vulkan-tools git-lfs

Additionally, you will need to install powershell.

SSL 1.1 is needed for the agent to work

Ubuntu 20.04 installs OpenSSL 3.1 which the Azure agent can cope with as it is using a very old .Net Core 3.1 version which only works with OpenSSL 1.1. The easiest way to solve this is by manually downgrading to 1.1 via these commands:

wget -c http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.16_amd64.deb
sudo apt install ./libssl1.1_1.1.1f-1ubuntu2.16_amd64.deb
wget -c http://security.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_1.1.1f-1ubuntu2.16_amd64.deb
sudo apt install ./openssl_1.1.1f-1ubuntu2.16_amd64.deb

Additionally, you will need to run these at least for the setup of the agent:

export AZP_AGENT_USE_LEGACY_HTTP=true
sudo sed -i 's/openssl_conf = openssl_init/#openssl_conf = openssl_init/g' /etc/ssl/openssl.cnf

lightdm needs to be the default window manager

By default, Ubuntu VMs have gdm3 as their default window manager. We need to switch to lightdm which allows us to create a headless monitor so that render tests are able to create a window / swapchain. One of these two commands should allow you to select the default window manager.

sudo dpkg-reconfigure gdm3
sudo dpkg-reconfigure lightdm

Install agent

The main azure page for this is here. Follow instructions and install agent. You can find the latest download link and instructions by going to the agent list, clicking New agent and selecting Linux. At the time of writing this would result in there commands:

mkdir agent && cd agent
wget https://vstsagentpackage.azureedge.net/agent/2.213.2/vsts-agent-linux-x64-2.213.2.tar.gz
tar zxvf vsts-agent-linux-x64-2.213.2.tar.gz
./config.sh

For configure you will need to select these values:

Run and test agent

Before registering the agent as a service, just run

sudo ./run.sh

The agent should appear as online in the agent pool list in azure devops. Run a linux pipeline on it and see what happens. If everything worked well, you can setup the agent as a service to autostart with the VM. You can see all the latest instructions here.

sudo ./svc.sh install

On the next boot, the agent should auto-start.

Register agent with azure function

We use an azure function called ezEngineCI to start and shutdown VMs when needed. As this function is currently stateless, the code needs to be modified in order to recognize the new VM. The code for the azure function can be found in the storage account under ez-container -> ezEngineCI.7z. Extract the archive locally and open it in VS. You will need to install the Azure feature set in VS to do so.

At the top of ShutdownWindowsVM.cs you will find a code section that lists each VM and the pipelines that use it like this:

static readonly string _windowsVmName = "Windows10-1809";
static readonly int[] _windowsDefinitionIDs = new int[] { 12, 10, 11 }; // android, x64 and UWP x64 build pipelines.

You can figure out the pipeline ID but clicking on it in Azure Devops and looking at the URL. You will need to duplicate the code for each new VM and duplicate the checks in StartVM and RunShutdownVM. This should eventually be moved into a storage account and work automatically. Until then this is necessary to make sure the VM is shutdown once no pipeline is queued up anymore that requires it.

Once changes are mode, you can publish the project from within VS. Afterwards zip up the project again and replace it in the storage acount.

Custom Mesa Build

As we can't afford running a GPU SKU in Azure we test the Vulkan renderer via the CPU implementation provided by Mesa, called lavapipe. The Mesa build in Ubuntu 22.04 is too old and the lavapipe build inside will not pass our tests, so we need to compile it from scratch.

Enable source apt-get

You need to add deb-src entries to your /etc/apt/sources.list. Either by manually editing the file and removing all the # before the deb-src lines or by running this:

sed -i~orig -e 's/# deb-src/deb-src/' /etc/apt/sources.list

Dependencies

Mainly we need python3, llvm, flex, bison. I'm not sure if apt-get build-dep mesa does all that already so I went a bit overboard perhaps. libncurses5 was needed for ez spirv compiler to load.

sudo apt-get update
sudo apt-get install mesa-utils libglu1-mesa-dev scons llvm-dev python-pip libyaml-dev libncurses5 flex bison
sudo apt install python3-pip
pip install prettytable Mako pyaml dateutils --upgrade
sudo apt-get build-dep mesa

Get Mesa

Latest is probably fine too but the commit here is used on CI right now:

git clone https://gitlab.freedesktop.org/mesa/mesa.git
cd mesa
git checkout 7b60414494bbc973fe65a565aec255c0cea99884
git submodule update
git submodule init

Configure Mesa

I'm sure this can be done in one line but whatever. I'm not sure all the parameters are correct but this one did compile at least even if some steps might be unnecessary.

  • -Db_ndebug: whether asserts are disabled
  • -Db_vscrt=mt -Dzlib:default_library=static : Windows only?
  • -Dllvm=enabled : llvm is needed for lavapipe
  • -Dshared-llvm=disabled : Windows only?
  • --buildtype=release : either release or debugoptimized
  • -Dgallium-drivers=swrast -Dvulkan-drivers=swrast : what drivers to build. We are only interested in the software rasterizer.
meson builddir/
meson setup --wipe builddir -Db_ndebug=true -Db_vscrt=mt -Dzlib:default_library=static -Dllvm=enabled -Dshared-llvm=disabled --buildtype=release -Dgallium-drivers=swrast -Dvulkan-drivers=swrast

Build

ninja -C builddir/

Install

You can use this which will put an icd json in /usr/local/share/vulkan/icd.d/lvp_icd.x86_64.json

sudo ninja -C builddir/ install

Alternatively you can put the mesa/builddir/src/gallium/targets/lavapipe/lvp_devenv_icd.x86_64.json file in one of the icd.d folders listed here, e.g. /etc/xdg/vulkan/icd.d. This has the benefit that the json points to the location of the build version so any re-build will update the icd. To use the correct version, you may have to force the driver selection by setting the environment variable:

export VK_ICD_FILENAMES=/usr/local/share/vulkan/icd.d/lvp_icd.x86_64.json

If everything worked, vulkaninfo --summary should list lavapipe. Make sure there are no mentions of non conformant vulkan driver, if that is the case the old lavapipe driver that ships with Ubuntu was loaded instead.

Mesa with Custom LLVM Version

If you are using a cutting-edge version of Mesa you may have to update LLVM as Ubunutu may not ship with v15 yet. This is not recommended unless you feel adventurous.

Automatic mode

Goto https://apt.llvm.org/ and use the script there to install llvm 15

wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh 15

Manual mode

If that fails, go to the Ubuntu section of that page and add the entries listed for your version to your /etc/apt/sources.list. For 22.04 it would be:

deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy-15 main
deb-src http://apt.llvm.org/jammy/ llvm-toolchain-jammy-15 main

Then run

apt-get update
apt-get install libllvm-14-ocaml-dev libllvm14 llvm-14 llvm-14-dev llvm-14-doc llvm-14-examples llvm-14-runtime

Specify LLVM version in Mesa

Create a text file named $HOME/.local/share/meson/native/my-llvm-x64 by repeatedly calling mkdir with all the steps and then gedit with the entire path and then put this in:

[binaries]
llvm-config = '/usr/lib/llvm-15/bin/llvm-config'

Edit meson.build and add 'passes' to the llvm_modules = [ array. This is necessary to make it compile.

Now run:

meson setup --wipe builddir -Dllvm=enabled --buildtype=debugoptimized -Dgallium-drivers=swrast -Dvulkan-drivers=swrast --native-file=$HOME/.local/share/meson/native/my-llvm-x64

Make sure the meson output lists:

        llvm:            yes
        llvm-version:    15.0.0

Compile again using:

ninja -C builddir/
Clone this wiki locally