← All TILs · ansible

A shared Ansible dev server for VS Code Remote-SSH, built with Ansible

ansible - 2026-09-30

Ninth entry in the Ansible development environment series. Part 4 listed Remote-SSH as the option for shops that only have VMs: VS Code stays on the laptop, and the code, the tools and the execution environments (EEs, the container images that Red Hat Ansible Automation Platform runs jobs in; part 3 builds one) live on a Linux server that developers reach over SSH. On the first connection, the Remote-SSH extension installs the VS Code Server into the user's home directory, ~/.vscode-server, and extensions such as Red Hat's Ansible extension run there.

This part builds that server with a playbook: Podman, one shared install of ADT (the Ansible Development Tools package from part 5), and one shared copy of the EE. It follows Alex Dworjan's two roles, shadowman_dev_vs_codeserver and shadowman_dev_shared_image_store, updated where testing showed they needed it. Everything below ran from ansible-core 2.21.4 against Rocky Linux 9.8, a RHEL 9 rebuild, booted with systemd, with Podman 5.8.2 and developers logging in over SSH.

The playbook

---
- name: Shared Ansible development server for VS Code Remote-SSH
  hosts: devservers
  become: true
  vars:
    dev_users: [alice, bob]
    adt_venv: /opt/adt
    shared_image_store: /var/lib/ee-shared
    ee_image: ghcr.io/ansible-community/community-ee-base@sha256:9f2836592ab92794e8b1982311d504ba3c28a2c09b3f8de221ca3564842c0902

  tasks:
    - name: Install Podman, fuse-overlayfs, git, Python 3.12, and packaging for the pip module
      ansible.builtin.dnf:
        name: [podman, fuse-overlayfs, git, python3.12, python3-packaging]
        state: present

    - name: Copy the ADT lock file
      ansible.builtin.copy:
        src: adt-requirements.txt
        dest: /opt/adt-requirements.txt
        mode: "0644"

    - name: Install the locked ADT into one venv for everyone
      ansible.builtin.pip:
        requirements: /opt/adt-requirements.txt
        virtualenv: "{{ adt_venv }}"
        virtualenv_command: python3.12 -m venv
        extra_args: --require-hashes

    - name: Put the venv on every login shell's PATH
      ansible.builtin.copy:
        dest: /etc/profile.d/adt.sh
        content: |
          case ":$PATH:" in
            *":{{ adt_venv }}/bin:"*) ;;
            *) PATH="{{ adt_venv }}/bin:$PATH" ;;
          esac
        mode: "0644"

    - name: Create the shared image store
      ansible.builtin.file:
        path: "{{ shared_image_store }}"
        state: directory
        mode: "0755"

    - name: Check whether the EE is already in the shared store
      ansible.builtin.command: podman --root {{ shared_image_store }} image exists {{ ee_image }}
      register: ee_present
      changed_when: false
      failed_when: ee_present.rc > 1

    - name: Pull the EE into the shared store
      ansible.builtin.command: podman --root {{ shared_image_store }} pull {{ ee_image }}
      when: ee_present.rc == 1
      changed_when: true

    - name: Make the shared store readable by every developer
      ansible.builtin.file:
        path: "{{ shared_image_store }}"
        mode: a+rX
        recurse: true

    - name: Check each developer has subordinate IDs for rootless Podman
      ansible.builtin.command: getsubids {{ item }}
      changed_when: false
      loop: "{{ dev_users }}"

    - name: Keep each developer's user services running between SSH sessions
      ansible.builtin.command: loginctl enable-linger {{ item }}
      args:
        creates: /var/lib/systemd/linger/{{ item }}
      loop: "{{ dev_users }}"

    - name: Create each developer's Podman and VS Code Server config folders
      ansible.builtin.file:
        path: /home/{{ item.0 }}/{{ item.1 }}
        state: directory
        owner: "{{ item.0 }}"
        group: "{{ item.0 }}"
        mode: "0700"
      loop: "{{ dev_users | product(['.config', '.config/containers', '.vscode-server', '.vscode-server/data', '.vscode-server/data/Machine']) }}"

    - name: Give each developer's rootless Podman the shared store, read-only
      ansible.builtin.copy:
        dest: /home/{{ item }}/.config/containers/storage.conf
        content: |
          [storage]
          driver = "overlay"

          [storage.options]
          additionalimagestores = ["{{ shared_image_store }}"]
          mount_program = "/usr/bin/fuse-overlayfs"
        owner: "{{ item }}"
        group: "{{ item }}"
        mode: "0644"
      loop: "{{ dev_users }}"

    - name: Point the Ansible extension at the shared venv
      ansible.builtin.copy:
        dest: /home/{{ item }}/.vscode-server/data/Machine/settings.json
        content: '{ "ansible.python.interpreterPath": "{{ adt_venv }}/bin/python" }'
        owner: "{{ item }}"
        group: "{{ item }}"
        mode: "0644"
        force: false
      loop: "{{ dev_users }}"

It passes ansible-lint 26.9.0 at the production profile. A second run reported changed=0.

One locked ADT, on Python 3.12

Dworjan's role installs ADT for each user with pip3.11 --user. As part 5 showed, Python 3.11 now resolves ADT to ansible-core 2.19, which reaches end of life in November 2026. RHEL 9's AppStream repository has Python 3.12 (3.12.14 on 9.8), which gets the current 2.21.

The playbook installs ADT once, into a venv (a directory with its own Python and packages) at /opt/adt, from a lock file made as in part 2:

echo 'ansible-dev-tools==26.9.0' > requirements.in
uv pip compile --python-version 3.12 --python-platform x86_64-manylinux_2_34 \
  --generate-hashes requirements.in -o adt-requirements.txt

Wheels are PyPI's pre-built packages, and their manylinux_2_NN tag names the oldest glibc, the GNU C library, they need. manylinux_2_34 matches RHEL 9's glibc 2.34, so uv picks wheels that run there. The file pinned 69 packages, including ansible-core 2.21.4. Everyone on the server gets the same versions, and an upgrade is a new lock file and one more run. Two details came from the test runs: - The pip module needs the packaging library in the server's system Python. The first run stopped with "Failed to import the required Python library (packaging) on devsrv's Python /usr/bin/python3". The python3-packaging package fixes it. - The case guard in adt.sh keeps the venv on PATH once. Without it, an SSH login shell had /opt/adt/bin on its PATH twice.

A shared image store for the EE

Rootless Podman runs containers as the developer, and keeps each user's images under their home directory. This site's rootless entry explains how. Fifteen developers means fifteen copies of the same EE. An additional image store avoids that: a read-only store that Podman searches besides the user's own. Root pulls the EE into it once with podman --root, and every user's Podman reads it from there.

Three things were needed before a developer's EE would run from it: 1. Read permission. Podman created the store's folders as 0700, root only. A user's Podman then failed with "open /var/lib/ee-shared/overlay-images/images.lock: permission denied". Dworjan's role runs chmod -R a+rx after each pull, and the playbook does the same with mode: a+rX. It adds read and search permission without removing any other bits. 2. A per-user storage.conf. Adding the store to /etc/containers/storage.conf had no effect for rootless users: podman info as a user showed no additional store. Podman's storage man page, on its main branch, lists a directory for all rootless users at once, /etc/containers/storage.rootless.conf.d/, but Podman 5.8.2 ignored it too. A ~/.config/containers/storage.conf for each user, as in Dworjan's role, worked. 3. fuse-overlayfs. Podman normally builds a container's filesystem with the kernel's overlay filesystem, which stacks the image's read-only layers under a writable one. With the shared store, every container failed before starting: "creating temporary passwd file for container … /etc/passwd: permission denied". The layers belong to the host's root, which isn't mapped into the developer's user namespace, so inside the container they're owned by nobody (UID 65534), and even the container's root can't change them. mount_program = "/usr/bin/fuse-overlayfs", the line Dworjan's template also has, mounts the layers through fuse-overlayfs instead: the same stacking, implemented as a user-space program. With it, the containers ran.

The result, as Alice, using part 7's ansible-navigator.yml with pull: policy: missing: - ansible-navigator run used the EE from the shared store and reported its ansible-core, 2.21.3. Alice's own image storage stayed at 196 KB. - Carol, a user without the shared store, needed her own copy of the image: 483 MB. - The Ansible extension's EE mode was checked by running its podman run flags, taken from the extension's language server source, with ansible-playbook --syntax-check. It passed with the shared store too. Unlike navigator, those flags run the image's own user, not root.

One difference remains. The EE's home directory /runner also shows as owned by nobody, so the image's entrypoint, the script that starts every container, set HOME=/tmp instead. ansible-navigator isn't affected: it mounts its own writable /runner.

If the EE comes from a registry that needs a login, don't copy Dworjan's podman pull --creds=USER:PASS. On a shared server, any user can read another process's command line with ps. Log in first, with podman login or the containers.podman.podman_login module and no_log: true.

Subordinate IDs and lingering

Rootless Podman maps a range of subordinate user and group IDs, reserved per user in /etc/subuid and /etc/subgid, into each container. useradd reserved one range per local user: alice:100000:65536 and bob:165536:65536. Users who come from a directory, such as LDAP or Red Hat Identity Management (IdM), don't get one that way. To test, Dave's entries were removed: - getsubids dave answered "Error fetching ranges". The playbook's check task fails on it, which is the point: better at provisioning time than in a developer's first podman run. - Podman itself logged "no subuid ranges found for user "dave" in /etc/subuid" and fell back to "rootless single mapping into the namespace. This might break some images."

For local users, usermod --add-subuids adds a range. For directory users, Podman's rootless tutorial notes that shadow-utils 4.9 and later, the version RHEL 9 has, can read them from SSSD, the service RHEL uses to look up directory users, with subid: sss in /etc/nsswitch.conf. Dworjan's role ships two small modules that write the files instead.

Systemd starts a per-user service manager, user@<UID>.service, and a runtime directory, /run/user/<UID>, when a user logs in, and removes them after the last session ends. Rootless Podman keeps its runtime state in that directory. Lingering keeps both running with no session open; this site's Caddy entry uses it for rootless services. In the test, Dave's /run/user/1003 was still there 2 seconds after his SSH session closed, and gone 14 seconds after. Alice's user service was already running before her first login. Podman's troubleshooting guide gives the symptom without it, "rootless containers exit once the user session exits", and loginctl enable-linger as the fix.

The machine settings file

Part 6 listed VS Code's settings scopes: default, user, remote, workspace. The remote scope is per server. VS Code Server reads it from ~/.vscode-server/data/Machine/settings.json. That's where ansible.python.interpreterPath belongs, since part 6 found its scope is machine-overridable. The playbook points it at /opt/adt/bin/python. force: false writes the file only once, so a developer's own remote settings survive the next run, and a repository's .vscode/settings.json still overrides it.

What the playbook leaves to VS Code

Dworjan's role also downloads the VS Code Server for one commit ID, and installs the Ansible extension with the server's own --install-extension. It's simpler to let Remote-SSH do both: - The server. Remote-SSH's install script downloads the server build matching the laptop's own VS Code commit, so a pre-installed one goes unused after the laptop's next VS Code update. If the server has no internet access, the default remote.SSH.localServerDownload: auto falls back to downloading on the laptop and copying the server over with scp. - Extensions. The repository's .vscode/extensions.json (part 6) prompts for them. Each developer can also list them under remote.SSH.defaultExtensions in their user settings to install them on every SSH host. Both need the server to reach the Visual Studio Marketplace over HTTPS; without that, VS Code's docs point to Extensions: Install from VSIX…. - The server's OS. Since VS Code 1.99 (March 2025), the prebuilt server needs glibc 2.28 or later, so RHEL 8 or newer. RHEL 7 is out.

Test notes

The server was a Rocky Linux 9.8 container running systemd, with sshd, reached by SSH from inside it. Four adjustments applied only to that test machine: - the sandbox proxy's CA certificate was trusted; - the file capabilities that newuidmap and newgidmap need were restored, because the container image had lost them (rpm -V shadow-utils flagged both); - /dev/fuse and /dev/net/tun were opened to users; - Ansible reached the server through the Docker connection plugin, since this sandbox has no SSH client.

On a RHEL 9 VM, the capabilities and device permissions are already set that way, and the proxy and the Docker connection don't apply.

Sources

Created 2026-09-30T17:00:39+02:00 · Edit