Set operations on lists: declared Proxmox guests against the cluster, and why the order changes between runs

ansible - 2026-10-02

Sixth entry in the Shaping data in Ansible series. Drift is the gap between what a configuration declares and what actually exists. Finding it means comparing two lists:

• what is declared but missing, to create;
• what exists but isn't declared, the strays, to investigate or remove.

ansible-core has five filters for this, from set theory: unique, union, intersect, difference and symmetric_difference. The example compares the Proxmox VE guests a cluster should have with the ones it reports. Everything below ran with ansible-core 2.21.4.

The five filters on Proxmox guests

Part 4 used six Proxmox guests, with VMIDs 100 to 105, from the output that community.proxmox.proxmox_vm_info returns. The VMID is each guest's unique number (part 2 explains why it beats the name as a key). The declared list has a guest not created yet, 106, and 101 twice by mistake:

declared_vmids: [106, 100, 101, 104, 101]
actual_vmids: "{{ proxmox_vms | map(attribute='vmid') | list }}"   # [100, 101, 102, 103, 104, 105]

• difference depends on the direction. a | difference(b) keeps what's in a and not in b. Both directions are needed: one gives the guests to create, the other the strays.
• symmetric_difference mixes the two. It answers "is there any drift?" in one list, but not which side each item came from.
• Every result drops duplicates. 101 appears once in intersect although it's declared twice. The docs, Selecting from sets or lists, say so: "all of the following filters imply uniqueness".

The order isn't the order you gave

strays came out as [105, 102, 103], neither sorted nor in the cluster's order. The source explains why. In ansible-core's plugins/filter/mathstuff.py, the four two-list filters are Python set operations, for example list(set(a) - set(b)) for difference. A Python set is an unordered collection that keeps each item once. It finds items by their hash, a number computed from the value, and the order you get back follows the hashes, not the input. unique is the exception: it keeps the order in which items first appear.

For integers the hash is the number itself, so the order is odd but the same on every run. For strings it changes from one run to the next. Python computes string hashes with a random seed chosen when each process starts, unless the PYTHONHASHSEED environment variable fixes it. Every ansible-playbook run is a new process. Four runs of the same expression on the guests' names:

"{{ declared_names | union(actual_names) }}"

['pg1', 'pxe.home.arpa', 'test-lxc.home.arpa', 'test1', 'test1-lxc.home.arpa']
['test-lxc.home.arpa', 'test1-lxc.home.arpa', 'pg1', 'pxe.home.arpa', 'test1']
['test-lxc.home.arpa', 'test1-lxc.home.arpa', 'test1', 'pxe.home.arpa', 'pg1']
['test-lxc.home.arpa', 'test1-lxc.home.arpa', 'pg1', 'pxe.home.arpa', 'test1']

Three different orders in four runs, with nothing changed. A task that writes this list into a file, a monitoring configuration or an inventory, changes the file on most runs. Ansible then reports the task as changed and runs its handlers, the tasks that only run on a change, such as a service restart. With | sort at the end, the list and the file were identical in every run.

Sort the result of a set operation before writing it anywhere, or before comparing it with something that has an order.

Items that can't go into a set, such as dicts, take a fallback in the same source file that keeps the first list's order: proxmox_vms | difference(proxmox_vms[:2]) returned the guests 102 to 105, in order.

A VMID is not "101"

The comparison is exact, type included. VMIDs from proxmox_vm_info are integers. Declared as strings, which is how they arrive from a form, a CSV file or an extra var on the command line, nothing matches:

There's no error or warning: the integer 101 and the string "101" are just different items. Convert one side with map('int') or map('string') before comparing.

Which one

• To create: declared | difference(actual).
• To investigate or remove: actual | difference(declared).
• Anything different at all: symmetric_difference, then the two differences to see which side.
• Order matters: add | sort. Only unique keeps the input order.
• Values from outside Ansible: check both lists hold the same type. The type_debug filter shows a value's type, for example int or str.

The example repository

The series' companion repository, abdelhousni/ansible-data-shaping-series, runs all of the above on the local machine, changing nothing outside its out/ directory. drift.yml compares the declared guests in vars/declared.yml with part 4's recorded data and writes every result to out/. run.sh runs it twice more with PYTHONHASHSEED=1 and PYTHONHASHSEED=2, standing for two separate runs, and shows the unsorted file of names differ while the sorted one doesn't. Its CI runs it on every push and compares the output with the expected one.

Sources

• ansible-core 2.21.4: plugins/filter/mathstuff.py (unique, union, intersect, difference, symmetric_difference, and their fallback for unhashable items).
• Ansible docs, from ansible/ansible-documentation: playbook_guide/playbooks_filters.rst, Selecting from sets or lists (set theory).
• Python documentation: PYTHONHASHSEED and set types.
• community.proxmox 2.0.0: tests/unit/plugins/modules/test_proxmox_vm_info.py, EXPECTED_VMS_OUTPUT, as recorded for part 4.
• Every result above came from ansible-core 2.21.4 on 2026-10-02, on the local machine, with the recorded Proxmox data. No Proxmox server was involved.

Created 2026-10-02T21:45:52+02:00 · Edit