← All TILs · ansible

A playbook hangs at fact gathering or `service_facts`: find the blocking call, then bound it

ansible - 2026-10-17

The play prints TASK [Gathering Facts] or TASK [ansible.builtin.service_facts] and nothing else, for hours. Ansible has no timeout on a task by default, so one command that never returns on one host holds up the whole run.

Some terms first:

Why it hangs

The module waits on a child process that doesn't end. Two common cases:

A prompt doesn't always hang. Ansible gives the module an empty input, so a script that reads one answer gets end-of-file and carries on. The script hangs when it asks again until it gets a non-empty answer: with no input it loops for ever, at 100% CPU. That's what I reproduced with this stand-in for such a script:

#!/bin/sh
# /etc/init.d/demo-prompt
case "$1" in
  status) sid=""
    while [ -z "$sid" ]; do printf 'Choose SID: '; read sid; done
    echo "running ($sid)";;
  *) exit 0;;
esac

With it in place, service_facts printed its task header and nothing more until I killed the run.

Find the host and the process

On the controller (the machine running ansible-playbook, or the AAP execution node), list the processes of the user Ansible runs as (awx on Tower):

ps -fww -u awx

The ssh ... hostname /bin/sh -c 'sudo ... python' line names the stuck host. Log in there and walk down from the Ansible Python process to the command that waits:

ps -ef --forest | less    # or: pstree -ap <pid of the ansible python process>

In my run the leaves were /usr/sbin/service --status-all and, under it, /etc/init.d/demo-prompt status.

Find the init.d script with a time limit

Run each script's status with a time limit and no input, and only report the ones that ran out of time:

for svc in /etc/init.d/*; do
  timeout 5 "$svc" status >/dev/null 2>&1 </dev/null
  [ $? -eq 124 ] && echo "HANG: $svc"
done

timeout exits with 124 when it had to stop the command. Test for that value only: a stopped service's status exits 3 by convention, which isn't a problem. The looser || echo form flagged five stopped services alongside the real culprit on the test host.

Fix the cause

For the script, in order of preference:

For fact gathering, leave out the subset that blocks. gather_subset takes subset names, with ! to exclude one:

# ansible.cfg
[defaults]
gather_subset = !hardware

Or per play, gather_subset: ['!hardware'] next to gather_facts. Tasks that need hardware facts (disks, memory, LVM) then won't find them. setup also has gather_timeout (10 seconds by default), but it only limits some of the hardware collectors, mount facts among them, so it doesn't catch every hang.

Bound the task so the run fails instead of waiting

Two ways, which don't clean up the same way. Tested with ansible-core 2.21.5:

- name: Gather service facts, at most 60 seconds
  ansible.builtin.service_facts:
  timeout: 60

- name: Or as a background job polled every 5 seconds
  ansible.builtin.service_facts:
  async: 60
  poll: 5

So async leaves a cleaner host for a single task such as service_facts. Add ignore_errors: true only if the play can do without the result; otherwise let it fail, so the host shows up in the recap.

On AAP, a job template's Timeout field (also timeout in /api/v2/job_templates/<id>/) and the global Default Job Timeout under the Jobs settings cancel the whole job after that many seconds. That ends the run, but doesn't tell you which host or command blocked.

Sources

The service_facts hang, the diagnostic loop and both time limits ran against a stand-in init.d script on an Ubuntu container without systemd, with connection: local. Not run: the third-party scripts themselves, an LVM or mount hang in fact gathering, systemd_service checks on a systemd host, and AAP.

Created 2026-10-17T20:31:27+02:00 · Edit