Runaway bash shell spawning

I recently ran into a problem on a production server which became sluggish and was showing load averages in the 20s. The server doesn’t usually do all that much and is overpowered for its job.

So this sluggishness was surprising.

Looking at top was even more surprising, as it showed many ps processes chewing up most of the CPU on the box. And running ps -fe revealed there was more than 200,000 bash processes running with the following arguments:

bash -ec IFS=:; paths=($PATH); for i in ${!paths[@]}; do if [[ ${paths[i]} == "/home/ubuntu/.pyenv/shims" ]]; then unset 'paths[i]'; fi; done; echo "${paths[*]}"

The only clue here was the reference to .pyenv, which did turn out to be the problem. I don’t really understand what happened, or why it showed up as ps gone wild, but a comment on a Github issue “pyenv init getting into an infinite loop when invoked from bashrc” pointed to the solution.

I modified the .bashrc to start with the line

[ -z "$PS1" ] && return

and almost instantly my load average dropped to normal levels. The bash process count gradually ticked down - I assume each shell was spawning another one and my change allowed that to unwind by stopping this process - and the machine became responsive again (except poor htop, which just couldn’t handle that many processes).

I guess the problem here is that I created the .bashrc in order to add Pyenv’s init steps, and didn’t realise that by default many .bashrc files start with the line I added as a fix.