[Supervisor-users] Thanks for a great tool --- and my first question

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[Supervisor-users] Thanks for a great tool --- and my first question

Luc Saffre
Hi Supervisor users and developers,

this is my first post to this list, so I'd like to first say thanks for
such a great tool. I discovered it a month ago and start to use it more
and more on my production servers. I also wrote about it in my own
documentation: http://www.lino-framework.org/admin/linod.html

But now I discovered my first problem where I hope for your help.
I have a single configuration file written by myself:

$ cat /etc/supervisor/conf.d/linod_prod.conf
[program:prod_linod]
command=/bin/bash /var/www/vhosts/prod/linod.sh
username = www-data

The mentioned bash script contains:

$ cat /var/www/vhosts/prod/linod.sh
#!/bin/bash
set -e  # exit on error
cd /var/www/vhosts/prod
. env/bin/activate
python manage.py linod

The process starts and works well. The problem is that Supervisor does
not correctly stop the process. Here is a console session:

$ ps aux | grep linod
1001     15828  0.0  0.0   8344   880 pts/0    S+   18:07   0:00 grep linod

$ sudo service supervisor start
Starting supervisor: supervisord.

$ ps aux | grep linod
root     16279  0.0  0.0  19720  1668 ?        S    18:31   0:00
/bin/bash /var/www/vhosts/prod/linod.sh
root     16284 58.8  0.8 328736 135368 ?       S    18:31   0:02 python
manage.py linod
1001     16309  0.0  0.0   8344   884 pts/0    S+   18:31   0:00 grep linod

Until here it looks good. I am a bit surprised to see two processes
here, but that might be caused by some magic.

The real problem is that Supervisor obviously does not see that spawned
second process and therefore doesn't terminate it. After every restart I
have one additional process running:

$ sudo service supervisor restart
Restarting supervisor: supervisord.
$ ps aux | grep linod
root     16284  6.0  0.8 328736 135368 ?       S    18:31   0:02 python
manage.py linod
root     16322  0.0  0.0  19720  1672 ?        S    18:32   0:00
/bin/bash /var/www/vhosts/prod/linod.sh
root     16327  106  0.7 306560 115280 ?       R    18:32   0:02 python
manage.py linod
1001     16352  0.0  0.0   8344   884 pts/0    S+   18:32   0:00 grep linod

Yes, I am not using the latest version. I have a default configuration
on a Debian Wheezy VPS:

$ uname -a
Linux SRV-LX1 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64 GNU/Linux

$ apt-cache show supervisor
Package: supervisor
State: installed
Version: 3.0a8-1.1+deb7u1
...

Any ideas?

Luc

_______________________________________________
Supervisor-users mailing list
[hidden email]
https://lists.supervisord.org/mailman/listinfo/supervisor-users
Reply | Threaded
Open this post in threaded view
|

Re: [Supervisor-users] Thanks for a great tool --- and my first question

Timothy Jones
I believe that supervisor is killing the 'bash' process, not the
underlying 'python manage.py linod' process.

There are two things you can do:
1)  add a 'trap' command to your linod.sh script to catch the TERM signal
that supervisor is second, and in the handler, kill the underlying 'python
manage.py linod' process.
2)  call the 'python manage.py linod' from python directly, taking care of
whatever the linod.sh program is doing elsewhere.

For example, if linod.sh just sets some environment variables before
launching 'python manage.py linod', then set them in the supervisor.conf
file (using 'environment' configuration variable).  Or you can set these
things in the environment that launches supervisor (and supervisor will,
naturally, inherit them, and so will your child process 'python manage.py
linod').

A second guess might be that your script linod.sh isn't "trap"ping SIGTERM
and passing it along to the 'python manage.py linod' child process.


t
[1] http://supervisord.org/configuration.html?highlight=stopsignal
...........................................................................
.............................
Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
+1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
Syniverse | We make mobile work.
[hidden email] | http://www.syniverse.com
<http://www.syniverse.com/>
 




On 7/28/16, 12:49 PM, "[hidden email] on
behalf of Luc Saffre" <[hidden email] on
behalf of [hidden email]> wrote:

>Hi Supervisor users and developers,
>
>this is my first post to this list, so I'd like to first say thanks for
>such a great tool. I discovered it a month ago and start to use it more
>and more on my production servers. I also wrote about it in my own
>documentation: http://www.lino-framework.org/admin/linod.html
>
>But now I discovered my first problem where I hope for your help.
>I have a single configuration file written by myself:
>
>$ cat /etc/supervisor/conf.d/linod_prod.conf
>[program:prod_linod]
>command=/bin/bash /var/www/vhosts/prod/linod.sh
>username = www-data
>
>The mentioned bash script contains:
>
>$ cat /var/www/vhosts/prod/linod.sh
>#!/bin/bash
>set -e  # exit on error
>cd /var/www/vhosts/prod
>. env/bin/activate
>python manage.py linod
>
>The process starts and works well. The problem is that Supervisor does
>not correctly stop the process. Here is a console session:
>
>$ ps aux | grep linod
>1001     15828  0.0  0.0   8344   880 pts/0    S+   18:07   0:00 grep
>linod
>
>$ sudo service supervisor start
>Starting supervisor: supervisord.
>
>$ ps aux | grep linod
>root     16279  0.0  0.0  19720  1668 ?        S    18:31   0:00
>/bin/bash /var/www/vhosts/prod/linod.sh
>root     16284 58.8  0.8 328736 135368 ?       S    18:31   0:02 python
>manage.py linod
>1001     16309  0.0  0.0   8344   884 pts/0    S+   18:31   0:00 grep
>linod
>
>Until here it looks good. I am a bit surprised to see two processes
>here, but that might be caused by some magic.
>
>The real problem is that Supervisor obviously does not see that spawned
>second process and therefore doesn't terminate it. After every restart I
>have one additional process running:
>
>$ sudo service supervisor restart
>Restarting supervisor: supervisord.
>$ ps aux | grep linod
>root     16284  6.0  0.8 328736 135368 ?       S    18:31   0:02 python
>manage.py linod
>root     16322  0.0  0.0  19720  1672 ?        S    18:32   0:00
>/bin/bash /var/www/vhosts/prod/linod.sh
>root     16327  106  0.7 306560 115280 ?       R    18:32   0:02 python
>manage.py linod
>1001     16352  0.0  0.0   8344   884 pts/0    S+   18:32   0:00 grep
>linod
>
>Yes, I am not using the latest version. I have a default configuration
>on a Debian Wheezy VPS:
>
>$ uname -a
>Linux SRV-LX1 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64 GNU/Linux
>
>$ apt-cache show supervisor
>Package: supervisor
>State: installed
>Version: 3.0a8-1.1+deb7u1
>...
>
>Any ideas?
>
>Luc
>
>_______________________________________________
>Supervisor-users mailing list
>[hidden email]
>https://lists.supervisord.org/mailman/listinfo/supervisor-users

_______________________________________________
Supervisor-users mailing list
[hidden email]
https://lists.supervisord.org/mailman/listinfo/supervisor-users
Reply | Threaded
Open this post in threaded view
|

Re: [Supervisor-users] Thanks for a great tool --- and my first question

Paul Lockaby-2
Another option is to call it like this:

    exec python manage.py linode


That has the added benefit of passing any error codes from your python script back to supervisor.

-Paul

> On Jul 28, 2016, at 10:44 AM, Timothy Jones <[hidden email]> wrote:
>
> I believe that supervisor is killing the 'bash' process, not the
> underlying 'python manage.py linod' process.
>
> There are two things you can do:
> 1)  add a 'trap' command to your linod.sh script to catch the TERM signal
> that supervisor is second, and in the handler, kill the underlying 'python
> manage.py linod' process.
> 2)  call the 'python manage.py linod' from python directly, taking care of
> whatever the linod.sh program is doing elsewhere.
>
> For example, if linod.sh just sets some environment variables before
> launching 'python manage.py linod', then set them in the supervisor.conf
> file (using 'environment' configuration variable).  Or you can set these
> things in the environment that launches supervisor (and supervisor will,
> naturally, inherit them, and so will your child process 'python manage.py
> linod').
>
> A second guess might be that your script linod.sh isn't "trap"ping SIGTERM
> and passing it along to the 'python manage.py linod' child process.
>
>
> t
> [1] http://supervisord.org/configuration.html?highlight=stopsignal
> ...........................................................................
> .............................
> Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
> +1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
> Syniverse | We make mobile work.
> [hidden email] | http://www.syniverse.com
> <http://www.syniverse.com/>
>
>
>
>
>
> On 7/28/16, 12:49 PM, "[hidden email] on
> behalf of Luc Saffre" <[hidden email] on
> behalf of [hidden email]> wrote:
>
>> Hi Supervisor users and developers,
>>
>> this is my first post to this list, so I'd like to first say thanks for
>> such a great tool. I discovered it a month ago and start to use it more
>> and more on my production servers. I also wrote about it in my own
>> documentation: http://www.lino-framework.org/admin/linod.html
>>
>> But now I discovered my first problem where I hope for your help.
>> I have a single configuration file written by myself:
>>
>> $ cat /etc/supervisor/conf.d/linod_prod.conf
>> [program:prod_linod]
>> command=/bin/bash /var/www/vhosts/prod/linod.sh
>> username = www-data
>>
>> The mentioned bash script contains:
>>
>> $ cat /var/www/vhosts/prod/linod.sh
>> #!/bin/bash
>> set -e  # exit on error
>> cd /var/www/vhosts/prod
>> . env/bin/activate
>> python manage.py linod
>>
>> The process starts and works well. The problem is that Supervisor does
>> not correctly stop the process. Here is a console session:
>>
>> $ ps aux | grep linod
>> 1001     15828  0.0  0.0   8344   880 pts/0    S+   18:07   0:00 grep
>> linod
>>
>> $ sudo service supervisor start
>> Starting supervisor: supervisord.
>>
>> $ ps aux | grep linod
>> root     16279  0.0  0.0  19720  1668 ?        S    18:31   0:00
>> /bin/bash /var/www/vhosts/prod/linod.sh
>> root     16284 58.8  0.8 328736 135368 ?       S    18:31   0:02 python
>> manage.py linod
>> 1001     16309  0.0  0.0   8344   884 pts/0    S+   18:31   0:00 grep
>> linod
>>
>> Until here it looks good. I am a bit surprised to see two processes
>> here, but that might be caused by some magic.
>>
>> The real problem is that Supervisor obviously does not see that spawned
>> second process and therefore doesn't terminate it. After every restart I
>> have one additional process running:
>>
>> $ sudo service supervisor restart
>> Restarting supervisor: supervisord.
>> $ ps aux | grep linod
>> root     16284  6.0  0.8 328736 135368 ?       S    18:31   0:02 python
>> manage.py linod
>> root     16322  0.0  0.0  19720  1672 ?        S    18:32   0:00
>> /bin/bash /var/www/vhosts/prod/linod.sh
>> root     16327  106  0.7 306560 115280 ?       R    18:32   0:02 python
>> manage.py linod
>> 1001     16352  0.0  0.0   8344   884 pts/0    S+   18:32   0:00 grep
>> linod
>>
>> Yes, I am not using the latest version. I have a default configuration
>> on a Debian Wheezy VPS:
>>
>> $ uname -a
>> Linux SRV-LX1 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64 GNU/Linux
>>
>> $ apt-cache show supervisor
>> Package: supervisor
>> State: installed
>> Version: 3.0a8-1.1+deb7u1
>> ...
>>
>> Any ideas?
>>
>> Luc
>>
>> _______________________________________________
>> Supervisor-users mailing list
>> [hidden email]
>> https://lists.supervisord.org/mailman/listinfo/supervisor-users
>
> _______________________________________________
> Supervisor-users mailing list
> [hidden email]
> https://lists.supervisord.org/mailman/listinfo/supervisor-users

_______________________________________________
Supervisor-users mailing list
[hidden email]
https://lists.supervisord.org/mailman/listinfo/supervisor-users
Reply | Threaded
Open this post in threaded view
|

Re: [Supervisor-users] Thanks for a great tool --- and my first question

Brent Tubbs
You can also use Supervisor's "stopasgroup" config option to have it stop all child processes at the same time as the parent, instead of relying on the parent to stop the children.


On Thu, Jul 28, 2016 at 11:47 AM, Paul Lockaby <[hidden email]> wrote:
Another option is to call it like this:

    exec python manage.py linode


That has the added benefit of passing any error codes from your python script back to supervisor.

-Paul

> On Jul 28, 2016, at 10:44 AM, Timothy Jones <[hidden email]> wrote:
>
> I believe that supervisor is killing the 'bash' process, not the
> underlying 'python manage.py linod' process.
>
> There are two things you can do:
> 1)  add a 'trap' command to your linod.sh script to catch the TERM signal
> that supervisor is second, and in the handler, kill the underlying 'python
> manage.py linod' process.
> 2)  call the 'python manage.py linod' from python directly, taking care of
> whatever the linod.sh program is doing elsewhere.
>
> For example, if linod.sh just sets some environment variables before
> launching 'python manage.py linod', then set them in the supervisor.conf
> file (using 'environment' configuration variable).  Or you can set these
> things in the environment that launches supervisor (and supervisor will,
> naturally, inherit them, and so will your child process 'python manage.py
> linod').
>
> A second guess might be that your script linod.sh isn't "trap"ping SIGTERM
> and passing it along to the 'python manage.py linod' child process.
>
>
> t
> [1] http://supervisord.org/configuration.html?highlight=stopsignal
> ...........................................................................
> .............................
> Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
> <a href="tel:%2B1%20%28813%29%20454.8643" value="+18134548643">+1 (813) 454.8643 work mobile | <a href="tel:%28813%29%20900.8131" value="+18139008131">(813) 900.8131 personal mobile
> Syniverse | We make mobile work.
> [hidden email] | http://www.syniverse.com
> <http://www.syniverse.com/>
>
>
>
>
>
> On 7/28/16, 12:49 PM, "[hidden email] on
> behalf of Luc Saffre" <[hidden email] on
> behalf of [hidden email]> wrote:
>
>> Hi Supervisor users and developers,
>>
>> this is my first post to this list, so I'd like to first say thanks for
>> such a great tool. I discovered it a month ago and start to use it more
>> and more on my production servers. I also wrote about it in my own
>> documentation: http://www.lino-framework.org/admin/linod.html
>>
>> But now I discovered my first problem where I hope for your help.
>> I have a single configuration file written by myself:
>>
>> $ cat /etc/supervisor/conf.d/linod_prod.conf
>> [program:prod_linod]
>> command=/bin/bash /var/www/vhosts/prod/linod.sh
>> username = www-data
>>
>> The mentioned bash script contains:
>>
>> $ cat /var/www/vhosts/prod/linod.sh
>> #!/bin/bash
>> set -e  # exit on error
>> cd /var/www/vhosts/prod
>> . env/bin/activate
>> python manage.py linod
>>
>> The process starts and works well. The problem is that Supervisor does
>> not correctly stop the process. Here is a console session:
>>
>> $ ps aux | grep linod
>> 1001     15828  0.0  0.0   8344   880 pts/0    S+   18:07   0:00 grep
>> linod
>>
>> $ sudo service supervisor start
>> Starting supervisor: supervisord.
>>
>> $ ps aux | grep linod
>> root     16279  0.0  0.0  19720  1668 ?        S    18:31   0:00
>> /bin/bash /var/www/vhosts/prod/linod.sh
>> root     16284 58.8  0.8 328736 135368 ?       S    18:31   0:02 python
>> manage.py linod
>> 1001     16309  0.0  0.0   8344   884 pts/0    S+   18:31   0:00 grep
>> linod
>>
>> Until here it looks good. I am a bit surprised to see two processes
>> here, but that might be caused by some magic.
>>
>> The real problem is that Supervisor obviously does not see that spawned
>> second process and therefore doesn't terminate it. After every restart I
>> have one additional process running:
>>
>> $ sudo service supervisor restart
>> Restarting supervisor: supervisord.
>> $ ps aux | grep linod
>> root     16284  6.0  0.8 328736 135368 ?       S    18:31   0:02 python
>> manage.py linod
>> root     16322  0.0  0.0  19720  1672 ?        S    18:32   0:00
>> /bin/bash /var/www/vhosts/prod/linod.sh
>> root     16327  106  0.7 306560 115280 ?       R    18:32   0:02 python
>> manage.py linod
>> 1001     16352  0.0  0.0   8344   884 pts/0    S+   18:32   0:00 grep
>> linod
>>
>> Yes, I am not using the latest version. I have a default configuration
>> on a Debian Wheezy VPS:
>>
>> $ uname -a
>> Linux SRV-LX1 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64 GNU/Linux
>>
>> $ apt-cache show supervisor
>> Package: supervisor
>> State: installed
>> Version: 3.0a8-1.1+deb7u1
>> ...
>>
>> Any ideas?
>>
>> Luc
>>
>> _______________________________________________
>> Supervisor-users mailing list
>> [hidden email]
>> https://lists.supervisord.org/mailman/listinfo/supervisor-users
>
> _______________________________________________
> Supervisor-users mailing list
> [hidden email]
> https://lists.supervisord.org/mailman/listinfo/supervisor-users

_______________________________________________
Supervisor-users mailing list
[hidden email]
https://lists.supervisord.org/mailman/listinfo/supervisor-users


_______________________________________________
Supervisor-users mailing list
[hidden email]
https://lists.supervisord.org/mailman/listinfo/supervisor-users
Reply | Threaded
Open this post in threaded view
|

Re: [Supervisor-users] Thanks for a great tool --- and my first question

Timothy Jones
In reply to this post by Paul Lockaby-2
Thanks, Paul, that's better (and simpler!) than my idea.


t
...........................................................................
.............................
Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
+1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
Syniverse | We make mobile work.
[hidden email] | http://www.syniverse.com
<http://www.syniverse.com/>
 




On 7/28/16, 1:47 PM, "Paul Lockaby" <[hidden email]> wrote:

>Another option is to call it like this:
>
>    exec python manage.py linode
>
>
>That has the added benefit of passing any error codes from your python
>script back to supervisor.
>
>-Paul
>
>> On Jul 28, 2016, at 10:44 AM, Timothy Jones
>><[hidden email]> wrote:
>>
>> I believe that supervisor is killing the 'bash' process, not the
>> underlying 'python manage.py linod' process.
>>
>> There are two things you can do:
>> 1)  add a 'trap' command to your linod.sh script to catch the TERM
>>signal
>> that supervisor is second, and in the handler, kill the underlying
>>'python
>> manage.py linod' process.
>> 2)  call the 'python manage.py linod' from python directly, taking care
>>of
>> whatever the linod.sh program is doing elsewhere.
>>
>> For example, if linod.sh just sets some environment variables before
>> launching 'python manage.py linod', then set them in the supervisor.conf
>> file (using 'environment' configuration variable).  Or you can set these
>> things in the environment that launches supervisor (and supervisor will,
>> naturally, inherit them, and so will your child process 'python
>>manage.py
>> linod').
>>
>> A second guess might be that your script linod.sh isn't "trap"ping
>>SIGTERM
>> and passing it along to the 'python manage.py linod' child process.
>>
>>
>> t
>> [1] http://supervisord.org/configuration.html?highlight=stopsignal
>>
>>.........................................................................
>>..
>> .............................
>> Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
>> +1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
>> Syniverse | We make mobile work.
>> [hidden email] | http://www.syniverse.com
>> <http://www.syniverse.com/>
>>
>>
>>
>>
>>
>> On 7/28/16, 12:49 PM, "[hidden email] on
>> behalf of Luc Saffre" <[hidden email] on
>> behalf of [hidden email]> wrote:
>>
>>> Hi Supervisor users and developers,
>>>
>>> this is my first post to this list, so I'd like to first say thanks for
>>> such a great tool. I discovered it a month ago and start to use it more
>>> and more on my production servers. I also wrote about it in my own
>>> documentation: http://www.lino-framework.org/admin/linod.html
>>>
>>> But now I discovered my first problem where I hope for your help.
>>> I have a single configuration file written by myself:
>>>
>>> $ cat /etc/supervisor/conf.d/linod_prod.conf
>>> [program:prod_linod]
>>> command=/bin/bash /var/www/vhosts/prod/linod.sh
>>> username = www-data
>>>
>>> The mentioned bash script contains:
>>>
>>> $ cat /var/www/vhosts/prod/linod.sh
>>> #!/bin/bash
>>> set -e  # exit on error
>>> cd /var/www/vhosts/prod
>>> . env/bin/activate
>>> python manage.py linod
>>>
>>> The process starts and works well. The problem is that Supervisor does
>>> not correctly stop the process. Here is a console session:
>>>
>>> $ ps aux | grep linod
>>> 1001     15828  0.0  0.0   8344   880 pts/0    S+   18:07   0:00 grep
>>> linod
>>>
>>> $ sudo service supervisor start
>>> Starting supervisor: supervisord.
>>>
>>> $ ps aux | grep linod
>>> root     16279  0.0  0.0  19720  1668 ?        S    18:31   0:00
>>> /bin/bash /var/www/vhosts/prod/linod.sh
>>> root     16284 58.8  0.8 328736 135368 ?       S    18:31   0:02 python
>>> manage.py linod
>>> 1001     16309  0.0  0.0   8344   884 pts/0    S+   18:31   0:00 grep
>>> linod
>>>
>>> Until here it looks good. I am a bit surprised to see two processes
>>> here, but that might be caused by some magic.
>>>
>>> The real problem is that Supervisor obviously does not see that spawned
>>> second process and therefore doesn't terminate it. After every restart
>>>I
>>> have one additional process running:
>>>
>>> $ sudo service supervisor restart
>>> Restarting supervisor: supervisord.
>>> $ ps aux | grep linod
>>> root     16284  6.0  0.8 328736 135368 ?       S    18:31   0:02 python
>>> manage.py linod
>>> root     16322  0.0  0.0  19720  1672 ?        S    18:32   0:00
>>> /bin/bash /var/www/vhosts/prod/linod.sh
>>> root     16327  106  0.7 306560 115280 ?       R    18:32   0:02 python
>>> manage.py linod
>>> 1001     16352  0.0  0.0   8344   884 pts/0    S+   18:32   0:00 grep
>>> linod
>>>
>>> Yes, I am not using the latest version. I have a default configuration
>>> on a Debian Wheezy VPS:
>>>
>>> $ uname -a
>>> Linux SRV-LX1 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64 GNU/Linux
>>>
>>> $ apt-cache show supervisor
>>> Package: supervisor
>>> State: installed
>>> Version: 3.0a8-1.1+deb7u1
>>> ...
>>>
>>> Any ideas?
>>>
>>> Luc
>>>
>>> _______________________________________________
>>> Supervisor-users mailing list
>>> [hidden email]
>>> https://lists.supervisord.org/mailman/listinfo/supervisor-users
>>
>> _______________________________________________
>> Supervisor-users mailing list
>> [hidden email]
>> https://lists.supervisord.org/mailman/listinfo/supervisor-users
>

_______________________________________________
Supervisor-users mailing list
[hidden email]
https://lists.supervisord.org/mailman/listinfo/supervisor-users
Reply | Threaded
Open this post in threaded view
|

Re: [Supervisor-users] Thanks for a great tool --- and my first question

Timothy Jones
In reply to this post by Luc Saffre
Also, I noticed your child process is still running as 'root', not
'www-data' as intended.

This is probably because you have 'username=www-data' in your
supervisor.conf file instead of 'user=www-data'.
 
http://supervisord.org/configuration.html?highlight=stopsignal#program-x-se
ction-settings (search for 'user').

'userNAME' is for authentication for the webserver portion of supervisor.


...........................................................................
.............................
Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
+1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
Syniverse | We make mobile work.
[hidden email] | http://www.syniverse.com
<http://www.syniverse.com/>
 




On 7/28/16, 12:49 PM, "[hidden email] on
behalf of Luc Saffre" <[hidden email] on
behalf of [hidden email]> wrote:

>Hi Supervisor users and developers,
>
>this is my first post to this list, so I'd like to first say thanks for
>such a great tool. I discovered it a month ago and start to use it more
>and more on my production servers. I also wrote about it in my own
>documentation: http://www.lino-framework.org/admin/linod.html
>
>But now I discovered my first problem where I hope for your help.
>I have a single configuration file written by myself:
>
>$ cat /etc/supervisor/conf.d/linod_prod.conf
>[program:prod_linod]
>command=/bin/bash /var/www/vhosts/prod/linod.sh
>username = www-data
>
>The mentioned bash script contains:
>
>$ cat /var/www/vhosts/prod/linod.sh
>#!/bin/bash
>set -e  # exit on error
>cd /var/www/vhosts/prod
>. env/bin/activate
>python manage.py linod
>
>The process starts and works well. The problem is that Supervisor does
>not correctly stop the process. Here is a console session:
>
>$ ps aux | grep linod
>1001     15828  0.0  0.0   8344   880 pts/0    S+   18:07   0:00 grep
>linod
>
>$ sudo service supervisor start
>Starting supervisor: supervisord.
>
>$ ps aux | grep linod
>root     16279  0.0  0.0  19720  1668 ?        S    18:31   0:00
>/bin/bash /var/www/vhosts/prod/linod.sh
>root     16284 58.8  0.8 328736 135368 ?       S    18:31   0:02 python
>manage.py linod
>1001     16309  0.0  0.0   8344   884 pts/0    S+   18:31   0:00 grep
>linod
>
>Until here it looks good. I am a bit surprised to see two processes
>here, but that might be caused by some magic.
>
>The real problem is that Supervisor obviously does not see that spawned
>second process and therefore doesn't terminate it. After every restart I
>have one additional process running:
>
>$ sudo service supervisor restart
>Restarting supervisor: supervisord.
>$ ps aux | grep linod
>root     16284  6.0  0.8 328736 135368 ?       S    18:31   0:02 python
>manage.py linod
>root     16322  0.0  0.0  19720  1672 ?        S    18:32   0:00
>/bin/bash /var/www/vhosts/prod/linod.sh
>root     16327  106  0.7 306560 115280 ?       R    18:32   0:02 python
>manage.py linod
>1001     16352  0.0  0.0   8344   884 pts/0    S+   18:32   0:00 grep
>linod
>
>Yes, I am not using the latest version. I have a default configuration
>on a Debian Wheezy VPS:
>
>$ uname -a
>Linux SRV-LX1 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64 GNU/Linux
>
>$ apt-cache show supervisor
>Package: supervisor
>State: installed
>Version: 3.0a8-1.1+deb7u1
>...
>
>Any ideas?
>
>Luc
>
>_______________________________________________
>Supervisor-users mailing list
>[hidden email]
>https://lists.supervisord.org/mailman/listinfo/supervisor-users

_______________________________________________
Supervisor-users mailing list
[hidden email]
https://lists.supervisord.org/mailman/listinfo/supervisor-users
Reply | Threaded
Open this post in threaded view
|

Re: [Supervisor-users] Thanks for a great tool --- and my first question

Luc Saffre
Thanks to everybody! Paul's suggestion of "exec" helped, and also that
username problem are now solved!
I wrote about all this in my blog:
http://luc.lino-framework.org/blog/2016/0728.html#supervisor-failed-to-terminate-linod

Luc

On 28/07/16 21:22, Timothy Jones wrote:

> Also, I noticed your child process is still running as 'root', not
> 'www-data' as intended.
>
> This is probably because you have 'username=www-data' in your
> supervisor.conf file instead of 'user=www-data'.
>  
> http://supervisord.org/configuration.html?highlight=stopsignal#program-x-se
> ction-settings (search for 'user').
>
> 'userNAME' is for authentication for the webserver portion of supervisor.
>
>
> ...........................................................................
> .............................
> Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
> +1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
> Syniverse | We make mobile work.
> [hidden email] | http://www.syniverse.com
> <http://www.syniverse.com/>
>  
>
>
>
>
> On 7/28/16, 12:49 PM, "[hidden email] on
> behalf of Luc Saffre" <[hidden email] on
> behalf of [hidden email]> wrote:
>
>> Hi Supervisor users and developers,
>>
>> this is my first post to this list, so I'd like to first say thanks for
>> such a great tool. I discovered it a month ago and start to use it more
>> and more on my production servers. I also wrote about it in my own
>> documentation: http://www.lino-framework.org/admin/linod.html
>>
>> But now I discovered my first problem where I hope for your help.
>> I have a single configuration file written by myself:
>>
>> $ cat /etc/supervisor/conf.d/linod_prod.conf
>> [program:prod_linod]
>> command=/bin/bash /var/www/vhosts/prod/linod.sh
>> username = www-data
>>
>> The mentioned bash script contains:
>>
>> $ cat /var/www/vhosts/prod/linod.sh
>> #!/bin/bash
>> set -e  # exit on error
>> cd /var/www/vhosts/prod
>> . env/bin/activate
>> python manage.py linod
>>
>> The process starts and works well. The problem is that Supervisor does
>> not correctly stop the process. Here is a console session:
>>
>> $ ps aux | grep linod
>> 1001     15828  0.0  0.0   8344   880 pts/0    S+   18:07   0:00 grep
>> linod
>>
>> $ sudo service supervisor start
>> Starting supervisor: supervisord.
>>
>> $ ps aux | grep linod
>> root     16279  0.0  0.0  19720  1668 ?        S    18:31   0:00
>> /bin/bash /var/www/vhosts/prod/linod.sh
>> root     16284 58.8  0.8 328736 135368 ?       S    18:31   0:02 python
>> manage.py linod
>> 1001     16309  0.0  0.0   8344   884 pts/0    S+   18:31   0:00 grep
>> linod
>>
>> Until here it looks good. I am a bit surprised to see two processes
>> here, but that might be caused by some magic.
>>
>> The real problem is that Supervisor obviously does not see that spawned
>> second process and therefore doesn't terminate it. After every restart I
>> have one additional process running:
>>
>> $ sudo service supervisor restart
>> Restarting supervisor: supervisord.
>> $ ps aux | grep linod
>> root     16284  6.0  0.8 328736 135368 ?       S    18:31   0:02 python
>> manage.py linod
>> root     16322  0.0  0.0  19720  1672 ?        S    18:32   0:00
>> /bin/bash /var/www/vhosts/prod/linod.sh
>> root     16327  106  0.7 306560 115280 ?       R    18:32   0:02 python
>> manage.py linod
>> 1001     16352  0.0  0.0   8344   884 pts/0    S+   18:32   0:00 grep
>> linod
>>
>> Yes, I am not using the latest version. I have a default configuration
>> on a Debian Wheezy VPS:
>>
>> $ uname -a
>> Linux SRV-LX1 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64 GNU/Linux
>>
>> $ apt-cache show supervisor
>> Package: supervisor
>> State: installed
>> Version: 3.0a8-1.1+deb7u1
>> ...
>>
>> Any ideas?
>>
>> Luc
>>
>> _______________________________________________
>> Supervisor-users mailing list
>> [hidden email]
>> https://lists.supervisord.org/mailman/listinfo/supervisor-users
>

_______________________________________________
Supervisor-users mailing list
[hidden email]
https://lists.supervisord.org/mailman/listinfo/supervisor-users
Reply | Threaded
Open this post in threaded view
|

Re: [Supervisor-users] Thanks for a great tool --- and my first question

Paul Lockaby-2
I feel compelled to add a bit of explanation for why it worked, based on what you wrote on your blog post.

Timothy was correct in the cause of your problem. Your bash script would call python so you'd see a process tree like this:

supervisor
\-> bash
    \-> python

When supervisor stops bash (with a TERM unless you've specified something different), bash doesn't send the signal to the programs that it is waiting for. This is pretty well explained here:

http://unix.stackexchange.com/questions/146756/forward-sigterm-to-child-in-bash


When you use "exec", your bash script completely replaces itself with the program that you exec. So your process starts like this:

supervisor
\-> bash

But turns into this after the exec call:

supervisor
\-> python

This does mean that after calling exec your bash script is done and disappears so I hope you don't have to do any cleanup after calling exec!

This is exactly the setup we do for our python programs. We have a small bash script that sets up the virtualenv for the program and then it execs python to run that program.

One other thing to be aware of that we ran into is that we were setting PYTHONPATH in our startup script for supervisor because supervisor was running in a non-standard location. This custom PYTHONPATH was being passed to our python programs by supervisor and royally screwing up the environment because we were running supervisor as python2 but our custom programs as python3.

-Paul


> On Jul 28, 2016, at 12:47 PM, Luc Saffre <[hidden email]> wrote:
>
> Thanks to everybody! Paul's suggestion of "exec" helped, and also that
> username problem are now solved!
> I wrote about all this in my blog:
> http://luc.lino-framework.org/blog/2016/0728.html#supervisor-failed-to-terminate-linod
>
> Luc
>
> On 28/07/16 21:22, Timothy Jones wrote:
>> Also, I noticed your child process is still running as 'root', not
>> 'www-data' as intended.
>>
>> This is probably because you have 'username=www-data' in your
>> supervisor.conf file instead of 'user=www-data'.
>>
>> http://supervisord.org/configuration.html?highlight=stopsignal#program-x-se
>> ction-settings (search for 'user').
>>
>> 'userNAME' is for authentication for the webserver portion of supervisor.
>>
>>
>> ...........................................................................
>> .............................
>> Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
>> +1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
>> Syniverse | We make mobile work.
>> [hidden email] | http://www.syniverse.com
>> <http://www.syniverse.com/>
>>
>>
>>
>>
>>
>> On 7/28/16, 12:49 PM, "[hidden email] on
>> behalf of Luc Saffre" <[hidden email] on
>> behalf of [hidden email]> wrote:
>>
>>> Hi Supervisor users and developers,
>>>
>>> this is my first post to this list, so I'd like to first say thanks for
>>> such a great tool. I discovered it a month ago and start to use it more
>>> and more on my production servers. I also wrote about it in my own
>>> documentation: http://www.lino-framework.org/admin/linod.html
>>>
>>> But now I discovered my first problem where I hope for your help.
>>> I have a single configuration file written by myself:
>>>
>>> $ cat /etc/supervisor/conf.d/linod_prod.conf
>>> [program:prod_linod]
>>> command=/bin/bash /var/www/vhosts/prod/linod.sh
>>> username = www-data
>>>
>>> The mentioned bash script contains:
>>>
>>> $ cat /var/www/vhosts/prod/linod.sh
>>> #!/bin/bash
>>> set -e  # exit on error
>>> cd /var/www/vhosts/prod
>>> . env/bin/activate
>>> python manage.py linod
>>>
>>> The process starts and works well. The problem is that Supervisor does
>>> not correctly stop the process. Here is a console session:
>>>
>>> $ ps aux | grep linod
>>> 1001     15828  0.0  0.0   8344   880 pts/0    S+   18:07   0:00 grep
>>> linod
>>>
>>> $ sudo service supervisor start
>>> Starting supervisor: supervisord.
>>>
>>> $ ps aux | grep linod
>>> root     16279  0.0  0.0  19720  1668 ?        S    18:31   0:00
>>> /bin/bash /var/www/vhosts/prod/linod.sh
>>> root     16284 58.8  0.8 328736 135368 ?       S    18:31   0:02 python
>>> manage.py linod
>>> 1001     16309  0.0  0.0   8344   884 pts/0    S+   18:31   0:00 grep
>>> linod
>>>
>>> Until here it looks good. I am a bit surprised to see two processes
>>> here, but that might be caused by some magic.
>>>
>>> The real problem is that Supervisor obviously does not see that spawned
>>> second process and therefore doesn't terminate it. After every restart I
>>> have one additional process running:
>>>
>>> $ sudo service supervisor restart
>>> Restarting supervisor: supervisord.
>>> $ ps aux | grep linod
>>> root     16284  6.0  0.8 328736 135368 ?       S    18:31   0:02 python
>>> manage.py linod
>>> root     16322  0.0  0.0  19720  1672 ?        S    18:32   0:00
>>> /bin/bash /var/www/vhosts/prod/linod.sh
>>> root     16327  106  0.7 306560 115280 ?       R    18:32   0:02 python
>>> manage.py linod
>>> 1001     16352  0.0  0.0   8344   884 pts/0    S+   18:32   0:00 grep
>>> linod
>>>
>>> Yes, I am not using the latest version. I have a default configuration
>>> on a Debian Wheezy VPS:
>>>
>>> $ uname -a
>>> Linux SRV-LX1 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64 GNU/Linux
>>>
>>> $ apt-cache show supervisor
>>> Package: supervisor
>>> State: installed
>>> Version: 3.0a8-1.1+deb7u1
>>> ...
>>>
>>> Any ideas?
>>>
>>> Luc
>>>
>>> _______________________________________________
>>> Supervisor-users mailing list
>>> [hidden email]
>>> https://lists.supervisord.org/mailman/listinfo/supervisor-users
>>
>
> _______________________________________________
> Supervisor-users mailing list
> [hidden email]
> https://lists.supervisord.org/mailman/listinfo/supervisor-users

_______________________________________________
Supervisor-users mailing list
[hidden email]
https://lists.supervisord.org/mailman/listinfo/supervisor-users
Reply | Threaded
Open this post in threaded view
|

Re: [Supervisor-users] Thanks for a great tool --- and my first question

Luc Saffre
Thanks very much for the valuable explanations!

Luc


On 28/07/16 23:06, Paul Lockaby wrote:

> I feel compelled to add a bit of explanation for why it worked, based on what you wrote on your blog post.
>
> Timothy was correct in the cause of your problem. Your bash script would call python so you'd see a process tree like this:
>
> supervisor
> \-> bash
>     \-> python
>
> When supervisor stops bash (with a TERM unless you've specified something different), bash doesn't send the signal to the programs that it is waiting for. This is pretty well explained here:
>
> http://unix.stackexchange.com/questions/146756/forward-sigterm-to-child-in-bash
>
>
> When you use "exec", your bash script completely replaces itself with the program that you exec. So your process starts like this:
>
> supervisor
> \-> bash
>
> But turns into this after the exec call:
>
> supervisor
> \-> python
>
> This does mean that after calling exec your bash script is done and disappears so I hope you don't have to do any cleanup after calling exec!
>
> This is exactly the setup we do for our python programs. We have a small bash script that sets up the virtualenv for the program and then it execs python to run that program.
>
> One other thing to be aware of that we ran into is that we were setting PYTHONPATH in our startup script for supervisor because supervisor was running in a non-standard location. This custom PYTHONPATH was being passed to our python programs by supervisor and royally screwing up the environment because we were running supervisor as python2 but our custom programs as python3.
>
> -Paul
>
>
>> On Jul 28, 2016, at 12:47 PM, Luc Saffre <[hidden email]> wrote:
>>
>> Thanks to everybody! Paul's suggestion of "exec" helped, and also that
>> username problem are now solved!
>> I wrote about all this in my blog:
>> http://luc.lino-framework.org/blog/2016/0728.html#supervisor-failed-to-terminate-linod
>>
>> Luc
>>
>> On 28/07/16 21:22, Timothy Jones wrote:
>>> Also, I noticed your child process is still running as 'root', not
>>> 'www-data' as intended.
>>>
>>> This is probably because you have 'username=www-data' in your
>>> supervisor.conf file instead of 'user=www-data'.
>>>
>>> http://supervisord.org/configuration.html?highlight=stopsignal#program-x-se
>>> ction-settings (search for 'user').
>>>
>>> 'userNAME' is for authentication for the webserver portion of supervisor.
>>>
>>>
>>> ...........................................................................
>>> .............................
>>> Tim Bailey-Jones | Senior Advisory Systems Engineer, Development
>>> +1 (813) 454.8643 work mobile | (813) 900.8131 personal mobile
>>> Syniverse | We make mobile work.
>>> [hidden email] | http://www.syniverse.com
>>> <http://www.syniverse.com/>
>>>
>>>
>>>
>>>
>>>
>>> On 7/28/16, 12:49 PM, "[hidden email] on
>>> behalf of Luc Saffre" <[hidden email] on
>>> behalf of [hidden email]> wrote:
>>>
>>>> Hi Supervisor users and developers,
>>>>
>>>> this is my first post to this list, so I'd like to first say thanks for
>>>> such a great tool. I discovered it a month ago and start to use it more
>>>> and more on my production servers. I also wrote about it in my own
>>>> documentation: http://www.lino-framework.org/admin/linod.html
>>>>
>>>> But now I discovered my first problem where I hope for your help.
>>>> I have a single configuration file written by myself:
>>>>
>>>> $ cat /etc/supervisor/conf.d/linod_prod.conf
>>>> [program:prod_linod]
>>>> command=/bin/bash /var/www/vhosts/prod/linod.sh
>>>> username = www-data
>>>>
>>>> The mentioned bash script contains:
>>>>
>>>> $ cat /var/www/vhosts/prod/linod.sh
>>>> #!/bin/bash
>>>> set -e  # exit on error
>>>> cd /var/www/vhosts/prod
>>>> . env/bin/activate
>>>> python manage.py linod
>>>>
>>>> The process starts and works well. The problem is that Supervisor does
>>>> not correctly stop the process. Here is a console session:
>>>>
>>>> $ ps aux | grep linod
>>>> 1001     15828  0.0  0.0   8344   880 pts/0    S+   18:07   0:00 grep
>>>> linod
>>>>
>>>> $ sudo service supervisor start
>>>> Starting supervisor: supervisord.
>>>>
>>>> $ ps aux | grep linod
>>>> root     16279  0.0  0.0  19720  1668 ?        S    18:31   0:00
>>>> /bin/bash /var/www/vhosts/prod/linod.sh
>>>> root     16284 58.8  0.8 328736 135368 ?       S    18:31   0:02 python
>>>> manage.py linod
>>>> 1001     16309  0.0  0.0   8344   884 pts/0    S+   18:31   0:00 grep
>>>> linod
>>>>
>>>> Until here it looks good. I am a bit surprised to see two processes
>>>> here, but that might be caused by some magic.
>>>>
>>>> The real problem is that Supervisor obviously does not see that spawned
>>>> second process and therefore doesn't terminate it. After every restart I
>>>> have one additional process running:
>>>>
>>>> $ sudo service supervisor restart
>>>> Restarting supervisor: supervisord.
>>>> $ ps aux | grep linod
>>>> root     16284  6.0  0.8 328736 135368 ?       S    18:31   0:02 python
>>>> manage.py linod
>>>> root     16322  0.0  0.0  19720  1672 ?        S    18:32   0:00
>>>> /bin/bash /var/www/vhosts/prod/linod.sh
>>>> root     16327  106  0.7 306560 115280 ?       R    18:32   0:02 python
>>>> manage.py linod
>>>> 1001     16352  0.0  0.0   8344   884 pts/0    S+   18:32   0:00 grep
>>>> linod
>>>>
>>>> Yes, I am not using the latest version. I have a default configuration
>>>> on a Debian Wheezy VPS:
>>>>
>>>> $ uname -a
>>>> Linux SRV-LX1 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1 x86_64 GNU/Linux
>>>>
>>>> $ apt-cache show supervisor
>>>> Package: supervisor
>>>> State: installed
>>>> Version: 3.0a8-1.1+deb7u1
>>>> ...
>>>>
>>>> Any ideas?
>>>>
>>>> Luc
>>>>
>>>> _______________________________________________
>>>> Supervisor-users mailing list
>>>> [hidden email]
>>>> https://lists.supervisord.org/mailman/listinfo/supervisor-users
>> _______________________________________________
>> Supervisor-users mailing list
>> [hidden email]
>> https://lists.supervisord.org/mailman/listinfo/supervisor-users
>

_______________________________________________
Supervisor-users mailing list
[hidden email]
https://lists.supervisord.org/mailman/listinfo/supervisor-users