I used to create a shell script to start a service if I have to pass arguments to the binary, since it does not support command line arguments in the field ExecStart in service file.

I worked well except that when I tried to restart a web server with systemctl restart XXX.service, it failed to listen on the port as the port was being used by another process. That was wired because that port was used by the web server right before I restarted the service. I checked the port state by using netstat -nalp, found that it was actually being listened by the web server. I did run restart and I was pretty sure the service is down because its status is inactive.

Manually I killed the web server and started the service again, I found that by using systemctl start it created two process.

The process 57699 was created by systemctl while the actual web server 57700 was launched by 57699. If I run systemctl stop or systemctl restart, it simply kills the process 57699, without checking the status of its child process.

I went back to Google and try to find out how to kill the child process as well, apparently systemd supports this by setting up KillMode.

If set to control-group, all remaining processes in the control group of this unit will be killed on unit stop (for services: after the stop command is executed, as configured with ExecStop=).
If set to process, only the main process itself is killed.
If set to mixed, the SIGTERM signal (see below) is sent to the main process while the subsequent SIGKILL signal (see below) is sent to all remaining processes of the unit’s control group.
If set to none, no process is killed. In this case, only the stop command will be executed on unit stop, but no process be killed otherwise. Processes remaining alive after stop are left in their control group and the control group continues to exist after stop unless it is empty.

Defaults to control-group.

Indeed, I had the KillMode set to process, which only kills the main process.

Good to know!