Using PythonService.exe to host python service while using virtualenv
Thanks very much for posting this question and a solution. I took a slightly different approach which might also be useful. It is pretty difficult to find working tips for Python services, let alone doing it with a virtualenv. Anyway...
Steps
This is using Windows 7 x64, Python 3.5.1 x64, pywin32-220 (or pypiwin32-219).
- Open an Administrator command prompt.
- Create a virtualenv.
C:\Python35\python -m venv myvenv
- Activate the virtualenv.
call myvenv\scripts\activate.bat
- Install pywin32, either:
- From Pypi:
pip install pypiwin32
, - From http://www.lfd.uci.edu/~gohlke/pythonlibs/:
pip install path\to\pywin32.whl
- From Pypi:
- Run the post-install script
python myvenv\Scripts\pywin32_postinstall.py -install
.- This script registers the DLL's in the system, and copies them to
C:\Windows\System32
. The DLL's are namedpythoncom35.dll
andpywintypes35.dll
. So virtual environments on the same machine on the same major Python point release will share these... it's a minor tradeoff :)
- This script registers the DLL's in the system, and copies them to
- Copy
myvenv\Lib\site-packages\win32\pythonservice.exe
tomyvenv\Scripts\pythonservice.exe
- On the service class (whatever subclasses win32serviceutil.ServiceFramework), set the class property
_exe_path_
to point to this relocated exe. This will become the service binPath. For example:_exe_path_ = os.path.join(*[os.environ['VIRTUAL_ENV'], 'Scripts', 'pythonservice.exe'])
.
- On the service class (whatever subclasses win32serviceutil.ServiceFramework), set the class property
Discussion
I think why this works is that Python looks upwards to figure out where the Libs folders are and based on that sets package import paths, similar to the accepted answer. When pythonservice.exe is in the original location, that doesn't seem to work smoothly.
It also resolves DLL linking problems (discoverable with depends.exe from http://www.dependencywalker.com/). Without the DLL business sorted out, it won't be possible to import from the *.pyd files from venv\Lib\site-packages\win32
as modules in your scripts. For example it's needed allow import servicemanager
; as servicemanager.pyd
is not in the package as a .py file, and has some cool Windows Event Log capabilities.
One of the problems I had with the accepted answer is that I couldn't figure out how to get it to accurately pick up on package.egg-link paths that are created when using setup.py develop
. These .egg-link files include the path to the package when it's not located in the virtualenv under myvenv\Lib\site-packages
.
If it all went smoothly, it should be possible to install, start and test the example win32 service (from an Admin prompt in the activated virtualenv):
python venv\Lib\site-packages\win32\Demos\service\pipeTestService.py installpython venv\Lib\site-packages\win32\Demos\service\pipeTestService.py startpython venv\Lib\site-packages\win32\Demos\service\pipeTestServiceClient.py
The Service Environment
Another important note in all this is that the service will execute the python code in a completely separate environment to the one you might run python myservice.py debug
. So for example os.environ['VIRTUAL_ENV']
will be empty when running the service. This can be handled by either:
- Setting environment variables from inside the script, e.g.
- Find current path starting from the sys.executable, as described in the accepted answer.
- Use that path to locate a config file.
- Read the config file and put them in the environment with
os.environ
.
- Add registry keys to the service with the environment variables.
- See Accessing Environment Variables from Windows Services for doing this manually with regedit.exe
- See REG ADD a REG_MULTI_SZ Multi-Line Registry Value for doing this from the command line.
It appears this used to work correctly with the virtualenv
module before virtual environments were added to Python 3.3. There's anecdotal evidence (see this answer: https://stackoverflow.com/a/12424980/1055722) that Python's site.py
used to look upward from the executable file until it found a directory that would satisfy imports. It would then use that for sys.prefix
and this was sufficient for PythonService.exe to find the virtualenv it was inside of and use it.
If that was the behavior, it appears that site.py
no longer does that with the introduction of the venv
module. Instead, it looks one level up for a pyvenv.cfg
file and configures for a virtual environment in that case only. This of course doesn't work for PythonService.exe which is buried down in the pywin32 module under site-packages.
To work around it, I adapted the activate_this.py
code that comes with the original virtualenv
module (see this answer: https://stackoverflow.com/a/33637378/1055722). It is used to bootstrap an interpreter embedded in an executable (which is the case with PythonService.exe) into using a virtualenv. Unfortunately, venv
does not include this.
Here's what worked for me. Note, this assumes the virtual environment is named my-venv and is located one level above the source code location.
import osimport sysif sys.executable.endswith("PythonService.exe"): # Change current working directory from PythonService.exe location to something better. service_directory = os.path.dirname(__file__) source_directory = os.path.abspath(os.path.join(service_directory, "..")) os.chdir(source_directory) sys.path.append(".") # Adapted from virtualenv's activate_this.py # Manually activate a virtual environment inside an already initialized interpreter. old_os_path = os.environ['PATH'] venv_base = os.path.abspath(os.path.join(source_directory, "..", "my-venv")) os.environ['PATH'] = os.path.join(venv_base, "Scripts") + os.pathsep + old_os_path site_packages = os.path.join(venv_base, 'Lib', 'site-packages') prev_sys_path = list(sys.path) import site site.addsitedir(site_packages) sys.real_prefix = sys.prefix sys.prefix = venv_base new_sys_path = [] for item in list(sys.path): if item not in prev_sys_path: new_sys_path.append(item) sys.path.remove(item) sys.path[:0] = new_sys_path
One other factor in my troubles - there is a new pypi wheel for pywin32 that is provided by the Twisted folks that makes it easier to install with pip. The PythonService.exe in that package was acting oddly (couldn't find a pywin32 dll when invoked) compared to the one you get when installing the official win32 exe package into the virtual env using easy_install.
I read all the answers, but no solution can fix my problem.
After carefully researched David K. Hess's code, I made some change, and it finally works.
But my reputation doesn't enough, so I just post the code here.
# 1. Custom your Project's name and Virtual Environment folder's name# 2. Import this before all third part models# 3. If you still failed, check the link below:# https://stackoverflow.com/questions/34696815/using-pythonservice-exe-to-host-python-service-while-using-virtualenv# 2019-05-29 by oraant, modified from David K. Hess's answer.import os, sys, siteproject_name = "PythonService" # Change this for your own project !!!!!!!!!!!!!!venv_folder_name = "venv" # Change this for your own venv path !!!!!!!!!!!!!!if sys.executable.lower().endswith("pythonservice.exe"): # Get root path for the project service_directory = os.path.abspath(os.path.dirname(__file__)) project_directory = service_directory[:service_directory.find(project_name)+len(project_name)] # Get venv path for the project def file_path(x): return os.path.join(project_directory, x) venv_base = file_path(venv_folder_name) venv_scripts = os.path.join(venv_base, "Scripts") venv_packages = os.path.join(venv_base, 'Lib', 'site-packages') # Change current working directory from PythonService.exe location to something better. os.chdir(project_directory) sys.path.append(".") prev_sys_path = list(sys.path) # Manually activate a virtual environment inside an already initialized interpreter. os.environ['PATH'] = venv_scripts + os.pathsep + os.environ['PATH'] site.addsitedir(venv_packages) sys.real_prefix = sys.prefix sys.prefix = venv_base # Move some sys path in front of others new_sys_path = [] for item in list(sys.path): if item not in prev_sys_path: new_sys_path.append(item) sys.path.remove(item) sys.path[:0] = new_sys_path
How to use it? It's simple, just paste it into a new python file, and import it before any third part model like this:
import service_in_venv # import at topimport win32serviceutilimport win32serviceimport win32eventimport servicemanagerimport timeimport sys, os........
And now you should fix your problem.