I have been trying to streamline my python development processes over the years. Earlier this year @rossjones introduced me to virtualenvwrapper which made things even simpler. I’m about to go into quite a concentrated period of development so I thought I’d take the opportunity to document what I do to make an easily reproduceable python environment. Please let me know of any errors or suggestions for better ways of doing this as I go along.
Note, I have just noticed this is not working on OS X Lion. Looking now at why. 20110816 17:28 BST
We are going to use pip to install virtualenvwrappper. If you have pip this should be as simple as
sudo pip install virtualenvwrappper
If you don’t have pip please read http://pypi.python.org/pypi/pip.
Assuming we have virtualenvwrapper installed and configured (read here for the nitty gritty) we can now create our virtual environment. My project is to sync some files to Dropbox automatically so I will call it dropbox-sync.
ian@vm:~$ mkvirtualenv dropbox-sync New python executable in dropbox-sync/bin/python2.7 Also creating executable in dropbox-sync/bin/python Installing setuptools............done. Installing pip...............done. etc etc etc etc etc
Now I am in my virtual environment, any packages I install will only be available to that environment. I have the habit of working within a projects directory of my home directory, so …
(dropbox-sync)ian@vm:~$ mkdir ~/projects/dropbox-sync (dropbox-sync)ian@vm:~$ cd ~/projects/dropbox-sync/ (dropbox-sync)ian@vm:~/projects/dropbox-sync$
To deactivate your working environment you can use ‘deactivate’.
(dropbox-sync)ian@vm:~/projects/dropbox-sync$ deactivate ian@vm:~/projects/dropbox-sync$
To reactivate the environment again use the workon command.
ian@vm:~/projects$ workon dropbox-sync (dropbox-sync)ian@vm:~/projects/dropbox-sync$
You can see from this that your virtual environment is not explicitly linked to the directory you’ll be working in. As I virtually always want to go to my working directory at the same time as I enable my working environment I can automate this.
By default the virtualenvwrapper files go into ~/.virtualenvs. Within that directory you’ll see a script called postactivate. This is run straight after an activation of an environment takes place. We can utilise this to change to our new directory. In my case I have modified it to read
[cc lang=”bash”]
#!/bin/bash
# This hook is run after every virtualenv is activated.
set envname=”$(basename $VIRTUAL_ENV)”
if [ ! -d “$HOME/projects/$envname” ]; then
mkdir ~/projects/$envname
fi
cd ~/projects/$envname
[/cc]
The first line of above sets our envname variable to be the name of our current virtual environment directory. I refer to this sometimes as the project name. The script checks if a similar directory exists in projects and if not creates it. Finally it changes to that directory.
Similarly when I deactivate an environment I want to move back to my projects directory (to remind myself I should no longer work in that directory when the virtual environment is not activated), so the script at ~/virtualenvs/postdeactivate reads simply
[cc lang=”bash”]
#!/bin/bash
# This hook is run after every virtualenv is deactivated.
cd ~/projects
[/cc]
If you look in the .virtualenvs directory you’ll see a lot of scripts that can be modified. The documentation explains this more and can be found at http://www.doughellmann.com/docs/virtualenvwrapper/. There are hooks you can also use to customise specific environments if you find you need to do something repeatedly for just certain projects when they are activated.
It may be that you already have a stack of libraries you have installed to your own site packages. Those will be available in your virtual environment. This may not be a bad thing as there may be certain packages you want to use repeatedly and not want to download and install each time. If however you want something clean you can use the “–no-site-packages” option. e.g.
mkvirtualenv --no-site-packages dropbox-sync
In my case, projects are either quick throwaway projects, or larger projects which need to be stored in a mercurial repository for source control and backup. I’m also always using Django (or very nearly always) and there are a number of utilities I can’t do without. I can use the hooks that virtualenvwrapper gives us to allow for these environments to be created very quickly.
Firstly I want any project I setup to be under mercurial control and to allow me to easily to push to the code to a bitbucket repository should I require. The following lines in our global postactivate script (the one we were editing before) do this. Add this below the code we already added.
[cc lang=”bash”]
if [ ! -e “$HOME/projects/$envname/.hgignore” ]; then
cp ~/.virtualenvs/.hgignore .
hg init
reset
echo ********************** IMPORTANT ***************************
echo “[paths]” > .hg/hgrc
echo “default = ssh://hg@bitbucket.org/zobbo/$envname” >> .hg/hgrc
echo if this is a long term project, create a repository on bitbucket
echo and then hg push. Check the value of default in paths in
echo .hg/hgrc – may need to switch from zobbo to bfs
echo ********************** IMPORTANT ***************************
hg add ./.hgignore
hg commit -m “Adding initial .hgignore”
fi
[/cc]
This looks for the presence of a file called .hgignore in our current directory. If it doesn’t exist it copies a template one from our ~/.virtualenvs directory which I have placed there. The contents of this file are currently:
syntax: glob *.pyc *~ bin .installed.cfg .DS_Store (.*/)?\#[^/]*\#$
This is a set of files that I do not wish to have checked into mercurial (e.g. .pyc files).
Having copied our .hgignore file into the project directory I then do a hginit to mark the directory as a mercurial repository. The ‘reset’ then clears the screen so I don’t miss any of the next output. I create a .hg/hgrc file which tells my repository where to push itself remotely (should I choose to do so). I then remind myself that I’ll need to setup bitbucket if I want to do that remote push. Finally I add .hgignore and commit it to my local repository.
The long and short of this is that I have a working project directory which is checked in and ready to go. And where I may have to do something I have reminders on the screen about what to do.
The next thing I wish to do is create a buildout file for this install. I’ll go into buildouts in more detail another time. For this article I’ll just show a very basic example. First of all some more lines are added to our postactivate script.
[cc lang=”bash”]
if [ ! -e “$HOME/projects/$envname/buildout.cfg” ]; then
cp ~/.virtualenvs/buildout.cfg .
hg add ./buildout.cfg
hg commit -m “Adding initial buildout.cfg”
echo
echo Now run pip install zc.buildout
echo
fi
[/cc]
This time we are checking for the presence of a file called buildout.cfg (unsurprisingly, the config file used by buildouts) and once again we copy over a template version of that file from our ~/.virtualenvs directory if it does not exist. We then add the file to mercurial, commit it and then tell ourselves we need to run a pip install command. The contents of the buildout.cfg file will be something like this:
[buildout]
parts = django
eggs = ipython
ipdb
eggs-directory = ../.buildout-eggs
download-cache = ../.buildout-downloads
unzip = true
[versions]
django=1.3
[django]
recipe = djangorecipe
settings = development
eggs = ${buildout:eggs}
extra-paths =
project = project
I don’t want to go into too much detail about this for the moment but just to say you’ll need to create a .buildout-eggs and .buildout-downloads directory in the parent directory of the project directory (in my case ~/projects) if you want to use that part of the recipe. It means that if you have multiple projects you can share the eggs and downloads without downloading them every time.
Now we have all our pieces together I can try a test run from start to finish. Note that if you’ve already created a dropbox-sync project you should get rid of it. That’s easy – something like:
(dropbox-sync)ian@vm:~/projects/dropbox-sync$ deactivate ian@vm:~/projects$ rmvirtualenv dropbox-sync ian@vm:~/projects$ rm -rf dropbox-sync/ ian@vm:~/projects$
I have also decided I want to store my code in a bitbucket repository so I create one there called dropbox-sync. Now I run the command
mkvirtualenv dropbox-sync
Some work is done, the screen is cleared and I see this
********************** IMPORTANT *************************** if this is a long term project, create a repository on bitbucket and then hg push. Check the value of default in paths in .hg/hgrc - may need to switch from zobbo to bfs ********************** IMPORTANT *************************** adding .hgignore .hgignore committed changeset 0:d3e247f59565 adding buildout.cfg buildout.cfg committed changeset 1:c270ecce5c5c Now run pip install zc.buildout
In this case I want to push the repository to bitbbucket. So I do
(dropbox-sync)ian@vm:~/projects/dropbox-sync$ hg push running ssh hg@bitbucket.org "hg -R zobbo/dropbox-sync serve --stdio" pushing to ssh://hg@bitbucket.org/zobbo/dropbox-sync searching for changes 2 changesets found remote: adding changesets remote: adding manifests remote: adding file changes remote: added 2 changesets with 2 changes to 2 files remote: bb/acl: zobbo is allowed. accepted payload.
The files are now on bitbucket. Now time to run the buildout. First do the pip install as our message says
(dropbox-sync)ian@vm:~/projects/dropbox-sync$ pip install zc.buildout
Downloading/unpacking zc.buildout
Downloading zc.buildout-1.5.2.tar.gz (331Kb): 331Kb downloaded
Running setup.py egg_info for package zc.buildout
Requirement already satisfied (use --upgrade to upgrade): setuptools in /home/ian/.virtualenvs/dropbox-sync/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg (from zc.buildout)
Installing collected packages: zc.buildout
Running setup.py install for zc.buildout
Skipping installation of /home/ian/.virtualenvs/dropbox-sync/lib/python2.7/site-packages/zc/__init__.py (namespace package)
Installing /home/ian/.virtualenvs/dropbox-sync/lib/python2.7/site-packages/zc.buildout-1.5.2-py2.7-nspkg.pth
Installing buildout script to /home/ian/.virtualenvs/dropbox-sync/bin
Successfully installed zc.buildout
Cleaning up...
And now run our actual buildout which will give us a working django installation
(dropbox-sync)ian@vm:~/projects/dropbox-sync$ buildout Creating directory '/home/ian/projects/dropbox-sync/bin'. Creating directory '/home/ian/projects/dropbox-sync/parts'. Creating directory '/home/ian/projects/dropbox-sync/develop-eggs'. Not upgrading because not running a local buildout command. Installing django. Generated script '/home/ian/projects/dropbox-sync/bin/django'. (dropbox-sync)ian@vm:~/projects/dropbox-sync$
We now have within our project bin directory a django command, which we can use as per the django-admin.py command. We will call our Django app without our project just syncer.
(dropbox-sync)ian@vm:~/projects/dropbox-sync$ bin/django startapp syncer
We now have a django environment created automatically with the following structure
(dropbox-sync)ian@vm:~/projects/dropbox-sync$ tree .
.
├── bin
│ └── django
├── buildout.cfg
├── develop-eggs
├── parts
└── project
├── development.py
├── development.pyc
├── __init__.py
├── __init__.pyc
├── media
├── production.py
├── settings.py
├── settings.pyc
├── syncer
│ ├── __init__.py
│ ├── models.py
│ ├── tests.py
│ └── views.py
├── templates
└── urls.py
You can see our created django app directory and (if you know Django) all the other usual Django components. We should probably commit this to our repository and push it remotely.
(dropbox-sync)ian@vm:~/projects/dropbox-sync$ hg stat ? project/__init__.py ? project/development.py ? project/production.py ? project/settings.py ? project/syncer/__init__.py ? project/syncer/models.py ? project/syncer/tests.py ? project/syncer/views.py ? project/urls.py
You can see the only items which are flagged as being missing from our repository are the actual items of code we want. So finally.
(dropbox-sync)ian@vm:~/projects/dropbox-sync$ hg add project adding project/__init__.py adding project/development.py adding project/production.py adding project/settings.py adding project/syncer/__init__.py adding project/syncer/models.py adding project/syncer/tests.py adding project/syncer/views.py adding project/urls.py (dropbox-sync)ian@vm:~/projects/dropbox-sync$ hg commit -m "Start of our new app" project/__init__.py project/development.py project/production.py project/settings.py project/syncer/__init__.py project/syncer/models.py project/syncer/tests.py project/syncer/views.py project/urls.py committed changeset 2:e8bd66e10c1a (dropbox-sync)ian@vm:~/projects/dropbox-sync$ hg push running ssh hg@bitbucket.org "hg -R zobbo/dropbox-sync serve --stdio" pushing to ssh://hg@bitbucket.org/zobbo/dropbox-sync searching for changes 1 changesets found remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 9 changes to 9 files remote: bb/acl: zobbo is allowed. accepted payload. (dropbox-sync)ian@vm:~/projects/dropbox-sync$
You can see the results of this on bit bucket here. I now have a fully working Django install (although Django config needs to be done). If you try and run bin/django syncdb you’d be told you haven’t setup your db settings but that’s a standard Django setup item.
This environment is self contained. Assuming you didn’t install zc.buildout outside of the environment, once you deactivate you’ll see the ‘buildout’ command no longer exists. Switch back again (“workon dropbox-sync”) and voila, it’s back again.
Now I have the environment setup I can create new projects very easily and consistently. However, there’s a lot more that can be done and having written this up I can see a lot of holes and places where things could be sweeter.
Note I have added the two virtualenv files we created into the repository at dropbox-sync, they can be found in the virtualenv-files directory