Jason C. McDonaldLike the articles? Buy the book! Dead Simple Python by Jason C. McDonald is available from No Starch Press.
Virtual environments. If you've ever done any meaningful work in Python, you've almost certainly heard of these. Chances are, you've even been told they're non-negotiable. Trouble is, you have no idea what they are, much less how to make them.
For my first several dozen attempts at using virtual environments, I managed to get something horribly wrong. They never worked. I hate to admit it, but I don't even know what I did anymore! Ever since I've learned how virtual environments worked, I haven't had a single problem with them.
Why Should I Care?
A virtual environment, or virtualenv as it is sometimes called, is a sandbox where you can install only the Python packages you need.
"Yeah, and what's a package?"
Well, Python is rather famous for being "batteries included." Most things "just work" with a simple import statement...
But what if you want something more than the built-in packages? For example, you might want to create a snazzy user interface, and you decide to use PySide2. PySide2, like thousands of other third-party libraries, aren't already built into Python - you have to install them.
Thankfully, installing most third-party libraries is easy! The library authors already bundled the whole library up into a package, which can be installed using a handy little Python tool called pip. (We'll get to that later.)
But this is where it gets tricky. Some packages require other packages to be installed first. Certain packages don't like certain other packages. Meanwhile, you can actually install specific versions of a package, depending on what exactly you need.
Did I mention that some of the applications and operating system components on your system rely on those Python packages?
If you're not careful, you'll end up with this mess...
This is why we have virtual environments! We can create a different little sandbox for each project, install only the packages we want in it, and keep everything neatly organized. Bonus, we never actually change what Python packages are installed on our system, so we avoid breaking important things that have nothing to do with our project.
Getting the Tools
Let's install pip and (if needed by your system, venv). Just for the sake of example, we'll also be installing Python 3. If you already have it, reinstalling doesn't hurt; otherwise, you can just skip that part.
Linux
On Linux, your distribution's package repository almost certainly has what you need.
- Debian/Ubuntu:
sudo apt install python3-venv python3-pip - Fedora:
sudo dnf python3-pip
Mac
On Mac, you can use either Macports or Homebrew to install.
Here's Macports, for Python 3.7. If you want 3.6, change all the instances of 37 to 36 below...
sudo port install python37 py37-pip
sudo port select --set python python37
sudo port select --set pip py37-pip
Here's Homebrew...
brew install python
Windows
On Windows, you'll only need to download and install Python. This should also automatically install pip and venv.
However, if you try to run pip in the command line, you should download get-pip.py onto your Desktop, navigate to that directory on your command line, and run it via python get-pip.py.
Whew! With that out of the way, let's get to the fun part...CREATING virtual environments!
Creating a Virtual Environment
Once again, a virtual environment is like a sandbox, containing only the packages you choose, and ignoring (by default) all the Python packages installed elsewhere on your system. Each virtual environment resides in a dedicated directory. Conventionally, we name this folder venv.
For each project, I typically like to create a dedicated virtual environment inside the project folder. (If you're using Git or another VCS, there's an additional setup step we'll get to in a moment.)
To create the virtual environment, we first change our working directory in our command line to our project folder. (Remember, that's the cd command.) Then, we create the virtual environment and its directory in one step.
On UNIX...
python3 -m venv venv
The last part of that command, venv, is the name of the directory you're creating for the virtual environment. Technically, you can call it whatever you want, but like I mentioned before, venv is the convention.
Note, we're explicitly specifying we want to use python3 here, although we can invoke venv with the specific Python executable we want to use (such as python3.6 -m venv venv)
If you look at your working directory, you'll notice that the directory venv/ has been created.
Activating a Virtual Environment
Great, so how do we use this thing?
Actually, it's deliciously simple.
On UNIX-like systems (Mac, Linux, etc.), just run...
source venv/bin/activate
On Windows, run...
venv\Scripts\activate.bat
Like magic, you're now using your virtual environment! On UNIX systems, you'll probably see (venv) at the start of your command line prompt, to indicate that you're using a virtual environment named venv.
Naturally, if you named your virtual environment something else, like bob, you'd need to change the activation command accordingly (source bob/bin/activate).
One of the wonderful things about virtual environments on a system with multiple versions of Python is that you no longer have to specify the path in your commands. While the virtual environment is activated, python whatever_your_command_is.py will use the version of Python you selected when you created the venv. Every time.
Introducing pip
Most of us have great expectations for Python's package system. (See what I did there? No? Sigh.)
pip is pretty easy to use, and much easier than it used to be in the bad-old-days. It used to be so clunky, in fact, that someone felt they needed to create something called easy_install, but pip is now very much painless to use.
Installing Packages
To install a package - say, pyside2, just run...
pip install PySide2
If you want to install a specific version of something, that's easy too.
pip install PySide2==5.11.1
Bonus, you can even use operators like >= ("at least this version, or greater"), and so forth. These are called requirement specifiers. So this...
pip install PySide2>=5.11.1
Would install the latest version of PySide2 that is at least version 5.11.1, or else later. This is really helpful if you want to make sure someone actually has access to a minimum version of a package (they might not).
requirements.txt
You can actually save even more time for yourself and others by writing a requirements.txt file for your project. On each line, list the name of the pip package, exactly as you would type it into the install command.
For example, if you had a requirements.txt file like this...
PySide2>=5.11.1
appdirs
...you could install all those packages in one shot with...
pip install -r requirements.txt
Easy, right?
Upgrading Packages
You can update an already installed package using the pip install command and the --upgrade flag. For example, to install the latest version of PySide2, just run...
pip install --upgrade PySide2
You can also upgrade all your required packages at once with...
pip install --upgrade -r requirements.txt
Removing Packages
Removing things is just as easy.
pip uninstall PySide2
Finding Packages
Great, so we can install, upgrade, and remove things. But how do we know what packages pip even has to offer?
There are two ways. The first is to use pip itself to run a search. Say you want a package for web scraping.
pip search web scraping
That will give you a whole ton of results to sift through, but it's helpful if you simply forgot the name of a package.
If you want something a little more browsable and informative, PyPI.org is the official Python Package Index.
Last Notes
Once you've installed the packages you need for your virtual environment, you're good to go! The next time you start the virtual environment, those packages will still be there, exactly as you left them, waiting for you.
One Warning About pip...
No matter who tells you to, never, ever ever, EVER use sudo pip on a UNIX system. It will do so many bad things to your system installation that your system package manager cannot correct, you will be regretting the decision for the lifetime of your system.
All the problems that sudo pip appears to fix can be solved with virtual environments.
Friends don't let friends sudo pip
Leaving a Virtual Environment
Great, so how do you get out of the virtual environment, and back to reality...er, ahem, the system.
You ready for this, UNIX users?
deactivate
I know! Simple, right?
Of course, things are just slightly more complicated on Windows...
venv\Scripts\deactivate.bat
Eh, still pretty painless. (Remember, like with activation, if you named your virtual environment something else, you'd have to change that line accordingly.)
The Whole She-Bang
So, one last little detail. You may have noticed that most Python files start with something like...
#!/usr/bin/python
First, this is called a "she-bang" (short for haSH-BANG, or #!), and it allows the script to be run without python being tacked onto the beginning of the terminal command.
Second, the above line is very, very wrong. It forces the computer to use a particular system-wide copy of Python, which more or less throws the whole virtual environment thing out the window.
Instead, you should always use the following she-bang for Python3 scripts:
#!/usr/bin/env python3
If you happen to have a script which runs on both Python2 and Python3, use:
#!/usr/bin/env python
(By the way, the rules about python vs. python2, vs. python3 officially come from PEP 394.)
Virtual Environments and Git
Remember that warning earlier, about venv if you're using a VCS like Git?
Within a virtual environment's directory are the actual packages you installed with pip. Those would clutter up your repository with big, unnecessary files, and you can't necessarily copy a virtual environment folder from one computer to another and expect it to work anyway.
Thus, we don't want to track these files in our VCS. In Git, you'll have a file called .gitignore in the root directory of your repository. Create or edit that file, and add this line somewhere in it...
venv/
Naturally, if you used a different name for your virtual environment, you'd change that line to match.
Conventionally, every developer who clones your repository will build their own virtual environment, probably using that requirements.txt file you created.
If you're using a different VCS, like Subversion or Mercurial, check the documentation to see how to ignore a directory like venv.
But...
A lot of Python developers will probably frown deeply at you for putting your virtual environment in the repository folder at all. The main disadvantage to my method above is, if you name your virtual environment anything but venv (or whatever you put in your .gitignore, it's going to get committed, and that's bad.
The best habit is actually to keep your virtual environment out of the repository directory altogether. But, if we're honest, most of us actually don't. It just feels more convenient to do it the "wrong" way.
That's why we add venv to the .gitignore. You, or someone else, is probably going to stick the virtual environment in your repository directory, so this just helps prevent some accidental commits.
A Few Extra Tricks
Some of my Python developer friends on Freenode IRC, as well as folks in the comments, pointed out a few extra tricks that will be helpful to virtual environment users.
Before Python 3.3
The venv command works only if you're using Python 3.3 or later. Before that, you'll need a package from pip called virtualenv. See the virtualenv documentation if you need to use that.
If you're on a system with both Python 2 and Python 3, be sure you use python3 -m venv or whatever is appropriate. This trick doesn't work on any version of Python before 3.6.
Using a Virtual Environment Without Activating
You can also use the binaries that are part of the virtual environment without actually activating it. For example, you can execute venv/bin/python to run the virtualenv's own Python instance, or venv/bin/pip to run its instance of pip. It'll actually work the same as if you had activated the virtual environment!
For example, I could do this (assuming my virtual environment is venv)...
venv/bin/pip install pylint
venv/bin/python
>>> import pylint
...and it works! Yet, import pylint will NOT work on the system-wide Python shell still...unless, of course, you installed it on the system. ;)
The Alternative
I've heard a lot of suggestions for using pipenv, including in the comments section. I won't be covering it in this article, but it has a neat looking workflow, and many vocal fans. You can find out more here: pipenv on PyPI
Wrapping Up
I hope this guide has completely demystified virtual environments and pip for you. Naturally, I recommend you keep the documentation under your pillow:
Chris Warrick also has an excellent article that covers some other facets of virtual environments: Python Virtual Environments in Five Minutes
Comics courtesy XKCD
Thank you to grym, deniska, and ChrisWarrick (Freenode IRC #python) for suggested revisions.
