PYTHON (do a time machine backup first)
The default Python on OS X shouldn’t be messed with as it’s used by the OS itself. If your default is 2.6.1 then you must have Snow Leopard.
If you just install from the standard 3.1 disk image then you can invoke it using python3.1
from the terminal (you don’t have to do any extra steps for this to work) and you can leave the plain python
as 2.6.1.
❯ python –version (shows mac default)Python 2.7.16
python3 –version (shows python 3 installed version)Python 3.8.2
If you don’t then go ahead and install it with the installer. Go the the python’s official site(https://www.python.org/downloads/), download the latest version
❯ pip list –outdated
Package Version Latest Type
UPDATES NEEDED:::
setuptools 41.2.0 57.0.0 wheelsix 1.15.0 1.16.0 wheelwheel 0.33.1 0.36.2 wheel~ ❯
PIP
❯ pip –versionpip 21.1.2 from /Users/spiffy/Library/Python/3.8/lib/python/site-packages/pip (python 3.8)
The Pip Python Package Manager
https://www.datacamp.com/community/tutorials/pip-python-package-manager
Learn about Pip, a powerful tool that helps you properly manage distribution packages in Python.
Before we begin, let’s do a quick glossary check and determine what a ‘Python package’ REALLY is. Package is a Python module which can contain other modules or recursively, other packages. It is the kind of Python package that you import in your Python code. However, this article is NOT about such packages.
This article is about ‘Distribution Package’, which is a versioned archive file that contains Python modules, packages and other resource files that are used to distribute a particular Release (think of it as versioned projects). The archive file is what you as an end-user download from the internet and install – thus, distribution packages are of major importance to the community to share and distribute their projects. This article aims to help you understand the various available tools that make your life easier for using and maintaining the many Python packages that you shall come across as a data scientist and a programmer in general.
A distribution package is more commonly referred to as ‘package’ or ‘distribution’. However the context is important, so as to not confuse it with ‘import Package’ – which is also commonly called just a ‘package’ or with another kind of distribution, like a Linux distribution or a computer language distribution – which are often referred to with the single term ‘distribution’.
So now that the meaning of ‘package’ is determined in the context of this article, let’s begin….
In this tutorial, you will see less executable code but more commands and concept discussion of the different flavors of technical Python jargon that is used in the Python community related to package management. You will begin at ground zero and see Python installation. Then you will see what Python Package Index is. Then you will discover the various tools that are available for package management, beginning with Distutils – where you will see how to register and upload your package.
Next you will see Setuptools, you will also learn about Easy Install – a package manager bundled with setuptools. You will also be introduced to some old and now abandoned tools such as Distribute and Distutils2. Then you will see Pip. You will also be introduced to two concepts: Python eggs and Wheels. And finally, you will see Conda, which is a package and environment management system that is included in Anaconda – a free and open source distribution of the Python and R programming languages for data science and machine learning related applications.
Be sure to check out our Intro to Python for Data Science course.
Installing Python
Well, the first step would be to actually make sure you have Python installed in your system. Ensure you can run Python from the command line. You can check this and the version of Python installed by typing:
python --version
If you get an error as below:
>>> python --versionTraceback (most recent call last): File "<stdin>", line 1, in <module>NameError: name 'python' is not defined
or this:
python --version---------------------------------------------------------------------------NameError Traceback (most recent call last)<ipython-input-3-a4637bbefc43> in <module>()----> 1 python --versionNameError: name 'python' is not defined
This is because the intended code was to be run in your operating system’s terminal (also called shell or console). When you do this correctly and you do have Python installed, you will get an output like this one: Python 3.6.3
. This determines that you have Python version 3.6.3 installed in your system.
If you do not have Python, please go ahead and install the latest 3.x version. You can use the Hitchhiker’s Guide to Python guide to walk you through the steps.
If you have Python installed but are using an enhanced shell like IPython or the Jupyter notebook, you can preface the command with a !
to see the version of Python you are working with:
!python --versionPython 3.5.2 :: Continuum Analytics, Inc.
Python Package Index
The Python Package Index (abbreviated as PyPI) and also known as the Cheese Shop is the official third-party software repository for Python. It primarily hosts Python packages in the form of archives called ‘sdists’ (source distributions) or precompiled wheels (you will see this later). In a sentence: PyPI is as a giant online repository of modules that are accepted by the Python community.
PyPI lets you submit any number of versions of your distribution to the index. If you alter the metadata for a particular version, you can submit it again and the index will be updated. PyPI holds a record for each (name, version) combination submitted. As an end-user you can search for packages by keywords or by filters against their metadata, and thus behaving as an index. Over 113,000 Python packages can be accessed through PyPI.
Why should you be aware of PyPI? Because it describes distributions packaged with ‘distutils’, as well as package data like distribution files if the package author wishes. It is also where ‘easy_install’ and ‘Pip’ search for available packages, by default (More on this coming up later).
Distutils
Distutils is the standard tool for packaging in Python. It is included in the standard library (Python 2 and Python 3.0 to 3.6). Distutils exposes two commands for submitting package data to PyPI: the register command for submitting metadata to PyPI and the upload command for submitting distribution files.
Register your package
The distutils command register
is used to submit your distribution’s metadata to the index. You can invoke it with the command:
python setup.py register
You will then be prompted to login or register to be able to submit your distribution package. You may submit any number of versions of your distribution to the index. If you alter the metadata for a particular version, you may submit it again and the index will be updated.
Upload your package
The distutils command upload
pushes the distribution files to PyPI. For more details on the steps and distutils, check out this page.
Setuptools
Setuptools is a package development process library (or tool, just like Distutils) designed to facilitate packaging Python projects. It is a collection of enhancements to the Python distutils and allow you to more easily build and distribute Python distributions, especially ones that have dependencies on other packages.
Why do Setuptools and Disutils coexist?
It is essentially because of the division of responsibility drafted by the Python core team. They reserved the “core standards” and “minimal necessary compilation” parts for themselves – developing the distutils while leaving all the features beyond that (extended compiler/package format/other support) to the 3rd parties. Setuptools is a third party library and is not developed by the core Python team. Thus, not included in the standard Python library.
Building and Distributing Packages
Check out the developer’s guide section to learn more about installing setuptools to build and distribute your packages. In this tutorial, let’s try to concentrate more on tools that let you manage Python packages to make your life easy.
Installing Packages ( hop down to Easy Install to check 1st)
https://pypi.org/project/setuptools/
https://packaging.python.org/tutorials/installing-packages
This section covers the basics of how to install Python packages.
It’s important to note that the term “package” in this context is being used to describe a bundle of software to be installed (i.e. as a synonym for a distribution). It does not to refer to the kind of package that you import in your Python source code (i.e. a container of modules). It is common in the Python community to refer to a distribution using the term “package”. Using the term “distribution” is often not preferred, because it can easily be confused with a Linux distribution, or another larger software distribution like Python itself.
Contents
Requirements for Installing Packages
This section describes the steps to follow before installing other Python packages.
Ensure you can run Python from the command line
Before you go any further, make sure you have Python and that the expected version is available from your command line. You can check this by running:
Unix/macOS
python3 --version
Windows
You should get some output like Python 3.6.3
. If you do not have Python, please install the latest 3.x version from python.org or refer to the Installing Python section of the Hitchhiker’s Guide to Python.
Note
If you’re a newcomer and you get an error like this:
>>>
>>> python --versionTraceback (most recent call last): File "<stdin>", line 1, in <module>NameError: name 'python' is not defined
It’s because this command and other suggested commands in this tutorial are intended to be run in a shell (also called a terminal or console). See the Python for Beginners getting started tutorial for an introduction to using your operating system’s shell and interacting with Python.
Note
If you’re using an enhanced shell like IPython or the Jupyter notebook, you can run system commands like those in this tutorial by prefacing them with a !
character:
In [1]: import sys !{sys.executable} --versionPython 3.6.3
It’s recommended to write {sys.executable}
rather than plain python
in order to ensure that commands are run in the Python installation matching the currently running notebook (which may not be the same Python installation that the python
command refers to).
Note
Due to the way most Linux distributions are handling the Python 3 migration, Linux users using the system Python without creating a virtual environment first should replace the python
command in this tutorial with python3
and the python -m pip
command with python3 -m pip --user
. Do not run any of the commands in this tutorial with sudo
: if you get a permissions error, come back to the section on creating virtual environments, set one up, and then continue with the tutorial as written.
Ensure you can run pip from the command line
Additionally, you’ll need to make sure you have pip available. You can check this by running:
Unix/macOS
python3 -m pip --version
Windows
If you installed Python from source, with an installer from python.org, or via Homebrew you should already have pip. If you’re on Linux and installed using your OS package manager, you may have to install pip separately, see Installing pip/setuptools/wheel with Linux Package Managers.
If pip
isn’t already installed, then first try to bootstrap it from the standard library:
Unix/macOS
python3 -m ensurepip --default-pip
Windows
If that still doesn’t allow you to run python -m pip
:
Securely Download get-pip.py 1
Run
python get-pip.py
. 2 This will install or upgrade pip. Additionally, it will install setuptools and wheel if they’re not installed already.Warning
Be cautious if you’re using a Python install that’s managed by your operating system or another package manager. get-pip.py does not coordinate with those tools, and may leave your system in an inconsistent state. You can use
python get-pip.py --prefix=/usr/local/
to install in/usr/local
which is designed for locally-installed software.
Ensure pip, setuptools, and wheel are up to date
While pip
alone is sufficient to install from pre-built binary archives, up to date copies of the setuptools
and wheel
projects are useful to ensure you can also install from source archives:
python3 -m pip install --upgrade pip setuptools wheel
Optionally, create a virtual environment
See section below for details, but here’s the basic venv 3 command to use on a typical Linux system:
Unix/macOS
python3 -m venv tutorial_envsource tutorial_env/bin/activate
Windows
This will create a new virtual environment in the tutorial_env
subdirectory, and configure the current shell to use it as the default python
environment.
Creating Virtual Environments
Python “Virtual Environments” allow Python packages to be installed in an isolated location for a particular application, rather than being installed globally. If you are looking to safely install global command line tools, see Installing stand alone command line tools.
Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python3.6/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.
Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.
Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.
In all these cases, virtual environments can help you. They have their own installation directories and they don’t share libraries with other virtual environments.
Currently, there are two common tools for creating Python virtual environments:
- venv is available by default in Python 3.3 and later, and installs pip and setuptools into created virtual environments in Python 3.4 and later.
- virtualenv needs to be installed separately, but supports Python 2.7+ and Python 3.3+, and pip, setuptools and wheel are always installed into created virtual environments by default (regardless of Python version).
The basic usage is like so:
Using venv:
Unix/macOS
python3 -m venv <DIR>source <DIR>/bin/activate
Windows
Using virtualenv:
Unix/macOS
python3 -m virtualenv <DIR>source <DIR>/bin/activate
Windows
For more information, see the venv docs or the virtualenv docs.
The use of source under Unix shells ensures that the virtual environment’s variables are set within the current shell, and not in a subprocess (which then disappears, having no useful effect).
In both of the above cases, Windows users should not use the source command, but should rather run the activate script directly from the command shell like so:
<DIR>\Scripts\activate
Managing multiple virtual environments directly can become tedious, so the dependency management tutorial introduces a higher level tool, Pipenv, that automatically manages a separate virtual environment for each project and application that you work on.
Use pip for Installing
pip is the recommended installer. Below, we’ll cover the most common usage scenarios. For more detail, see the pip docs, which includes a complete Reference Guide.
Installing from PyPI
The most common usage of pip is to install from the Python Package Index using a requirement specifier. Generally speaking, a requirement specifier is composed of a project name followed by an optional version specifier. PEP 440 contains a full specification of the currently supported specifiers. Below are some examples.
To install the latest version of “SomeProject”:
Unix/macOS
python3 -m pip install "SomeProject"
Windows
To install a specific version:
Unix/macOS
python3 -m pip install "SomeProject==1.4"
Windows
To install greater than or equal to one version and less than another:
Unix/macOS
python3 -m pip install "SomeProject>=1,<2"
Windows
To install a version that’s “compatible” with a certain version: 4
Unix/macOS
python3 -m pip install "SomeProject~=1.4.2"
Windows
In this case, this means to install any version “==1.4.*” version that’s also “>=1.4.2”.
Source Distributions vs Wheels
pip can install from either Source Distributions (sdist) or Wheels, but if both are present on PyPI, pip will prefer a compatible wheel. You can override pip`s default behavior by e.g. using its –no-binary option.
Wheels are a pre-built distribution format that provides faster installation compared to Source Distributions (sdist), especially when a project contains compiled extensions.
If pip does not find a wheel to install, it will locally build a wheel and cache it for future installs, instead of rebuilding the source distribution in the future.
Upgrading packages
Upgrade an already installed SomeProject
to the latest from PyPI.
Unix/macOS
python3 -m pip install --upgrade SomeProject
Windows
Installing to the User Site
To install packages that are isolated to the current user, use the --user
flag:
Unix/macOS
python3 -m pip install --user SomeProject
Windows
For more information see the User Installs section from the pip docs.
Note that the --user
flag has no effect when inside a virtual environment – all installation commands will affect the virtual environment.
If SomeProject
defines any command-line scripts or console entry points, --user
will cause them to be installed inside the user base’s binary directory, which may or may not already be present in your shell’s PATH
. (Starting in version 10, pip displays a warning when installing any scripts to a directory outside PATH
.) If the scripts are not available in your shell after installation, you’ll need to add the directory to your PATH
:
- On Linux and macOS you can find the user base binary directory by running
python -m site --user-base
and addingbin
to the end. For example, this will typically print~/.local
(with~
expanded to the absolute path to your home directory) so you’ll need to add~/.local/bin
to yourPATH
. You can set yourPATH
permanently by modifying ~/.profile. - On Windows you can find the user base binary directory by running
py -m site --user-site
and replacingsite-packages
withScripts
. For example, this could returnC:\Users\Username\AppData\Roaming\Python36\site-packages
so you would need to set yourPATH
to includeC:\Users\Username\AppData\Roaming\Python36\Scripts
. You can set your userPATH
permanently in the Control Panel. You may need to log out for thePATH
changes to take effect.
Requirements files
Install a list of requirements specified in a Requirements File.
Unix/macOS
python3 -m pip install -r requirements.txt
Windows
Installing from VCS
Install a project from VCS in “editable” mode. For a full breakdown of the syntax, see pip’s section on VCS Support.
Unix/macOS
python3 -m pip install -e git+https://git.repo/some_pkg.git#egg=SomeProject # from gitpython3 -m pip install -e hg+https://hg.repo/some_pkg#egg=SomeProject # from mercurialpython3 -m pip install -e svn+svn://svn.repo/some_pkg/trunk/#egg=SomeProject # from svnpython3 -m pip install -e git+https://git.repo/some_pkg.git@feature#egg=SomeProject # from a branch
Windows
Installing from other Indexes
Install from an alternate index
Unix/macOS
python3 -m pip install --index-url http://my.package.repo/simple/ SomeProject
Windows
Search an additional index during install, in addition to PyPI
Unix/macOS
python3 -m pip install --extra-index-url http://my.package.repo/simple SomeProject
Windows
Installing from a local src tree
Installing from local src in Development Mode, i.e. in such a way that the project appears to be installed, but yet is still editable from the src tree.
Unix/macOS
python3 -m pip install -e <path>
Windows
You can also install normally from src
Unix/macOS
python3 -m pip install <path>
Windows
Installing from local archives
Install a particular source archive file.
Unix/macOS
python3 -m pip install ./downloads/SomeProject-1.0.4.tar.gz
Windows
Install from a local directory containing archives (and don’t check PyPI)
Unix/macOS
python3 -m pip install --no-index --find-links=file:///local/dir/ SomeProjectpython3 -m pip install --no-index --find-links=/local/dir/ SomeProjectpython3 -m pip install --no-index --find-links=relative/dir/ SomeProject
Windows
Installing from other sources
To install from other data sources (for example Amazon S3 storage) you can create a helper application that presents the data in a PEP 503 compliant index format, and use the --extra-index-url
flag to direct pip to use that index.
./s3helper --port=7777python -m pip install --extra-index-url http://localhost:7777 SomeProject
Installing Prereleases
Find pre-release and development versions, in addition to stable versions. By default, pip only finds stable versions.
Unix/macOS
python3 -m pip install --pre SomeProject
Windows
Installing Setuptools “Extras”
Install setuptools extras.
Unix/macOS
python3 -m pip install SomePackage[PDF]python3 -m pip install SomePackage[PDF]==3.0python3 -m pip install -e .[PDF]==3.0 # editable project in current directory
Windows
-
“Secure” in this context means using a modern browser or a tool like curl that verifies SSL certificates when downloading from https URLs.
-
Depending on your platform, this may require root or Administrator access. pip is currently considering changing this by making user installs the default behavior.
-
Beginning with Python 3.4,
venv
(a stdlib alternative to virtualenv) will create virtualenv environments withpip
pre-installed, thereby making it an equal alternative to virtualenv. -
The compatible release specifier was accepted in PEP 440 and support was released in setuptools v8.0 and pip v6.0
Easy Install
Windows
Easy Install (easy_install) is a package manager for Python bundled with setuptools. It automatically downloads, builds, installs and manages Python packages for you. For download links and installation instructions for each of the supported platforms, head over to the setuptools PyPI page.
For basic use of easy_install, you only need to supply the filename or URL of a source distribution. Easy Install accepts URLs, filenames, PyPI package names (distutils “distribution” names), and package+version specifiers. It will attempt to locate the latest available version that meets your criteria. When downloading or processing downloaded files, Easy Install recognizes distutils source distribution files with extensions of .tgz, .tar, .tar.gz, .tar.bz2, or .zip. It also handles already-built .egg distributions as well as .win32.exe installers built using distutils.
By default, packages are installed to the running Python installation’s site-packages
directory. site-packages is by default part of the python search path and is the target directory of manually built python packages. Modules installed here can be imported easily afterwards. You can override this using the -s
or --script-dir
option.
Let’s see some commands to download, install, upgrade or even delete a package using easy_install:
-
Install a package by name, searching PyPI for the latest version. This automatically downloads, builds, and installs the package:
>>easy_install PackageName
-
Install or upgrade a package by name and version by finding links on a given download page:
>>easy_install -f URL PackageName
-
Upgrade an already-installed package to the latest version listed on PyPI:
>>easy_install --upgrade PackageName
-
Else to upgrade to a specific version, you can type the package name followed by the required version:
>>easy_install "PackageName==2.0"
-
If you have upgraded a package, but want to revert to a previously installed version, you can use the command:
>>easy_install PackageName==1.3.4
-
To uninstall a package, first run the command:
>>easy_install -m PackageName
This ensures that Python will not search for a package you are planning to delete. After you’ve done this, you can safely delete the .egg files or directories, along with any scripts you wish to remove. If you have replaced a package with another version, then you can just delete the package(s) you don’t need by deleting the PackageName-versioninfo.egg file or directory (found in the installation directory).
Deprecated/abandoned tools
Distribute
Distribute was a fork of setuptools and merged back into setuptools 0.7. It shared the same namespace, so if you had distribute installed, import setuptools would actually import the package distributed with Distribute. You don’t need to use Distribute any more. In fact, the version available on Pypi is just a compatibility layer that installs setuptools.
Distutils2
Distutils2 was an attempt to take the best of Distutils, Setuptools and Distribute and become the standard tool included in Python’s standard library. The idea was that Distutils2 would be distributed for old Python versions, and that Distutils2 would be renamed to packaging for Python 3.3, which would include it in its standard library. These plans did not go as intended and currently Distutils2 is an abandoned project. The latest release was in March 2012, and its Pypi home page has finally been updated to reflect its death with a tl;dr: “keep using setuptools and pip for now, don’t use distutils2”.
Pip
Pip is one of the most famous and widely used package management system to install and manage software packages written in Python and found in Python Package Index (PyPI). Pip is a recursive acronym that can stand for either “Pip Installs Packages” or “Pip Installs Python”. Alternatively, pip stands for “preferred installer program”.
Python 2.7.9 and later (on the python2 series), and Python 3.4 and later include pip (pip3 for Python 3) by default. It is an explicit replacement and indirect successor to easy_install that you saw earlier. Check out Python Packaging User Guide’s page on pip vs easy_install for a detailed discussion.
To ensure you can run pip from the command line, type:
>>pip --version
If pip isn’t installed, you can do so through the system package manager or by invoking cURL (a client-side data transfer tool):
>>curl https://bootstrap.pypa.io/get-pip.py | python
While you are at it, it is a good idea to update pip, setuptools and wheel:
>>python -m pip install --upgrade pip setuptools wheel
While pip alone is sufficient to install from pre-built binary archives (the end file(s) that are ready to be executed. The output is the machine instructions that are loaded into the CPU and executed), updated copies of the setuptools and wheel projects are useful to ensure you can also install from source archives.
Let’s check out some handy commands to use pip:
-
To install the latest version of a package:
>>pip install 'PackageName'
-
To install a specific version, type the package name followed by the required version:
>>pip install 'PackageName==1.4'
-
To upgrade an already installed package to the latest from PyPI:
>>pip install --upgrade PackageName
-
Uninstalling/removing a package is very easy with pip:
>>pip uninstall PackageName
Pip has a feature to manage full lists of packages and corresponding version numbers through a requirements file: requirements.txt
. Typically, this file outlines all the pip packages that that project uses. You can install everything in that file by using:
>>pip install -r requirements.txt
You can read more about the requirement files here.
Packaging formats: Egg and Wheel
There has been a substantial amount of mentions of the two terms: ‘Python egg’ and ‘wheel’. They are both packaging formats that aim to support the use case of needing an install artifact that doesn’t require building or compilation. Building and compilations can be costly in testing and production workflows.
The egg format (.egg) was introduced by setuptools in 2004. It is a logical structure embodying the release of a specific version of a Python project, comprising its code, resources, and metadata. Basically, a .zip folder with metadata. They follow the same concept as .jar file in Java.
A wheel is a ZIP-format archive with a specially formatted filename and the .whl extension. The Wheel format was introduced by PEP 427 in 2012. Earlier, installing a python package using pip or easy_install could require you to compile a bunch of underlying code, making the import longer. Wheels provide the option to pre-compile the code for a target architecture and operating system. Wheels are the new standard of Python distribution and are intended to replace eggs. Support is offered in pip >= 1.4 and setuptools >= 0.8.
You can read the important differences between Wheel and Egg here.
Conda
Conda is an open source package management system and environment management system. It is maintained by Continuum Analytics. Conda can quickly install, run and update packages and their dependencies. It is also an environment manager and can easily create, save, load and switch between environments on your local computer.
And although Conda was created for Python programs, it can package and distribute software for many language. The conda package and environment manager is included in all versions of Anaconda – a free and open source distribution of the Python and R programming languages for data science and machine learning related applications. Additionally, Anaconda still has the useful interaction with pip that allows you to install any additional libraries which are not available in conda.
Hence, it is a good idea to download and work with Anaconda. This page will walk you through Anaconda installation, the whole process is pretty straightforward. Once you successfully have Anaconda installed, if you are working on Windows – head to the Start menu and search for, open Anaconda Prompt. If you are working on the MacOS or Linux, open the terminal window. Then go ahead and type:
>>conda --version
This will verify that conda is installed and running on your system and displays the version number that you have installed. To update conda to the current version, type the following:
>>conda update conda
Conda allows you to to create separate environments containing files, packages and their dependencies that will not interact with other environments. When you begin using conda, you already have a default environment named base. But don’t put programs into your base environment. Rather, create separate environments to keep your programs isolated from each other. You can learn more about creating environment and managing python and python packages in Conda here.
Packaging it all up
You have been introduced to quite a lot of terminologies and tools in this tutorial. Take a break and let it all sink in. This is a general overview of all the tools there are to manage your Python packages, what you finally use depends a lot on the task at hand and the environment you are working on. In the end, there isn’t one package manager that suits everyone and you have to pick your own poison.
Be sure to check out DataCamp’s Intermediate Python for Data Science course, to learn more about Python.
Once upon a time…