General Python questions should still go to python-list or tutor or similar resources, such as StackOverflow or the #python IRC channel on Freenode.
The python-ideas mailing list is specifically intended for discussion of new features and language changes. Please don’t be disappointed if your idea isn’t met with universal approval: as the long list of Rejected and Withdrawn PEPs in the PEP Index attests, and as befits a reasonably mature programming language, getting significant changes into Python isn’t a simple task.
If the idea is reasonable, someone will suggest posting it as a feature request on the issue tracker, or, for larger changes, writing it up as a draft PEP.
Sometimes core developers will differ in opinion, or merely be collectively unconvinced. When there isn’t an obvious victor then the Status Quo Wins a Stalemate as outlined in the linked post.
For some examples on language changes that were accepted please read Justifying Python Language Changes.
See also the Changing the Python Language section of this guide.
The Python Mentors program is specifically about encouraging developers and others that would like to contribute to Python development in general, rather than necessarily being focused on one particular issue. Some core developers are also available on the #python-dev IRC channel on Freenode.
Specific problems should be posted to the issue tracker.
The general Python help locations listed above are the best place to start with that kind of question. If they agree it looks like a bug, then the next step is to either post it to the issue tracker or else to ask further on the core development mailing list, python-dev.
First, take some time to consider any comments made in association with the resolution of the tracker issue. On reflection, they may seem more reasonable than they first appeared.
If you still feel the resolution is incorrect, then raise the question on python-dev. If the consensus there supports the disputed resolution, please take any further objections to python-ideas (or some other forum). Further argument on python-dev after a consensus has been reached amongst the core developers is unlikely to win any converts.
You can check their name against the full list of developers with commit rights to the main source control repository.
On the issue tracker, most core developers will have the Python logo appear next to their name.
We try to foster environments of mutual respect, tolerance and encouragement, as described in the PSF’s Diversity Statement. Abiding by the guidelines in this document and asking questions or posting suggestions in the appropriate channels are an excellent way to get started on the mutual respect part, greatly increasing the chances of receiving tolerance and encouragement in return.
The following FAQs are intended for both core developers and contributors.
Mercurial‘s (also known as hg) official web site is at http://mercurial.selenic.com/. A book on Mercurial published by O’Reilly Media, Mercurial: The Definitive Guide, is available for free online. Another resource is Hg Init: a Mercurial tutorial by Joel Spolsky.
With Mercurial installed, you can run the help tool that comes with Mercurial to get help:
hg help
The man page for hg provides a quick refresher on the details of various commands, but doesn’t provide any guidance on overall workflow.
While the main workflow for core developers requires Mercurial, if you just want to generate patches with git diff and post them to the issue tracker, there is a semi-offical read-only git mirror of the main CPython repository. To create a local clone based on this mirror rather than the main repository:
git clone git://github.com/python/cpython
The mirror’s master branch tracks the main repository’s default branch, while the maintenance branch names (2.7, 3.4, etc) are mapped directly.
Please only use this approach if you’re already an experienced Git user and don’t require assistance with the specifics of version control commands. All other parts of this developer’s guide assume the use of Mercurial for local version control.
First, you need to download Mercurial. Most UNIX-based operating systems have binary packages available. Most package management systems also have native Mercurial packages available.
If you have push rights, you need OpenSSH. This is needed to verify your identity when performing commits. As with Mercurial, binary packages are typically available either online or through the platform’s package management system.
Mercurial does not use its own compression via SSH because it is better to enable compression at the SSH level. Enabling SSH compression can make cloning a remote repository much faster. You can configure it in your ~/.ssh/config file; for example:
Host hg.python.org
Compression yes
The recommended option on Windows is to download TortoiseHg which integrates with Windows Explorer and also bundles the command line client (meaning you can type hg in a DOS box). Note that most entries in this FAQ only cover the command line client in detail - refer to the TortoiseHg documentation for assistance with its graphical interface.
If you have push rights, you need to configure Mercurial to work with your SSH keys. For that, open your Mercurial configuration file (you can do so by opening the TortoiseHg Global Settings dialog and then clicking “Edit File”). If there is no [ui] section, create it by typing just that on a line by itself. Then add the following line:
ssh = TortoisePlink.exe -ssh -2 -C -i C:\path\to\yourkey.ppk
where C:\path\to\yourkey.ppk should be replaced with the actual path to your SSH private key.
Note
If your private key is in OpenSSH format, you must first convert it to PuTTY format by loading it into PuTTYgen.
Mercurial is a “distributed” version control system. This means that each participant, even casual contributors, download a complete copy (called a clone, since it is obtained by calling hg clone) of the central repository which can be treated as a stand-alone repository for all purposes. That copy is called in the FAQ the local repository, to differentiate with any remote repository you might also interact with.
But you don’t modify files directly in the local repository; Mercurial doesn’t allow for it. You modify files in what’s called the working copy associated with your local repository: you also run compilations and tests there. Once you are satisfied with your changes, you can commit them; committing records the changes as a new revision in the local repository.
Changes in your local repository don’t get automatically shared with the rest of the world. Mercurial ensures that you have to do so explicitly (this allows you to experiment quite freely with multiple branches of development, all on your private computer). The main commands for doing so are hg pull and hg push.
Typing hg branches displays the open branches in your local repository:
$ hg branches
default 93085:030fda7b1de8
2.7 93060:7ba47bbfe38d
3.4 93082:5fd481150b35 (inactive)
3.3 93079:cda907a02a80 (inactive)
3.2 92975:eac54f7a8018 (inactive)
Assuming you get the following output:
$ hg branches
default 93085:030fda7b1de8
3.4 93082:5fd481150b35 (inactive)
This means all changesets in the “3.4” branch have been merged into the “default” branch (or any other branch, if such exists).
Use:
$ hg branch
default
Or to get more information:
$ hg summary
parent: 68026:f12ef116dd10 tip
In FTP.close() method, make sure to also close the socket object, not only the file.
branch: default
commit: (clean)
update: (current)
Simply use hg update to checkout another branch in the current directory:
$ hg branch
default
$ hg update 3.4
86 files updated, 0 files merged, 11 files removed, 0 files unresolved
$ hg branch
3.4
Adding the -v option to hg update will list all updated files.
Note that, due to some previously built executables being used as a part of the build process, you may sometimes run into issues when attempting to switch between Python 2.x and Python 3.x branches. In these cases, it is best to run a make distclean to ensure that all previously built files are removed.
There are two ways:
If you want to use the second way, you can do:
$ hg clone cpython py34
updating to branch default
3434 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd py34
$ hg update 3.4
86 files updated, 0 files merged, 11 files removed, 0 files unresolved
The current branch in a working copy is “sticky”: if you pull in some new changes, hg update will update to the head of the current branch.
Your local repository is linked by default to the remote repository it was cloned from. If you created it from scratch, however, it is not linked to any remote repository. In .hg/hgrc file for the local repository, add or modify the following section:
[paths]
default = ssh://hg@hg.python.org/devguide
This example is for a local repository that mirrors the devguide repository on hg.python.org. The same approach works for other remote repositories.
Anywhere that <remote repository> is used in the commands in this FAQ, hg will use the default remote repository if you omit the parameter.
In your global .hgrc file add a section similar to the following:
[paths]
dg = ssh://hg@hg.python.org/devguide
This example creates a dg alias for the devguide repository on hg.python.org. This allows “dg” to be entered instead of the full URL for commands taking a repository argument (e.g. hg pull dg or hg outgoing dg).
Anywhere that <remote repository> is used in the commands in this FAQ, hg should accept an alias in place of a complete remote URL.
To display the list of changes that are in your local repository, but not in the remote, use:
hg outgoing <remote repository>
This is the list of changes that will be sent if you call hg push <remote repository>. It does not include any uncommitted changes in your working copy!
Conversely, for the list of changes that are in the remote repository but not in the local, use:
hg incoming <remote repository>
This is the list of changes that will be retrieved if you call hg pull <remote repository>.
Note
In most daily use, you will work against the default remote repository, and therefore simply type hg outgoing and hg incoming.
In this case, you can also get a synthetic summary using hg summary --remote.
Run:
hg pull <remote repository>
from the repository you wish to pull the latest changes into. Most of the time, that repository is a clone of the repository you want to pull from, so you can simply type:
hg pull
This doesn’t update your working copy, though. See below:
Do:
hg update
This will update your working copy with the latest changes on the current branch. If you had uncommitted changes in your working copy, they will be merged in.
If you find yourself typing often hg pull followed by hg update, be aware that you can combine them in a single command:
hg pull -u
If you want to try out or review a patch generated using Mercurial, do:
patch -p1 < somework.patch
This will apply the changes in your working copy without committing them. If the patch was not created by Mercurial (for example, a patch created by Subversion and thus lacking any a/b directory prefixes in the patch), replace -p1 with -p0.
If the patch contains renames, deletions or copies, and you intend committing it after your review, you might prefer using:
hg import --no-commit somework.patch
If you want to work on the patch using mq (Mercurial Queues), type instead:
hg qimport somework.patch
This will create a patch in your queue with a name that matches the filename. You can use the -n argument to specify a different name. To have the patch applied to the working copy, type:
hg qpush
Finally, to delete the patch, first un-apply it if necessary using hg qpop, then do:
hg qdelete somework.patch
The standard patch command, as well as hg import, will produce unhelpful *.rej files when it fails applying parts of a patch. We suggest you try the mpatch utility, which can help resolve a number of common causes of patch rejects.
To make use of mpatch transparent, you can define a shell alias in one of your startup files. For example, if you want it to open the kdiff3 merge program to fix failing patch hunks:
alias patch='mpatch --merge=kdiff3'
or if you want it to automatically solve conflicts by using heuristics:
alias patch='mpatch --auto --no-merge'
Simply specify the path to the file or directory to add and run:
hg add PATH
If PATH is a directory, Mercurial will recursively add any files in that directory and its descendants.
If you want Mercurial to figure out by itself which files should be added and/or removed, just run:
hg addremove
Be careful though, as it might add some files that are not desired in the repository (such as build products, cache files, or other data).
You will then need to run hg commit (as discussed below) to commit the file(s) to your local repository.
To split a file into several files (e.g. a module converted to a package or a long doc file divided in two separate documents) use hg copy:
hg copy module.rst module2.rst
and then remove the parts that are not necessary from module.rst and module2.rst. This allows Mercurial to know that the content of module2.rst used to be in module.rst, and will make subsequent merges easier. If necessary, you can also use hg copy several times.
If you simply create module2.rst, add it with hg add, and copy part of the content from module.rst, Mercurial won’t know that the two file are related.
Specify the path to be removed with:
hg remove PATH
This will remove the file or the directory from your working copy; you will have to commit your changes for the removal to be recorded in your local repository.
Running:
hg status
will list any pending changes in the working copy. These changes will get committed to the local repository if you issue an hg commit without specifying any path.
Some key indicators that can appear in the first column of output are:
A Scheduled to be added R Scheduled to be removed M Modified locally ? Not under version control
If you want a line-by-line listing of the differences, use:
hg diff
Running:
hg revert PATH
will revert PATH to its version in the repository, throwing away any changes you made locally. If you run:
hg revert -a
from the root of your working copy it will recursively restore everything to match up with the repository.
You want:
hg annotate PATH
This will output to stdout every line of the file along with which revision last modified that line. When you have the revision number, it is then easy to display it in detail.
To see the history of changes for a specific file, run:
hg log -v [PATH]
That will list all messages of revisions which modified the file specified in PATH. If PATH is omitted, all revisions are listed.
If you want to display line-by-line differences for each revision as well, add the -p option:
hg log -vp [PATH]
If you want to view the differences for a specific revision, run:
hg log -vp -r <revision number>
In Mercurial repositories, changesets don’t form a simple list, but rather a graph: every changeset has one or two parents (it’s called a merge changeset in the latter case), and can have any number of children.
The graphlog extension is very useful for examining the structure of the changeset graph. It is bundled with Mercurial.
Graphical tools, such as TortoiseHG, will display the changeset graph by default.
Run:
hg tags
to get a list of tags. To update your working copy to a specific tag, use:
hg update <tag>
hg bisect, as the name indicates, helps you do a bisection of a range of changesets.
You need two changesets to start the search: one that is “good” (doesn’t have the bug), and one that is “bad” (has the bug). Usually, you have just noticed the bug in your working copy, so you can start with:
hg bisect --bad
Then you must update to a previous changeset that doesn’t have the bug. You can conveniently choose a faraway changeset (for example a former release), and check that it is indeed “good”. Then type:
hg bisect --good
Mercurial will automatically bisect so as to narrow the range of possible culprits, until a single changeset is isolated. Each time Mercurial presents you with a new changeset, re-compile Python and run the offending test, for example:
make -j2
./python -m test -uall test_sometest
Then, type either hg bisect --good or hg bisect --bad depending on whether the test succeeded or failed.
Mercurial comes with many bundled extensions which can be explicitly enabled. You can get a list of them by typing hg help extensions. Some of these extensions, such as color, can prettify output; others, such as fetch or graphlog, add new Mercurial commands.
There are also many configuration options to tweak various aspects of the command line and other Mercurial behaviour; typing man hgrc displays their documentation inside your terminal.
In the end, please refer to the Mercurial wiki, especially the pages about extensions (including third-party ones) and the tips and tricks.
These FAQs are intended mainly for core developers.
To commit any changes to a file (which includes adding a new file or deleting an existing one), you use the command:
hg commit [PATH]
PATH is optional: if it is omitted, all changes in your working copy will be committed to the local repository. When you commit, be sure that all changes are desired by reviewing them first; also, when making commits that you intend to push to public repositories, you should not commit together unrelated changes.
To abort a commit that you are in the middle of, leave the message empty (i.e., close the text editor without adding any text for the message). Mercurial will then abort the commit operation so that you can try again later.
Once a change is committed to your local repository, it is still only visible by you. This means you are free to experiment with as many local commits you feel like.
Note
If you do not like the default text editor Mercurial uses for entering commit messages, you may specify a different editor, either by changing the EDITOR environment variable or by setting a Mercurial-specific editor in your global .hgrc with the editor option in the [ui] section.
The easiest way is to install KDiff3 — Mercurial will open it automatically in case of conflicts, and you can then use it to solve the conflicts and save the resulting file(s). KDiff3 will also take care of marking the conflicts as resolved.
If you don’t use a merge tool, you can use hg resolve --list to list the conflicting files, resolve the conflicts manually, and the use hg resolve --mark <file path> to mark these conflicts as resolved. You can also use hg resolve -am to mark all the conflicts as resolved.
Note
Mercurial will use KDiff3 automatically if it’s installed and it can find it — you don’t need to change any settings. KDiff3 is also already included in the installer of TortoiseHg. For more information, see http://mercurial.selenic.com/wiki/KDiff3.
If you committed something (e.g. on 3.4) that shouldn’t be ported on newer branches (e.g. on default), you have to do a null merge:
cd 3.x
hg merge 3.4
hg revert -ar default
hg resolve -am # needed only if the merge created conflicts
hg ci -m '#12345: null merge with 3.4.'
Before committing, hg status should list all the merged files as M, but hg diff should produce no output. This will record the merge without actually changing the content of the files.
If you see this message while pushing, it means that you committed something on a clone that was not up to date, thus creating a new head. This usually happens for two reasons:
First of all you should pull the new changesets using hg pull. Then you can use hg heads to see which branches have multiple heads.
If only one branch has multiple heads, you can do:
cd default
hg heads .
hg up csid-of-the-other-head
hg merge
hg ci -m 'Merge heads.'
hg heads . will show you the two heads of the current branch: the one you pulled and the one you created with your commit (you can also specify a branch with hg heads <branch>). While not strictly necessary, it is highly recommended to switch to the other head before merging. This way you will be merging only your changeset with the rest, and in case of conflicts it will be a lot easier.
If more than one branch has multiple heads, you have to repeat these steps for each branch. Since this creates new changesets, you will also have to merge them between branches. For example, if both 3.4 and default have multiple heads, you should first merge heads in 3.4, then merge heads in default, and finally merge 3.4 with default using hg merge 3.4 as usual.
In order to avoid this, you should always remember to pull and update before committing.
First, this should not happen if you take the habit of reviewing changes before committing them.
In any case, run:
hg backout <revision number>
This will modify your working copy so that all changes in <revision number> (including added or deleted files) are undone. You then need to commit these changes so that the backout gets permanently recorded.
Note
These instructions are for Mercurial 1.7 and higher. hg backout has a slightly different behaviour in versions before 1.7.
All generated SSH keys should be sent to hgaccounts@python.org for adding to the list of keys.
Run:
ssh-keygen -t ed25519
This will generate two files; your public key and your private key. The public key is in file ending in .pub.
Use PuTTYgen to generate your public key. Choose the “SSH2 DSA” radio button, have it create an OpenSSH formatted key, choose a password, and save the private key to a file. Copy the section with the public key (using Alt-P) to a file; that file now has your public key.
Use ssh-agent and ssh-add to register your private key with SSH for your current session. The simplest solution, though, is to use KeyChain, which is a shell script that will handle ssh-agent and ssh-add for you once per login instead of per session.
The Pageant program is bundled with TortoiseHg. You can find it in its installation directory (usually C:\Program Files (x86)\TortoiseHg\); you can also download it separately.
Running Pageant will prevent you from having to type your password constantly. If you add a shortcut to Pageant to your Autostart group and edit the shortcut so that the command line includes an argument to your private key then Pageant will load the key every time you log in.
You can make commits from any machine, since they will be recorded in your local repository.
However, to push these changes to the remote server, you will need proper credentials. All you need is to make sure that the machine you want to push changes from has both the public and private keys in the standard place that ssh will look for them (i.e. ~/.ssh on Unix machines). Please note that although the key file ending in .pub contains your user name and machine name in it, that information is not used by the verification process, therefore these key files can be moved to a different computer and used for verification. Please guard your keys and never share your private key with anyone. If you lose the media on which your keys are stored or the machine on which your keys are stored, be sure to report this to pydotorg@python.org at the same time that you change your keys.
If a change is made to Python which relies on some POSIX system-specific functionality (such as using a new system call), it is necessary to update the configure script to test for availability of the functionality.
Python’s configure script is generated from configure.ac using Autoconf. Instead of editing configure, edit configure.ac and then run autoreconf to regenerate configure and a number of other files (such as pyconfig.h.
When submitting a patch with changes made to configure.ac, it is preferred to leave out the generated files as differences between Autoconf versions frequently results in many spurious changes cluttering the patch. Instead, remind any potential reviewers on the tracker to run autoreconf.
Note that running autoreconf is not the same as running autoconf. For example, autoconf by itself will not regenerate pyconfig.h.in. autoreconf runs autoconf and a number of other tools repeatedly as is appropriate.
Python’s configure.ac script typically requires a specific version of Autoconf. At the moment, this reads: version_required(2.65)
If the system copy of Autoconf does not match this version, you will need to install your own copy of Autoconf.
test_gdb attempts to automatically load additional Python specific hooks into gdb in order to test them. Unfortunately, the command line options it uses to do this aren’t always supported correctly.
If test_gdb is being skipped with an “auto-loading has been declined” message, then it is necessary to identify any Python build directories as auto-load safe. One way to achieve this is to add a line like the following to ~/.gdbinit (edit the specific list of paths as appropriate):
add-auto-load-safe-path ~/devel/py3k:~/devel/py32:~/devel/py27