Process for Plone core’s development¶

Dealing with pull requests on GitHub¶

Before we can merge a pull request, we have to check that the author has signed the contributor’s agreement.

If they’re listed in https://github.com/plone?tab=members, the author has signed so we can go ahead and merge.

If they aren’t listed there, there’s still a chance they have signed the contributor’s agreement. Check on IRC #plone-framework.

Pull requests without contributor’s agreement can only be merged in trivial cases, and only by the release manager.

Version Support Policy¶

If you are triaging or fixing bugs, keep in mind that Plone has a version support policy.

Dependencies¶

• Git
• Subversion
• Python 2.6 or 2.7 including development headers.
• If you are on Mac OSX, you will need to install XCode. You can do this through the app store or several other soul-selling methods. You will likely want to install your own python 2.6 as well since they strip out all the header files which makes compiling some extensions weird. You can ignore this advice to start, but have faith, you’ll come back to it later. They always do...
• Python Imaging Library (PIL). Make sure to install this into the proper python environment.
• VirtualEnv in the proper python environment.
• GCC in order to compile ZODB, Zope and lxml.
• libxml2 and libxslt, including development headers.

Setting up Your Development Environment¶

The first step in fixing a bug is getting this buildout running. We recommend fixing the bug on the latest branch and then backporting as necessary. Github by default always points to the currently active branch. More information on switching release branches is below.

To set up a plone 4.2 development environment:

> cd ~/buildouts # or wherever you want to put things
> git clone -b 4.2  https://github.com/plone/buildout.coredev ./plone42devel
> virtualenv --no-site-packages plone42devpy
> cd plone42devel
> ../plone42devpy/bin/python bootstrap.py # (where "python" is your python 2.6 or 2.7 binary).
> bin/buildout -v

If you run into issues in this process, please see the doc Troubleshooting.

This will run for a long time if it is your first pull (~20 mins). Once that is done pulling down eggs, You can start your new instance with:

> ./bin/instance fg

The default username/password for a dev instance is admin/admin.

> git checkout -t origin/4.1

> git checkout 4.1

> git branch

Checking out Packages for Fixing¶

Most packages are not in src/ by default, so you can use mr.developer to get the latest and make sure you are always up to date. It can be a little daunting at first to find out which packages are causing the bug in question, but just ask on irc if you need some help. Once you [think you] know which package(s) you want, we need to pull the source.

You can get the source of the package with mr.developer and the checkout command, or you can go directly to editing checkouts.cfg. We recommend the latter but will describe both. In the end, checkouts.cfg must be configured either way so you might as well start there.

At the base of your buildout, open checkouts.cfg and add your package if it’s not already there:

auto-checkout =
        # my modified packages
        plone.app.caching
        plone.caching
        # others
        ...

Then rerun buildout to get the source packages:

> ./bin/buildout

Altternatively, we can manage checkouts from the command line, by using mr.developer’s bin/develop command to get the latest source. For example, if the issue is in plone.app.caching and plone.caching:

> ./bin/develop co plone.app.caching
> ./bin/develop co plone.caching
> ./bin/buildout

Don’t forget to rerun buildout! In both methods, mr.developer will download the source from github (or otherwise) and put the package in the src/ directory. You can repeat this process with as many or as few packages as you need. For some more tips on working with mr.developer, please read more here.

Testing Locally¶

In an ideal world, you would write a test case for your issue before actually trying to fix it. In reality this rarely happens. No matter how you approach it, you should ALWAYS run test cases for both the module and plone.org before commiting any changes.

If you don’t start with a test case, save yourself potential problems and validate the bug before getting too deep into the issue!

To run a test for the specific module you are modifying:

> ./bin/test -m plone.app.caching

These should all run without error. Please don’t check in anything that doesn’t! If you haven’t written it already, this is a good time to write a test case for the bug you are fixing and make sure everything is running as it should.

After the module level tests run with your change, please make sure other modules aren’t affected by the change by running the full suite:

> ./bin/alltests

Updating CHANGES.rst and checkouts.cfg¶

Once all the tests are running locally on your machine, you are ALMOST ready to commit the changes. A couple housekeeping things before moving on.

First, please edit CHANGES.rst (or CHANGES.txt, or HISTORY.txt) in each package you have modified and add a summary of the change. This change note will be collated for the next Plone release and is important for integrators and developers to be able to see what they will get if they upgrade. New changelog entries should be added at the very top of CHANGES.txt.

Most importantly, if you didn’t do it earlier, edit checkouts.cfg file in the buildout directory and add your changes package to the auto-checkout list. This lets the release manager know that the package has been updated so that when the next release of Plone is cut a new egg will be released and Plone will need to pin to the next version of that package. READ: this is how your fix becomes an egg!

Note that there is a section seperator called “# Test Fixes Only”. Make sure your egg is above that line or your egg probably won’t get made very quickly. This just tells the release manager that any eggs below this line have tests that are updated, but no code changes.

Modifying checkouts.cfg file also triggers the buildbot, jenkins, to pull in the egg and run all the tests against the changes you just made. Not that you would ever skip running all tests of course... More on that below.

If your bug is in more than one release (e.g. 4.1 and 4.2), please checkout both branches and add to the checkouts.cfg file.

Committing and Pull Requests¶

Phew! We are in the home stretch. How about a last minute checklist:

• Did you fix the original bug?
• Is your code consistent with our Style Guide?
• Did you remove any extra code and lingering pdbs?
• Did you write a test case for that bug?
• Are all test cases for the modules(s) and for Plone passing?
• Did you update CHANGES.rst in each packages you touched?
• Did you add your changed packages to checkouts.cfg?

If you answered YES to all of these questions, you are ready to push your changes! A couple quick reminders:

• Only commit directly to the development branch if you’re confident your code won’t break anything badly and the changes are small and fairly trivial. Otherwise, please create a pull request (more on that below).
• Please try to make one change per commit. If you are fixing three bugs, make three commits. That way, it is easier to see what was done when, and easier to roll back any changes if necessary. If you want to make large changes cleaning up whitespace or renaming variables, it is especially important to do so in a separate commit for this reason.
• We have a few angels that follow the changes and each commit to see what happens to their favourite CMS! If you commit something REALLY sketchy, they will politely contact you, most likely after immediately reverting changes. There is no official people assigned to this so if you are especially nervous, jump into #plone and ask for a quick eyeball on your changes.

Committing to Products.CMFPlone¶

If you are working a bug fix on Products.CMFPlone, there are a couple other things to take notice of. First and foremost, you’ll see that there are several branches. At the time of writing this document, there are branches for 4.1, 4.2, and master, which is the implied 4.3.

Still with me? So you have a bug fix for 4.x. If the fix is only for one version, make sure to get that branch and party on. However, chances are the bug is in multiple branches.

Let’s say the bug starts in 4.1. Pull the 4.1 branch and fix and commit there with tests.

If your fix only involved a single commit, you can use git’s cherry-pick command to apply the same commit to a different branch.

First check out the branch:

> git checkout 4.2

And then cherry-pick the commit (you can get the SHA hash from git log).

> git cherry-pick b6ff4309

There may be conflicts; if so, resolve them and then follow the directions git gives you to complete the cherry-pick.

If your fix involved multiple commits, cherry-picking them one by one can get tedious. In this case things are easiest if you did your fix in a separate feature branch.

In that scenario, you first merge the feature branch to the 4.1 branch:

> git checkout 4.1
> git merge my-awesome-feature

Then you return to the feature branch and make a branch for rebasing it onto the 4.2 branch:

> git checkout my-awesome-feature
> git checkout -b my-awesome-feature-4.2
> git rebase ef978a --onto 4.2

(ef978a happens to be the last commit in the feature branch’s history before it was branched off of 4.1. You can look at git log to find this.)

At this point, the feature branch’s history has been updated, but it hasn’t actually been merged to 4.2 yet. This lets you deal with resolving conflicts before you actually merge it to the 4.2 release branch. Let’s do that now:

> git checkout 4.2
> git merge my-awesome-feature-4.2

> cd src/plone.app.caching
> git checkout -b my_descriptive_branch_name

> git push origin my_descriptive_branch_name

Jenkins¶

You STILL aren’t done! Please check jenkins to make sure your changes haven’t borked things. It runs every half an hour and takes a while to run so checking back in an hour is a safe bet. Have a beer and head over to the Jenkins control panel.

Finalizing Tickets¶

If you are working from a ticket, please don’t forget to go back to the ticket and add a link to the changeset. We don’t have integration with github yet so it’s a nice way to track changes. It also lets the reporter know that you care. If the bug is really bad, consider pinging the release manager and asking him to make a release pronto.

FAQ¶

• How do I know when my package got made?

  You can follow the project on github and watch its timeline. You can also check the CHANGES.txt of every plone release for a comprehensive list of all changes and validate that yours is present.

Documentation of Plone¶

As a community, Plone maintains several types of documentation:

• Curated documents. This is a limited set of documentation that is intended to be carefully managed and regularly updated.

  • User Manual
  • Installing Plone
  • Theme Reference
  • Developer Manual

  Improvements to the curated documents can be discussed on the plone-docs mailing list.

• Community-edited documents. These are open for contributions by anyone. This leads to a wealth of information that is of more widely ranging quality.

  • Knowledgebase on plone.org. Anyone with a plone.org account is free to edit.
  • Collective Plone developer documentation. Anyone may contribute.

Documenting a package¶

Naming Conventions¶

Above all else, be consistent with any code your are modifying! Historically the code is all camel case, but many new libraries are in the PEP8 convention. The mailing list is exploding with debate over what is better so we’ll leave the excersize of deciding what to do with the user.

File Conventions¶

In Zope 2, file names used to be MixedCase. In Python, and thus in Plone going forward, we prefer all-lowercase filenames. This has the advantage that you can instantly see if you refer to a module / file or a class:

from zope.pagetemplate.pagetemplate import PageTemplate

compare that to:

from Products.PageTemplates.PageTemplate import PageTemplatePageTemplate

Filenames should be short and descriptive. Think about how an import would read:

from Products.CMFPlone.utils import safe_hasattr

compare that to:

from Products.CMFPlone.PloneUtilities import safe_hasattr

The former is obviously much easier to read, less redundant and generally more aesthetically pleasing.

Note This example is just about as terrible as they come. We need a better one.

Concrete Rules¶

• Do not use tabs in Python code! Use spaces as indenting, 4 spaces for each level. We don’t “require” PEP8, but most people use it and it’s good for you.

• Indent properly, even in HTML.

• Never use a bare except. Anything like ‘except: pass’ will likely be reverted instantly

• Avoid tal:on-error, since this swallows exceptions

• Don’t use hasattr() - this swallows exceptions, use getattr(foo, ‘bar’, None) instead. The problem with swallowed exceptions is not just poor error reporting. This can also mask ConflictErrors, which indicate that something has gone wrong at the ZODB level!

• Never, ever put any HTML in Python code and return it as a string

• Do not acquire anything unless absolutely necessary, especially tools. For example, instead of using ‘context.plone_utils’, use:

  from Products.CMFCore.utils import getToolByName
  plone_utils = getToolByName(context, 'plone_utils')
  

• Do not put too much logic in ZPT (use Views instead!)

• Remember to add i18n tags in ZPTs and Python code

All about PLIPS¶

What is a PLIP?

A PLIP is a Plone Improvement Proposal. It is a change to a Plone package that would affect everyone. PLIPs go through a different process than bug fixes because of their broad reaching effect. The Plone 4.x Framework Team reviews all PLIPs to be sure that it’s in the best interest of the broader community to be implemented and that it is of high quality.

Is it a PLIP or a bugfix?

In general, anything that changes the API of Plone in the backend or UI on the front end should be filed as a PLIP. When in doubt, submit it as a PLIP. The framework team is eager to reduce its own workload and will re-classify it for you.

Who can submit PLIPs?

Anyone who has signed a Plone core contributor agreement can work on a PLIP. Don’t let the wording freak you out: signing the agreement is easy and you will get access almost immediately. You do not have to be the most amazing coder in the entire world to submit a PLIP. The Framework Team is happy to help you at any point in the process. Submitting a PLIP can be a great learning process and we encourage people of all backgrounds to submit. When the PLIP is accepted, a Framework Team member will “champion” your PLIP and be dedicated to seeing it completed. PLIPs are not just for code monkeys. If you have ideas on new interactions or UI your ideas are more than welcome. We will even help you pair up with implementors if needed.

What is a PLIP champion?

When you submit your PLIP and it is approved, 1 Framework Team member who is especially excited about seeing the PLIP completed will be assigned to your PLIP as a champion. They are there to push you through completion as well as answer any questions and provide guidance.

A champion should:

• Answer any questions the PLIP implementor has, technically and otherwise
• Encourage the PLIP author by constantly giving feedback and encouragement
• Keep the implementor aware of timelines and push to get things done on time
• Assist with finding additional help when needed to complete the implementation in a timely matter

Keep in mind that champions are in passive mode by default. If you need help or guidance, please reach out to them as soon as possible to activate help mode.

I’m still nervous. Can I get involved other ways at first?

If you want to feel the process and how it works, help us review PLIPs as the implementations finish up. Simply ask on of the Framework Team members what PLIPs are available for review or check the status of PLIPs at the following link. Make sure to let us know you intend to review the PLIP by joining the Framework Team mailing list and sending a quick email. Then, follow the simple instructions for reviewing a PLIP. Thank you in advance!

When can I submit a PLIP?

Today, tomorrow, any time! After the PLIP is accepted, the Framework Team will try to judge complexity and time to completion and assign it to a milestone. You can begin working immediately, and we encourage submitting fast and furious.

When is the PLIP due?

Summary: As soon as you get it done. Technically, we want to see it completed for the release to which it’s assigned. We know that things get busy and new problems make PLIPs more complicated and we will push it to the next release. In general, we don’t want to track a PLIP for more than a year. If your PLIP is accepted and we haven’t seen activity in over a year, we will probably ask you to restart the whole process.

You don’t like my PLIP :( What now?

Just because a PLIP isn’t accepted in core doesn’t mean it’s a bad idea. It is often the case that there are competing implementations and we want to see it vetted as an add on before “blessing” a preferred implementation.

Process Overview¶

1. Submit a PLIP (at any time)
2. PLIP is approved for inclusion into core for a given release
3. Developer implements PLIP (code, tests, documentation)
4. PLIP is submitted for review by developer
5. Framework Team reviews the PLIP and gives feedback
6. Developer addresses concerns in feedback and re-submits if necessary. This may go back and forth a few times until both the FWT and developer are happy with the result.
7. PLIP is approved for merge. In rare circumstances, a PLIP will be rejected. This is usually the result of the developer not responding to feedback or dropping out of the process. Hang in there!
8. After all other PLIPS are merged, a release is cut. Standby for bugs!

How to Submit a PLIP¶

Whether you want to update the default theme or rip out a piece of architecture, everyone should go through the PLIP process. If you need help at any point in this process, please contact a member of the framework team personally or ask for help on the FWT mailing list.

A PLIP is just a ticket with a special template. To get started, open a new ticket and select “PLIP” as the ticket type. A new ticket template will reload and you should plan to fill in all of the fields.

When writing a PLIP, be as specific and to-the-point as you can. Remember your audience - to get support for your proposal, people will have to be able to read it! A good PLIP is sufficiently clear for a knowledgeable Plone user to be able to understand the proposed changes, and sufficiently detailed for the release manager and other developers to understand the full impact the proposal would have on the codebase. You don’t have to list every line of code that needs to be changed, but you should also give an indication that you have some idea that how the change can be feasibly implemented.

If your change is minor then a ticket in the tracker will be sufficient, added as an enhancement. The key point here is that each change needs documentation so other users can see what it is. This can be in the form of an issue tracker entry, or a PLIP in the case of a bigger change. A bug or minor change does normally not need to go through a review process - a PLIP does.

After your PLIP is written, solicit feedback on your idea on the plone-developers mailing list. In this vetting process, you want to make sure that the change won’t adversely affect other people on accident. Others may be able to point out risks or even offer up better or existing solutions.

When you are happy with the feedback, submit a PLIP. Please use the template provided (XXX: put the template here? Can we just have a custom ticket type?). Please note a few things. It is very rare that the “Risks” section will be empty or none. If you find this is the case and your PLIP is anything more than trivial, maybe some more vetting should be done.

The seconder field is REQUIRED. We will send the PLIP back to you if it is not filled in. Currently, this is just someone else who thinks your PLIP is a good idea, a +1. In the near future, we will start asking that the seconder is either a coding partner, or someone who is willing and able to finish the PLIP should something happen to the implementor.

Everything else should be self explanatory. That or I got lazy when writing these docs. I’m betting on the latter.

Implementing Your PLIP¶

You can start the development at any time - but if you are going to modify Plone itself, you might want to wait to see if your idea is approved first to save yourself some work if it isn’t.

[buildout]
extends = plipbase.cfg
auto-checkout +=
    plone.somepackage
    plone.app.someotherpackage

[sources]
plone.somepackage = git git://github.com/plone/plone.somepackage.git branch=plip-1234-widget-frobbing
plone.app.someotherpackage = git git://github.com/plone/plone.app.somepackage.git branch=plip-1234-widget-frobbing

[instance]
eggs +=
    plone.somepackage
    plone.app.someotherpackage
zcml +=
    plone.somepackage
    plone.app.someotherpackage

Buildout Issues¶

Buildout can be frustrating for those unfamiliar with parsing through autistic robot language. Fear not! These errors are almost always a quick fix and a little bit of understanding goes a long ways.

...
 File "/usr/local/lib/python2.6/site-packages/distribute-0.6.13-py2.6.egg/pkg_resources.py", line 556, in resolve
    raise VersionConflict(dist,req) # XXX put more info here
 pkg_resources.VersionConflict: (zc.buildout 1.5.1 (/usr/local/lib/python2.6/site-packages/zc.buildout-1.5.1-py2.6.egg), Requirement.parse('zc.buildout==1.5.2'))

> python bootstrap.py --version=1.5.1

> rm -rf /usr/local/lib/python2.6/site-packages/zc.buildout-1.5.1-py2.6.egg

mr.developer: Can't update package 'Products.CMFPlone' because its URL doesn't match.

Link to http://sphinx.pocoo.org/ ***BLOCKED*** by --allow-hosts

Getting distribution for 'Sphinx==1.0.7'.
Got Sphinx 1.0.7.

allow-hosts += sphinx.pocoo.org

``ln -s plips/.mr.developer.cfg``

Other Random Issues¶