path: root/needs-checking/docco.txt

Patch management scripts
Andrew Morton <akpm@digeo.com>
18 October 2002

This is a description of a bunch of shell scripts which I use for
managing kernel patches.  They are quite powerful.  They can be used on
projects other than the linux kernel.  They are easy to use, and fast.

You end up doing a ton of recompiling with these scripts, because
you're pushing and popping all the time.  ccache takes away the pain of
all that.  http://ccache.samba.org/ - be sure to put the cache
directory on the same fs as where you're working so that ccache can use
hardlinks.

The key philosophical concept is that your primary output is patches. 
Not ".c" files, not ".h" files.  But patches.  So patches are the
first-class object here.

Installation
============

You place all the scripts somewhere in your path, or in
/usr/lib/patch-scripts.

Terminology
===========

The patch scripts require three special directories called "pc",
"patches" and "txt".

If the environment variable PATCHSCRIPTS is set, it is taken to to be
the directory in which those three directories reside.  Typically, it
would be a relative pathname.  So

	setenv PATCHSCRIPTS ./i-put-them-here

would tell the patch scripts to look in ./i-put-them-here/pc, etc.

If PATCHSCRIPTS is not set, and the directory ./patch-scripts is
present then the patch scripts will us ./patch-scripts/pc/,
./patch-scripts/patches/ and ./patch-scripts/txt/.

Otherwise, the patch scripts use ./pc, ./patches and ./txt.

In this document, the symbol $P is used to describe the directory which
holds the pc/, patches/ and txt/ directories, as determined by the
above search.

It is expected that $P will always expand to a relative path.

Concepts
========

All work occurs with a single directory tree.  All commands are invoked
within the root of that tree.  The scripts manage a "stack" of patches.

Each patch is a changeset against the base tree plus the preceding patches.

All patches are listed, in order, in the file ./series.  You manage the
series file.  Lines in the series file which start with `#' are ignored.

Any currently-applied patches are described in the file
./applied-patches.  The patch scripts manage this file.

Each patch affects a number of files in the tree.  These files are
listed in a "patch control" file.  These .pc files live in the
directory $P/pc/

Patches are placed in the directory $P/patches/

Documentation for the patches is placed in $P/txt/

So for a particular patch "my-first-patch" the following will exist:

- An entry "my-first-patch.patch" in ./series

- An entry "my-first-patch" in ./applied-patches (if it's currently applied)

- A file $P/pc/my-first-patch.pc which contains the names of the
  files which my-first-patch modifies, adds or removes

- A file $P/txt/my-first-patch.txt which contains the patch's
  changelog.

- A file $P/patches/my-first-patch.patch, which is the output of the
  patch scripts.

Operation
=========

When a patch "my-patch" is applied with apatch, or with pushpatch
(which calls apatch), all the affected files (from $P/pc/my-patch.pc)
are copied to files with ~my-patch appended.  So if $P/pc/my-patch.pc
contained

	kernel/sched.c
	fs/inode.c

then apatch will copy those files into kernel/sched.c~my-patch and
fs/inode.c~my-patch.  It will then apply the patch to kernel/sched.c
and fs/inode.c

When a diff is regenerated by refpatch (which calls mpatch), the diff
is made between kernel/sched.c and kernel/sched.c~my-patch.  How do the
scripts know to use "~my-patch"?  Because my-patch is the current
topmost patch.  It's the last line in ./applied-patches.

In this way, the whole thing is stackable.  If you have four patches
applied, say "patch-1", "patch-2", "patch-3" and "patch-4", and if
patch-2 and patch-4 both touch kernel/sched.c then you will have:

	kernel/sched.c~patch-2		Original copy, before patch-2
	kernel/sched.c~patch-4		Copy before patch-4.  Contains changes
					from patch-2
	kernel/sched.c			Current working copy.  Contains changes
					from patch-4.

This means that your diff headers contain "~patch-name" in them, which
is convenient documentation.

Walkthrough
===========

Let's start.

Go into /usr/src/linux (or wherever)

	mkdir pc patches txt

Now let's generate a patch

	fpatch my-patch kernel/sched.c

OK, we've copied kernel/sched.c to kernel/sched.c~my-patch.  We've
appended "my-patch" to ./applied-patches and we've put "kernel/sched.c"
into the patch control file, pc/my-patch.pc.

	Now edit kernel/sched.c a bit.

Now we're ready to document the patch

	Now write txt/my-patch.txt

Now generate the patch

	refpatch

This will generate patches/my-patch.patch.  Take a look.

Now remove the patch

	poppatch

applied-patches is now empty, and the patch is removed.

Now let's add a file to my-patch and then generate my-second-patch:

	Add "my-patch.patch" to ./series (no blank lines in that file please)

	pushpatch

OK, the patch is applied again.  Let's add another file

	fpatch kernel/printk.c

Note that here we gave fpatch a single argument.  So rather than
opening a new patch, it adds kernel/printk.c to the existing topmost
patch.  That's my-patch.

	Edit kernel/printk.c

Refresh my-patch (you end up running refpatch a lot)

	refpatch

Now start a second patch:

	fpatch my-second-patch kernel/sched.c

Now take a look at applied-patches.  Also do an `ls kernel/sched*'.

	Edit kernel/sched.c, to make some changes for my-second-patch

Generate my-second-patch:

	refpatch

Take a look in patches/my-second-patch.patch

Don't forget to add "my-second-patch.patch" to the series file.

And remove both patches:

	poppatch
	poppatch


That's pretty much it, really.


Command reference
=================

Generally, where any of these commands take a "patch-name", that can be
of the form txt/patch-name.txt, patch-name.pc, just patch-name or
whatever.  The scripts will strip off a leading "txt/", "patches/" or
"pc/" and any trailing extension.  This is so you can do

	apatch patches/a<tab>

to conveniently use shell tabbing to select patch names.



added-by-patch

  Some internal thing.

apatch [-f] patch-name

  This is the low-level function which adds patches.  It does the
  copying into ~-files and updates the applied-patches file.  It
  applies the actual patch.

  apatch will do a patch --dry-run first and will refuse to apply the
  patch if the dryrun fails.

  So when you are getting rejects you do this:

	pushpatch		# This fails, due to rejects.  Drat.
	apatch -f patch-name	# Force the patch
  (or)  pushpatch -f		# Force the patch

  OK, you've now applied patch-name, but you have rejects.  Go fix
  those up and do

	refpatch

  And you're ready to move on.

combine-series output-file

  It incrementally combinediffs all the patches in series to make a
  complete patch for the series.  Requires combinediff frmo patchutils.

  See http://cyberelk.net/tim/patchutils/ (Don't download the
  "experimental" patchutils - it seems to only have half of the
  commands in it.  Go for "stable")

cvs-take-patch

  I forget.

export_patch

  export the patches listed in ./series to a set of files which
  are named in such a way that the sort order is the same as the
  order of the series file. 

  Usage:  export_patch directory [prefix]

  Example:
	
	Suppose ./series contains
	
	mango.patch
	orange.patch
	banana.patch
	apple.patch
	pear.patch

	export_patch ../mypatches fruit 
	
	The patches would be copied to

	../mypatches/p00001_fruit_mango.patch
	../mypatches/p00002_fruit_orange.patch
	../mypatches/p00003_fruit_banana.patch
	../mypatches/p00003_fruit_banana.patch
	../mypatches/p00003_fruit_banana.patch

	Named in this way, someone may easily apply them:

	cat mypatches/p*fruit* | patch -p1

	If prefix is omitted, the patchnames will be transformed
	such that "original.patch" becomes "pXXXXX_original.patch".

fpatch [patch-name] foo.c

  If patch-name is given, fpatch will start a new patch which
  modifies (or adds, or removes) the single file foo.c.  It updates
  ./applied-patches and creates pc/patch-name.pc.  fpatch will copy
  foo.c to foo.c~patch-name in preparation for edits of foo.c.

  If patch-name is not given then fpatch will add foo.c to the
  current topmost patch.  It will add "foo.c" to $P/pc/$(toppatch).pc. 
  It will copy foo.c to foo.c~$(toppatch).

import_patch

  Imports a set of patch files, creating $P/pc, $P/txt, $P/patches and
  ./series as necessary.  It also creates $P/txt/*.txt by stripping 
  off the top of the patches (and removes any diffstat output it finds, 
  so that it can eat refpatch output and export_patch output.)  The 
  imported patch names are appended to the series file.

  In creating the $P/txt/*.txt files, mail headers are stripped with 
  formail, preserving the "From:" and "Subject:" lines.  "DESC" and
  "EDESC" markers are added if they are not already present, using the
  "From:" and "Subject:" lines for the DESC portion, if they are present.
  (See "patchdesc" command, below, for more on these markers.)

  Also, it can rename the patch file as it is imported by stripping out
  a pattern.  This is useful if, as often is the case, you have patch 
  sets with filenames designed to help sort the patches into the correct 
  order, such as "p001_xxx_funky_stuff.patch" you can have it automatically
  renamed to funky_stuff.patch on import, and let the series file manage
  the ordering.

  Import_patch will uncompress patches (*.Z, *.bz2, *.gz) as necessary.

  Usage:

  import_patch [-p pattern] patchfile ...

  Example:

	% ls ../fruit/p*patch
	../fruit/p00001_northern_apple.patch
	../fruit/p00001_tropical_mango.patch
	../fruit/p00002_northern_pear.patch
	../fruit/p00002_tropical_orange.patch
	../fruit/p00003_tropical_banana.patch
	% import_patch -p 'p[0-9]*_tropical_' ../fruit/p*tropical*
	Recreated pc/mango.pc
	Recreated pc/orange.pc
	Recreated pc/banana.pc
	% import_patch -p 'p[0-9]*_northern_' ../fruit/p*northern*
	Recreated pc/apple.pc
	Recreated pc/pear.pc

  Then you can "pushpatch; refpatch" 5 times.

inpatch

  List the names of ths files which are affected by the current
  topmost patch.

  This is basically

	cat pc/$(toppatch).pc

join-patch patchname

  "joins" the named patch to the current topmost patch.

  Use this when you want to merge two patches into one.  All the
  files which `patchname' affects are added to pc/$(toppatch).pc (if
  they are not already there) and patch `patchname' is applied.  The
  top patch remains unchanged.  You'll need to run refpatch afterwards.

mpatch

  A low-level thing to generate patches

new-kernel

  Some thing I use for importing a new kernel from kernel.org

p0-2-p1

  Internal thing to convert patch -p0 form into patch -p1

patchdesc

  Generates a single-line description of a patch.

  The txt/my-patch.txt files have the following format:

  <start of file>
  DESC
  some short description
  EDESC

  The long description
  <end of file>

  I use

	patchdesc $(cat series)

  to generate short-form summaries of the patch series.

patchfns

  Internal utilities

pcpatch

  Standalone tool to generate a .pc file from a patch.

  Say someone sends you "his-patch.diff".  What you do is:

	cp ~/his-patch.diff patches/his-patch.patch
	pcpatch his-patch

  This generates $P/pc/his-patch.pc and you're all set.  Add
  "his-patch.patch" to ./series in the right place and start pushing.

p_diff

  I forget

poppatch

  Remove one or more patches from the current stack.  This command
  does *not* use the series file.  It works purely against
  applied-patches.

  Usage:

	poppatch
		Remove the topmost patch
	poppatch 10
		Remove ten patches
	poppatch some-patch-name[.patch]
		Remove patches until "some-patch-name" is top patch

pstatus

	Shows status of patches

  Usage:
	pstatus [patchfile ...]
	
	One line per patch is output showing:
	1: Patch number in the series file
	2: Whether the patch is currently applied
	3: Name of patch
	4: Status of the patch (needs pcpatch, changelog, refpatch)

	If no patchfiles are specified, $P/patches/*.patch
	are assumed.

  Caveats:
	A patch set which contains separate patches to add a file
	and modify that same file may give spurious "Needs refpatch"
	status for the patch which adds the file or the topmost patch.

ptkdiff

  Two modes:

	ptkdiff -

               Run tkdiff against all the file affected
               by $(toppatch).  The diff is only for the changes made
               by the top patch! ie: it's between "filename" and
               "filename~toppatch-name".

	ptkdiff filename

               Just run tkdiff against that file,
               showing the changes which are due to toppatch.

pushpatch [-f]

  Apply the next patch, from the series file.

  This consults ./applied-patches to find out the top patch, then
  consults ./series to find the next patch.  And pushes it.

    pushpatch

      Apply the next patch

    pushpatch 10
 
      Apply the next ten patches

    pushpatch some-patch-name

      Keep pushing patches until "some-patch-name" is toppatch

    pushpatch -f

      Push the next patch, ignoring rejects.

refpatch

    regnerates the topmost patch.  Reads all the affected files
    from pc/$(toppatch).pc and diffs them against their tilde-files.

    Also pastes into the patch your patch documentation and
    generates a diffstat summary.

removed-by-patch

  Some thing.

rename-patch

  CVS rename for patches.

rolled-up-patch

  Bit of a hack.  Is designed to generate a rolled-up diff of all
  currently-applied patches.  But it requires a ../linux-2.x.y tree to
  diff against.  Needs to be redone.

rpatch

  Internal command

split-patch

  Some thing someone write to split patches up.  I don't use it.

tag-series

  Assuming you keep pc/*, patches/* and txt/* under CVS revision
  control, tag-series allows you to tag a patchset's individual
  components.  I use

	tag-series s2_5_44-mm3 pc/2.5.44-mm3-series

  which will attach the cvs tag "s2_5_44-mm3" to every .pc, .patch
  and .txt file which is mentioned in the series file
  "pc/2.5.44-mm3-series".

  It will also tag pc/2.5.44-mm3-series, which is a bit redundant
  given that I use a different series file for each patchset release..


toppatch

  Print the name of the topmost patch.  From ./applied-patches

touched-by-patch patch-filename

  List the names of files which are affected by a diff.

unitdiff.py

  Rasmus Andersen's script to convert a diff into minimum-context
  form.  This form has a better chance of applying if you're getting
  nasty rejects.  But patch can and will make mistakes when fed
  small-context input.


Work Practices
==============

I keep the kernel tree, the $P/pc/, $P/patches/ and $P/txt/ contents under
CVS control.  This is important...

I have several "series" files.  I keep these in $P/pc/foo-series and use

	ln -s pc/foo-series series

when I'm working on foo.

If someone sends me a patch I'll do:

	cp ~/whatever patches/his-patch.patch
	pcpatch his-patch
	apatch his-patch

  If apatch fails then run `apatch -f his-patch' and fix the rejects.

	refpatch

  to clean up any fuzz.

	poppatch
	cvs add pc/his-patch.pc patches/his-patch.patch
	cvs commit pc patches

  Now edit ./series and place "his-patch.patch" in the appropriate place.


If you're working on a particular patch (say, "dud-patch") and you
balls something up, just run:

	refpatch	# Generate the crap patch
	poppatch	# Remove it all
	rm patches/dud-patch.patch
	cvs up patches/dud-patch.patch

and all is well.


Getting updates from Linus
==========================

What I do is to grab the latest -bk diff from
http://www.kernel.org/pub/linux/kernel/people/dwmw2/bk-2.5/
and do:

	gzip -d < cs<tab> > patches/linus.patch
	pcpatch linus
	apatch linus | grep diff

               Now fix up all the files which got deleted,
               because there's something wrong with bitkeeper diffs:

	cvs up -ko <missing files from the above diff>

	apatch linus
	$EDITOR linus/linus.txt
	
		Add the changeset number to txt/linus.txt

	refpatch
	poppatch

  Now add "linus.patch" as the first entry in your ./series file and
  start pushing your other patches on top of that.

BUGS
====

Tons and tons.  The scripts are fragile, the error handling is ungraceful and
if you do something silly you can end up in a pickle.

Generally the scripts are very careful to not wreck your files or your
patches.  But they can get the ./applied-patches and ~-files into an
awkward state.

Usually you can sort it out by copying the ~-files back onto the originals
and removing the last line from ./applied-patches.  Or do a "refpatch ;
poppatch ; rm patches/troublesome-patch.patch ; cvs up patches".

If it's really bad, just blow away the entire tree and do a new CVS checkout.


Working on non-kernel projects
==============================

Well it's the same thing.  Say you've downloaded a copy of util-linux
and you want to make a change:

	cd /usr/src
	tar xvfz ~/util-linux.tar.gz
	cd util-linux
	mkdir pc patches txt
	fpatch my-patch sys-utils/rdev.c
	fpatch sys-utils/ipcs.8
	<edit, edit>
	refpatch
	<ship patches/my-patch.patch>

How to balls things up
======================

Well here's one way.  Suppose you have 20 patches applied, and three of
them (say, "p1", "p6" and "p11") all modify "foo.c".

Now you go and change foo.c.

Well, to which patch does that change belong?  You need to decide. 
Let's say you decide "p6".

If you run `refpatch' when "p11" is toppatch then you lose.  The diff
went into p11.

What you can do is:

1:
	poppatch p6
	<edit>
	refpatch
	pushpatch p11
	<test>

  (See why ccache is looking good?)

or

2:
	<edit>
	<test>
	poppatch p6	<hope like hell that the other patches remove cleanly>
	refpatch


Another good way of ballsing up is to cheat.  Say "oh I just want to make
this one-line change".  And "oh, and this one".

Now you're getting in a mess.  It's much, much better to just use the system:

	fpatch junk file1
	fpatch file2
	<edit>
	<play>
	refpatch
	poppatch
	rm pc/junk.pc patches/junk.patch

Merging with -mm kernels
========================

Haven't tried this, but it should work:

- Grab all the patches from broken-out/, place them in your $P/patches/

- Copy my series file into ./series (or $P/pc/akpm-series and symlink it)

- pushpatch 99

And you're off and running.  The nice thing about this is that you can
send me incremental diffs to diffs which I already have.

Or whatever.  I'm fairly handy with diffs nowadays.  Rejects are
expected.  I just prefer to have "one concept per diff".