We've had two serious bugs on the NSLU2 recently that both caused the system not too boot anymore. The first one was related to initramfs-tools. Kevin Price reported that after upgrading to initramfs-tools 0.92 the system would hang during boot waiting for the root filesystem. When I booted my NSLU2 and looked at the output, I noticed that no root device was set. When I mentioned this to maks, the maintainer of the initramfs-tools package, he immediately knew what was wrong and uploaded a fix. On the NSLU2, the boot loader doesn't pass a root device to the kernel so we set one in the initramfs itself. The code that loads this config variable is not used by many devices so nobody noticed that this got broken in 0.92 until the package moved to testing.

The second problem, reported on Saturday by Paul Collins and diagnosed today with some help from Kevin Price is also related to the strange way the NSLU2 boots. The NSLU2 has 6 MB available for the ramdisk but due to a bug in APEX (the second stage loader used on the NSLU2 by Debian) only 4 MB are actually used. When you write a ramdisk that is larger than 4 MB but smaller than 6 MB, flash-kernel will happily write the full image, but APEX will only tell the kernel about the first 4 MB. As it turns out, everything works fine on arm with 2.6.25, but on armel some additional SCSI modules are enabled that result in the ramdisk becoming too large. The short term solution is to disable these modules, but really APEX should finally be fixed (patches have been available for quite a while thanks to Gordon Farquharson). Fortunately 2.6.25 hasn't moved to testing yet and this problem only occurs on armel, so few people were impacted.

The nice thing about the NSLU2 is that you can write a new flash image via the network, so you can flash a good image in case your machine no longer boots. Nevertheless, we have to make sure such serious bugs don't occur in the future or that they are at least caught very early. Special thanks go to testers like Kevin Price.

Yesterday I completed the first successful installation of Debian on a QNAP TS-209 using a custom debian-installer image with 2.6.25 from unstable. I made the installer image for QNAP TS-109 and TS-209 available as an experimental developer release and today I also uploaded images for the TS-409. More on my plans to support the QNAP TS-109/TS-209 and TS-409 in the upcoming release of Debian (starting with beta3 of debian-installer) soon.

Benjamin Kosnik of Red Hat put together a guide on the GCC web site with information on porting applications to the upcoming GCC 4.3 release. He integrated some information from my blog and added a number of new items. If you run into any issues while moving to GCC 4.3, I suggest you consult this guide.

The release release team of Debian released 4.0r2 today which fixes the bug that kept the installer from working on ARM based systems, such as the Linksys NSLU2. This means that installations are finally possible again. I've updated my installation instructions accordingly.

I've also updated the tar ball used for the manual installation to 4.0r2 but I doubt many people are interested in this method now that the installer is working again.

I joined HP's Open Source and Linux Organization (OSLO) at the beginning of the month as an Open Source Community Expert. I spent the last three weeks in Ft. Collins for some training and to get to know the team, but normally I'll be working from home in Europe. The main focus of my work will be governance issues related to the deployment of FOSS in enterprise environments. However, I've already started to get involved in a number of other areas I'm interested in, such as community relations. I'm pretty excited about my new position because it is a good fit for my skills and interests.

GCC 4.3 now prints an error message when C++ code contains duplicate function parameter names in function prototypes. You'd think that's a pretty obvious problem, but we still have about 15 packages in the archive that contain such code.

Anyway, for code like this:

class foo {
    foo(int w, int w);
};

GCC will as of 4.3 show the following error message:

t.cc:2: error: multiple parameters named 'w'

Daniel Smolik reported problems with NFS on a Thecus N2100 a few days ago. I confirmed the bug with 2.6.18 and 2.6.22 but found that the problem didn't occur with 2.6.23. I started a git bisect and a few hours later located a fix that Francois Romieu, the maintainer of the r8169 driver, had put in. Apparently the driver had some confusion regarding hardware and IP header alignment. The good news is that the patch is very small and can easily be backported to our 2.6.18 kernel in stable. Also, packages of 2.6.23 were uploaded today and should be in the archive soon.

David Härdeman has contributed two really valuable guides to my Debian on NSLU2 site that will be of great interest to many NSLU2 users:

• The first guide is for people running Linux on a flash device, such as a USB flash stick. Since flash only supports a limited number of writes, it is important to reduce the wear and tear on the underlying flash device. David's guide gives some practical advice of how to achieve this.
• The second guide gives a few tips and tricks that can be used to reduce the amount of RAM a system uses. This is particularly interesting to NSLU2 users since 32 MB of RAM is not a lot.

These guides mention Debian but most of the advice applies to other NSLU2 firmware flavours and to Linux in general.

I updated the tar ball for the manual Debian/NSLU2 installation to Debian 4.0r1 (including the new 2.6.18-5 kernel) today. The proprietary NPE microcode from Intel is now included in the tar ball since the code no longer has a click-through license.

Having passed all requirements for the Doctor of Philosophy, I had my graduation today.

Update: This change has been reverted from 4.3 again and won't be in the final release of 4.3.

GCC 4.3 changed pedantic preprocessor warnings in C++ code into errors, in line with the C++ front-end. Because of this change, preprocessor warnings are now errors.

This change has caused at least the twofollowing errors which are all fairly common:

error: "xxx" redefined

It's not allowed to #define something twice, as in this example:

#define TEST "foo"
#define TEST "bar"

This code will lead to the following error:

test2.cc:2:1: error: "TEST" redefined
test2.cc:1:1: error: this is the location of the previous definition

Most typically, this occurs because something is defined in a header that is included and then redefined in the current source file. It can also happen when a #define is put in the code but also on the command line, e.g. via -DTEST="bar". This will lead to:

test2.cc:1:1: error: "TEST" redefined
<command line>:1:1: error: this is the location of the previous definition

The correct solution is to avoid redefinitions directly or to use #ifndef to make sure that something has not been defined already.

extra tokens at end of #endif directive

While an #ifdef obviously needs a second parameter, it's not allowed to give one to #else and #endif. Unfortunately, this is commonly done as a reminder for what is being tested for:

#ifdef TEST
#endif TEST

This code will lead to the following error:

test3.cc:2:8: error: extra tokens at end of #endif directive

In GCC 4.3 the C++ header dependencies have been cleaned up. In the past, compilation would often be quite slow because including a simple header would indirectly include a lot of other headers, even if they were not needed to compile the current code. Many headers have now been cleaned up with the result that compilation is quicker. The downside is that programmers cannot rely on indirect inclusion of headers they may need anymore. Instead, everything that is needed has to be directly referenced with an #include.

Typical errors look like these:

error: 'find' is not a member of 'std'
error: 'exit' was not declared in this scope

Below is a table showing which header needs to be included for a number of common functions:

Even though GCC 4.2 has seen many delays and still hasn't been released (although it's finally getting close), a lot of development has already been done on trunk for the release of GCC 4.3. I started testing 4.3 in March and in addition to finding and reporting compiler issues I have filed bugs on packages that fail to build with GCC 4.3. The idea is to give people an advance warning and to be ready for GCC 4.3 when it will be released.

We have almost 500 bugs related to GCC 4.3 already, mostly due to a clean-up of C++ headers in GCC that has exposed sloppy programming in many programs. Fortunately, these bugs are easy to fix. As I'm finding common mistakes in packages, I will document these problems and way to make sure a program is ready for GCC 4.3.

Common problems

• Missing #include directives
• Duplicate parameters in function prototypes

Update: there is now a GCC 4.3 porting guide that describes the most common problems people will face when moving to GCC 4.3.

Here are a number of links to various postings and other information related to my research about release management in free software projects:

• Bruce Byfield wrote an article about my research for Linux.com.
• I recently gave a presentation about my research at Google as part of their Tech Talks series. A video of this talk is now available.
• MORITA Hajime kindly translated my blog postings about my research to Japanese.
• Matt Asay briefly discussed my work in his blog and relates it to commercial and proprietary software development. Roberto Galoppini relates my findings to his work and concludes that hierarchy is a must. Murray Cumming summarizes what he believes are the key findings of my research.

In previous times, most Linux distributions and other free software projects relied on the XFree86 system. Over the years, the structures within the XFree86 project became were rigid and the project failed to innovate and keep up with the pace of the wider free software community. When XFree86 changed its license in February 2004, the active community and the majority of vendors quickly moved to X.org. X.org is a very active community and they decided to break up the monolithic code base and to adopt a more modern build system. As of X.org 7.0, the project moved to the modular system in which components are developed and released separately. Effectively, the project introduced a development mechanism which features two release mechanisms: individual components can be released as needed and there is an overall release of X.org in which all stable components are put together. These roll-up releases take place every six months.

Past problems

• XFree86 made only infrequent releases every few years, had no plan, and the project's structures were very rigid.
• The code base was huge and monolithic. It had an archaic build system that few new developers were comfortable with. This made it hard for new contributors to get involved and was bad for testing.

Solutions

• X.org moved from a monolithic to a modular system. This made it easier to perform testing and it made it possible to give contributors write access to specific components.
• The move to the modular system allowed them the introduction of two release mechanisms: individual components can make releases on an ongoing basis and roll-up releases take place every six months.
• The roll-up releases have a fall back mechanism in case a specific component is not ready for release: the former release of this component can be incorporated.

Outstanding problems

• The project needs to work the interface between the server and drivers, so updates to hardware drivers can be released more often than other components.

Plone is a content management system that is built on the powerful Zope application server. The project experienced many delays with its 2.1 release. This made it difficult for Plone consultants to choose whether to use the old release or wait for the new one, and users faced many upgrade issues when 2.1 was finally released and had many changes. Partly to address these problems and partly in order to sync their releases with Zope, they decided in 2005 to move to a six month time based release cycle.

Past problems

• Releases, in particular version 2.1, took a long time to get out.
• Releases had many changes and caused some migration problems.
• Many Plone developers work as consultants building web sites. Because of the unpredictability of Plone, it was difficult for them to decide which version to use for future projects.

Solutions

• Plone moved to a time based release, partly because the Zope framework on which it builds has done so. This allows them to make use of new Zope features in their software.
• Time based releases allowed the project to implement more structure. For example, new features have to be proposed as a Plone Improvement Proposals (PLIP) which the framework team reviews.
• Deadlines have motivated developers to get their features done within a certain time frame.
• Plone consultants can decide in advance which version of Plone to use for future commercial projects.

Outstanding problems

• As the project has moved to time based releases only recently, they still need to show whether they can consistently release on time.

OpenOffice.org is an office suite offering various integrated applications, such as a word processor and a spreadsheet. Originally developed by StarDivision, StarOffice was acquired by Sun who released it as free software as OpenOffice.org in July 2000. While Sun still maintains fairly tight control over the development of OpenOffice.org, many other vendors, in particular Novell, are important contributors to the project. The project had a fairly long release cycle of about 18 months to accommodate StarOffice, the commercial product from Sun. There were many delays, making it hard for vendors to decide which version to include. OpenOffice.org moved to a three month release cycle after their long-delayed 2.0 release, published 26 months after 1.1. The new release cycle is viewed as a positive development by contributors who get their features and fixes out to users faster. Nevertheless, at the end of 2006 a discussion took place in which a six month interval was suggested. Apparently users didn't want new features every three months and the short interval between releases put a lot of pressure on the QA team.

Past problems

• The long release cycle of 18 months, bound to the commercial StarOffice product, meant that little testing occurred for a long time because developers believed the release was far away.
• Many changes accumulated during the long development phase, making testing towards the end very difficult, and leading to a `big bang' release.
• Features were put in very late, even during the beta cycle, because of the perceived 18 month delay to the next release.
• There was very little code review. The QA team only tested the program.
• Vendors shipped unreleased versions because the significant delays during the 2.0 cycle made planning impossible.

Solutions

• After the 2.0 release, the project moved to a 3 month release interval. This model promises a tight feedback loop with users.
• Because planning is now possible, collaboration between vendors on the same code base is far easier.
• The faster release cycle and more collaboration among vendors has promoted code review.
• Motivation in the project has increased because people see their contributions getting out to users within a reasonable time.
• The release process has become more transparent, allowing voluntary contributors to take a more active part in release preparations.

Outstanding problems

• There are discussions about changing the release interval to six months. There is some evidence that some users do not want new features every three months and that the aggressive release cycle of three months puts a lot of pressure on the QA team.

The Linux kernel project has seen major changes to its development and release strategy in the last few years, in particular since the first stable release of the 2.6 series in December 2003. This series was opened almost three years after the 2.4 series in January 2001, which many felt was too long. A problem that resulted from the long interval between major releases was that vendors had to back- and forward-port a lot of patches. Nowadays, major development happens on the 2.6 series and new releases are published every three to four months. This new development model has faced much controversy. While some people, in particular developers of the kernel, claim that the new model is working very well, some users are worried about the number of significant changes and lack of stability in the kernel. Andrew Morton, a lead developer, has expressed several times that he believes the kernel is getting buggier.

Past problems

• Because of the long release cycle, many changes accumulated. It was hard to get the development stable and there were few testers.
• Features got out very slowly because of the long release cycle.
• Hardware support and crucial features had to be backported to the latest stable kernel.
• Vendors backported many features to their own releases. The code base from different vendors diverged a lot from each other and from the official development version.

Solutions

• New versions are now released every two or three months. There is a two week merge window after each release during which new features are accepted; subsequently, the focus is on stabilization.
• There is now a steady flow of code into production and many people get to test the new code.
• Features get out more quickly.
• Vendors can directly work with current releases and get their changes into official versions easily.

Outstanding problems

• There is no long-term stable version based on the 2.6 kernel. This is being addressed with Adrian Bunk's 2.6.16 long-term maintenance tree but whether it will have a big impact remains to be seen.
• Regressions between versions are introduced more frequently. Better control and tracking of regressions are needed.
• The Bugzilla bug tracker needs to be integrated better into the development process and it would be helpful to have a QA person. In January 2007, Google announced that they're looking for a kernel QA person who'd work together with Andrew Morton.

GNOME provides a complete desktop environment that is easy to use. The project experienced major problems, such as delays, during its 1.x series and in particular during the preparation of its 2.0 release. GNOME introduced time based releases after its 2.0 release and has significantly improved release management. The project has published time based releases every six months for a number of years now and is considered as the reference model for a good implementation of time based release management. They have shown that it's possible to set and meet deadlines in volunteer projects and to release on time.

Past problems

• The main goal of version 2.0 was to change internal interfaces but after more than a year of work the project felt they had to deliver more user-visible changes, therefore leading to delays.
• Developers were disappointed with delays and that their work was not available to users.
• It was not clear what was going on. Developers constantly had to ask about the release status.
• Freezes were announced and people worked towards them but then they were delayed. Freezes also often came unexpectedly.
• Vendors had deadlines but the GNOME schedule was unpredictable. Vendors did not know whether they should aim for the next version or focus on the previous version and backport fixes.
• Vendors had to backport many changes to a version that the GNOME project considered as old.

Solutions

• The project introduced a rigorous schedule promising a release every six months.
• There was a core team so few people had to agree to the introduction of a schedule.
• GNOME introduced policies to keep the development tree fairly stable.
• The project introduced the idea of reverting: if a feature was not ready on a certain cut-off date, it would be taken out again.
• The project gained credibility because releases were actually performed on time.
• The schedule allowed vendors better planning, and hence vendors could implement their features on the main development tree.

Outstanding problems

• The six month schedule has been successful in the delivery of incremental updates. There are some concerns whether this release cycle makes the project less innovative and ambitious regarding major changes that would lead to GNOME 3.0.

The GNU Compiler Collection (GCC) is a compiler suite which supports a number of programming languages, such as C and C++. It is a very important development tool and is the standard compiler among free software projects. GCC was quite stagnant in the middle of the 1990s until the EGCS project formed. EGCS took over development of the official GCC in October 1998 and instituted rigorous processes, such as high levels of peer review, and created a steering committee which has the power to appoint maintainers and make important decisions. In theory, the project follows a time based release with an interval of six months. In practice, the project has released only one new version every year in recent times.

Past problems

• The GCC project suffered from a closed development style in the past: few people could make code changes and the mailing list was by invitation only.
• There was a long time between releases and development snapshots, which contained bug fixes and features, were not available to the public.
• When development opened and picked up, significant code changes were often made which required a long stabilization phase.

Solutions

• The project moved to a more open development style and established a steering committee which has the power to appoint new maintainers and make important decisions.
• The development process was divided into three stages in order to coordinate code submissions and keep the development tree reasonable stable. Major changes that developers wish to make during stage 1 and stage 2 have to be proposed on the project's wiki. Proposals are reviewed by the release manager who also assign a sequence to proposed projects which says when they can be applied to the development tree.
• All patches are peer reviewed on the development mailing list and need approval before they can be applied. The steering committee appoints maintainers who can approve patches that touch particular areas. There are maintainers for the C front-end, various port maintainers, and maintainers for other areas.
• Each development stage lasts two months, theoretically yielding a release every six months. A regular release cycle ensures that it is not the end of the world if a feature does not make it into the following release.

Outstanding problems

• The release manager is busy and has not pushed the release forwards as much as would be possible.
• The branch criteria may need revision to make it easier to create a branch which leads to the next stable version. At the moment, the branch criteria is to have 100 or less regressions on the development tree. However, this number may include regressions that are also present in previous releases, which makes it fairly hard to reach this criteria.