The European Open Source Observatory and Repository (OSOR)

I attended the Open Nordic Conference in Norway in June, a conference that brought together people from Norway, Sweden, Finland and Denmark. Attending the conference allowed me to find out what's going on with FOSS in Northern Europe. What I found interesting is that there was a lot of talk about using FOSS in the public sector. A number of countries are working on repositories to exchange software, in particular for public administration.

One example is the Software Exchange (softwareborsen.dk), a project by the Danish National Software Knowledge Centre to promote the use of FOSS in Danish public administration. The Norwegian software exchange (delingsbazaren.no) plays a similar role in Norway.

It seems as if every country is working on their own software repository to share FOSS. As such, it's great to see that the European Commission is taking a step towards bringing everyone together by introducing the Open Source Observatory and Repository (OSOR) for European public administrations. The goal of OSOR is to provide a platform for the exchange of information, experiences and FOSS code for the use in public administrations.

OSOR will officially launch at the Open Source World Conference in Malaga in October. I've always thought that there is not enough cooperation and communication between European countries, so I have high hopes for OSOR.

(Originally published on FOSSBazaar)

Starting to test GCC 4.4

I've been too busy to follow GCC development recently and haven't done any testing of development snapshots during the GCC 4.4 development phase so far. However, GCC 4.4 moved to stage 3 yesterday, the development stage dedicated to bug fixes and documentation. Given that GCC is now in stage 3 and that Debian unstable has fairly little churn because of the freeze, it's an ideal time to do some archive rebuilds.

I started archive rebuilds on amd64 and powerpc last night and added ia64 today. I also tried to build GCC on armel, alpha and sparc but there are compiler errors during bootstrap. mips builds fine but I haven't started any archive rebuilds.

I found and reported 9 compiler errors so far. Most of them have already received some attention and one of them has been fixed in the meantime. I'm not looking at build failures of packages at the moment since that's very time consuming and I'm not sure whether I'll find the time to report build failures later as GCC becomes more stable, but we'll see.

Cannot connect to debian-installer via SSH: mystery solved?

I finally found the time today to dig into oldsys-preseed and what I found wasn't pleasant. In fact, I may have solved the mystery why some people trying to install Debian on their NSLU2 cannot connect to the installer via SSH.

oldsys-preseed is used on certain NAS devices to read the network configuration from the original firmware and these values are used to bring up SSH in debian-installer. If you used DHCP in the original firmware to obtain an IP address, debian-installer will do the same. If you have a static IP config, debian-installer will use it. In theory, it's all pretty simple, but users often run into troubles when a static IP config is used (as is the case on the NSLU2). While you only need an IP address, netmask and gateway to bring up SSH (and then you could configure the rest, such as DNS and hostname), debian-installer expects everything at the same time. I've seen many cases where users couldn't connect to the installer because they forgot to configure their DNS in the original firmware or some other values they didn't think were important.

Anyway, right now oldsys-preseed will use a static configuration if the firmware is configured this way even if it's clear that netcfg won't like this configuration because of missing values. The IP address and DNS are always required, but the netmask and gateway can be figured out automatically. I wanted to add a sanity check that would choose DHCP in case the IP address or DNS were not specified.

While working on this sanity check, I noticed two things: First, the "empty" value oldsys-preseed preseeds when there's no netmask or gateway is not accepted by netcfg and leads to an error message (which the user cannot see because SSH is not up yet). I'm pretty sure I tested this scenario when I wrote oldsys-preseed but it clearly doesn't work today and I don't see how it could ever have worked. This is pretty significant because having an empty gateway is a valid configuration (if hopefully not too common). Second, it seems that it's impossible to create a preseed file that tells netcfg that no gateway should be used. If you preseed an empty value, netcfg will simple use .1 as the gateway.

I've fixed the first problem and proposed a patch to put in a special case to allow preseeding of no gateway. The sanity check that chooses DHCP when the network configuration is incomplete is also there now.

Please test daily debian-installer images

debian-installer is now using the 2.6.26 kernel (the version the kernel team wants to release lenny with) and 2.6.26 has moved to testing a few days ago. As such, it's a great time to test the daily installer images to make sure our installer for lenny is in a good state. There's still time to fix bugs in the installer, but we need your feedback.

Daily images can be obtained from the debian-installer web site which also contains instructions about how to submit installation reports. I'm particularly interested in reports about installations on ARM and MIPS. If you want to test debian-installer on these architectures, please read my messages on debian-arm and debian-mips.

initramfs-tools MODULES=dep default on ARM

Frans Pop reported recently that the initramfs did not fit in the flash of the QNAP TS-109 when you used LVM. We had problems with initramfs images that were too large before (on the NSLU2). By default, initramfs-tools uses the MODULES=most setting which puts all modules that might be of interest to booting a machine in the initramfs. This is a good idea on a PC where the hardware can change a lot, but it makes less sense on most NAS devices.

Frans has now changed debian-installer so the MODULES policy can be chosen during the installation. His change allowed architecture specific defaults, so I've changed ARM to use MODULES=dep. debian-installer will now create the file /etc/initramfs-tools/conf.d/driver-policy with a MODULES setting that will override that from /etc/initramfs-tools/initramfs.conf. On QNAP devices, I intend to only support MODULES=dep installations, so please change your configuration if you've installed Debian already. On the NSLU2, MODULES=most will of course continue to be supported.

OSI and License Proliferation

License proliferation has been a topic for discussion for quite a while now in the FOSS community and many would like to see the Open Source Initiative (OSI) fix this problem for good. In a license proliferation report, the OSI lists three problems that people generally see with license proliferation:

1. Too many different licenses makes it difficult for licensors to choose: it's difficult to choose a good license for a project because there are so many.
2. Some licenses do not play well together: some open source licenses do not inter-operate well with other open source licenses, making it hard to incorporate code from other projects.
3. Too many licenses makes it difficult to understand what you are agreeing to in a multi-license distribution: since a FOSS application typically contains code with different licenses and people use many applications which each contain one or several licenses, it's difficult to see what your obligations are.

License proliferation is a hard problem because nobody wants to give up their favourite license and because it can be extremely difficult to relicense a piece of code when there are multiple copyright holders, as is the case in the majority of community projects.

It's a particularly hard problem for the OSI because of two conflicting interests. One the one hand, the OSI approves licenses as open source based on the Open Source Definition (OSD). Therefore, it seems natural that a license that fulfils the OSD should be approved as open source. On the other hand, because of the problems of license proliferation, it's beneficial to keep the number of open source approved licenses low.

At the OSI meeting in Portland a few weeks ago, the Board decided to tackle this problem again and split licenses into two tiers. The specific wording has not been finalized yet, but "recommended" and "compliant" have been suggested. This would make it clear which open source licenses are "merely" OSD compliant whereas a limited number are recommended for use in projects.

The discussion about the two tiers has just started, so there are no results yet. But it's definitely an important discussion that needs to take place.

(Originally published on FOSSBazaar)

TODO list for Debian on D-Link DNS-323

Someone asked me the other day what it would take to add Debian support for the D-Link DNS-323. Since we support a number of Orion based devices in debian-installer already, adding support for another device is typically fairly easy. I don't have a D-Link DNS-323 myself, but I looked around the useful DNS-323 wiki and this is what I came up. I'm sharing this list in the hope that other people are interested in working on DNS-323 support.

• The Orion kernel in Debian has support for the D-Link DNS-323 but it needs testing. Also, some patches, such as support for the fan speed chip, are not included in the mainline kernel yet.
• flash-kernel (which writes the kernel and ramdisk to flash) needs support for the DNS-323. I actually implemented this in the meantime but it's completely untested.
• oldsys-preseed reads the network configuration from your existing firmware and uses it to configure the network for debian-installer. This also needs support for the DNS-323.
• Some tools to generate proper firmware images need to be packaged for Debian.
• The debian-installer needs to generate a boot image for the DNS-323 (easy once the tools are packaged).
• Apparently the MAC address is not set automatically in u-boot and you have to run a tool called mac_read to set it. This is problematic because at the moment there's no code to set the MAC address in d-i and to make sure the newly installed system will automatically set the MAC address. This needs some work.

2.6.26 ARM and MIPS udebs on the way...

Otavio has updated kernel-wedge in SVN to 2.6.26, so I updated ARM and MIPS today. The udebs built after minor modifications and I can also build installer images after having fixed some bugs in the build process. I tested a 2.6.26 based image on the QNAP TS-409 (armel) as well as in QEMU (mipsel).

2.6.26 will bring major improvements on ARM. Orion support is much better and our 2.6.26 has a number of patches from Marvell that help with performance on Orion. Our 2.6.26 also has Riku Voipio's LED driver for Thecus N2100 which has been accepted for the 2.6.27 mainline kernel.

I haven't actually uploaded the udebs yet because I'm waiting for the go-ahead as well as for the new version of the 2.6.26 kernel package to build. But at least we're getting closer to having 2.6.26 as the kernel for lenny.

Debian on HP mv2120 web site

I've created a web site with information for running Debian on the HP mv2120. The HP mv2120 is a NAS device based on Marvell's Orion platform and has a 500 MHz ARM CPU, 128 MB RAM and two SATA slots.

All components needed to support Debian on the HP mv2120 have been integrated into Debian and Debian installer. However, a new release of the Debian installer is required so installations of Debian testing are actually possible.

Daily installer image available for QNAP TS-109, TS-209 and TS-409

Daily installer images are now available for the QNAP TS-109, TS-209 and TS-409. This means that complete installations of Debian testing are possible now.

You can consider the daily image as a beta release of RC1 of the debian-installer for Debian lenny. It should only be used by experienced users and only if you can make a serial console in case something goes wrong. As the name suggests, the daily images change on a daily basis and they install Debian testing which also changes on a daily basis (although testing should be fairly stable since we're close to the freeze of testing in preparation for the release of Debian lenny).

If you install Debian with these installer images, please send an installation report to debian-boot send an email to me so I can let you know in case there are any important news you should be aware of.

There is a web site now with complete installation instructions and other information about Debian on QNAP Turbo Station. If you have any questions or suggestions, please let me know.

New behaviour for oldsys-preseed regarding default settings

I've just changed the behaviour of oldsys-preseed in the case when a static IP address is set in the configuration of the device that is the default setting from the vendor. In the past, oldsys-preseed would try to be smart in this case and do DHCP in the assumption that the users never configured the static IP address themselves and might not want to have that setting. However, Mike (mwester) convinced me that doing DHCP when the configuration clearly says to use a static IP is a bad idea. Apparently quite a few users got confused by the old behaviour of oldsys-preseed. The new behaviour exists as of version 2.0 of oldsys-preseed which will be in rc1 of debian-installer for lenny. This change affects the Linksys NSLU2 and Thecus N2100.

Debian support for HP mv2120: putting everything together

I managed to get my hands on a HP Media Vault mv2120, a nice ARM based NAS device, a few months ago with the intentions of porting Debian to it. Unfortunately, I have been really busy lately and most of my time was spent on adding support for the QNAP TS-109/TS-209 and TS-409 (which required a lot of generic work to get Marvell Orion support into Debian, a new SoC used in many NAS devices, including the QNAP TS-x09 and HP mv2120).

There were a number of things that had to be worked out before Debian would run on the mv2120. The good news is that Marc Singer and Eugene San have done all of the heavy lifting in the last few weeks in figuring out how the mv2120 works and that now it's just a matter of putting everything together for Debian to work.

Here are the issues that had to be worked out:

• Figure out how the rescue mode works and how to construct a rescue image that the device can boot: I knew that the mv2120 had a rescue mode that can be activated by keeping the the power and reset buttons pressed when starting that would request a rescue image via the network. This rescue mode can be used to start the Debian installer. Eugene San figured out how to construct a rescue image that the mv2120 would accept. Marc Singer has also written a tool, uphpmvault, to serve such rescue images from Linux.
• Figure out the best way to make one image that contains the kernel and ramdisk: the mv2120 only loads a kernel and no ramdisk, but a ramdisk is needed to start Debian. I discussed some really ugly hacks with Marc Singer to get the mv2120 to accept an image consisting of kernel and ramdisk but Marc wanted to try something else first. u-boot, the boot loader used on the mv2120, can accept something called a multipart image which can consist of a kernel and a ramdisk. When Eugene San heard that we were trying to load a multipart image, he went ahead and figured out how to construct an image that the mv2120 would boot.

Now that these two issues are resolved, I simply need to put everything together and add support for the mv2120 to a number of debian-installer components. We already have Orion kernel images in unstable that support the HP mv2120 (along with a number of other Orion based NAS devices) and the rest shouldn't take too long.

NSLU2 Debian installer for lenny beta 2

The Debian installer team has published the second beta release of the installer for lenny (what will become Debian 5.0). I repacked the image for the Linksys NSLU2 with the proprietary IXP4xx microcode so Ethernet will work.

The installer is now available in two flavours for the ARM architecture: arm is the old ARM port whereas armel is the new ARM port using the EABI which offers better support for floating point and other features. Both the arm and armel architectures will be supported for lenny but the old arm port will be dropped after the release of lenny. At this point, I therefore recommend that people use armel for new installations.

The beta 2 installer image for NSLU2 can be found on my site: armel image (recommended) and arm image.

The NSLU2 causing trouble recently...

We've had two serious bugs on the NSLU2 recently that both caused the system not too boot anymore. The first one was related to initramfs-tools. Kevin Price reported that after upgrading to initramfs-tools 0.92 the system would hang during boot waiting for the root filesystem. When I booted my NSLU2 and looked at the output, I noticed that no root device was set. When I mentioned this to maks, the maintainer of the initramfs-tools package, he immediately knew what was wrong and uploaded a fix. On the NSLU2, the boot loader doesn't pass a root device to the kernel so we set one in the initramfs itself. The code that loads this config variable is not used by many devices so nobody noticed that this got broken in 0.92 until the package moved to testing.

The second problem, reported on Saturday by Paul Collins and diagnosed today with some help from Kevin Price is also related to the strange way the NSLU2 boots. The NSLU2 has 6 MB available for the ramdisk but due to a bug in APEX (the second stage loader used on the NSLU2 by Debian) only 4 MB are actually used. When you write a ramdisk that is larger than 4 MB but smaller than 6 MB, flash-kernel will happily write the full image, but APEX will only tell the kernel about the first 4 MB. As it turns out, everything works fine on arm with 2.6.25, but on armel some additional SCSI modules are enabled that result in the ramdisk becoming too large. The short term solution is to disable these modules, but really APEX should finally be fixed (patches have been available for quite a while thanks to Gordon Farquharson). Fortunately 2.6.25 hasn't moved to testing yet and this problem only occurs on armel, so few people were impacted.

The nice thing about the NSLU2 is that you can write a new flash image via the network, so you can flash a good image in case your machine no longer boots. Nevertheless, we have to make sure such serious bugs don't occur in the future or that they are at least caught very early. Special thanks go to testers like Kevin Price.

QNAP TS-109/TS-209 and TS-409 experimental developer release

Yesterday I completed the first successful installation of Debian on a QNAP TS-209 using a custom debian-installer image with 2.6.25 from unstable. I made the installer image for QNAP TS-109 and TS-209 available as an experimental developer release and today I also uploaded images for the TS-409. More on my plans to support the QNAP TS-109/TS-209 and TS-409 in the upcoming release of Debian (starting with beta3 of debian-installer) soon.

FLOSSInclude wants to foster FOSS in developing countries

The EU is sponsoring a new two year project called FLOSSInclude to explore the use of FOSS as a development tool. In particular, the aim of FLOSSInclude is to study what is needed to "increase the deployment, development and societal impact of FOSS in Africa, Asia and Latin America". The project starts with the premise that FOSS provides a number of benefits to developing countries, such as low cost, adaptability, and a free-of-charge high quality training environment.

The project will identify and analyze key problem areas and areas of blocked potential in the target regions. Furthermore, pilot studies will be conducted to ensure that FOSS solutions, tools and services can be cost-effective and practical for each environment. At the end of the project, FLOSSInclude intends to develop a roadmap for future EU development research cooperation in order to "ensure that the impact of FLOSSInclude will be sustained far beyond the duration of the project". I think it's great that there is a research project now with the focus on including developing countries in FOSS and I hope this project will be very successful.

(Originally published on FOSSBazaar)

Obstacles for making FOSS development truly global

Li Yang, who works for Freescale Semiconductor, gave a very interesting talk at linux.conf.au a few weeks ago about the obstacles of getting people from all around the world involved in FOSS projects. He started off by saying that he'd like FOSS to be really global, and not just global in the sense that developers from North America, Europe and Australia are involved.

He started by comparing Internet usage and FOSS involvement of different parts of the world. The US, Europe and Australia make up about 45% of Internet users while 55% come from other parts of the world. He also noted that Internet usage is growing particularly fast in Brazil, China, India and Russia. When you look at FOSS development, you see a different picture. He looked at the attendees of the Linux Kernel Summit 2007 and found that 51% were from the US, 32% from Europe, 9% from Australia and only 8% from other countries. He also presented some statistics from SourceForge that showed a similar picture: 36% of developers are from the US, 40% from Europe, 3% from Australia and 21% from other parts of the world. His conclusion was that it's important to get more people from Asia, Africa and other underrepresented regions involved.

His presentation followed with obstacles associated with the participation in an international community. Unsurprisingly, language is a big problem. Learning English can be hard for people from China and other countries whose native language is not part of the Indo-European family. While many educated people in China can actually read English at a basic level, written and spoken English is often poor because of lack of practice. This makes communication slow, misunderstandings are more common and it's difficult to express yourself clearly.

The problem is not just with language, though. Culture is also an important factor. For example, direct criticism is taken as an insult in China. This is a problem since the patch review process can be very direct. Furthermore, Li Yang argued that Chinese people are more effective in close-knit teams whereas the FOSS community prefers to form loosely-knit teams.

Time difference is another problem. For example, China has a 8-12 hour time difference to Europe and the US. This makes it hard to use real-time communication, like IRC. While e-mail works fairly well, communication is quite slow since each round of discussion takes a whole day. Finally, developers in China have less time for hobbies and so it's often hard to get more people involved. There are also some problems with net access. For example, sourceforge.net and freebsd.org used to be banned by the government and wikipedia.org is still banned.

Li Yang finished his presentation by sharing his experience with local communities. The idea is to establish a local community in a country with members who can talk to each other in their own language. A local community should also have some experts who are part of the international community. Those experts can then help other members with sending bug reports or patches to the international community for feedback. This way, it's ensured that the local community has at least some connections to the global community and hopefully over time members of the local community will get more experience interacting with the global community.

(Originally published on FOSSBazaar)

Models and tools for FOSS quality

There has been interest in quality models for FOSS for a long time. There are various concerns about FOSS and the quality thereof. Given that a lot of FOSS is produced by volunteers, how can we rely on the software? Is software developed in the public more secure, or can people use the source code to find exploits? It's important to have an objective assessment of the quality of a piece of software to address such questions. Furthermore, having good metrics allows users to choose between different software that offers comparable functionality. Given the large number of FOSS projects, this problem is of growing concern.

Software quality is a tough nut to crack. When you see and use a product, you will usually form a judgement as to its quality pretty quickly. However, if you try to develop rules for assessing the quality of a product you'll find that it's really hard. This is partly because there are so many different components that make up quality, and that different people put different emphasis on these components or see them in a different way. While quality has a subjective component, there are several objective components that can be measured.

There are number of researchers who are interested in developing tools and models that can be used for empirical studies of quality in FOSS. The EU has recognized the need for such models and tools and is funding not only one but several projects that study quality in FOSS:

• QUALity in Open Source Software (QUALOSS): QUALOSS will develop a high level methodology to benchmark the quality of FOSS in order to ease the strategic decision of integrating adequate FOSS components into software systems. The QUALOSS platform uses tools to analyze two types of data: source code and project-repository information. The goal is to estimate the evolvability and robustness of the evaluated software products.
• Software Quality Observatory for Open Source Software (SQO-OSS): This project is developing a comprehensive suite of software quality assessment tools. These tools will enable the objective analysis and benchmarking of FOSS. SQO-OSS aims to remove one of the key barriers to entry for FOSS by providing scientific proof of its quality.
• Qualification and Selection of Open Source Software (QSOS): QSOS is a method designed to qualify, select and compare free and open source software in an objective, traceable and argued way. It consists of four different steps.
• Quality Platform for Open Source Software (QualiPSo): QualiPSo is a huge project which focuses on a number of different areas. As part of their activities on trustworthy processes, they are interested in developing a Capability Maturity Model (CMM) like model for FOSS.

These projects are very ambitious but they certainly have the potential to make a great contribution. There is also quite a bit of overlap between these projects, which is why some of them have united and formed the Flossquality initiative (FLOSS stands for "free, libre and open source software"). These projects are relatively young, but I look forward to their results.

(Originally published on FOSSBazaar)

GCC 4.3 porting guide

Benjamin Kosnik of Red Hat put together a guide on the GCC web site with information on porting applications to the upcoming GCC 4.3 release. He integrated some information from my blog and added a number of new items. If you run into any issues while moving to GCC 4.3, I suggest you consult this guide.

Driving forces of FOSS adoption and development

Matt Asay of Alfresco recently observed differences in the adoption of FOSS. He sees a lot of adoption in Europe by governments who support the ideals of FOSS whereas he claims that adoption in North America is mainly driven by enterprises "seeking to lower costs and maximize innovation". His comments remind me of a discussion I once had with someone a few years ago. We didn't talk so much about adoption, but rather about development of FOSS. Our conclusion was that the main driving forces behind FOSS are not the same in different parts of the world. Specifically, it seemed that FOSS development in North America was heavily driven by corporations, whereas FOSS development in Europe was mostly community driven. Finally, governments in Asia and other emerging markets, like Brazil, saw FOSS as a great opportunity to establish a healthy software development industry in their own countries and as a way to become more independent from the US.

Today, this picture may be different and clearly there are multiple driving forces in a region. The EU in particular has been pushing FOSS in recent times for public administration and they are funding a number of research projects related to FOSS, such as FLOSSMETRICS, SQO-OSS and QualiPSo (more on these later). This may explain why Matt Asay is seeing so much adoption of FOSS by governments in Europe. Nevertheless, in terms of development I'd still say that the community is the driving force in Europe. I find the differences between the FOSS community in Europe and in the US quite amazing. It seems to me that the FOSS community is much more tightly connected in Europe than in the US. In Europe, we have a lot of community events for FOSS (both for users and developers). FOSDEM is one of the largest FOSS events in Europe, but almost every country in Europe has their own event (or even several) related to Linux and FOSS. Many of them are organized by the community and entrance is often free of charge. Contrast this to the US, which has much fewer events and the most prominent one, OSCON, sets you back around $1000. I'm aware of OLS and SCALE, and have heard many good things about the LinuxFest Northwest and Ohio LinuxFest, but it's still not comparable to what you can find in Europe.

Some countries in Asia are very keen on participating in the FOSS community but I don't see many individual contributions from there (with the major exception of Japan, which has a very strong FOSS community). However, governments are providing incentives for companies to get involved. Unfortunately, there are many obstacles. For example, the open way in which most FOSS projects are performed may be in conflict to how some Asian cultures work. Also, FOSS development is often quite hostile (with heated discussions and arguments), which may be a problem to some. Finally, proficiency of English and access to broadband Internet connections are a problem in some Asian countries. Nevertheless, it's important to engage with developers in Asia and to help them contribute to and join our world-wide community.

In summary, I think there are important differences with regards to the adoption of FOSS and the way development communities work in different parts of the world. These differences will have an important impact on the governance of FOSS. Since FOSS is a world-wide phenomenon, we need to develop a better understanding of issues that various people face. I think that FOSSBazaar will be a great venue to hear from people all around the world and to learn what the obstacles are to wide adoption of FOSS in their countries.

(Originally published on FOSSBazaar)