My Ideal Open Source Distribution

I've discussed what I'd want from an ideal operating and why it's so hard for me to switch to completely Open Source solutions. With the advent of operating systems like Metro (Windows 8), Windows also becomes a less than ideal solution. Curtailing backward compatibility and the ability to run legacy software is a great loss. However, virtual machines are quickly taking a place in solving the issue. My main issue, any system that locks down what applications you're allowed to install is quickly becoming an appliance not a computer. I have enough appliances such as hi fi systems, refrigerator, etc. I do not want yet another one. I love the ability of the computer to solve multiple problems and perform multiple tasks and give me the freedom to solve them myself if there's not a solution currently available.

I've been looking at quite a bit of Open Source code and also experimenting with what I could build myself and what I can't. After looking at what's currently out there for Open Source, I've come up with my Ideal Distribution. It doesn't currently exist, but I've tried to limit my expectations to what would be very feasible given enough time to develop it. I've looked at several Open Source distributions out there and none of them are anything like what I've come up with. Perhaps I'm the only one in the world who would like to see a solution like this implemented. If I'm not, I'd really appreciate hearing from you.

What follows is my vision of a feasible design for an Open Source distribution. I'd be very interested in brainstorming the ideas further with others. I don't want to replace the operating system itself. I do, however, want the flexibility to do the tasks I need on whatever machine I want to get my job done on. My ideal distribution would be a complement to any system already out there. It's not designed to fully replace an operating system or all the functionality you currently have with that operating system. My design priorities and goals are probably unlike most average users, so I think it's prudent to clarify some of them up front.

With all that in mind, let's take a look at the layers needed for the design. The first layer is the operating system and whatever basic support you need such as drivers, etc. One could use FreeBSD, a simple Debian install from the netinst CD, FreeDOS or whatever system desired as a base as long as necessary compilers and interpreters are available to build applications. The next layers are detailed below.

Sysbase Alpha

My original design idea was to use the basic system tools already available on a particular operating system. That doesn't work quite so well in practice. FreeBSD, Linux and Windows MinGW with msys all have versions of sed, but they don't all work quite the same. To be able to replicate building across multiple platforms, one needs consistent tools that work the same way. The design was created to allow for swapping one tool out for a similar tool. However, if there are too many differences between the tools performing the same function, the swap will fail. There needs to be at least one set of tools as a fallback that works the same on all systems.

Without looking into it further, going with the GNU core tools seemed like a good plan. Most Linux distributions do this. However, when I looked at the actual code and tried building the tools on systems other than POSIX (or even other than Linux), I quickly had a mess on my hands. I also do not like the way the GNU cp functions. If you use it in one particular way, it overwrites all your files. DOS copy has very similar functionality and doesn't cause this problem. Switching between the two systems, it's too easy to mistype and wipe out files with the GNU cp. I began to look for other solutions. The next most widely mentioned Open Source solution is Busybox. It creates one monolithic executable that can be given different program names via links. This saves space on an operating system that makes good use of links, but not on operating systems like DOS/FreeDOS. Also, for me, it seems to complicate the coding to have an entire world of functions and applications all in one set of source code. You have to dig through everything to find what you want. Other options, I looked at were Toybox (a BSD style licensed solution very similar to Busybox) and the Heirloom Project. The Heirloom Project seems to have the best UTF-8 support. It uses classic algorithms but rewrites common tools from the ground up with support for internationalization. The biggest issue I had with all these solutions was that I could not get them to build well on non-POSIX systems. I finally stumbled across similar views to mine when I found a thread from one of the Suckless.org mailing lists about sbase. Now here was a design that was similar to the goals I had in mind. Unfortunately, at the time, development had stalled. (I'm told the project has become active again.) Also, the project was again very much a POSIX implementation with no interest in support for other non-POSIX platforms. Sbase is under an MIT license, so here finally was a good base to start with.

Sbase did not provide everything one might need to replace the basic system tools, plus there was no UTF-8 support. I'm told this has now been added. So, I started scouring other sources such as PicoBSD, OpenBSD, Minix, ELKS, etc. for code that was listed as BSD or MIT licensed or in the public domain. I have a UTF-8 library in the works to add some internationalization support. I want to start with working code that's already out there and available, add in UTF-8 support and other functionality and eventually come up with a working set of tools to replace the base tools. Most of the tools I've located are under very Open licenses, but in a few cases such as for man page support or with an application like minised, the license is GPL. I've dubbed the collection of tools Sysbase Alpha. It references their sbase origin and also sort of references a sf show.

At one time, I thought that using GPL licensed software would not be an issue. All one has to do is supply the source that you link in with the GPL software and you comply with the license. I've heard some people compare the GNU GPL license to a virus and at the time, I never understood why. After building quite a few applications, the issues involved become much more evident. Linking to non-GPL code from a GPL licensed application is a lot more tricky and restrictive than I first believed. Just supplying the source isn't enough. For instance, many GPL applications need a special disclaimer in their license in order to use the openssl library (which has an older BSD license). There's a conflict between the two licenses and you cannot legally use openssl with a GPL licensed application without some type of exception added for it. I've seen some projects use libedit or other BSD licensed readline alternatives to avoid similar issues with the GNU licensed readline library. There's a mention at the Audacity wiki about using an alternative sample rate library with a LGPL license instead of a GPL licenced alternative because of legal issues with using commercial plugins if the GPL licensed library was linked into audacity. I think it's very unfortunate that with the limited resources we have for Open Source development, we're splitting those resources even further because of incompatibility of licenses. The way I see it, when building software and trying to comply properly with licenses, it's easier to go with a GNU system or with a BSD/MIT licensed system as much as possible. Mixing both in the same application can be a recipe for disaster. So, for my own personal needs, simply because I feel BSD, MIT and similar licenses have far less restrictions, my preference is to go that route whenever possible.

One major missing piece is the shell. I searched all over for a cross-platform shell and was unable to find one. Bash is a good default and most build tools work with it. However, bash builds well mainly in POSIX environments. Msys is using bash with a POSIX emulation layer similar to Cygwin. Djgpp supplies an older bash version that's but modified quite a bit to provide support for the DOS platform. I did eventually get a version of BASH to compile natively on MinGW using ideas from the djgpp solution and a native Windows NT port. However, compiling isn't everything. It wouldn't run because of memory issues. Looking at what other shell solutions were out there, I decided to approach it from the point of which shells provide vfork support. If a shell could work with vfork, it can definitely work with forks and it might be able to work with spawn in place of it. The Blackfin Linux distribution typically only has vfork support. On one of their mailing lists, I saw mention of what shells they were using. The most bash-like shell was msh (the Minix SHell). Busybox documents that they used msh as well at one point, but has since switched completely to other shells. PicoBSD has a version of msh available under a BSD license. I had already managed to get it to compile on MinGW and FreeBSD. I think msh would make a good starting point for a shell. My other option would be to write a shell completely from scratch. I do have the libraries available to do that in C++, but I don't feel familiar enough with the Bash syntax to completely recreate it. I've found a few BNF descriptions of Bash, but they don't seem very complete. I've been able to get msh to run applications on MinGW and to pipe information from one application to another. The biggest issue is UTF-8 support. There's code specifically in msh that appears to assume a character doesn't use the highest bit. It would need a rewrite to store parsing flags externally to the character strings it's trying to parse. I've been working on the rewrite and am it's almost complete. There are also some issues with saving environment information during forks. I think I have most of that worked out except for handling asynchronous forks such as when the user runs a command with an ampersand at the end of the command line. Basically, it needs work before it's functional enough to be a practical solution. However, I've put a few months into the effort and it's slowly getting to the point where I might be able to replace bash with it when building applications.

The plan is to use whatever native build system an application requires. My preference is for simple makefiles, but not all applications use them. If a makefile doesn't work, it's very simple to adjust it. If the gnu autotools fail, you can't even get to a makefile to work around the issue because it's never created. I was able to get the make program itself to bootstrap a build (build without a make tool available) in MinGW. I had to write my own shell script to do it, because the one that came with it failed in the msys (bash) environment. After several months, I've finally managed to bootstrap cmake to build in the msys environment, but there are several issues one may need to look out for when doing so. While I've used cmake on a Linux (ulibc) environment successfully, in the MinGW environment when you use the supplied executable, it appears to be looking for libraries and other dependencies in all the wrong places. I currently have over a two hundred build scripts that attempt to build applications in a repeatable manner across platforms and attempt to use the build tools those application were designed to work with. I've managed to get some applications building that haven't built in years and I've patched issues like memory bugs that I found annoying. However, I still have a long way to go.

Package Manager

I've looked at quite a number of package managers out there and haven't liked any of them. My personal preference is for portable apps that are easy to install and uninstall and can keep track of preference settings in a way that's easy to move or back up. Of the package managers out there that I investigated, the one I felt most comfortable with was the one used on Slackware systems. Taking ideas from the scripts used by Slackware systems, I've come up with my own build scripts. I've kept these as scripts rather than creating an executable, because I want to be able to change the code easily (without a recompile). I've factored out and consolidated common code for maintainability purposes. The lowest level of the design is scripted and does the basic building of applications, install and uninstall.

I've decided to use the Linux Filesystem Hierarchy (straight from the spec) as much as possible, since most Open Source applications attempt to honor this. However, my goal is portability, so I've added code to applications to be able to move them wherever I need (a subdirectory, another drive, flash drive or CD). I try to match the LFH file structure when possible. Similar to the way Cygwin and MinGW/msys move the LFH under a subdirectory, I've added the capability to move the entire file structure to another location (not necessarily the top level on the drive). This is sort of like what chroot does, but it's not hardware dependent.

The next level of the package manager design needs to be able to handle two things, installing software from other sources (not just building it yourself) and dealing with dependencies. I picture the issue as similar to set operations. Whether you want to install software from an online repository or from another person using the distribution, it would be nice to bring up a description of what's available on both systems side by side (similar to a directory difference tool that shows two directories side by side). This would give the opportunity to decide which of the other libraries and applications you want to install to your own system. The technique will work for sneaker-netting as well as for systems with fast Internet access. The tools at this level will display dependencies, differences between two versions of the distribution, information about the applications (what they do, licenses, etc.). My original intention was to write these tools in JavaScript so they could be web-based or local. However, I'm not finding enough support in the available JavaScript server side tools to do this. I'm now reconsidering using modified web browsers and writing the main logic in C++.

I do use a few of the freedesktop.org specifications such as their XDG_CONFIG_HOME option. However, I don't find the freedesktop.org requirements particularly suitable to non-POSIX operating systems. I don't even find them particularly suitable for lightweight desktops on POSIX systems. I've read some of the issues lightweight desktop developers have had with some of the standards. I'd like to build in support for easy to access and customizable menus, how-to directions for completing simple, common tasks and other easy ways access to documentation. I'm looking at doing a lot of this with shell scripts and modified web browsers. HTML is a great medium for documentation. It can be internationalized. It can also provide a highly customizable interface for starting applications if used appropriately.

Libraries

I've tried to stay mainly with applications that will build using lightweight cross-platform libraries. I've read some users prefer their applications to all have the same look and feel. That wasn't one of my design goals. I want applications that get the job done. I don't care how they look, but they should be easy to work with. If one wants a similar look and feel, one can stick with mostly one GUI library. For command line applications, front-ends can be written using modified web browsers (find one that uses your GUI library of choice). They can be used a lot like dialog, gtkdialog, zenity, yad, etc. to create GUI front-ends to commonly used command line applications.

Besides command line applications, I'm considering curses based applications. There are three versions of curses and I'd like to get these applications to build with all three depending on what system they're run on. There's ncurses on POSIX systems. There's also a recent port of ncurses to Windows. There's pdcurses on non-POSIX systems like Windows and DOS. Both of these run console based on their respective systems. One can build pdcurses on top of SDL on POSIX systems and there's a Windows port of PDCurses. These options let you run applications that aren't console based (although they have the same look and feel). Another very good cross-platform solution is SDL. It works on most systems even when X Windows doesn't. There are options to run SDL using directfb on Linux systems. It works on Win32 or X Windows. There's even some support for Wayland on Linux systems. I'm also looking into the option of running it using Nano-X. One potential problem with SDL is that most applications still run on the 1.2.x line and current development is now centered on the 2.0 line. That might require supporting two sets of SDL libraries and helper libraries for each. On the other hand, if applications aren't ported to 2.0, it might be possible to continue with just the earlier line. However, it's likely at least a few applications will be switched to 2.0 at some point and many will remain working with the older library. One other GUI library worth considering is FLTK. It runs on Win32 or X Windows. It also works with Nano-X and the XFDOS distribution is using it to port FLTK applications to FreeDOS. I'm having issues with FLTK running with X Windows on a Windows platform, but am still working on the issue. Another nice thing about the patched version of FLTK I'm using, there's some small support for changing skins via environment variable. The patches add two extra skins not available in the regular FLTK source.

It would be nice if all the functionality one needs could be covered by applications built with just these interfaces, but I think that's going to be unlikely. I am considering other options, but haven't come to any decisions yet. From a portability standpoint on POSIX systems, one would need to use LSB libraries for X, distribute a lightweight self-contained version of X or avoid X altogether. If Nano-X can provide the functionality to run a wide variety of applications, it might be a good solution even if it's not too actively developed today. One advantage for systems like Windows and DOS is that they don't have this type of portability issue. However, now that I have X working on Windows, if I decide to distribute applications that require X on a Windows system, I'll hit the same distributions issues I have with POSIX systems. Am considering distributing X along with the applications at this point and trying to make X as portable (similar to portable apps) as possible so it's not confined to particular directory locations. While I don't want to have to support a ton of different GUI libraries and toolkits, I still want to provide choices and enough software selection to get a job done.

Fox Toolkit is a lightweight GUI that works well on older systems. The main drawback is that there are currently two versions to contend with. The 1.7 version has current development support and has some support for internationalization. Most applications (except the ones that come as examples and FXite) appear to only work on the 1.6 version. I didn't see any information on porting guidelines and the project does not to appear as active as it once was. I like FXite as a possible lightweight alternative to SciTE, but I'd like to find a better solution. Textadept using PDCurses might fill in some the gap, but I'd still like to find some other options.

I tried Qt. There are plenty of good applications available for it. My main issue with it is that it hardcodes development locations which I find difficult to work with. It's also in the process of a major code change (going from version 4 to version 5). I can't help wondering how many applications will be left working after the switch. What I've read so far about porting wasn't good news. Many applications that use Qt also use cmake and I haven't had much luck with it to date. Some developers use a minimal Qt version for development. There's also a port of an older version of Qt to Nano-X. Plus, projects like wkhtmltopdf need to make modifications to Qt to get advanced functionality to work. If Qt were to become an option for my ideal distribution, I'd like to look at the possibility of consolidating a lightweight Qt port that worked on top of Nano-X and was still capable of running the applications I wanted most.

WxWidgets appears on the surface to be a great choice for a cross-platform GUI option. However, when I looked into it from a practical perspective, it didn't pan out as such. It works great on Win32 and has some very good applications written to use it. However, on the POSIX side, it's written to work on top of GTK which requires X. So, that's two layers beneath it and applications typically run slow on low resource machines. There are several alternative platform options such as for embedded systems. The port that has the most work completed (99%), running directly on top of X11, failed to run any of the applications I tried on it. So, either I'm only using applications that need something in the 1% not completed or these ports haven't been maintained in so long that they're really not as close to being useable as indicated. A possible option might be the Win32 version of WxWidgets on top of Nano-X's microwindows support. I haven't tried it yet, but I'm not sure it would work either.

Win32 would actually make a useful GUI library if there was better cross-platform support. I looked into using the Wine libraries, but they weren't very user friendly to build. The microwindows part of Nano-X provides nice support for a Win32 subset. Was able to get a simple Windows program to build and run. It doesn't appear to have good support for resource files. Although, it is supposed to have limited support with the makedlg program. Trying to get X applications to build and work on Windows outside of Cygwin is even harder. There is a standalone X server called Xming. What there is not is easy access to an X11 library. There's a fascinating discussion of an attempt to get Wine working to no avail under Windows at the Wine web site. There's no port of Nano-X to Windows. There's only a port of Nano-X to DOS. So, I can get an X11 based program like xfireworks to build and run in DOS, but not in native Windows at this point. It will run fine if built with the Cygwin libraries and run in conjunction with a Cygwin or Xming server though.

I've ruled GTK out mainly because of issues with the developers involved. They don't seem at all open to fixing things for non-POSIX systems and were not very polite about it either. It's also in the midst of transition from 2 to 3, so backward compatibility is an issue. On the other hand, I did see mention of a FL-GTK library that purportedly created a compatibility layer so that GTK+ 1 applications could run using FLTK. If that was available for use and actually worked, there are several GTK+ 1.2 applications out there, some still used by distributions that cater to older computers. It would provide a lot of useful applications to choose from. They'd need to be maintained and updated, but in essence all the applications need that to some extent anyway.

I've also ruled out the Enlightenment libraries because of issues with the developers involved. They only seem interested in supporting specific platforms and are not open to making their libraries work on other platforms. Applications would require building of several libraries before they'd work, so it's not exactly a lightweight solution. Another drawback is the lack of applications available using those libraries.

For more information on what applications run with which screen libraries, check out the Open Source Screen Libraries page.

What's Next

That's an overview of some of the design decisions I've been looking at when brainstorming the creation of my Ideal Open Source Distribution. I've been working on this for quite a while now in my free time and will continue to do so. Even if the final distribution doesn't see the light of day, I've learned about many new and useful applications and I've learned a lot about the code behind the scenes. I would really like to make a distribution of this nature a reality. So, I have to ask, is there anyone else out there that would be interested in something of this nature? If so, if you'd like to volunteer to help make it a reality, let me know. It would be very helpful to have others with similar needs and goals to help brainstorm ideas and solutions and to find useful applications and create this particular vision of an ideal Open Source distribution.

 

To the main page.

 

Validate XHTML



The information on these pages is copyrighted by the author with all rights reserved. Reproduction of anything without the author's permission is in violation of copyright laws.
All original material is copyrighted:
(c) Copyright 2012, 2015 by Laura Michaels
All Rights Reserved
Last Update: 20150927