Open Source Pet Peeve No. 1: We don’t do formal releases

This is hardly a problem specific to the open source community. This happens to webapps whenever developers have access to the production database. However, I am picking on open source productions here, because as a user I am shielded from a web apps release process unless I catch a maintance screen. For closed source products that run on my desktop or my server, you have to at least give me a zip file with compiled binaries so your forced to do a formal release of some sort.

A specific example of where this occurs is ffmpeg. Apparently they do the occasional formal release, but highly recommend that you grab the latest svn repo and build that. Having used ffmpeg to process video files uploaded to a website, and of course transform them to flv files in a manner much like youtube, I have some experience dealing with this issue.

Basically, ffmpeg is the netcat of  digital A/V. If you want to take a stream of A/V data from one source and tranform it, this is the program to use. Generally this means converting video from one format to another, but so is video capture, changing resolution, and separating audio and video streams.

The only small issue is as their download page says:

FFmpeg has always been a very experimental and developer-driven project. It is a key component in many multimedia projects and has new features added constantly. New, official “releases” are few and far between. In short, if you want to work with FFmpeg, you are advised to go along with SVN development rather than relying on formal releases. SVN snapshots work really well 99% of the time so people are not afraid to use them.

Basically, they don’t have time to do formal releases but generally speakin, the code from their svn repo will build and run. I must say that after getting the code to run the first time, that statement is generally true.

Lets deal with my personal  experiences before I pontificate on potential problem for other people. Generally speaking, updating svn to the head revision and doing the usual unix ./configure –a-bunch-of-configure-arguments && make && make install gave me working binaries. Sometimes I had to twiddle with configure options as some features, including support for specific file formats, would break. Being my goal was to convert from any possibly format to flv, I reasonably expected support for certain esoteric formats to wax and wane, even if formal releases were performed. So other than the problem of having to pick specific svn revisions to move from the test sever to production, this posed no practical issues to me at first glance.

Now had I been a unix admin that did not program myself, I might have been unfamiliar with svn.  This is alleviated by nightly snapshots of the repo. Their is plenty of documentation on building ffmpeg, and the distro’s use older formal releases. So, the average user will be able to easily get a ffmpeg binary that will convert from video in a common format to flv to display on their website if they want to host video without involving youtube.

However, in all of these tasks are hidden wastes. If ffmpeg had formal releases that just worked, I’d probably never attempted to recompile it, or join the mailing list. I would probably not document the process on the ConchShell wiki, or a more detailed document in my companies internal wiki dealing with the specific intricacies of ffmpeg on our setup.

Also, I believe if the developers invested time in formal releases, they would spend less time on support issues.  The mailing list was very high traffic. Many people asked questions and few people got answers. I felt I asked “proper” questions on the list, yet I rarely got answers. If the developers did formal releases, and managed to reduce the number of support requests as a result, they might be able to spend more time actively listening to their users instead of actively ignoring them. This might have led to a better product. They also might be able to tap into the talents of their power users and centralize some of the documentation scattered across various peoples personal home pages.

Now I was starting a website where encoding videos to flv and playing them back was a primary function of the site. So it is very likely I would have become involved in the mailing lists eventually. However, if I could have afforded more time to non ffmpeg tasks initially, the business venture might have afforded me the success to have the need, and resources to sponsor ffmpeg development.

I’m not blaming ffmpeg for the failure of that project, just illustrating the point that I spent much more time dealing with ffmpeg then i really needed to. When I got down to encoding the video I was basically having ffmpeg autodetect the codec of the file that was uploaded and convert it to a flv. I was happy enough that ffmpeg supported many codecs including ones most people haven’t heard of. Had the CentOS rpm been anywhere near up to date, I would have been happy with using that initially. Had the mailing list been useful to me, I would probably spend more time on it, but wouldn’t complain about it.

Bad journalism in the Wired article on Sun buying MySQL

Its hard for me to write an article like this without sounding trite. I’m sure I make many journalistic mistakes in this fine publication. I can’t say “well wired should know better,” since that is tantamount to saying, “I admit no one should read what I say.”

That being said I really think that the wired article on Sun buying MySQL had one really bad flaw. The either misunderstand or misrepresent of facts.

Even with Ruby on Rails and Django gaining in popularity, MySQL remains a popular and common choice of database for today’s startups.

Ruby on Rails and Django are frameworks that can use MySQL or several other databases to store their data. If anything, they would help MySQL gain popularity as many people use MySQL as the database for applications that use that framework. Non technical readers would get the impression that Ruby on Rails would replace MySQL as opposed to complementing it.

The importance of changing ones mind

I do not consider myself an expert on Microsoft SQL server. I prefer Postgresql. However, I am a fan of referential integrity and seek to enforce it at the database level, regardless of the engine. I like my databases to reject invalid data like DMV clerks rejecting incorrectly filled out forms.

When I work with Microsoft SQL server I struggle to enforce referential integrity in the same manner as I do with Postgres. Recently, I wanted to restrict the values that may be stored in a column in a table. A mapping table was not the answer in this case. I thought the answer was to user T-SQL RULES. I implemented a few and prepared to add them to my standard practices. However, about a week later I discovered this nugget on the MSDN:

CREATE RULE will be removed in a future version of Microsoft SQL Server. Avoid using CREATE RULE in new development work, and plan to modify applications that currently use it. We recommend that you use check constraints instead. Check constraints are created by using the CHECK keyword of CREATE TABLE or ALTER TABLE. For more information, see CHECK Constraints.

All of a sudden I needed to unlearn what I had learned. I do this on a regular basis. I consider myself quite stubborn. And I quite value this stubbornness. I will argue my position to the death if I believe it is right. I pride myself on caring about the things no one else cares about, and being able to explain why you should care, if necessary. However, I can be persuaded to change my mind. It’s not so much that I bend or break, it’s more like I recrystallize. Sometimes the transition is instantaneous. Sometime my faith in a concept is broken and I spend a while looking for a replacement. This transition was rather instantaneous as I was presented with a clear message: “What you are using is depreciated and doing it this other way will work better anyway.”

So what is the lesson here? The lesson is simple. Be unwavering in your beliefs, until you have a good reason to change them. Then be unwavering in the new beliefs until they are demonstratively proven inferior.

If you have too much free time, use Java.

Now, I am no superstar programmer, this I admit. However, I do believe I am fairly intelligent, and have a decent breadth of knowledge and ability when it comes not just to programming, but IT as a whole.

This weekend, I decided to spend a little time attempting to re-invent an intranet app I created using Java in Applet form, as opposed to the present PHP/Ajax solution. To provide a little background, the present application is basically an administrative tool which keeps track of computers & users. It presents several tables identifying e-mail accounts, Southware users, PCs, and the relationships of these various items to each other. For now, I use PHP5 with Smarty templating, and use XMLHttpRequest calls to update just certain portions of the page instead of reloading the whole thing each time you want to sort a table, or edit a field.

I am pretty pleasd with the solution thus far, however I like to experiment. Previously, I tried using Flex — and did OK until the point that I hit what I believe was a limitation of Flex itself, having to do with binding controls to the gridview and having changes propagate back to the control that the edit controls were bound to. So this time around, I said to myself “Hey, I took Java back in college and even did my senior project in that!”.

Can’t be too hard right? Taking some data, displaying it on a grid, edits, etc. Boy was I wrong!

CHALLENGE #1 — Finding a decent IDE

Good luck! I tried probably 5 different IDEs, and ditched them all for various reasons — mostly because they weren’t free. Finally I ended up with Netbeans, which I happened upon completely by accident, as I was browsing around Sun’s website. It’s actually a pretty nice IDE. It has several built in project templates, a drag & drop WYSIWYG designer, code completion, etc. This IDE also had a cool feature where you can easily set up a JDBC connection to several database servers, in my case PostgreSQL. I was able to connect, browse around the tables (however, you must have a primary key before any tables can be used!?), and bind the tables to various controls for example a list or a data grid. I had some problems with the IDE though — I had a tough time wrestling for control. It wouldn’t let me delete function declarations, variables, etc.. They were just simply a different shade of gray, and somehow untouchable.

CHALLENGE #2 — Getting anything done

Once I had finally gotten everything set up with my little project, I tried to compile it. UHOH it seems it couldn’t find javax.persistence. WTF is that you may ask? Well, apparently it’s necessary (at least with this IDE) to bind the database to the control. After another half an hour of searching, I resigned myself to having to download the enterprise edition of the SDK (100+ megs..) and then trying again. After that was finally done, I tried to run it again… It compiled, but now it was giving exceptions on start, and it STILL wouldn’t work..

CHALLENGE #3 — Debugging!

There may be a nice quick solution for this (if there is, please let me know). If you’re writing an applet you should be debugging in your browser right? Well if you do, then your browser will cache the Applet, and your subsequent rebuilds will not be represented.. I am using firefox, and to counter this, it is suggested you load up your Tools -> Java Console, and hit ‘x’ to clear the java cache.. So I did it. And did it again. And again and again. NO EFFECT! I discovered I had to exit firefox in its entirety after clearing the cache, then re-launching, and THEN it would show the new applet. SO, what kind of way to debug is that?? As I said above, i’m no superstar.. I like to code a little, then check it. Then code, then check. I just can’t work like that. Yes, you can run the program as a regular application then switch to be an applet, but then you have various security permissions issues which may come back to bite you if you’re not careful no?

The above may not seem particularly frustrating, but all of this came after several hours of playing around with SOAP. I had gotten one of the functions of my app set up via SOAP, and was going to test that with java. I was looking around the web, and all the samples I saw were horrible… They were hand-crafting the request XML, OR using some 3rd party framework to get the job done. I didn’t want to do either of those things, so eventually I gave up on SOAP, and that is when I tried the direct database connection.

In closing, those of you out there who are PROs at Java will read this and laugh — no problem there. What should be taken away from this though, is that while Java may be perfectly good once you immerse yourself in the intricacies, it is not for the faint of heart, and in the age of integration, polished languages & IDEs, it’s just not up to the hype.

These are a Few of My Favorite Open Source Sites

I’m a big fan of sites about open source. Especially those that categorize, index, and report information about them. I’ve probably spent more time marketing my pet Open Source project PlaneDisaster.NET on these sites then writing code for it. So I’ve compiled this list of sites I use.

SourceForge (http://www.sourceforge.net)

The true power of open source is allowing anyone to contribute your project. In order to have a successful open source projects, you need to build a community where users and contributors can collaborate. This means a source code repository, forums, bug tracking, a wiki, project webspace, and of course a system of mirrors to download project releases. One place that allows you to do all that for free is SourceForge. In recent years new players have entered the market including Microsoft CodePlex and Google Code. However, SourceForge is still the most popular.

Freshmeat.NET (http://www.freshmeat.net)

While Sourceforge will host anyones project, few released are handpicked to appear on their frontpage. The site that fills that void is FreshMeat. If you release a new version of your software, you can notify FreshMeat, and the announcement will appear on the SourceForge home page. Unfortunately for myself, it does not accept windows only software. They are a site that lists unix software, not open source software. This means that they will list closed source souftware, but not windows only open source software. Despite this policy, they will list open source operating systems, and you can specify that your software will work on windows if it also happens to have a version that works on some unix variant. Finally there is a special category for OSX software.

ohloh (http://www.ohloh.net)

Ohloh is a unique site. It is a combination of social networking and open source software metrics. If you write open source software you can list it on the site, and have it scan your version control repository. It will report metrics about your software. It will also generate metrics about the lines of code you write across all open source projects on the site. You can also list software you use, and give other users on the site “kudos” if you enjoy their work. Finally, all these metrics are used to sort every user on the site by a single ranking system. The exact formula is a secret like those used to calculate credit scores.

koders (http://www.koders.com)

A simple code search engine. I can honestly say I have yet to find source code with the site that I’ve used. However, I think the idea has potential. While there are better forms of code reuse such as static and shared libraries, sometimes you need a small snippet of code.

osalt Open Source as Alternative (http://www.osalt.com/)

This is a strange site. The way the site is supposed to work is that you select a piece of commercial software listed on the site and it lists equivalent open source software. While they have a form on the site for suggesting software there site, the content is highly editorialized. The content on most other sites on this list is more directly user generated. I think a more open format would lead to a site with more information. However, until a more open competing site opens up, this is best in breed by virtual of being the only one of its breed.

Conclusion

These are my favorite sites. Feel free to list yours in the comments. If I get enough feedback I will post a follow up article.

Microsoft and the OLPC - Red Tape and Misconceptions

It would appear that Microsoft is making significant progress in porting XP to the XO laptop. While I think the third world is better off learning Linux, I have no problem with them learning on a Microsoft platform, provided they make it suitable for their needs. They seem to be attempting to do that. I can’t say for sure that the OLPC is what the third world needs regardless of operating system, therefore I really can’t make a judgment as to which OS better serves said undefined needs. However, this is not a post to judge the merits of the OLPC or any OS that would run on it. Rather, its about red tape and misconceptions.

First the red tape. The technet article states that Microsoft cannot contribute to the OLPC project directly because of the open source license. It then seems to imply that Microsoft must reverse engineer the hardware because their engineers would somehow be tainted if they read the specs. I am quite ignorant of the OLPC project licenses, as well as open source hardware in general. So I have to assume that their lawyers are correct in their interpretation of whatever licenses govern the spec of the hardware.

This seems to be quite a hindrance. Granted the hindrance might be intentional. It seems to be similar to the hindrance that the GPL places on you reusing the code in a non open source project. I would expect many people that have worked on the OLPC project to be glad of this hindrance, just as many are glad of the hindrance that the GPL provides to their code being reused in a closed source project. However, many companies such as Sun and IBM are able to donate their programmers time and talents to Open Office, Apache, Firefox and other open source projects without “tainting” them. I think Microsoft could do similar. Even if it was impossible not to “taint” there developer, they are not in the business of making hardware or drivers. I doubt it would hurt Microsoft if the drivers they created for the custom XO hardware were all released as open source.

Now to tackle the misconception. James Utzschneider, the author of the technet blog, seems to think that the projects plan for supporting the laptop is to have the children patch the source code themselves.

And we have a different support model than OLPC is envisioning: we are not expecting K-6 school children to access the source code and do their own programming in the event they have to fix a problem in the computer. Certainly, we think there is a role for students in the support of school computers — in fact, as part of our Partners in Learningprogram we have trained over a million kids in a student helpdesk program (like in this case study from Brazil) — but we also think that local entrepreneurs and businesses need to play an important role here when you are talking about deployments involving tens of thousands of computers.

The angry young idealist in me started screaming of FUD immediately when I read this one. Just because the source is available, doesn’t necessarily mean you have to edit it. There has got to be a better support model than this. So I looked on the OLPC wiki page linked to by the article for more information and I found the following.

The laptop is an open-source machine: free software gives children the opportunity to fully own the machine in every sense. While we don’t expect every child to become a programmer, we don’t want any ceiling imposed on those children who choose to modify their machines. We are using open document formats for much the same reason: transparency is empowering. The children—and their teachers—will have the freedom to reshape, reinvent, and reapply their software, hardware, and content.

The article contains no explicit statement of their support model. It does allow for students and teachers to modify the laptop, but acknowledges that not every child will become a programmer. A cursory search does not turn up a formal support model. However, very few support issues require a code change, especially for commodity software that have a much larger usage in testing scenarios than most code gets in production. So while the support model may be lacking, I don’t see where they assume the children will be hacking their own laptops except as a learning exercise.

I honestly hope this port of XP is successful, and that they ship some OLPCs running Windows. First of all it gives people more choices. Secondly, I’d like to see that translate in to a first world consumer oriented $200 laptop.

Comparing the Ohloh.net to Koders.com

When I start or join a new open source project, I usually make sure they are registered on several websites relating to open source that I frequent. Two of them are http://koders.com and http://ohloh.net. These sites offer different services. However, they both take the code from your version control repository, download it periodically, and report information derived from that data on their website. What the derived information is different.

Koders is a source code search engine. It lets you search for source code. Ohloh is a website that calculates and reports various metrics about open source projects. While you cannot compare the services they offer directly, you can compare how their project submission and user interactions work.

The first thing you will notice if you register with both sites is that ohloh is totally automated and koders reviews all projects individually. Ohloh will let you register in minutes. If you are lucky and server load is low enough, your project will be imported in less than an hour and you will have what it calls “shallow metrics,” or statistics about your code such as code to comment ratios and the breakdown of languages used in it. Koders, on the other hand, will inform you that your project must be reviewed by hand. My personal experience is you will have to wait over a month to be informed that your project was approved. They claim the purpose of the review is to ensure the code is legitimately open source.

Feedback mechanisms are similar for both sites. They both have forums and the site admins read and respond to support requests on them. Both sites are small in terms of forum volume at the moment. I’ve used Ohloh’s forums at various time, but never had a need to use Koders’. I predict Ohloh’s forum volume will grow faster than Koders’ simply because Ohloh is a community and Koders is a search engine. There are various activities that you can do at Ohloh, such as review projects, give other users “kudos” and updaste your “stack”, a list of open source applications you use. However, at koders, your options are to register a project or search source code. Therefore, there is more to talk about on the Ohloh forums than the Koders ones.

Another thing you will notice about Ohloh is the extensive use of the honors system. For example, I can login, find a project with a developer not associated with an Ohloh account, and click on the button that says “I am this user.” I’ve not tried to see what happens if I claim to be Linus Torvalds, but I do know if click on that developer profile for linux or git I see the “I am this user” button. There is no honors system or trust in the community at Koders. They hand reviews projects, and once the project is in the system, its in the system.

One final thing I noticed about Koders is unlike Ohloh, they seem to seek open source project to add to the repository themselves. In a blog article they proudly announce they have “partnered” with IBM to add all their developer works projects to the repository. Ohloh is using crowdsourcing to get all their data, where koders uses it as one avenue, but also seeks out projects on their own.

To put all this in perspective we must look at each websites motivation, or to more clearly define it, business model. Ohloh is building a community. It is a social networking site of sorts. I expect either ads to start appearing, the site to start selling aggregate data, or some other income source that basically revolves around selling their demographic. Koders, on the other hand, sells a professional version of their product. You can buy Koders Pro Edition, install it on your network, point it at your repositories and search your internal code. Ohloh needs active members that constantly go to the site, so they offer things to do there. Koders, on the other hand, needs people to find their search engine useful enough to want to use it internally, so they would want all the source code to be legitimately under the license it is listed under and would want to make sure that the majority of the source code on their site is worth reading and reusing.

What Qualifies an Open Source Project as Successful?

The title is actually deceiving. I’m really going to talk about my pet open source project. I will talk about its low download rate and the lack of feedback it gets. I will then discuss if that makes it successful.

My pet open source project is called PlaneDisaster.NET. It is a SQL frontend for Microsoft Access and SQLite databases. Basically it allows you to query and manipulate Microsoft Access databases through the JetSQL engine. I wrote it originally to teach myself C#, and then eventually began to rely on it extensively as part of a contracting job where my data was being stored in an access database.

So how can I classify this project as unsuccessful. First of all the project gets little traffic and even less feedback. The most downloads I got in a calendar month was 234 in March of 2007, and there were 2 trouble tickets opened about it anonymously that I didn’t open myself. So obviously the program gets used. However, I have yet to have anyone testify that, “Yes I downloaded your project and I use it regularly.” People I know have downloaded the program and given me feedback on it, but none of them have any actual use for the program.

There is some other data supporting the conclusion that some people actually use my project. A similar console based project called JetSQLConsole gets far fewer downloads on SourceForge than my project does. There is one person that gets notified when I post a new release on SourceForge. Traffic jumps a bit when I make a new release. So in all likelihood the project has limited success in terms of being utilitarian to others, but fails miserably in giving the author a sense of benefiting others.

There are also other measures that I can qualify this open source project as successful..  This was originally my “teach myself C#” project. I have succeeded in that task. It later became my “teach myself NSIS” and “teach myself WiX” project when I needed to learn how to write an installer.  It was immensely successful in those regards. It has also been useful for when I has to deal with Microsoft Access databases. So it has been of great academic value and utilitarian value to myself.

The conclusion I’ve come to is that you cannot define the success of an open source project by a single set of benchmarks. It depends on many factors. First and foremost is what purpose the software serves. Like  closed source software, some programs serve a very small audience, and some could benefit most users if installed on their workstations. Some are designed by the programmer for their own personal use and released as open source in the unlikely event others will benefit.

As one final postscript, I hope my readership does not discount JetSQLConsole if they need a SQL editor for Microsoft Access Databases. It has a lot of potential. I have submitted a patch to it a while back that is part of the current binary release. Some people might find a command line based program more useful. With a bit more work the program could become serious competition to PlaneDisaster.NET.

Getting the Attention of Redmond.

One of my more insignificant accomplishments as a software developer is discovering an digression between the actual error code two Windows API functions returned and the ones the MSDN claimed they returned. These two functions are NetUserGetGroups and NetUserGetLocalGroups. It took Microsoft three months and one day to get back to me with there response informing me of the changes made and thanking me for my feedback. Also, towards the bottom of there response was the following text.

A recent change on the MSDN Library has ended our ability to provide the
e-mail based feedback system you have been using for the past few years.
In the future, please take advantage of the two new mechanisms provided
by MSDN:

(1) Ratings and Feedback - a feature that pops-up when you hover near
the stars in the upper right hand corner of the Windows SDK
documentation pages on MSDN.

(2) Community Content - a feature added to the bottom of each page to
allow users registered with MSDN using a LiveId to ask questions and add
content

We invite you to use the Ratings and Feedback features to comment on and
give your rating for the topic itself. Please use the Community Content
option to ask questions and add your own developer guidance on the
topic. And continue to use the MSDN Forums for developer questions. We
appreciate your patience with this change to our system and very much
want your continued feedback, suggestions, and critique of our developer
content.

When I originally sent the bug report in there was no Community Content section at the bottom of every API doc page on MSDN. However, it was there when I received my response. Normally one would think that their response queue was backed up, and as a result it takes 3 months for them to get to these minor issues. However, it should be noted that on March 25th, the day before I received my response, I sent an email to the WiX users mailing list about some issues with their documentation and received a response the next day from Microsoft Technical Evangelist Bob Arson.

Many of you will say its coincidence. However, personally I feel Bob pulled the Microsoft equivalent of my FBI file to see if I had any negative marks in the eyes of Microsoft. After seeing this was not the case he noticed I sent an as yet unanswered email to the MSDN SDK feedback team, and he made a phone call to bump that to the top of the queue.

While I don’t believe Microsoft has a file on me, I do think that Bob or another Redmond employee could easily search for all correspondence I’ve had with Microsoft’s feedback mechanisms under a given email address. I know someone that that works for Microsoft. He has found bugs in products his team is not responsible for. Since he is a project manager, he can file a bug report internally and assign it directly to the appropriate programmer. The bug is usually closed unresolved and he is requested to file the bug through the public feedback mechanisms. This tells me two things. The first is their programmers don’t enjoy getting bug reports and hide behind bureaucracy. The second is that there is a unified bug tracking system at Microsoft. Assuming that public feedback mechanisms all filter into this, it would make sense that an appropriately entitled person could search for all tickets that reference my email address. If follows of course I assume that Bob was appropriately entitled.

So tell me folks, does anyone have any experiences with Microsoft Feedback teams that support or detract from my theory? Leave a comment because I would love to hear from you.

Just an Idea

Things always start out with ideas. Webbloging is a new idea to me.
As programmers we are all surrounded by ideas.

But how do you know an idea is good? How do you know if your idea is good?
From what I’ve learned the only way to find out is to share.

So I rounded some friends and decided to start a place to let our ideas be shared. No matter how eccentric.
Welcome to the site.