Yesterday, Chuck Reeve’s tweeted an article from Daniel Lemire’s blog entitled Who will need database administrators in 2020?. The thesis is that with the advent of all these NoSQL technologies, the role of DBA will become unnecessary. I disagree with this for two reasons. First of all, NoSQL will not replace SQL. Secondly, your NoSQL data store probably needs a DBA, even if he has a different title.

Just a quick note, I’ve worked with enough SQL databases to make broad generalizations about them. The only NoSQL database I have experience with is MongoDB.

NoSQL will not completely replace RDBMSes

SQL databases are the primary practical implementation of the relational model. Most of the “trade school” explanations of the relational model and normal forms use SQL syntax as an example. Relational databases are great at storing data in an organized fashion. Through constraints you can enforce most business rules. Triggers will allow you to do the rest. Relational databases also usually have fine grained access control systems, and mechanisms for auditing changes. Finally, if you have to build a report from your data in a way you never did before or planned to, its usually nice to be able to start out with your data normalized.

Now there are a lot of things a NoSQL database like MongoDB does better than most RDBMSes. For example, MongoDB would be better suited for hosting a simple blog than MySQL. However, MongoDB has not been around all that long, and before MongoDB, relational databases did a good enough job. Now there are many things SQL is better at than MongoDB. For example, I would never use mongo for a complex inventory system. However, many technologists, like Daniel, have been focused on thing that NoSQL is good at, like blogs and simple ecommerce sites. These technologists recognize NoSQL as disruptive technology in the data management field. However, they make the mistake of assuming NoSQL will usurp the role of relational databases completely.

To put it another way, in the brave new MaybeSQL future, we will use SQL for some things and NoSQL for others. The things we will use SQL for, like complex inventory systems, will have complex schemas and need specialists to manage all that data. We already call those specialists DBAs.

Your NoSQL Database needs a DBA

Ok I lied. Your NoSQL database might not need a DBA, just like your relational database might not need one. In relational database shops without formal DBA positions,  there are usually defacto DBAs, senior developers who’ve made it their business to manage the companies databases because management would not allocate a dedicated salary to that function. Currently, I am serving as a defacto DBA for some small databases.

Now I’ve also been playing with MongoDB a lot. I’ve contributed to mongo, spoke about mongo, and been to three mongo conferences. I’ve talked to a lot of people using mongo, and I’ve made a lot of observations. My primary observation is mongo tends to get used in startups. These startups don’t have dedicated DBAs. However, they do have well rounded senior developers that perform DBA and sysadmin functions. Many of these NoSQL programmers also know more about relational databases than I do, which is why they didn’t fight “the mongo way” tooth and nail before accepting it like I did. Now as is the nature of startups, most of the businesses these programmers work at will fail. However, a few will succeed and get big enough to have to hire technologists with more specialized roles. I expect to see a mongo specialists role thats part sysadmin and part programmer evolving at these companies. For companies that use a a combination of a relational databases and MongoDB, I expect a DBA to be hired, learn MongoDB, and take ownership of managing the data stored in that companies MongoDB instances.

Conclusion

NoSQL databases were designed for different problems than relational databases. Relational databases were not designed for things like blogs and massive sites lie facebook. They were used for this role because they were the best tool at the time for the job. MongoDB on the other hand was founded by a founder of doubleclick, who wanted to build a database that scaled the way a database for websites should scale. MongoDB is taking a piece of the pie from relational databases, but not all of it. Also, just like not all relational databases have a full time DBA to maintain them, not all NoSQL databases have a full time administrator. However, that does not mean that a role similar to DBA for NoSQL databases is unnecessary.

Paul Randal wrote a What three events brought you here article on his blog and asked some others in the SQL server community to do the same. I of course was not on that list. However, no one ever invited me to blog about anything before so why should I be shy now.

I have decided to not tell this story in terms of professional accomplishments. For me, my hobbies have always led my career.

Event 1: A trip to Borders

The year was 1999. At 18 years old I spent my last summer before college doing contract work at 590 Madison Avenue, a.k.a. The IBM building. Of course Lou Gerstner had already sold the IBM building. I had heard of this linux thing. Deciding that I needed something better than Windows 95 (read: wanting to be more 1337), I bought myself a linux book and Mandrake Linux. I only had a 14.4k modem at home, and had no way of downloading an ISO easily.

Try as I might, I could not get linux working on that computer. Sure it installed, but no modem, no printer and no X11. That September I brought the machine to college and bough a NIC that proudly claimed to be linux compatible. Of course it required a special patched version of the tulip driver that I could not get compiled against the kernel. Someone told me about the even more awesome FreeBSD which actually worked with my NIC. Eventually, a version of the 2.4 kernel was released that supported my NIC out of the box.

Event 1 Epilogue

In college I learned a lot more about linux than I did in the month previous. All I really managed to learn in the few weeks on my own in my parents basement was how to use vi, and what sort of hardware tended to not be compatible with linux (modems, printers and damn near everything). However, had I not struggled those weeks on my own, I would have never sought acceptance with the other unix geeks at college. That trip to borders changed the course of my time in college.

Event 2: Learning SQL from Stan in a bowling alley.

I would like to note that I never finished college. This part of the story takes place in the time period after I stopped going to school full time but while I was taking classes at what one would call a junior college.

So I am sitting one Saturday at my local bowling alley with and old Toshiba laptop. I believe it was a 133mhz pentium. This was probably 2002 so it was old at the time. At the time I knew half the employees and my brother was on a league there. So I spent Saturdays at the bowling alley, mostly because I could get free food.

I would hack at perl on the machine. It was running Wndows 95 with an ActiveState Perl install. I don’t think any of my perl programs ever did anything useful, but I had fun writing them. For whatever reason, the founder of this blog Stan, who worked the cash register at the time, decided to teach me SQL. It was relevant to whatever project I was working on at the time. I had recently finished a course in VB6 at this point in which a lesson at the end involved talking to an Access Database but no SQL was involved.

I ended up not using this knowledge until I began taking a SQL class that was taught with Microsoft Access. Because I had a basic understanding of SQL from Stan, and also had a clerical job at the time, I took to Access like a semi-employed, quasi-college dropout to free chicken fingers.

Event 2 Epilogue

Once again this was a catalyst event. Had I not learned SQL from Stan I might not have taken that MS Access course, and I might not have turned a clerical job into a development job.

Event 3: My first LIPHP meeting

March 2006 and I am back in a semi-employed state, this time by choice. I walked away from a great unix administration position to take a programming contract and try to start a website. I didn’t take much of a paycut, and the next full time job I got was a big raise. However, this is not about my professional accomplishments.

So there I was programming on the side for money and the rest of the time for room, board, and potential millions. Unfortunately I was the only programmer in this boiler room startup. I had no one to talk about computers with. I was programming PHP at the time and became aware of a PHP user group in Suffolk county. So I traveled one Monday night from my bedroom/office in Manhattan to an office in Suffolk county less than a mile from the job I walked away from.

I don’t remember the topic, but it was a good time with good people. It was a great first experience at LIPHP, a group that I eventually became the organizer of. I had been to one user group before, a LILUG meeting in 2001. However, it was not something I saw any value in at the time. I will note that I am an occasional LILUG attendee, these days and the meetings in recent years have been very good.

Event 3 Epilogue

Its not easy to qualify the impact of LIPHP. It never got me a job. It did get me jobs and contracting positions I turned down. However, it has lead me to more user groups, more mailing lists, and some networking. I learned many things from the group. I learned about giving presentations, running a user group. Most importantly I learned that its not that hard to get outside speakers to your user group.

I do not consider myself an expert on Microsoft SQL server. I prefer Postgresql. However, I am a fan of referential integrity and seek to enforce it at the database level, regardless of the engine. I like my databases to reject invalid data like DMV clerks rejecting incorrectly filled out forms.

When I work with Microsoft SQL server I struggle to enforce referential integrity in the same manner as I do with Postgres. Recently, I wanted to restrict the values that may be stored in a column in a table. A mapping table was not the answer in this case. I thought the answer was to user T-SQL RULES. I implemented a few and prepared to add them to my standard practices. However, about a week later I discovered this nugget on the MSDN:

CREATE RULE will be removed in a future version of Microsoft SQL Server. Avoid using CREATE RULE in new development work, and plan to modify applications that currently use it. We recommend that you use check constraints instead. Check constraints are created by using the CHECK keyword of CREATE TABLE or ALTER TABLE. For more information, see CHECK Constraints.

All of a sudden I needed to unlearn what I had learned. I do this on a regular basis. I consider myself quite stubborn. And I quite value this stubbornness. I will argue my position to the death if I believe it is right. I pride myself on caring about the things no one else cares about, and being able to explain why you should care, if necessary. However, I can be persuaded to change my mind. It’s not so much that I bend or break, it’s more like I recrystallize. Sometimes the transition is instantaneous. Sometime my faith in a concept is broken and I spend a while looking for a replacement. This transition was rather instantaneous as I was presented with a clear message: “What you are using is depreciated and doing it this other way will work better anyway.”

So what is the lesson here? The lesson is simple. Be unwavering in your beliefs, until you have a good reason to change them. Then be unwavering in the new beliefs until they are demonstratively proven inferior.