I recently received a press release from UnboundID announcing the availability of a new “synchronization server”. This software keeps two LDAP servers in sync (as the name suggests) – bidirectionally. In theory very useful, and it’s free too. But there’s a small trick: the synchronization server supports both Sun’s DSEE (Directory Server Enterprise Edition) and the new Unbound ID Directory Server. In the release, Unbound ID makes no secret of what this software should be used for: to migrate away from Sun’s directory toward Unbound ID’s competing solution.

UnboundID is a start-up based out of Austin, TX. It was founded by several ex-Sun employees, including Neil Wilson, author of the “slamd” load generation engine, and formerly one of the key people behind Sun’s OpenDS. I have already raved about their new LDAP SDK for Java, in my opinion the finest and most complete LDAP development kit for any language ever written.

The company is going after the very lucrative Telco and large service provider market, and launched a frontal attack on Sun Microsystems, who is the market leader in that space. UnboundID is offering a 30-40% reduction in yearly maintenance costs if customers switch from DSEE to their solution. Of course there is the usual fine print, and this offer is limited to medium-sized directories with less than two million entries. Why should Sun customers switch from DSEE to UnboundID Directory? According to UnboundID, their server is faster, has less footprint and is supported on a wider platform range.

It is not really obvious to me however why Telcos and large service providers would want to switch. For one, DSEE has been the de-facto market leader for massive-scale directory services, and customer satisfaction is high (not just if you believe the marketing – I’ve personally heard the same from Telcos using the product). A directory server running in a Telco is an absolutely super-critical component, and ripping it out and replacing it is akin to heart surgery. DSEE is very mature after having been around for many years and the kinks have been ironed out in many very large deployments a long time ago already (in fact, I was in one of those deployments in 2002 – that was fun). UnboundID would obviously need to make a very good case and give organisations a high level of assurance for them to switch over. The Telco sector is much more innovative than others, and tends to be on the bleeding edge of technology – but even so, there is a reluctance to switch from a very mature product that “just works” to a brand-new product.

That’s why UnboundID offers the “synchronization server” in order to try to entice organisations to run both directory servers next to each other for a longer period – to evaluate and eventually become comfortable enough with the UnboundID server to make the switch. It seems that the “synchronization server” has been written specifically for this purpose.

Which, personally speaking, I think is a bit of a pity, but hopefully UnboundID will realise the immense value that this synchronisation server could have once they’ve gotten over their frontal attack on Sun. A generic synchronization server that would keep multiple directories from multiple vendors synchronised is a fantastic value proposition, and I’m sure many organisations would jump at it. Especially when it comes from such brilliant minds like Neil Wilson’s who is known for his awesome LDAP stuff.

I’ve recently been to Sun’s directory labs in the the beautiful city of Grenoble, France to talk about what Sun has in store with their two directory servers: DSEE and OpenDS. I’ve used many predecessors of DSEE (starting with the good old Netscape Directory Server) on several projects over the last decade and used to know it inside out. I’ve grown quite fond of it, and so has everybody else I know who has used the product. I wasn’t exactly sure why Sun embarked on its OpenDS project. Why reinvent from scratch what is already a perfectly great product? This question was on my mind, and I was eager to find out why.

When it comes to directory servers, most analysts like to classify them according to the market segments they address. In no particular order, they are: operating system/network, telco and service provider, enterprise and embedded. When it comes to the operating system/network directory servers, Active Directory rules – not necessarily because it is the best for this purpose (and just to be clear: it’s not bad either!), but – well – it’s so intrinsically linked to Windows that you don’t really have a choice. When Novell Netware was around, NDS and e-Directory was another candidate in that area, but it’s pretty much down to AD at this point in time. It’s in the other segments where it gets really interesting because there is some very active development and strong competition.

The Telco/Service provider directory segment is particularly interesting because only the highest scalable directory servers can even attempt to survive in this area. Sun has been very strong in this area for many years, and for a good reason: experience and continuous improvement. I’ve been involved first hand in several very large deployments of Sun Directory Server 5.0 (I think it was during the time when Sun called it “iPlanet Directory Server”). At that time, in the early years of this millennium, we deployed the server for hosting several hundreds of millions of entries. Yes indeed, about 120 Million entries! This was 2002, and at the time the sheer scale was pushing the envelope quite a bit -  but it didn’t just work, it actually worked quite well! Performance, Multi-master replication, and resilience were absolutely key for these types of installations. And sure – in the early versions of 5.0 there were some kinks that had to be ironed out of the replication protocol, but even then it was quite amazing how scalable the directory was, and how well it could actually be managed with such an impressive number of entries. Over the last 7 years, the directory server evolved even further – multimaster replication is rock solid and Sun has tinkered continously with the software to increase scalability way beyond what was already impressive in 2002. Nowadays, there are quite a few reference customers who run Sun directory server with literally billions of entries (incidentally, many of them in China – why am I not surprised ), and this is considered perfectly normal.

When it comes to reliability, a key to deploying very large directories is redundancy, and the possibility to balance loads and fail over between multiple instances. In the early days, load balancing appliances were used to do this (Alteon was really good at this in its days), but unless those applicances had specialised proxy features to handle the instrinsics of the LDAP protocol, this by itself wasn’t a very good option for large deployments. Sun had acquired a company called Innosoft a decade ago, and with it came a product called “DAR” – Directory Access Router – a fully fledged LDAP proxy. Over the years, Sun has enhanced DAR and bundled its next generation into Directory Server (now known as “DSEE”, Directory Server Enterprise Edition”) at no additional cost. Being an important cornerstone of very large and complex directory deployments, it fits like a glove into the directory service and extends it by offering extensive request routing functionality, high availability and performance features and simple mapping features. Previously, only the CA eTrust directory had these features.

I can talk all day about deploying telco directory services, because I’ve used to do it for a living, and am still fascinated by the sheer volume and raw power involved But there’s another two very glorious aspects of directory services, and they can be found in the enterprise and in the still fairly recent embedded directory segment.

The enterprise directory segment is where most of the innovation is happening. Enterprises are typically not as focused on performance, and often more interested in integration, security and manageability. Integration is a very big topic, because the directory service is a crucial piece in any identity management infrastructure. And we’re usually not talking about “a” directory either – most enterprises have many different directory servers, containing either different user populations, or part of the same users but for different purposes. It is in the integration area where much innovation has happened in the directory area. Is doesn’t surprise me that most enterprise directories nowadays feature simple virtual directory functions. That was not the case five years ago, when I worked for a virtual directory vendor. At that time directory service vendors did not foresee virtualisation features as being an important part of their portfolio – perhaps because some of those vendors were also selling an “identity manager” type provisioning system and thought that any directory integration could be solved by deploying a full-blown provisioning system and brute force copying data around Well, this wasn’t really a feasible solution in all cases, so it is only natural that virtual directory companies such as OctetString and Maxware were acquired, and other vendors are “rolling their own” virtualisation features.

Some of the features that are not obvious, but extremely useful in the enterprise scenario are exactly those that allow a directory server to easily interoperate with provisioning, virtualisation and synchronisation products. Technically, the features in LDAP server that are relevant here are persistent queries, incremental updates and proxy auth. These are low-level features that are absolutely crucial when identity “managers” and provisioning services interface with directory servers.

Some other desired features within the enterprise directory segment are about password services and policies. In the vast list of featureds to be found in most modern directory servers are sophisticated access control lists that are expressive enough to configure a finely grained access control policy for deciding who gets access to what type of information. This used to be very important in the past but is getting less important as access control rules on the directory servers tend to be simpler nowadays, because changes typically ocurr through provisioning systems, and not that much any more directly to the LDAP server. Password policies are also a typical feature used in enterprise directory servers (you know – minimum length, character combination, auto-lockout,auto-expiry, and all those things). And of course, keeping track of when users last logged on – very helpful in order to identity dormant accounts.

Another important detail is also how passwords are stored, and how they can be migrated from one server to the other. As a general rule, it’s always good to offer administrators choice. Obviously passwords need to be well protected. But the approach of some directory vendors (specifically Microsoft and Novell) to “secure” their directories has backfired – the directory servers hoard the passwords and don’t even offer any possibility for administrators to export encrypted password hashes. You may wonder whether this “secure” feature is actually a hidden “lock-in trap”! That has created a secondary market around password “synchronisation solutions” in order to overcome the deficiency in the product itself, where the product’s designers thought they had to be smarter than the poor administrators who actually need to deploy, migrate and maintain them.

Last but not least, let’s not forget about one of the very important aspects of enterprise directory services. They need to be simple to deploy, administer and maintain! In the telco area it may be considered acceptable if the directory administrator team features several fully trained relational database administrators, but in enterprise environments that can be too much overhead. Directory servers that make use of relational databases for storing their directory data, such as Oracle’s OID and IBM’s Tivoli Directory Server can point to the advantages of running a directory services platform on a rock-solid database foundation (in these cases, Oracle and DB2 respectively). But the extra administration overhead can be considerable. CA has traditionally used the Ingres relational database for its eTrust Directory Server, but has now in the latest Version 12 switched to something called “DXgrid” – a revolutionary internal memory-mapped storage that not only offers incredible throughput, but also eliminated a significant portion of administration. Sun has since always used a simpler, but very fast and highly scalable data store for its directory server called BerkeleyDB – the same used also in most installations of OpenLDAP.

After mumbling on for quite a discourse I actually wanted to get to the point of Sun’s OpenDS, and the question that I wrote in the beginning of this entry. Why reinvent from scratch (OpenDS) what is already a perfectly great product (Sun DSEE)? As it turns out, there’s been a new segment for directory server that is steadily growing: the one of embedded directory services. For example, packaged solutions that require a directory server internally. Or “black box” appliances with a provisioning interface that contain – guess what – a directory server. A few years back, it was OpenLDAP that was typically shipped with those solutions, because it was free, open and could be embedded easier than other full-fledged directory server products. Now it is OpenDS that is continuously gaining ground, and for good reason. With its incredibly easy set-up, minimal administration, OpenDS epitomises what an embedded directory stands for. And on top of that, the scalability and performance are world-class. Development on OpenDS is, as the name implies, well – open. The development team features Sun employees and others outside Sun, just like OpenSSO. The release cycle is short and new features list is growing at an incredible rate.

So will OpenDS one day replace DSEE? Most likely. But this is still far in the future – for the next few years Sun is actively investing in DSEE as its flagship directory whilst continuing to nurture OpenDS and offering it as an embedded directory server, as well as to anyone interested in quickly deploying a directory server. Now, when I say “quickly” – I’ve managed to install it, extend the schema and load some data into it in less than fifteen minutes! Now that’s what I would call “quickly”. And once I had it up and running on my slow and overloaded laptop, I ran the “slamd” LDAP benchmark tool against it on the same laptop, and got back thousands of searches per second. Not bad at all! Now that’s what I call innovation in the world of LDAP

I’ll be speaking at TEC on Wednesday the 25th of March, on the topic “Cool LDAP Innovations”. OpenDS will definitely get a mention. On the presentation, I’ll also talk about some other real innovations that happened over the last few years in the directory services area. If you’re there, be sure to drop by!

An interesting conversation is taking place within the blogsphere about meta-directories, with Dave Kearns and Kim Cameron on both sides of the argument. This was all inspired by a blog entry on the 4th of March from Jackson Shaw called “You won’t have to kick me around anymore!”. That musing was about HP’s retreat from the identity management market, but makes a statement about meta-directory technology:

Let’s be honest. The meta-directory is dead. Approaches that look like a meta-directory are dead. We talk about Identity 2.0 in the context of Web services and the evolution of digital identity but our infrastructure, enterprise identity “stuff” is decrepit and falling apart. I have visions of identity leprosy with this bit and that bit simply falling off because it was never built with Web services in mind.

I started in this area in 1993 and some of the same architectures are still out there.

The certainly struck a chord with me when I read it. Dave Kearns picked up the topic in his newsletter when he wrote about Optimal IDM, the new virtual directory kid on the block, and made the case that meta-directories have “finally given way to the virtual directory”. Kim Cameron picked up Dave’s entry and disagreed. Up to now, this has lead to an interesting ping-pong of opinions between Dave and Kim, which has not exactly been easy to follow, not just because new contributions are being made on a daily basis up to now, and also because Kim uses the term “meta-directory” to mean something different than what Dave (and myself included) understand. I am going to take this opportunity to jump into the commotion as well, knife not freshly sharpened, but armour freshly polished!

First of all, to clarify what “meta-directory” means (at least, to me!). I am thinking about “Via” (Kim’s baby, the product that Microsoft acquired in 1999 together with Kim’s company, Zoomit). I’m also thinking about Novell Dir-XML, Siemens DirXmetahub and the Critical Path Meta-Directory Server. Old products, created many years ago. You don’t really see much happening with this technology any more, because it has its share of problems, and unless assisted with other technologies, does not fit well into today’s much more dynamic identity and access models. The only exception to that is probably MIIS, but I’ll get to that in a minute.

The old traditional “meta-directory” technology works by creating one big “centralised directory” (or “metaverse” as it’s known in MS-speak), pulling data from everywhere into that centralised directory and then pushing data out into all directions either. This approach is usually not a good fit by itself, because it has several significant shortcomings. I would not go as far as call the technology “dead” (it’s impossible to ignore the many MIIS installations out there), but I’ll call it something else: “quaint”. Now that word has several meanings according to the dictionary, but I sure don’t mean “marked by skillful design, beauty or elegance”!!!

Microsoft has made an investment into that technology by rewriting MIIS pretty much from scratch. And Siemens to this date probably has the most comprehensive and advanced meta-directory implementation with its DirXmetahub component that is part of its Dir-X offering. Nevertheless, meta-directories are arguably still around mostly because Microsoft forces this technology onto its customers for what I think are political reasons: Several people working for Microsoft in the field have told me that it was in Microsoft’s interest to have Active Directory as a central component, and believe it against Microsoft’s interest to have a “filtered access”, such as a virtual directory in front of AD, abstracting information away from what should be the authoritative source. I never really understood this fear, but recently it seems that this brick wall may be slowly starting to crumble (see below).

Some experts in the field still obstinately (in my opinion) push meta-directory technology as the only way to integrate multiple sources of identity information. I think this is very short-sighted. This might have been true in the last century, which is not even that far ago. But in a truly dynamic environment, meta-directory technology and a “synchronisation-only” approach just tends to get into the way. Likewise, the idea that virtual directories by themselves could solve all integration issues is wrong. It’s never been only one or the other, unless you had a specific problem to solve. It’s not synchronisation or virtualisation. You need both, at least if you are in a dynamic identity environment, or have a vision to get there.

So what is the solution for the future? Some people believe that virtual directories will eventually fully supplant meta-directories. Coming from the virtual directory world myself (I worked for Symlabs before joining Kuppinger Cole), I never truly believed that – at least not the virtual directories that were around at that time. Virtual directories and meta-directories could co-exist, and the combination of both had in the past shown great benefits. Think of it as the screwdriver vs. the hammer. Sure, with some brute force you might argue that you can use a hammer to put a screw in, and with some agility you might use a screwdriver to hammer in a nail. But you’re likely to damage something in the way, or at best, not be very practical about it.

I think the future is definitely in the convergence of traditional directory servers, virtual directories and synchronisation solutions to provide rock-solid dynamic directory infrastructure. To a certain extent we can already see this. Maxware (before getting acquired by SAP) and Radiant Logic have already released early, basic versions of synchronisation solutions that harness the power of virtualisation and combine synchronisation with dynamic, abstracted multiple views of data, rather than the static meta-directory approach.

In the future I believe we will see “super-directories” that combine traditional data storage with LDAP access, virtual views and synchronisation features. Some of the players in this space are gearing up to do this already. As synchronisation is usually well-established technology by most of the large players in the identity management space, the missing part is currently still virtualisation, and especially the integration of virtualisation and synchronisation.

Sun and the OpenLDAP foundation, for example, have already added some basic virtualisation features to their directory servers. Oracle has acquired OctetString a while back, and has arguably the most complete, all-around implementation of directory services, synchronisation and virtualisation. Novell, IBM and Microsoft are still lagging behind in this space, with some of the “old guard” defiantly resisting directory virtualisation and hanging on to last century’s belief that synchronisation can solve everything. But there are signs that this resistance is crumbling. It better be. Recently, at DEC2008, Microsoft’s Stuart Kwan presented Microsoft’s vision of a truly dynamic identity infrastructure based on an “identity bus”, where applications could plug in, and “transformers allow us to fold, spindle and mutilate the data in any way we want” – changing internal claims into any other format required by applications. Surely virtualisation is not the only piece that is needed to fulfill such vision, but it is an important (and still missing!) piece. Kim Cameron has not been known to be a big fan of virtual directories – and he still shows some scepticism for the “virtual only” approach, but seems to be warming to virtualisation in combination with synchronisation in one of his recent postings:

So we are led to the conclusion that we need a spectrum of synchronization and remote access capabilities. We should be able to use policy to define what information is stored where, and how to get to information that is not stored locally – e.g., combine metadirectory and virtual directory functionality.

I pretty much agree with Dave and Jackson in that traditional meta-directory technology just doesn’t cut it anymore, at least by itself, and is at best “quaint”. I very much agree with Kim in what I think is his vision of a future “super directory service” that integrates synchronisation and virtualisation with traditional directory services. Where I completely have to disagree with Kim however, is his use of the term “meta-directory” for this new type of “super-directory” technology. OK, I agree that “super directory” sounds a bit tawdry. A better term should be found. But c’mon Kim, “meta-directory” is sooooo… 20th century