The last stable version of RavenDB 1.0 was released about six months ago, ever since then, we were hard at work adding new features, improving things and in general doing Good Work.

Looking back into the last half a year of work, it is actually quite hard to pick out the major stuff. There was so much we did. That said, I think that I can pick up some things to salivate on for 2.0.

First & foremost, we drastically improved the RavenDB Management Studio (RDBMS). We spent a lot of time there, and you can now do pretty much everything you want in RavenDB through the studio. This seems like a stupid major feature, right? After all, this is just the UI that was updated, and RavenDB is actually the server stuff. But it provides you with at least an order of magnitude better tooling and ability to work more easily with RavenDB.

And that is really just the tip of the iceberg in terms of what is new in the studio.

But even though the changes to the studio are probably the most obvious ones, we have done a tremendous amount of work of work the server itself. Here are some of the highlights.

Operational Support – We spent a lot of time on making sure that ops people will have a lot of reasons to be happy with this new release. You can monitor this using any standard monitoring tool (SCOM, MOM, HP OpenView, etc). We expose a lot more data through performance monitors and logs. And we even added dedicated endpoints that you can hit to gather monitoring information (which database is currently doing what, for example) that would give ops the full view about what is actually going on there.

Core Bundles -  We always had bundles, and we implement a lot of features through them. But in 2.0, we took a lot of the bundles and move them to the core, so now you can configure & use them easily.

Setting them up on new DB setup:

Setting up replication through the UI:

We have management UI support for all of the core bundles now, which makes using them a lot easier.

Changes() API – this allows you to get PUSH notifications from the RavenDB Server, you can subscribed to events from a particular documents, a set of documents or an index. That allows you to notify the user if something have changes without the need to do any expensive polling.  The studio was actually doing a lot of polling, but we changed pretty much all of that to be PUSH based now.

Here is an usage sample:

   1: store.Changes()

   2:     .ForDocument("users/1")

   3:     .Subscribe(notification =>

   4:         {

   5:             using(var session = store.OpenSession())

   6:             {

   7:                 var user = session.Load<User>(notification.Name);

   8:                 Console.WriteLine("Wow! " + notification.Name + " changed. New name: " + user.Name);

   9:             }

  10:         });

Yes, it is as easy as this .

Eval Patching – you can now run JS scripts against your objects, to modify them on the server side. This is perfect if you want to do migrations (if you actually need to, usually you don’t), want to run some complex modification on the server side or just need to do something as an administrator.

More authentication options & control – We now have a far easier time defining and controlling who can access the server, and what databases they can touch.

Here is an example of allowing the network service to have access to the tryout database:

And here we have an example of defining API Keys:

This allows your to define an API Key for a particular application very easily (vs. defining users ,which is usually how you handle admin / ops people coming in).

Indexing Performance – We have spent a lot of time on optimizing the way we are handling indexing. In particular, we now do a lot of work to make sure that we don’t wait for IO and we use as many cores as we can to get things done even faster. Even when you throw a lot of data at RavenDB, indexing catch up very quickly and the indexing latency is far lower.

Better map/reduce – Our map/reduce implementation have been drastically improved, allowing us to re-process and update existing results with a lot less computational & IO needs at the large scale of things.

Better facets – We have completely remapped the facets support, reducing the per facet value cost that used to be there. Now we are able to generate facets quickly regardless of how many facet values you have in a facet, and we even support paging & sorting of facets.

Better IN Query – This sounds silly, but supporting an efficient IN query is important for a lot of scenarios, especially when the number of items in the IN is large. We have a dedicated support for doing that efficiently and easily now.

Async – We matched all the standard client capabilities in our Async API, that means that we support async sharding, async replication failover, and the whole shebang. It means that using RavenDB with C# 5.0 is just as easy as you can imagine, it is all been done for you.

Sharding improvements – In addition to the async sharding support, we worked on improving sharding itself, giving you more integration & extensions points and made the whole thing just a tad bit smarter by default.

Cloud Backup – Backing up is hard to do, and we have decided to make it easier. In addition to supporting all enterprise backup tools, and having the ability to manually trigger backups or export (full or incremental), we now have the ability to schedule automatic backups to the cloud.

You can setup period backups to Amazon Glacier or Amazon S3, and it will incrementally backup your database there as we go along.

CSV Import / Export – Silly, but data still goes around mostly in flat files, and RavenDB now support importing & exporting data in CSV format, so you can easily pull some data into excel or push some data that you got from some other source.

Debuggability – We now expose a lot more hooks for you to use when you debug things. You can look directly into how the index is built (index entries, stored fields, etc), you can inspect the intermediate steps of RavenDB’s map/reduce processes, how the IO processes for loading document work, indexing times and a lot more.

There are more, but I think that this is enough for now.

We use RaveDB to handle our entire internal infrastructure. That has several reasons, to start with, RavenDB is a joy to work with, so it cuts down on the dev time for new internal features. More importantly, however, it let us try RavenDB in real world conditions. That seems obvious, because as much as you try, you can never really simulate real production environments.

But even more important is the concept of production data. The important thing about production data is that it is dirty. You don’t get data that is all nice and simple and all the same as you have when you generate the data, or when you are creating unit and system tests.

In this case, we had an interesting problem. We had a map reduce index similar to this one:

   1: public class UsersByCountry : AbstractIndexCreationTask<User, UsersByCountry.Result>

   2: {

   3:     public class Result

   4:     {

   5:         public string Country { get; set; }

   6:         public int Count { get; set; }

   7:     }

   8:  

   9:     public UsersByCountry()

  10:     {

  11:         Map = users =>

  12:               from user in users

  13:               select new {user.Country, Count = 1};

  14:         Reduce = results =>

  15:                  from result in results

  16:                  group result by result.Country

  17:                  into g

  18:                  select new

  19:                  {

  20:                      Country = g.Key,

  21:                      Count = g.Sum(x => x.Count)

  22:                  };

  23:     }

  24: }

This is a pretty standard thing to have, but I noticed that we had a difference from the 1.0 results in our production data. Investigating further, it appeared that this was the root issue:

   1: using (var session = store.OpenSession())

   2: {

   3:     session.Store(new User { Country = "Israel" });

   4:     session.Store(new User { Country = "ISRAEL" });

   5:     session.Store(new User { Country = "israel" });

   6:     session.SaveChanges();

   7: }

With RavenDB 1.0, due to fairly esoteric implementation details, this would generate the following values:

• {"Country": "Israel", "Count": 1}
• {"Country": "ISRAEL", "Count": 1}
• {"Country": "israel", "Count": 1}

This matches what you’ll get using Linq to Objects, and that was fine by me. I could see an argument for doing the reduce using case insensitive reduce, but then you have the argument about which or the representation is the one you should use, etc.

In the version that I tested, I actually got:

• {"Country": "Israel", "Count": 3}
• {"Country": "ISRAEL", "Count": 3}
• {"Country": "israel", "Count": 3}

Now, that was wrong. Very wrong.

As it turned out, once I managed to recover from the palpitations that this issue gave me, the actual reason was pretty easy to figure out. Some of our code was case sensitive, some of our code was not. That meant that under this condition, we would feed the map/reduce engine with duplicate entries, per the number of various casing combinations that we had.

Spooky bug, but once we narrowed down what the actual problem was, very easy to resolve.

When we started full fledged active development on the next version of RavenDB, we used the tag name “1.2” for that release version. We wanted to take the time and punch some big tickets items that were going to be very hard to do with small incremental releases. I think we did quite a good job of that. But one of the major things that we wanted to do was to be able to go back and fix some of the early on design decisions that we made.

Some of them were minor (null handling improvements during indexing), some of them were pretty major (changing indexing format for dates to human readable ones). Some are obvious (moving things around in the client API to make it easier to avoid mistakes), some are not so much (far superior handling of DateTime and DateTimeOffset). Interestingly enough, we didn’t get any big breaking changes, only a lot of minor course corrections. For the most part, I expect people to move from 1.0 to 2.0 without even noticing that things got much better.

We stilled called this release RavenDB 1.2, but we have some breaking changes, and we have done tremendous amount of work on the system. We started getting pushback from users and customers about the version number. This isn’t just some minor release, this is a major step up.

Thus, what we were calling 1.2 became 2.0. And I’ll be posting about what exactly is new in RavenDB 2.0 in just a bit…

Well, we tried to upgrade to RavenDB 2.0 build 2152, and then we quickly had to roll things back. The reason for that, by the way, was authentications issues.

In RavenDB 2.0, we did a lot of stuff to tighten security. In particular, we actively try to push you away from using the system database, and into using named databases. And we also made it easier to specify which databases which users has access to (and what kind of an access).

Unfortunately, while it worked perfectly during testing, going to production revealed that we had a few… issues in it.

Surprisingly enough, those issues weren’t in the logic or how it worked. Those issues boiled down to…. case sensitivity. It turned out that both the user names and the database names were case sensitive in the permission configuration, leading to… well, you probably saw that the site threw a 401 error.

Fixing that config issue was pretty high on my list of things to fix. If only because case sensitivity in this place was just a trap waiting to happen. Once we figured out what was going on, it was relatively easy to fix. This post is now written on a system running the very latest version of RavenDB 2.0.

Well, one thing that we put a lot of focus on was performance. In order to test that, I had a dataset of 4.66 million documents (IMDB data set, if you care) as well as two indexes defined.

The results for RavenDB 2.0 (drum roll):

Loading 4.66 millions records in 44 minutes. Average rate of less then half a millisecond per document.

But wait, what about the indexes? Well, RavenDB index stuff as they come, and as we were inserting the documents, they were indexed along the way. That meant that 11 seconds after we were done putting 4.66 millions documents to RavenDB, we were done indexing (across all indexes).

Pretty nice perf, even if I say so myself.

In my previous post about the command pattern, I gushed about how much I loved it. That doesn’t mean that the command pattern as originally envisioned is still completely in fashion.

In particular, the notion of “Undo” was  one of the major features in the command pattern’s cap. Today, that is almost never the case. Sure, if you are building an application such as an editor. Something like Photoshop or a text editor would find the notion of commands with the ability to have an undo stack very compelling. Other than that, however, that is a very rare need.

In most business scenarios, there really is no good way to undo things. How would you implement SendEmailCommand.Undo(), for example? But even something like PlaceOrder.Undo() is a lot more complex and hard to do than you would think. The mere notion of undoing the operation assumes that this isn’t going to have any side affects. But cancelling an order may result in cancellation fees, require you to ship back things you got back, end. It is not “Undoing PlaceOrder”, rather that is a whole different and distinct business process, usually represented by another command: CancelOrder.

Another common issue that people have is the degeneration of the entire architecture to something like:

CommandExecuter.Execute(Command cmd);

To that I answer, more power to you! I love code that is composed of a lot of small classes all doing things about the same way. There is no easier way to look at a system, and that allows you to quite easily add additional functionality to the system easily. That said, mind how you handle routing in that scenario. I have seen people go into the “when a request comes to this URL, let us invoke the following commands” in XML. One of the reasons that people dislike this approach is how you actually call this. If just getting to the command executer is hard and involved, you lose a lot of the advantages.

This popped up in the mailing list, and I really dislike it. The notion of Composite Command. A command that can execute multiple commands. Now, from a programming point of view, I can easily see why you would want to do that. The PlaceOrderCommand operation is composed of a lot of smaller commands. But, and this is important, the notion of Composite Commands basically mean that you get the same thing as the PlaceOrderCommand, but you just lost the name. And naming is important. Almost as important, error handling is quite different between different business scenarios, and you sometimes end up with something like:

   1: var placeOrderCommand = new CompositeCommand(

   2:    new RegisterOrderCommand(),

   3:    new ReserveStockCommand(),

   4:    new ChargeCardCommand(),

   5:    new ShipOrderCommand()

   6: )

   7: {

   8:    Sequential = true,

   9:    StopOnError = true

  10: }

And this is simple, how do you handle an error in the ShipOrderCommand after you already charged the card, for example?

Well, it is about this time. We run out of work with RavenDB 2.0, and now we pretty much have no choice but to get into the release cycle.

We aren’t done yet, but we are close. Close enough that we started running the current build in our own production systems. There is still stuff to be done, in particular, performance and bug fixes.

You can get the latest stuff here. We would love to get some people to hammer at things and see if they can break it.

One of the things that we really pride ourselves with Hibernating Rhinos is the level of support. Just to give you some idea, today & yesterday we had core team members on the phone with people (not customers, yes) who are having problems with RavenDB for quite some time.

Now, I understand that you may not always have the information to give, but what you have, give me! So I can help you.

From a recent exchange in the mailing list:

var clickCount = session.Query<TrackerRequest>().Where(t => t.TrackerCreated == doc.Created).Where(t => t.Type == Type.Click).Count();

This gives:

"Non-static method requires a target"

To which I replied:

What is the full error that you get? Compile time? Runtime?

The answer I got:

This is a document I'm trying to 'Count':

{

  "Created": "2012-11-15T16:12:42.1775747",

  "IP": "127.0.0.1",

  "TrackerCreated": "2012-11-15T14:12:16.3951000Z",

  "Referrer": "http://example.com",

  "Type": "Click"

}

Raven terminal gives:

Request # 172: GET     -     3 ms - <system>   - 200 - /indexes/Raven/DocumentsByEntityName?query=Tag%253ATrackerRequests&start=0&pageSize=30&aggregation=None&noCache=-1129797484

        Query: Tag:TrackerRequests

        Time: 2 ms

        Index: Raven/DocumentsByEntityName

        Results: 3 returned out of 3 total.

By the way, you might note that this ISN’T related in any way to his issue. This query (and document) were gotten from the Studio. I can tell by the URL.

Then there was this:

I mean, seriously, I am happy to provide support, even if you aren’t a customer yet, but don’t give me some random bit of information that has absolutely nothing to do to the problem at hand and expect me to guess what the issue is.

Relevant information like the stack trace, what build you are on, what classes are involved, etc are expected.

The command pattern is a behavioral design pattern in which an object is used to represent and encapsulate all the information needed to call a method at a later time.

More about this pattern.

I adore this pattern. If this pattern had a paypal account, I would donate it money on a regular basis.

In general, the notion of encapsulating the method call into an object (like the functor sin C++) is an incredibly powerful idea, because is separate the idea of selecting what to invoke and when to invoke it. Commands are used pretty much every where, WPF is probably the most obvious place, because it actually have the notion of Command as a base class that you are supposed to be using.

Other variations, like encapsulating a bunch of code to be executed later (job / task), or just being able to isolate a complex behavior into its own object, is also very useful. I base quite a lot of my architectural advice on the notion that you can decompose a system to a series of commands that you can compose and shuffle at will.

Recommendation: Use it. Often. In fact, if you go so far as to say that the only reason we have classes is to have a nice vehicle for creating commands, you wouldn’t be going far enough.

Okay, I am kidding, but I really like this pattern, and it is a useful one quite often. The thing that you want to watch for are commands that are too granular. IncrementAgeCommand that is basically wrapping Age++ is probably too much, for example. Commands are supposed to be doing something meaningful from the scope of the entire application.

In one of our production deployments, we occasionally get a complete server process crash. Investigating the event log, we have this:

Exception: System.InvalidOperationException

Message: Collection was modified; enumeration operation may not execute.

StackTrace:    at System.Collections.Generic.Dictionary`2.KeyCollection.Enumerator.MoveNext()

   at System.Web.Hosting.ObjectCacheHost.TrimCache(Int32 percent)

   at System.Web.Hosting.HostingEnvironment.TrimCache(Int32 percent)

   at System.Web.Hosting.HostingEnvironment.TrimCache(Int32 percent)

   at System.Web.Hosting.ApplicationManager.TrimCaches(Int32 percent)

   at System.Web.Hosting.CacheManager.CollectInfrequently(Int64 privateBytes)

   at System.Web.Hosting.CacheManager.PBytesMonitorThread(Object state)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)

   at System.Threading._TimerCallback.PerformTimerCallback(Object state)

As you can see, this is a case of what appears to be a run of the mill race condition, translated to a process killing exception because it was thrown from a separate thread.

This thread, by the way, is the ASP.Net Cache cleanup thread, and we have no control whatsoever over that. To make things worse, this application doesn’t even use the ASP.NET Cache in any way shape or form.

Any ideas how to resolve this would be very welcome.