+972 52-548-6969
NuGet Perf, The Final Part – Load Testing – Results ^ 2
After seeing how well RavenDB does in perf testing, I decided to take it up a notch.
- Starting from 10 users, with a step duration of 1 sec, add 50 users for each step, all the way to 3,000.
- Start with a warm up period of 20 seconds, then run the test for 10 minutes.
Let us see what happens, okay?
Just to be clear, this is a RavenDB application running with three thousands concurrent users, on an off the shelve laptop while I was busy doing other stuff.
One word of warning before hand, because I run everything on a single machine, just running so many users on the machine significantly slowed down how RavenDB is reacting. Basically, the code for managing the perf test took so many resources that RavenDB had to fight to get some to actually answer the queries.
Scared yet, because here are the results in graph form.
Now you can actually see that we have some fluctuations in the graphs, the number of users grows and grows until we get to 3,000 and we have 0.37 seconds response times.
Again, I remind you, we have done zero optimizations and this is idiomatic RavenDB code. And we were able to serve requests at a frankly pretty amazing rate of speed.
And here are they in their full details:
| Load test name | LoadTest1 |
| Description | |
| Start time | 04/09/12 15:28:48 |
| End time | 04/09/12 15:38:48 |
| Warm-up duration | 00:00:20 |
| Duration | 00:10:00 |
| Controller | Local run |
| Number of agents | 1 |
| Run settings used | Load |
| Max User Load | 3,000 |
| Tests/Sec | 196 |
| Tests Failed | 0 |
| Avg. Test Time (sec) | 14.3 |
| Transactions/Sec | 0 |
| Avg. Transaction Time (sec) | 0 |
| Pages/Sec | 741 |
| Avg. Page Time (sec) | 0.37 |
| Requests/Sec | 741 |
| Requests Failed | 0 |
| Requests Cached Percentage | 0 |
| Avg. Response Time (sec) | 0.37 |
| Avg. Content Length (bytes) | 3,080 |
| URL (Link to More Details) | 95% Page Time (sec) |
|---|---|
| Page 1 | 0.83 |
| Page 0 | 0.82 |
| Page 2 | 0.82 |
| Page 1 | 0.82 |
| http://localhost:52688/api/search | 0.81 |
| Name | 95% Test Time (sec) |
|---|---|
| Browsing | 20.8 |
| BrowseAndSearch | 19.8 |
| Searching | 12.9 |
| Name | Scenario | Total Tests | Failed Tests (% of total) | Avg. Test Time (sec) |
|---|---|---|---|---|
| Browsing | Load | 31,843 | 0 (0) | 17.4 |
| BrowseAndSearch | Load | 33,989 | 0 (0) | 16.8 |
| Searching | Load | 51,650 | 0 (0) | 10.8 |
| URL (Link to More Details) | Scenario | Test | Avg. Page Time (sec) | Count |
|---|---|---|---|---|
| Page 2 | Load | Browsing | 0.40 | 32,338 |
| Search yui | Load | Searching | 0.39 | 52,597 |
| Page 1 | Load | Browsing | 0.39 | 32,627 |
| http://localhost:52688/api/search | Load | BrowseAndSearch | 0.39 | 68,576 |
| Page 0 | Load | Browsing | 0.38 | 32,803 |
| Search grid | Load | Searching | 0.38 | 52,283 |
| Page 1 | Load | BrowseAndSearch | 0.37 | 34,766 |
| Page 0 | Load | BrowseAndSearch | 0.36 | 34,982 |
| Search debug | Load | Searching | 0.35 | 51,991 |
| Search ravendb | Load | Searching | 0.33 | 51,846 |
| Name | Scenario | Test | Response Time (sec) | Elapsed Time (sec) | Count |
|---|
| Machine Name | % Processor Time | Available Memory at Test Completion (Mb) |
|---|
| Machine Name | % Processor Time | Available Memory at Test Completion (Mb) |
|---|---|---|
| RAVEN | 85.4 | 1,203 |
| Type | Subtype | Count | Last Message |
|---|
Note that the reason fro the high CPU usage is that the tests and RavenDB were running on the same machine.
Comments
These numbers are so great that i would like to see what would came out of a production environment with an even higher number of users like... i mean... vulgar display of power :)
Too bad ravendb can't serve odata or this would be a no-brainer for the nuget team to implement.
This is a great series of posts. I think this clearly demonstrates how a traditional rdbms is not designed to handle the average use case of today's applications. Yes, you could optimize the heck out of your model and db to try and make it perform but the out of the box experience of ravendb just works, and works well. Even with an optimized rdbms things like search are always going to perform poorly just because they are not designed for this use case. A rdbms still has its place, but its not for the traditional web app anymore.
Reviewing the test plan it seems only 6 search terms are used: orm, nhibernate, yui, grid, debug, ravendb. My analysis shows there are over 3,400 unique terms in the Tags field, 6,500 in ReleaseNotes and 9,000 in Description.
No phrase queries, no variation in sort order and only 6 term queries does not seem like an adequate simulation of the types of queries that nuget.org is likely to experience in high volume.
With such a small number of unique queries, the server can simply return cached results without experiencing virtually any memory pressure or CPU spikes.
Since no packages are being added to the index nor are download counters being incremented, the data store is effectively in read-only mode, making the simulation even less accurate in comparison to real traffic.
With all of those considerations, I don't think this load test fairly compares RavenDB to the SQL Server/EF solution in use on nuget.org.
Comment preview