Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redis overall performance improvements #94

Open
rehia opened this issue Nov 18, 2016 · 2 comments
Open

Redis overall performance improvements #94

rehia opened this issue Nov 18, 2016 · 2 comments

Comments

@rehia
Copy link
Contributor

rehia commented Nov 18, 2016

While heavily using this awesome lib with Redis, I have a couple of performance improvements in mind.
I'd like to share them here, to open the discussion on it.
Obviously, this will lead to some PRs.

  • the way events are retrieved by scanning all the keys with a match pattern forces to get all the keys before any mget, because scan does not guarantee the order. maybe using a sorted set to store keys could help scanning them more efficiently, and even stream events (I know, node-eventstore does not support streaming... yet!)

  • snapshots are stored forever, although they can be useful for some time only. It could be useful to be able to clear the oldest, or having a rolling strategy which only keeps the nth latest.

I'll add other improvements in this thread. This is an open discussion.
So don't hesitate to make suggestions or remarks.
@adrai
Copy link
Contributor

adrai commented Nov 19, 2016

Nice ideas...
would mean when changing the redis implementation it will not be backwards compatible (just keep in mind)
For the rolling snapshot I wish that feature would be working for the other db implementation too ;-)

@rehia
Copy link
Contributor Author

rehia commented Nov 30, 2016

I still think about this issue.
And I had performance issues, even after using snapshots (I had about 40K snapshots).
It took a very long time (3 sec ?) loading the last one.

The main concern I see are still the ones mentioned in the issue description.

  • Find a way to better index snapshots and events to scan a large number of them. This could be done even backwards compatible, by just creating new indexes next to the current ones.
  • Remove old snapshots, definitely, with a default strategy of keeping them all, to be backwards compatible as well.

Don't know yet when I'll have time for this :/
Just applying quick fixes on my data for the moment...

@rehia rehia mentioned this issue Jan 17, 2017
Closed
@rehia rehia mentioned this issue Jan 28, 2017
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@adrai @rehia