Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use Apache Arrow / Parquet as communication between intermittent data #1

Open
mpearmain opened this issue Jun 30, 2018 · 2 comments
Open

Use Apache Arrow / Parquet as communication between intermittent data #1

mpearmain opened this issue Jun 30, 2018 · 2 comments

Comments

@mpearmain
Copy link

More an idea than an issue, but rather than JSON structures for intermittent data, apache arrow (https://arrow.apache.org/) would be another option for the data structures. This would naturally mean better support for other languages in the future and in the short term be significantly faster than having to serialize and deserialize data between languages.
@sean-mcclure
Copy link
Owner

Thanks for the comment. I’ll look into Apache Arrow. Apparently ~80% of processing time can be wasted on serialization/deserialization, so between that and better support across languages Apache Arrow might be a nice fit.

@mpearmain
Copy link
Author

As an update this just was merged
apache/arrow#2489

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mpearmain @sean-mcclure