Field Report on the Kernel Community Workshop
From May 27th to May 29th 2019, thirty developers from the Jupyter community met in Paris for a three-days workshop on the Jupyter kernel protocol.
For three days, attendees worked full time on the Jupyter project, including hacking sessions and discussions on improvements to Jupyter protocols and standards.
We were lucky to count five core developers to the project and several more regular contributors to the larger ecosystem in the group!
Beyond the hacking sessions, each day was concluded with a series of presentations and demos of the progress made during the workshop.
Why a workshop on Jupyter kernels?
The Jupyter Kernel protocol is one of the main extension points of the Jupyter ecosystem. Dozen of language kernels have been developed by the community, and these languages can then leverage the other components of the stack, such as the notebook, the console, and interactive widgets.
One of the objectives of this event was to foster collaboration between kernel developers and core contributors and develop common tools used across all language kernels. Another goal was to work improving the protocol, to support visual debugging, parameterized kernels, etc.
Technical Achievements
A lot of work was done during the workshop. Several people were working on new Jupyter kernels for programming languages.
We were amazed to see how much was achieved in just three days. Today, we are still working on projects that stemmed from this community workshop.
JupyterLab Debugger
One area in which the Jupyter team is actively working is support for visual debugging in JupyterLab. This is a major endeavor spanning from the front-end to the back-end, including changes to the Jupyter protocol.
The kernel workshop was the occasion for several developers (Wolf Vollprecht, Maarten Breddels, Johan Mabille, Loic Gouarin) to get together and hack on the implementation of the frontend. A lot of work had been done ahead of the workshop with a new Python kernel based on the Xeus library. This new kernel includes a backend to the Debug Adapter Protocol over kernel messages.
There are other key contributors to the debugger project who did not attend the community workshop. We should mention Jeremy Tuloup and Afshin Darian who are spearheading the frontend development.
A Calculator Jupyter Kernel based on xeus
Vasavan Thirusittampalam and Thibault Lacharme were in the middle of their summer internships as scientific software developers at QuantStack when they attended the workshop. During the event, they set themselves up to implement a new Jupyter kernel in C++!
At the end of the workshop, they were able to demonstrate a functional calculator kernel based on xeus, a C++ implementation of the Jupyter protocol! This was later polished and resulted in the publication of a blog post on the Jupyter blog: "Building a Calculator Jupyter Kernel". This example may serve as an example for people interested in creating new language kernels with xeus.
A prototype Julia backend to Jupyter interactive Widgets.
During the workshop Sebastian Gutsche (who is the author of the Singular Jupyter kernel and a co-author of the GAP kernel), set himself to develop a backend to Jupyter interactive widgets for the Julia progamming language. The current state of the code can be found here.
The Julia backend to jupyter widgets is still early-stage but should this project be completed, it would enable other Jupyter widget libraries for the users of the Julia Jupyter kernel, such as ipyvolume, ipyleaflet, bqplot, etc.
Parameterized Kernelspecs
Paul Ivanov and Kyle Kelley worked on ironing out the proposal for the support of parameterized kernelspecs. The objective is to make it possible to pass user-specified arguments to kernels upon launch.
In the proposal, the parameters expected by the kernel executable and the possible values for these parameters may be specified in the kernelspec file or a companion file to the kernelspec in the form of a JSON schema.
Paul and Kyle produced extensive notes on various aspects on the subjects such as
- the storing of previously used values of the parameters (in notebooks)
- the web UI design on how to specify parameters
- the definition of default values
More details will be shared at a later stage on the subject of parameterized kernelspecs.
Note: the subject of parameterized kernelspecs was also central in the earlier community workshop about the Jupyter server organized at IBM by Luciano Resende.
Forking of Jupyter Kernels
Starting of kernels is an expensive operation, and executing a whole notebook even more. For certain use cases (such as dashboards based on Voilà), it is useful to have a pre-executed notebook ready, so that dashboards can be presented instantly to a user. To enable this, Maarten Breddels came up with the idea of forking kernels and put together a proof-of-concept implementation. The implementation can be found in pull requests to jupyter_client (PR #441) and ipykernel (PR #410).
Forking kernels will also allow interesting features such as an undo/rollback of cell execution in the notebook. While it is still a proof of concept, this subject has already sparked interest in the community.
Acknowledgments
This event would not have been possible without the generous support provided by Bloomberg, who made this workshop series possible
We are grateful to CFM for gracefully hosting the event.
We should also thank Ana Ruvalcaba from Project Jupyter for her incredible work on the logistics and finances of the Jupyter Community Workshop series.
