Hey lobsters. @jspahrsummers and I created this at Anthropic. I hope I didn’t cross any lines of self-promotion here, but I am super excited about this and wanted to share it.
We will work on remote connectivity. The current “implementation” is focused on local-only, but we hope the underlying primitives will hold true for remote connections.
My understanding is that Cohere Connectors and others aim to solve something very different. MCP is trying to solve the NxM problem of applications to context providers and build in the open. Anybody is free to implement it both on the client as well as on the server side. It is also focused on general interaction primitives rather than pure data providers (hence the separation of prompts and resources for example). As such it’s fundamentally different from proprietary APIs that are bound to a specific product surface. I personally feel LSP is a much closer related concept and something that inspired us.
Why json schema and json rpc and stdio? I understand that json is the lowest common denominator (somewhat pejorative!) for data interchange with all kind of clients. Was making the transport work over http and stdio that important? Why couldn’t local agents just use http? Surely there must be a schema-based approach to specifying the entire protocol.
I work at Whimsical. We have a GPT that people use to create diagrams and flowcharts, and I’d love to build something similar for Claude. The Model Context Protocol is very cool but it seems like it currently requires every context provider to write a server that will run locally on a user’s machine.
It would be great if there was a server-side protocol that we could implement in our existing API which wouldn’t require writing a server. There are two reasons for this:
Writing, maintaining, and distributing an MCP server is more work than implementing new API endpoints in our existing app.
More importantly, it is much lower trust and effort for users to configure their LLM to point to a web service versus downloading and running a server that has local code execution.
Do you have plans for something like this in the future?
We are working on thinking through “remote MCP”, which would solve this (as I understand it). We started with local MCP because we wanted to see if the concepts hold up while we work on hard parts of remote transports, particularly getting authentication right. We will tackle that aspect next and certainly recognize that this would unlock a lot of use cases.
(Also a good part of MCP comes from our own internal need and usage, where we can easily distribute servers)
For example in my code I have this function to get a config filename from the environment variable:
let get_config_filename: string =
try
let path = Sys.getenv "CONFIG" in
path
with Not_found ->
"./websites.yaml"
That’s not a function - that’s a named value. It’ll be evaluated only once. To make it a function, you’d need to take an argument (usually unit, i.e. ()), like:
let get_config_filename () = (* ... *)
I think this confusion is why some ML-syntax-like languages try to differentiate it with a different keyword for functions (like Rust’s fn) or using lambda syntax (like Roc’s name = \arg -> result.)
Apart from that, I agree with the author’s conclusion that OCaml is hard to get into. It’s hard to figure out how you’re supposed to solve common problems, and there’s scarce few “blessed” approaches to do common things. This is amplified by the OCaml 5 and domains situation because the earlier popular approaches for things aren’t necessarily how things should be done in the post-5 world. There’s been a few times where I look up how to do something, and decide on library/approach foo, but while I’m working on it I face issues like what the author did with crawl_website, and I’ll run into threads on reddit where people will be like “Obviously you shouldn’t use foo, use ppx_bar instead” and I’ll give up and resolve not to use OCaml again…
I haven’t looked into it too deeply, but even the newest edition of RW OCaml is not written for OCaml 5+, so it won’t cover domains and whatnot. Add to that the usual Async vs Lwt argument and I don’t recommend it right now.
It’s me, author of this article.
Thanks a lot for clarification. Does that mean even if env var value changes it won’t be reflected because it is executed only once?
kivikakk already covered your question but I’ll throw in a helpful tip for knowing the difference in the future: all functions in OCaml must have an -> in their type. You can use your editor’s LSP integration (e.g. in neovim I have to press K on a symbol to see its docs) to see that the inferred type of get_config_filename is string, while a function would be () -> string.
Yes — you can prove this to yourself by trying something like this:
let main () =
let get_config_filename: string =
print_endline "evaluating get_config_filename";
try
let path = Sys.getenv "CONFIG" in
path
with Not_found ->
"./websites.yaml"
in
print_endline get_config_filename;
print_endline get_config_filename
in
main ()
That said, how are you anticipating the environment variable changing? OCaml’s stdlib itself doesn’t expose setenv or unsetenv — note aborted attempt here.
edit: oh, never mind me! Unix.putenv is mentioned right there, which of course works on macOS. So you can try this:
#load "unix.cma";;
let main () =
let get_config_filename: string =
print_endline "evaluating get_config_filename";
try
let path = Sys.getenv "CONFIG" in
path
with Not_found ->
"./websites.yaml"
in
print_endline get_config_filename;
Unix.putenv "CONFIG" "hello";
print_endline get_config_filename
in
main ()
Which outputs the same as above. Now note the change here:
#load "unix.cma";;
let main () =
let get_config_filename () =
print_endline "evaluating get_config_filename";
try
let path = Sys.getenv "CONFIG" in
path
with Not_found ->
"./websites.yaml"
in
print_endline (get_config_filename ());
Unix.putenv "CONFIG" "hello";
print_endline (get_config_filename ())
in
main ()
This is amplified by the OCaml 5 and domains situation because the earlier popular approaches for things aren’t necessarily how things should be done in the post-5 world.
Keep in mind that OCaml 5 is very very young and that 5.0, 5.1 and even 5.2 are not considered ready yet. I agree that OCaml often lacks a one-true way but OCaml 5 is not a good example of that.
I agree that it is young, but at this point it objectively is the future of OCaml, so for someone just getting into it it doesn’t make sense to not start with it. Doubly so because it affects IO, which is what most basic and common programs do! The author of this article started out writing a scraper, my first attempt was to write a discord bot, and many people kick the tires with a web framework like Dream. So the problem of pre-4 approaches not being the future is more of a problem: as a newcomer, is what I’m learning going to become outdated soon? Especially when it’s the core of what I’m doing?
I don’t want to be overly negative abotu OCaml because I really like the language otherwise, so I’ll point out that there are people like @leostera on twitter who are working really hard on this problem and have made great progress. For people who want to get started with OCaml, I think https://github.com/leostera/minttea is natively OCaml 5 and really fun for making TUIs.
I believe it does make sense to start with Lwt rather than with Eio now. There’s simply more learning resources and more Lwt-compatible libraries. Lwt won’t become obsolete in the near future and the knowledge of OCaml+Lwt will transfer good to OCaml+Eio — same language after all.
Does it support osc52 for copy/paste, which is very useful to propagate clipboard across ssh/tmux/nvm or is that superseded by having wezterm available remotely and multiplexing (no tmux)?
does ghostty supports graphics in terminal? with kitty (and tmux) I can render graphviz directly onto terminal which I find handy when debugging ragel grammars, for example
Pushup https://pushup.adhoc.dev is not JSX exactly, but it’s HTML-first blended with Go for control flow and variable access. (I’m the Pushup creator.)
Thanks. Not sure what to think about the syntax though. Why not make it like JSX instead? Would also make it easier to port tooling and editor support to.
There’s a bittorrent protocol extension where you can distribute a public key that points to a mutable torrent, but I don’t know if it has been ported to webtorrent.
As far as I understand, you can’t use DHT in a web browser, as nodes do not support WebRTC. The Webtorrent project includes a DHT library that works with Node.js (which is used by the desktop application).
I see FunSQL.jl is already mentioned in the discussion but I still want to
highlight how it works as I think FunSQL represents a very interesting point in design space of “SQL replacements”.
FunSQL is a Julia library which presents a
combinator-based API to generate SQL. The query construction looks like:
I’ve myself ported the lib (with some deviations) to Python and OCaml (ended up
using this one) and built a concrete syntax on top of it, so at the end it looks
very similar to PRQL, albeit I’ve kept it conservative syntax-wise (closer to
SQL). Same query as above in the concrete syntax:
from person
where year_of_birth >= 1930 and year_of_birth <= 1940
select person_id
(I’ll use this syntax going forward in this post)
Now the feature I’ve wanted to highlight (and which I think makes FunSQL very
useful) is how it treats group by and aggregate expressions. An example first:
from users as u
join (from comments group by user_id) as c on c.user_id = u.id
select
u.username,
c.count() as comment_count,
c.max(created_date) as comment_last_created_date
Here the subrelation c which is joined to u is being grouped by user_id
column. The c itself doesn’t specify any aggregates though as you can see. All
aggregates are located in the single select below.
FunSQL treats grouped relations as a special kind of namespace from which you
can either select columns you grouped by or refer to aggregate functions.
In my mind, this feature is very powerful as it allows one to incrementally
construct reusable query fragments and then use them to build final queries.
Same query in two steps:
let users_with_comments =
from users as u
join (from comments group by user_id) as c on c.user_id = u.id
from users_with_comments
select u.username, c.array_agg(text) as comments
In other words: you can build a nested namespace from relations / grouppings and
then later decide what you want to select from this namespace.
This surprisingly works well if you want to implement visual query builders (I
have one for my OCaml port).
Check out JuliaCon 2021 presentaton about FunSQL if interested, the main
author Kyrylo (who’s by the way the co-creator of HTSQL, another query language
mentioned in the discussion here) talks about motivation and design of FunSQL.
It is a line of text visible in the editor but is not present in the actual text buffer. Previously nvim supported virtual text inline, used to show warnings or other information after a line of code for example. Now entire lines can be inserted virtually.
That is legitimately awesome. Does anyone have any idea what plugin / library enables that? And, if it works for other proof assistants like Isabelle? (I’m not a Coq user)
Proof assistants are amazing tools, and some of them do offer a way to print out the full proof that they end up automating. Showing intermediate proof states is definitely nice, but some tactics still take very large leaps (like tauto used in the paper). In those cases, I want to see insight into how the step was taken in between goals as well.
Also thinking a bit about a new programming language I want to work on for fun - A combination of standard ML drawing heavy inspiration from the https://github.com/janet-lang/janet runtime.
So many things to do - clearly I need to avoid spending time on the less important things - I just have trouble reigning in what my mind is currently obsessing over.
Also thinking a bit about a new programming language I want to work on for fun - A combination of standard ML drawing heavy inspiration from the https://github.com/janet-lang/janet runtime.
Do you mean you want to reimplement StandardML but on top of Janet’s like runtime? Or is there something specific to Janet which can influence the SML language itself?
I’m myself contemplating a compile to LuaJIT ML-like language: the efficiently of LuaJIT and its convenient FFI + ergonomics of ML (though I’d want to experiment with adding modular implicits to the language).
Is this related to Hermes (sorry for lots of questions but you have so many interesting projects)? Are you still using/developing it?
Some time ago I was working on designing and implementing esy which is a package manager + meta build system (invoking package specific build system in hermetic environments) for compiled languages (OCaml/Reason/C/C++/…). It looks like Nix but has integrated SAT solver for solving deps, we rely on package.json metadata and npm as a registry of package sources (though we can install from git repos as well).
Personally, I think there’s a real opportunity to make a “lightweight” version of Nix/Guix which could be used widely and Hermes seems to be aimed to at this exact spot.
Do you mean you want to reimplement StandardML but on top of Janet’s like runtime? Or is there something specific to Janet which can influence the SML language itself?
The way janet has great CFFI and a few things like compile time evaluation and the language compilation model.
I also enjoy how janet can be distributed as a single amalgamated .c file like sqlite3.
My main criticism of janet is perhaps the lack of static types - and standard ML might be one of the simplest ‘real’ languages that incorporates a good type system, so I thought it might be a good place to start for ideas.
I’m myself contemplating a compile to LuaJIT ML-like language: the efficiently of LuaJIT and its convenient FFI + ergonomics of ML (though I’d want to experiment with adding modular implicits to the language).
Yeah, the way it complements C is something I would love to capture. I am not familiar with modular implicits at all - but it sounds interesting!
Is this related to Hermes (sorry for lots of questions but you have so many interesting projects)? Are you still using/developing it?
Yes and no - p2pkgs is an experiment to answer the question - ‘what if we combined ideas from Nix with something like homebrew in a simple way?’ I think the answer is something quite compelling but it still has a lot of tweaking to get right. p2pkgs uses a combination of a traditional package model - so far less patching is needed to build packages - it also is conceptually easier to understand than nix/hermes - while providing a large portion (but not all) of the benefits. You could consider p2pkgs like an exploratory search of ideas for ways to improve and simplify hermes. The optional p2p part was kind of an accident that seems to work so well in practice that I feel it is also important in it’s own way.
while providing a large portion (but not all) of the benefits
Could you possibly elaborate on which benefits are carried over, and which are not? I’m really interested in your explorations in this area of what I see as “attempts to simplify Nix”, but in this particular case, to the extent I managed to understand the repository, it’s currently very unclear to me what it really brings over just using redo to build whatever packages? Most notably, the core benefits I see in Nix (vs. other/older package managers), seem to be “capturing complete input state” of a build (a.k.a. pure/deterministic build environment), “perfectly clean uninstalls”, and “deterministic dependencies” including the possibility of packages depending on different versions of helper package. Does p2pkgs have any/all of those? It’s ok if not, I understand that this is just a personal exploration! Just would like to try and understand what’s going on there :)
seem to be “capturing complete input state” of a build (a.k.a. pure/deterministic build environment)
Yes it does, builds are performed in an isolated sandbox and use none of the host system.
“perfectly clean uninstalls”, and “deterministic dependencies” including the possibility of packages depending on different versions of helper package.
Packages are currently used via something I called a venv, this is more like nix shell, so has clean uninstalls - each venv can use different versions of packages, but within a venv you cannot - this is one of the downsides.
it’s currently very unclear to me what it really brings over just using redo to build whatever packages?
It uses redo + isolated build sandboxes + hashing of the dependency tree in order to provide transparent build caching, this is not so far removed from NixOS, which is why i feel nixos might be over engineered.
One thing p2pkgs does not have is atomic upgrades/rollback unless it is paired with something like docker.
All that being said, I think i oversimplified it to the point where the UX is not as good as it should be, so i hope to shift it back a bit to look a bit more like the nix cli - I think that will make things more clear.
Thanks! I was confused how redo works (had some wrong assumptions); now I start to understand that the main entry point (or, core logic) seems to be in the pkg/default.pkg.tar.gz.do file. I’ll try to look more into it, though at a first glance it doesn’t seem super trivial to me yet.
As to venv vs. NixOS, does “a linux user container” mean some extra abstraction layer?
Also, I don’t really understand “container with top level directories substituted for those in the requested packages” too well: is it some kind of overlayed or merged filesystem, where binaries running in the container see some extra stuff over the “host’s” filesystem? If yes, where can I read more about the exact semantics? If not, then what does it mean?
Back to “input completeness”: could you help me understand how/where can I exactly see/verify that e.g. a specific Linux kernel version was used to build a particular output? similarly, that a specific set of env variables was used? that a specific hash of a source tarball was used? (Or, can clearly see that changing one of those will result in a different output?) Please note I don’t mean this as an attack; rather still trying to understand better what am I looking at, and also hoping that the “simplicity” goal would maybe mean it’s indeed simple enough that I could inspect and audit those properties myself.
As to venv vs. NixOS, does “a linux user container” mean some extra abstraction layer?
Like Nixos, it uses containers to build packages, they are not needed to use packages - but they are helpful
Also, I don’t really understand “container with top level directories substituted for those in the requested packages” too well: is it some kind of overlayed or merged filesystem, where binaries running in the container see some extra stuff over the “host’s” filesystem? If yes, where can I read more about the exact semantics? If not, then what does it mean?
The build inputs are basically put into a chroot with the host system /dev/ added with a bind mount - this is quite similar to nixos - You can see it in default.pkg.tar.do
Back to “input completeness”: could you help me understand how/where can I exactly see/verify that e.g. a specific Linux kernel version was used to build a particular output?
Nixpkgs does not control the build kernel, not sure why you seem to think it does. Regardless - You can run redo pkg/.pkghash to compute the identity of a given package - which is the hash of all the build inputs including build scripts - again, much like nix. I suppose to control the build kernel we could use qemu instead of bwrap to perform the build. To see the inputs for a build you can also inspect the .bclosure file which is the build closure.
similarly, that a specific set of env variables was used? that a specific hash of a source tarball was used?
Env variables are cleared - this can be seen by the invocation of bwrap which is a container - much like nixpkgs.
I get the impression you might be misunderstanding the trust model of NixOS - nixos lets you run package builds yourself - but it still relies on signatures/https/trust for the binary package cache - You can’t go back from a given store path and workout the inputs - you can only go forward from an input to verify a store path.
also hoping that the “simplicity” goal would maybe mean it’s indeed simple enough that I could inspect and audit those properties myself.
The entire implementation is probably less than 700 lines of shell, i think you should be able to read them all - especially default.pkghash.do and default.pkg.tar.gz.do .
Thank you for your patience and bearing with me! I certainly might misunderstand some things from NixOS/Nixpkgs - I guess it’s part of the allure of p2pkgs that their simplicity may make those things easier to understand :) Though also part of my problem is that I’m having trouble expressing some things I’m thinking about here in precise terms, so I’d be super grateful if you’d still fancy having some more patience to me trying to continue searching for better precision of expression! And sorry if they’re still confused or not precise enough…
Like Nixos, it uses containers to build packages, they are not needed to use packages - but they are helpful
Hm; so does it mean I can run a p2pkgs build output outside venv? In Nixpkgs, AFAIU, this typically requires patchelf to have been run & things like make-wrapper (or what’s the name, I seem to never be able to remember it correctly). (How) does p2pkgs solve/approach this? Or did I misunderstand your answer here?
The build inputs are basically put into a chroot with the host system /dev/ added with a bind mount - this is quite similar to nixos - You can see it in default.pkg.tar.do
What I was asking here about was the “Running packages in venv” section - that’s where the “container with top level directories substituted (…)” sentence is used in p2pkgs readme. In other words: I’m trying to understand how during runtime any “runtime filesystem dependencies” (shared libraries, etc.; IIRC that’d be buildInputs in nixpkgs parlance) are merged with “host filesystem”. I tried reading bwrap’s docs in their repo, but either I couldn’t find the ultimate reference manual, or they’re just heavily underdocumented as to precise details, or they operate on some implicit assumptions (vs. chroot? or what?) that I don’t have.
In other words: IIUC (do I?), p2pkgs puts various FHS files in the final .pkg.tar.gz, which do then get substituted in the chroot when run with venv (that’s the way buildInputs would be made available to the final build output binary, no?). For some directory $X present in .pkg.tar.gz, what would happen if I wanted to use the output binary (say, vim), run via venv, to read and write a file in $X on host machine? How does the mechanism work that would decide whether a read(3) sees bytes from $X/foo/bar packed in .pkg.tar.gz vs. $X/foo/baz on host machine’s filesystem? Or, where would bytes passed to write(3) land? I didn’t manage to find answer to such question in bwrap’s docs that I found till now.
Do I still misunderstand something or miss some crucial information here?
Nixpkgs does not control the build kernel, not sure why you seem to think it does. (…)
Right. I now realize that actually in theory the Linux kernel ABI is stable, so I believe what I’m actually interested here in is libc. I now presume I can be sure of that, because the seed image contains gcc and musl (which I currently need to trust you on, yes?), is that so?
Env variables are cleared (…)
Ah, right: and then any explicitly set env vars result in build script changes, and then because it’s hashed for bclosure (or closure, don’t remember now), which is also included in (b?)closures of all dependees, the final (b?)closure depends on env vars. Cool, thanks!!
Hm; so does it mean I can run a p2pkgs build output outside venv? In Nixpkgs, AFAIU, this typically requires patchelf to have been run & things like make-wrapper (or what’s the name, I seem to never be able to remember it correctly). (How) does p2pkgs solve/approach this? Or did I misunderstand your answer here?
It replaces /bin /lib but keeps the rest of the host filesystem when you run the equivalent to a nix shell. This seems to work fine and lets you run programs against the host filesystem. This works because on modern linux kernels you can create containers and do bind mounts without root.
If we designed a package installer tool (and a distro?), it would also be possible to just install them like an alpine linux package.
I now presume I can be sure of that, because the seed image contains gcc and musl (which I currently need to trust you on, yes?), is that so?
You can rebuld the seed image using the package tree itself, the seed image is reproducible so you can check the output seed is the same as the input seed. You need to trust me initially though before you produce your own seed.
Ah, right: and then any explicitly set env vars result in build script changes, and then because it’s hashed for bclosure (or closure, don’t remember now), which is also included in (b?)closures of all dependees, the final (b?)closure depends on env vars. Cool, thanks!!
Some time ago I was working on designing and implementing esy which is a package manager + meta build system (invoking package specific build system in hermetic environments) for compiled languages (OCaml/Reason/C/C++/…). It looks like Nix but has integrated SAT solver for solving deps, we rely on package.json metadata and npm as a registry of package sources (though we can install from git repos as well).
I feel like it should be possible to use the same solver ideas or something like go MVS in order to make a distributed package tree - this is another idea I really want to try to integrate in something simpler than Nix. I agree that it seems like a great thing and I definitely want it to be built.
edit: I will investigate esy more - it definitely has most of what I want - The big difference seems to be how p2pkgs simply overrides / using user containers and installs them using DESTDIR.
I feel like it should be possible to use the same solver ideas or something like go MVS in order to make a distributed package tree
The depsolver is an interesting beast. I’m not satisfied with how it ended up in esy (though we had constraints to operate within, see below) — the feature I miss the most is the ability to create a separate “dependency graph” for packages which only expose executables (you don’t link them into some other apps) — dependencies from those packages shouldn’t impose constraints outside their own “dependency graphs”.
Ideally there should be some “calculus of package dependencies” developed which could be used as an interface between depsolver and a “metabuildsystem”. That way the same depsolver could be used with nix/hermes/esy/… Not sure how doable it is though — people don’t like specifying dependencies properly but then they don’t like to have their builds broken either!
edit: I will investigate esy more - it definitely has most of what I want - The big difference seems to be how p2pkgs simply overrides / using user containers and installs them using DESTDIR.
Keep in mind that we had our own set of constrains/goals to meet:
esy is usable on Linux/macOS/Windows (for example we ship Cygwin on Windows, this is transparent to the users)
esy uses npm as a primary package registry (appeal to people who know how to publish things to npm, an open world approach to managing packages)
esy doesn’t require administrator access to be installed/used (the built artefacts are inside the home directory)
esy strives to be compatible with OCaml ecosystem thus the depsolver is compatible with opam constraints and esy can install packages from opam
The depsolver is an interesting beast. I’m not satisfied with how it ended up in esy (though we had constraints to operate within, see below) — the feature I miss the most is the ability to create a separate “dependency graph” for packages which only expose executables (you don’t link them into some other apps) — dependencies from those packages shouldn’t impose constraints outside their own “dependency graphs”.
This is much like in a general package tree - statically linked programs really don’t care - but some programs don’t support static linking, or provide dynamic libraries.
Another challenge is when you don’t have a monolithic repository of all packages you now have double the versioning problems to tackle - Each version is really two - the package version and the packaged software version.
My goals for a general package tree are currently:
Linux only (it worked for docker).
Doesn’t require administrator for building or using packages.
Allows ‘out of tree packages’ or combining multiple package trees.
Transparent global build caching from trusted sources (like nixos, I don’t think esy has this).
Hey lobsters. @jspahrsummers and I created this at Anthropic. I hope I didn’t cross any lines of self-promotion here, but I am super excited about this and wanted to share it.
Let us know if you have any questions.
Congrats on launching. There a couple of weird things about this protocol launch:
Wehn faced with a similar need in my hobby project, https://chatcraft.org I found webrtc to be a much more suitable transport layer for llms as it can work for both local native apps and connect over network securely. https://gist.github.com/tarasglek/ff3353169d94e82cbd91218ac43188d6 might be sufficient to grasp my approach
We will work on remote connectivity. The current “implementation” is focused on local-only, but we hope the underlying primitives will hold true for remote connections.
My understanding is that Cohere Connectors and others aim to solve something very different. MCP is trying to solve the NxM problem of applications to context providers and build in the open. Anybody is free to implement it both on the client as well as on the server side. It is also focused on general interaction primitives rather than pure data providers (hence the separation of prompts and resources for example). As such it’s fundamentally different from proprietary APIs that are bound to a specific product surface. I personally feel LSP is a much closer related concept and something that inspired us.
Were you inspired by LSP?
Yes.
Why json schema and json rpc and stdio? I understand that json is the lowest common denominator (somewhat pejorative!) for data interchange with all kind of clients. Was making the transport work over http and stdio that important? Why couldn’t local agents just use http? Surely there must be a schema-based approach to specifying the entire protocol.
I work at Whimsical. We have a GPT that people use to create diagrams and flowcharts, and I’d love to build something similar for Claude. The Model Context Protocol is very cool but it seems like it currently requires every context provider to write a server that will run locally on a user’s machine.
It would be great if there was a server-side protocol that we could implement in our existing API which wouldn’t require writing a server. There are two reasons for this:
Do you have plans for something like this in the future?
We are working on thinking through “remote MCP”, which would solve this (as I understand it). We started with local MCP because we wanted to see if the concepts hold up while we work on hard parts of remote transports, particularly getting authentication right. We will tackle that aspect next and certainly recognize that this would unlock a lot of use cases.
(Also a good part of MCP comes from our own internal need and usage, where we can easily distribute servers)
That’s not a function - that’s a named value. It’ll be evaluated only once. To make it a function, you’d need to take an argument (usually unit, i.e.
()), like:I think this confusion is why some ML-syntax-like languages try to differentiate it with a different keyword for functions (like Rust’s
fn) or using lambda syntax (like Roc’sname = \arg -> result.)Apart from that, I agree with the author’s conclusion that OCaml is hard to get into. It’s hard to figure out how you’re supposed to solve common problems, and there’s scarce few “blessed” approaches to do common things. This is amplified by the OCaml 5 and domains situation because the earlier popular approaches for things aren’t necessarily how things should be done in the post-5 world. There’s been a few times where I look up how to do something, and decide on library/approach
foo, but while I’m working on it I face issues like what the author did withcrawl_website, and I’ll run into threads on reddit where people will be like “Obviously you shouldn’t usefoo, useppx_barinstead” and I’ll give up and resolve not to use OCaml again…Would following a guide like https://dev.realworldocaml.org/ help with the no-blessed-ways problem, in your opinion?
(I ask because I think I want to try going down the OCaml rabbit hole using that book.)
I haven’t looked into it too deeply, but even the newest edition of RW OCaml is not written for OCaml 5+, so it won’t cover domains and whatnot. Add to that the usual Async vs Lwt argument and I don’t recommend it right now.
Oh, darn. Very good to know…
nooooooooooo! I was hoping I could find a good resource to read.
Not a guide, but recently read this user experience of somebody switching to Eio which might be nice for comparison: https://tarides.com/blog/2024-09-19-eio-from-a-user-s-perspective-an-interview-with-simon-grondin/ - I don’t have any personal experience with OCaml’s async ecosystem though.
The readme is really good and worth a read https://github.com/ocaml-multicore/eio?tab=readme-ov-file
oh neato!
It’s me, author of this article. Thanks a lot for clarification. Does that mean even if env var value changes it won’t be reflected because it is executed only once?
kivikakk already covered your question but I’ll throw in a helpful tip for knowing the difference in the future: all functions in OCaml must have an
->in their type. You can use your editor’s LSP integration (e.g. in neovim I have to pressKon a symbol to see its docs) to see that the inferred type ofget_config_filenameisstring, while a function would be() -> string.Yes — you can prove this to yourself by trying something like this:
Output:
That said, how are you anticipating the environment variable changing? OCaml’s stdlib itself doesn’t expose
setenvorunsetenv— note aborted attempt here.edit: oh, never mind me!
Unix.putenvis mentioned right there, which of course works on macOS. So you can try this:Which outputs the same as above. Now note the change here:
And the output:
Keep in mind that OCaml 5 is very very young and that 5.0, 5.1 and even 5.2 are not considered ready yet. I agree that OCaml often lacks a one-true way but OCaml 5 is not a good example of that.
I agree that it is young, but at this point it objectively is the future of OCaml, so for someone just getting into it it doesn’t make sense to not start with it. Doubly so because it affects IO, which is what most basic and common programs do! The author of this article started out writing a scraper, my first attempt was to write a discord bot, and many people kick the tires with a web framework like Dream. So the problem of pre-4 approaches not being the future is more of a problem: as a newcomer, is what I’m learning going to become outdated soon? Especially when it’s the core of what I’m doing?
I don’t want to be overly negative abotu OCaml because I really like the language otherwise, so I’ll point out that there are people like @leostera on twitter who are working really hard on this problem and have made great progress. For people who want to get started with OCaml, I think https://github.com/leostera/minttea is natively OCaml 5 and really fun for making TUIs.
I believe it does make sense to start with Lwt rather than with Eio now. There’s simply more learning resources and more Lwt-compatible libraries. Lwt won’t become obsolete in the near future and the knowledge of OCaml+Lwt will transfer good to OCaml+Eio — same language after all.
I have a very similar experience, every time I want to try some networking related I am torn between Eio and Lwt.
Does it support osc52 for copy/paste, which is very useful to propagate clipboard across ssh/tmux/nvm or is that superseded by having wezterm available remotely and multiplexing (no tmux)?
Yup, it supports OSC 52.
Really love code mirror 6, I’ve used it to build a repl and a notebook for BQN.
does ghostty supports graphics in terminal? with kitty (and tmux) I can render graphviz directly onto terminal which I find handy when debugging ragel grammars, for example
Yes, It supports the Kitty graphics protocol.
Also, I’ve been planning to start working on sixel - I’ve implemented color decoding, and just left it after that
So we have JSX-like language support for:
Any others?
Technically Elixir has something similar to that thanks to sigil macros and HEEx.
Rust: https://yew.rs/docs/concepts/html#tag-structure
Haskell: https://hackage.haskell.org/package/ihp-hsx-0.18.0#readme
Scala 2 has XML literals, perhaps they could be used.
Pushup https://pushup.adhoc.dev is not JSX exactly, but it’s HTML-first blended with Go for control flow and variable access. (I’m the Pushup creator.)
Thanks. Not sure what to think about the syntax though. Why not make it like JSX instead? Would also make it easier to port tooling and editor support to.
This is pretty cool, I’m also doing AOC2022 partly in BQN, partly in my own K.jl (K on top of Julia).
This is a nice way to test software! By the way OCaml’s default build system dune has this feature built-in.
There’s also Lemon parser generator (which is used by SQLite, from the same authors).
Lemon is LALR(1), so it’s completely different.
Sure, just wanted to point out a good alternative, also LALR is superior to LR! :-)
SHENZHEN I/O
I remember this was useful for some Advent Of Code puzzle few years ago.
For those who like LALR parser generators there’s Lemon, the one used in SQLite.
I wonder if you could support mutability somehow?
I’m partly imagining torrent websites hosted on bittorrent (because it’s kinda meta) but could be generally useful/interesting perhaps.
There’s a bittorrent protocol extension where you can distribute a public key that points to a mutable torrent, but I don’t know if it has been ported to webtorrent.
The reference implementation for BEP0046 is done with webtorrent, don’t know if/how it works in browser though.
As far as I understand, you can’t use DHT in a web browser, as nodes do not support WebRTC. The Webtorrent project includes a DHT library that works with Node.js (which is used by the desktop application).
I’ve worked at the company which developed HTSQL.
It was used by software engineers and by (data/business) analysts.
In my experience HTSQL was very good, especially for data exploration / creating ad-hoc reports. It is easy to pick up and very concise.
Now the core developers of HTSQL are working on FunSQL, I’ve posted about it some days ago.
Besides that piece of tech they have Rex Deploy — an Ansible-style database migration system — https://github.com/prometheusresearch/baseline-codebase/blob/baseline/master/src/rex.deploy/README.rst —
I see FunSQL.jl is already mentioned in the discussion but I still want to highlight how it works as I think FunSQL represents a very interesting point in design space of “SQL replacements”.
FunSQL is a Julia library which presents a combinator-based API to generate SQL. The query construction looks like:
I’ve myself ported the lib (with some deviations) to Python and OCaml (ended up using this one) and built a concrete syntax on top of it, so at the end it looks very similar to PRQL, albeit I’ve kept it conservative syntax-wise (closer to SQL). Same query as above in the concrete syntax:
(I’ll use this syntax going forward in this post)
Now the feature I’ve wanted to highlight (and which I think makes FunSQL very useful) is how it treats
group byand aggregate expressions. An example first:Here the subrelation
cwhich is joined touis being grouped byuser_idcolumn. Thecitself doesn’t specify any aggregates though as you can see. All aggregates are located in the singleselectbelow.FunSQL treats grouped relations as a special kind of namespace from which you can either select columns you grouped by or refer to aggregate functions.
In my mind, this feature is very powerful as it allows one to incrementally construct reusable query fragments and then use them to build final queries. Same query in two steps:
In other words: you can build a nested namespace from relations / grouppings and then later decide what you want to select from this namespace.
This surprisingly works well if you want to implement visual query builders (I have one for my OCaml port).
Check out JuliaCon 2021 presentaton about FunSQL if interested, the main author Kyrylo (who’s by the way the co-creator of HTSQL, another query language mentioned in the discussion here) talks about motivation and design of FunSQL.
Doing AoC in BQN!
They’ve added virtual lines! Would love to see REPL plugins support this to show output inline. Also theorem prover plugins!
What is a virtual line?
It is a line of text visible in the editor but is not present in the actual text buffer. Previously nvim supported virtual text inline, used to show warnings or other information after a line of code for example. Now entire lines can be inserted virtually.
I might be poking at this in the future – editor integrations for annotated code.
I wish I could buy JetBrains products packaged as LSP servers (or with extensions or maybe using some other protocol) so I can use them with Vim.
Seems like they’re leveraging LSP, so this might not be too far fetched!
https://www.jetbrains.com/help/fleet/1.0/architecture-overview.html
I’ve enjoyed how they render Coq proofs with all the prove state extracted from Coq and available on hover.
That is legitimately awesome. Does anyone have any idea what plugin / library enables that? And, if it works for other proof assistants like Isabelle? (I’m not a Coq user)
https://github.com/cpitclaudel/alectryon/
Awesome, thank you. Wish it worked for Isabelle, but that’s great to have a tool to reference.
Ahh, based on the README it seems like it has preliminary support for Lean 3, which is neat!
You might also enjoy this presentation by the creator, and the paper that goes with it: Untangling mechanized proofs.
Yes, that’s great content, thank you.
Proof assistants are amazing tools, and some of them do offer a way to print out the full proof that they end up automating. Showing intermediate proof states is definitely nice, but some tactics still take very large leaps (like
tautoused in the paper). In those cases, I want to see insight into how the step was taken in between goals as well.Planning the next release of https://bupstash.io/ / https://github.com/andrewchambers/bupstash . I am way behind schedule but want to do a closed beta of managed repositories.
Also thinking a bit about a new programming language I want to work on for fun - A combination of standard ML drawing heavy inspiration from the https://github.com/janet-lang/janet runtime.
I also have some ideas kicking around for my peer to peer package tree - https://github.com/andrewchambers/p2pkgs .
So many things to do - clearly I need to avoid spending time on the less important things - I just have trouble reigning in what my mind is currently obsessing over.
Do you mean you want to reimplement StandardML but on top of Janet’s like runtime? Or is there something specific to Janet which can influence the SML language itself?
I’m myself contemplating a compile to LuaJIT ML-like language: the efficiently of LuaJIT and its convenient FFI + ergonomics of ML (though I’d want to experiment with adding modular implicits to the language).
Is this related to Hermes (sorry for lots of questions but you have so many interesting projects)? Are you still using/developing it?
Some time ago I was working on designing and implementing esy which is a package manager + meta build system (invoking package specific build system in hermetic environments) for compiled languages (OCaml/Reason/C/C++/…). It looks like Nix but has integrated SAT solver for solving deps, we rely on
package.jsonmetadata and npm as a registry of package sources (though we can install from git repos as well).Personally, I think there’s a real opportunity to make a “lightweight” version of Nix/Guix which could be used widely and Hermes seems to be aimed to at this exact spot.
The way janet has great CFFI and a few things like compile time evaluation and the language compilation model. I also enjoy how janet can be distributed as a single amalgamated .c file like sqlite3. My main criticism of janet is perhaps the lack of static types - and standard ML might be one of the simplest ‘real’ languages that incorporates a good type system, so I thought it might be a good place to start for ideas.
Yeah, the way it complements C is something I would love to capture. I am not familiar with modular implicits at all - but it sounds interesting!
Yes and no - p2pkgs is an experiment to answer the question - ‘what if we combined ideas from Nix with something like homebrew in a simple way?’ I think the answer is something quite compelling but it still has a lot of tweaking to get right. p2pkgs uses a combination of a traditional package model - so far less patching is needed to build packages - it also is conceptually easier to understand than nix/hermes - while providing a large portion (but not all) of the benefits. You could consider p2pkgs like an exploratory search of ideas for ways to improve and simplify hermes. The optional p2p part was kind of an accident that seems to work so well in practice that I feel it is also important in it’s own way.
Could you possibly elaborate on which benefits are carried over, and which are not? I’m really interested in your explorations in this area of what I see as “attempts to simplify Nix”, but in this particular case, to the extent I managed to understand the repository, it’s currently very unclear to me what it really brings over just using redo to build whatever packages? Most notably, the core benefits I see in Nix (vs. other/older package managers), seem to be “capturing complete input state” of a build (a.k.a. pure/deterministic build environment), “perfectly clean uninstalls”, and “deterministic dependencies” including the possibility of packages depending on different versions of helper package. Does p2pkgs have any/all of those? It’s ok if not, I understand that this is just a personal exploration! Just would like to try and understand what’s going on there :)
Yes it does, builds are performed in an isolated sandbox and use none of the host system.
Packages are currently used via something I called a venv, this is more like nix shell, so has clean uninstalls - each venv can use different versions of packages, but within a venv you cannot - this is one of the downsides.
It uses redo + isolated build sandboxes + hashing of the dependency tree in order to provide transparent build caching, this is not so far removed from NixOS, which is why i feel nixos might be over engineered.
One thing p2pkgs does not have is atomic upgrades/rollback unless it is paired with something like docker.
All that being said, I think i oversimplified it to the point where the UX is not as good as it should be, so i hope to shift it back a bit to look a bit more like the nix cli - I think that will make things more clear.
Thanks! I was confused how redo works (had some wrong assumptions); now I start to understand that the main entry point (or, core logic) seems to be in the pkg/default.pkg.tar.gz.do file. I’ll try to look more into it, though at a first glance it doesn’t seem super trivial to me yet.
As to venv vs. NixOS, does “a linux user container” mean some extra abstraction layer?
Also, I don’t really understand “container with top level directories substituted for those in the requested packages” too well: is it some kind of overlayed or merged filesystem, where binaries running in the container see some extra stuff over the “host’s” filesystem? If yes, where can I read more about the exact semantics? If not, then what does it mean?
Back to “input completeness”: could you help me understand how/where can I exactly see/verify that e.g. a specific Linux kernel version was used to build a particular output? similarly, that a specific set of env variables was used? that a specific hash of a source tarball was used? (Or, can clearly see that changing one of those will result in a different output?) Please note I don’t mean this as an attack; rather still trying to understand better what am I looking at, and also hoping that the “simplicity” goal would maybe mean it’s indeed simple enough that I could inspect and audit those properties myself.
Like Nixos, it uses containers to build packages, they are not needed to use packages - but they are helpful
The build inputs are basically put into a chroot with the host system /dev/ added with a bind mount - this is quite similar to nixos - You can see it in default.pkg.tar.do
Nixpkgs does not control the build kernel, not sure why you seem to think it does. Regardless - You can run redo pkg/.pkghash to compute the identity of a given package - which is the hash of all the build inputs including build scripts - again, much like nix. I suppose to control the build kernel we could use qemu instead of bwrap to perform the build. To see the inputs for a build you can also inspect the .bclosure file which is the build closure.
Env variables are cleared - this can be seen by the invocation of bwrap which is a container - much like nixpkgs. I get the impression you might be misunderstanding the trust model of NixOS - nixos lets you run package builds yourself - but it still relies on signatures/https/trust for the binary package cache - You can’t go back from a given store path and workout the inputs - you can only go forward from an input to verify a store path.
The entire implementation is probably less than 700 lines of shell, i think you should be able to read them all - especially default.pkghash.do and default.pkg.tar.gz.do .
Thank you for your patience and bearing with me! I certainly might misunderstand some things from NixOS/Nixpkgs - I guess it’s part of the allure of p2pkgs that their simplicity may make those things easier to understand :) Though also part of my problem is that I’m having trouble expressing some things I’m thinking about here in precise terms, so I’d be super grateful if you’d still fancy having some more patience to me trying to continue searching for better precision of expression! And sorry if they’re still confused or not precise enough…
Hm; so does it mean I can run a p2pkgs build output outside venv? In Nixpkgs, AFAIU, this typically requires patchelf to have been run & things like make-wrapper (or what’s the name, I seem to never be able to remember it correctly). (How) does p2pkgs solve/approach this? Or did I misunderstand your answer here?
What I was asking here about was the “Running packages in venv” section - that’s where the “container with top level directories substituted (…)” sentence is used in p2pkgs readme. In other words: I’m trying to understand how during runtime any “runtime filesystem dependencies” (shared libraries, etc.; IIRC that’d be
buildInputsin nixpkgs parlance) are merged with “host filesystem”. I tried reading bwrap’s docs in their repo, but either I couldn’t find the ultimate reference manual, or they’re just heavily underdocumented as to precise details, or they operate on some implicit assumptions (vs. chroot? or what?) that I don’t have.In other words: IIUC (do I?), p2pkgs puts various FHS files in the final .pkg.tar.gz, which do then get substituted in the chroot when run with venv (that’s the way
buildInputswould be made available to the final build output binary, no?). For some directory $X present in .pkg.tar.gz, what would happen if I wanted to use the output binary (say, vim), run via venv, to read and write a file in $X on host machine? How does the mechanism work that would decide whether aread(3)sees bytes from $X/foo/bar packed in .pkg.tar.gz vs. $X/foo/baz on host machine’s filesystem? Or, where would bytes passed towrite(3)land? I didn’t manage to find answer to such question in bwrap’s docs that I found till now.Do I still misunderstand something or miss some crucial information here?
Right. I now realize that actually in theory the Linux kernel ABI is stable, so I believe what I’m actually interested here in is libc. I now presume I can be sure of that, because the seed image contains gcc and musl (which I currently need to trust you on, yes?), is that so?
Ah, right: and then any explicitly set env vars result in build script changes, and then because it’s hashed for bclosure (or closure, don’t remember now), which is also included in (b?)closures of all dependees, the final (b?)closure depends on env vars. Cool, thanks!!
It replaces /bin /lib but keeps the rest of the host filesystem when you run the equivalent to a nix shell. This seems to work fine and lets you run programs against the host filesystem. This works because on modern linux kernels you can create containers and do bind mounts without root.
If we designed a package installer tool (and a distro?), it would also be possible to just install them like an alpine linux package.
You can rebuld the seed image using the package tree itself, the seed image is reproducible so you can check the output seed is the same as the input seed. You need to trust me initially though before you produce your own seed.
Thats right :).
I feel like it should be possible to use the same solver ideas or something like go MVS in order to make a distributed package tree - this is another idea I really want to try to integrate in something simpler than Nix. I agree that it seems like a great thing and I definitely want it to be built.
edit: I will investigate esy more - it definitely has most of what I want - The big difference seems to be how p2pkgs simply overrides / using user containers and installs them using DESTDIR.
The depsolver is an interesting beast. I’m not satisfied with how it ended up in esy (though we had constraints to operate within, see below) — the feature I miss the most is the ability to create a separate “dependency graph” for packages which only expose executables (you don’t link them into some other apps) — dependencies from those packages shouldn’t impose constraints outside their own “dependency graphs”.
Ideally there should be some “calculus of package dependencies” developed which could be used as an interface between depsolver and a “metabuildsystem”. That way the same depsolver could be used with nix/hermes/esy/… Not sure how doable it is though — people don’t like specifying dependencies properly but then they don’t like to have their builds broken either!
Keep in mind that we had our own set of constrains/goals to meet:
This is much like in a general package tree - statically linked programs really don’t care - but some programs don’t support static linking, or provide dynamic libraries.
Another challenge is when you don’t have a monolithic repository of all packages you now have double the versioning problems to tackle - Each version is really two - the package version and the packaged software version.
My goals for a general package tree are currently: