To characterize that SQLite started out as a TCL extension as a “fun fact” is to undermine the value of TCL and its importance to SQLite. SQLite was birthed from TCL and has been inexorably intertwined with it ever since. Without TCL, SQLite would not exist.
I was surprised that Hipp says SQLite’s code style is like Tcl’s, because Hipp’s style is really unusual. So I went to check Tcl and it’s much more conventional. So now I am differently surprised.
I wrote a fairly-easy to read full featured IRC client in a little over 1000 lines of TCL. (https://github.com/dlowe-net/irken) It was extremely fun to write and I learned a lot. With tcl9, I’m looking forward to having emoji actually show up. This is about the complexity limit of what I would use it for, though. The structure of the thing makes it difficult for me to make non-leaky abstractions.
Tcl is one of those languages that I wish I had a good use case for. I remember someone talking fondly about using it as an embedded command language (I think it was for a smart oscilloscope) and it sounded like a lovely improvement over bash or some kind of hacked-together little custom language a la early Unrealscript. I’ve played with it a few times and it does indeed have the Lisp style of fun where you can rewrite programs, without the shell style of hell where the contents of your program wobbles around depending on context and quoting style. But most of the times when I’d want a sweet little embedded language (mostly gamedev) I already would be using Lua or Fennel…
I’ve never enjoyed writing Tcl. So when I learned about Lua, found it fit my brain better, and learned that it was just as easy (maybe easier!) to embed as Tcl, I started to prefer it for every scenario where I would have considered Tcl without Tk.
Leaving aside situations where you need/want Tk, do you see any particular functional advantage to Tcl over Lua?
lua is a pretty poor bash alternative. the standard library lacks basic functionality for manipulating directories, you have to shell out for basic stuff like mkdir, introducing security vulnerabilities.
Kind of. This is both a strength and weakness of Lua. The standard library is minimal, which is good because it means it’s easy to embed Lua (last time I did the comparison, the Lua interpreter was under half the size of bash, not including the readline dependency in bash). It’s bad because it means that common things are often missing. Lua is easy to extend, so if you have a use case that depends on shell-like functionality, it’s easy to add process manipulation things. This is good because you can easily create tailored Lua environments, it’s bad because code won’t be portable between them.
I would probably use Lua a lot less if Sol3 didn’t exist. It makes exposing C++ types into Lua trivial (it bridges shared pointers and so on automatically, so memory management just works), so creating something with a C++ core and all of the business logic in Lua is very easy.
Tcl also has an event-based concurrent IO system. A friend of mine used Tcl to write anti-spam Software Against UCE because it could fit in a single process, and thereby handle many tarpitted connections. (Well, two processes, he also wrote an asynchronous DNS resolver because Tcl lacks one.)
I had a somewhat similar experience with Tcl. It took me several attempts to grasp the semantics of the language, a classical example of “you’re holding it wrong”, as Tcl is more an offspring of old-school shell scripting than of the C-like languages like Lua is one, and you have to treat it as such. I would argue that both languages are easy to embed. I’ve written bindings to both and find them well implemented each in their own way: the elegant stack-oriented approach of Lua and the “fat” object-based API of Tcl that features such niceties like linked variables.
However, I can see the following advantages of Tcl:
Tk: probably the greatest of the toolkits, makes it painless to add a platform-independent graphical user interface to an existing program.
Batteries included, at least if you install tcllib: covers the most important tasks while not being brimful like Python’s standard library. Lua, in contrast, is literally naked.
SQLite: ships with excellent Tcl bindings which are the perfect language enhancement for SQL programming.
For a larger engineering project, I’ve settled with Lua as the embedded scripting language in the end, due to Lua tables being the better data structure and because of the smaller interpreter size. Dictionaries came quite late to Tcl and are rather ugly inside a custom configuration file format (of course, I could have written something alike Lua tables in Tcl – everything is a command after all).
The Tcl Programming Language: a Comprehensive Guide is the most up-to-date book I’m aware of, and was a fun read for me a few years ago. It won’t cover the Tcl 9 stuff, but everything in the book I can remember ought to work as-is (and the Tcl 9 release notes will kind of just make sense afterwards, too).
This is also my favorite reference but it doesn’t cover Tk. The second edition of the original seminal text, Tcl and the Tk Toolkit (Ousterhout, et al) is an excellent reference.
I second this. I used that book to learn it too and it is indeed a great resource. Once you get the concepts the official documentation is a pretty good reference.
It has been really long since I did any Tcl, but since the commands are… well, commands, I got really far with just reading the official docs for them. On top of that their Wiki has really nice examples.
It’s really fun especially in combination with Tk.
They use Tcl all over the place apparently. I think he mentioned that he just e-mailed Tcl creator Ousterhout right when it was created, started hacking on it, and kept using it ever since.
They started the company back in 2004 when Python was still pretty niche. My reading is that it is kind of a Unix-y data ingestion problem, which needs to integrate with C, and Tcl is very good for that.
Python used to be like that too – it was a language for C devs to write in 2 languages – with Ousterhout’s dichotomy!
But these days I think most Python programmers use package managers and don’t write their own C extensions (which is progress in some ways, but I recommend writing a C extension to see how much power and simplicity it opens up. Although integrating ASAN is non-trivial so there are the usual C coding pitfalls pitfalls related to using Python’s reference counting API correctly.)
(This video happens to be from Houston Functional Programmers, which I spoke at, but that’s a coincidence. I just searched “Tcl” on YouTube, and this came up, and I ended up watching the whole video !! It is more for an “advanced” audience and thus very interesting/useful to me
I found that this is coherent language - Forth/Tcl/Lisp can be meaningfully integrated into a shell-like language :) Main difference from Tcl is that not everything is a string. Catbrain has 2 types – strings and lists. One primitive and one compound.
But these days I think most Python programmers use package managers and don’t write their own C extensions (which is progress in some ways, but I recommend writing a C extension to see how much power and simplicity it opens up. Although integrating ASAN is non-trivial so there are the usual C coding pitfalls.)
It’s even simpler just to plug Rust in and not worry about C coding pitfalls. pyo3 and maturin make it trivial, even.
If your coworkers write a lot of C/C++, as is the case at FlightAware, Google, etc.
Being able to wrap these things yourself opens up a lot of power …
Also how do pyo3 and maturin handle the Python reference counting protocol? My experience with SWIG is that it works 98% of the time, with some good heuristics, and then there is a memory leak now and then
You’re supposed to “fix” the heuristics with your own annotations, and then you are never sure if there is a bug, because the mechanism is so large and opaque
In fact, the reason I learned the Python/C API is because SWIG generated a binding with a memory leak
I suspect you probably have to add a bunch of manual annotations even in Rust, which can be wrong. i.e. they need to be unit tested. Interfacing between languages is precisely where Rust helps the least, because CPython does not follow Rust’s type or ownership system
But yes, if you have a bunch of Rust code and want to wrap it, obviously those solutions become more appealing
Also how do pyo3 and maturin handle the Python reference counting protocol? My experience with SWIG is that it works 98% of the time, with some good heuristics, and then there is a memory leak now and then
Correctly, probably? Like most FFI, it’s up to the side that allocates the memory to de-allocate it. I haven’t seen any issues with resource usage from using Polars, as an example. Nor from my other Rust-implemented, Python-binded tools I use in my everyday work.
Also, memory leaks are safe, according to Rust.
According to everyone? What UB are you triggering by leaking memory?
in my experience, Tk is by far the easiest way to throw together a quick gui. even if you already know a different programming language, it’s often faster to just learn Tcl than to try and get something else working.
also, it can go from source code to an open window way faster than anything else.
it’s fairly niche, but for it’s niche, it is without competition.
I wonder why it isn’t used more. It’s been around since forever and people who know it seem to really love it, but it doesn’t seem very widely used. Is it like “the Lisp curse” where everyone rolls their own stuff, and it’s hard to standardize things and it doesn’t scale well to very large teams, so the corporate suits forbid it?
I know this is a very incomplete answer, but a large part of what’s been going on is that we’re spoilt for choice, and already were even when TCL was young!
set a foo
set b bar
# With apply
set closure [list {{a b} {puts "I captured $a and $b"}} $a $b]
apply {*}$closure ;# will print "I captured foo and bar"
# With dict with
set closureVars [dict create a $a b $b]
set closureBody {puts "I captured $a and $b"}
dict with closureVars $closureBody ;# will print "I captured foo and bar"
With a small convenience function you can avoid repeating variable names:
proc capture {varNames body} {
list [list $varNames $body] {*}[lmap varName $varNames {
upvar $varName var
set var
}]
}
set closure [capture {a b} {
puts "I captured $a and $b"
}]
apply {*}$closure ;# will print "I captured foo and bar"
OK, but what if I want to pass some regular arguments to my function, and I want to make it look like a regular function instead of something that has to be applyed. Then I have the same problem, since now my wrapper function has to close on the original closed arguments and function body. Maybe this could be automated…
Fun fact: SQLite started out as a li’l embedded database engine for TCL.
The primary SQLite authors have immense amounts of interesting Tcl code, including a custom and comprehensive test suite.
I’ve really enjoyed using Tcl to build a SQLite search index for a website and the CGI search. They work really nicely together.
To characterize that SQLite started out as a TCL extension as a “fun fact” is to undermine the value of TCL and its importance to SQLite. SQLite was birthed from TCL and has been inexorably intertwined with it ever since. Without TCL, SQLite would not exist.
https://www.tcl.tk/community/tcl2017/assets/talk93/Paper.html
I was surprised that Hipp says SQLite’s code style is like Tcl’s, because Hipp’s style is really unusual. So I went to check Tcl and it’s much more conventional. So now I am differently surprised.
https://github.com/sqlite/sqlite/blob/master/src/expr.c
https://github.com/tcltk/tcl/blob/main/generic/tclExecute.c
(OMG form feeds as section separators, retro <3)
I wrote about this in my story about the release of Tcl/Tk 12, 27 years after Tcl/Tk 8.
https://www.theregister.com/2024/10/02/tcltk_version_9/
That is a fun little tidbit of info!
I wrote a fairly-easy to read full featured IRC client in a little over 1000 lines of TCL. (https://github.com/dlowe-net/irken) It was extremely fun to write and I learned a lot. With tcl9, I’m looking forward to having emoji actually show up. This is about the complexity limit of what I would use it for, though. The structure of the thing makes it difficult for me to make non-leaky abstractions.
Tcl is one of those languages that I wish I had a good use case for. I remember someone talking fondly about using it as an embedded command language (I think it was for a smart oscilloscope) and it sounded like a lovely improvement over bash or some kind of hacked-together little custom language a la early Unrealscript. I’ve played with it a few times and it does indeed have the Lisp style of fun where you can rewrite programs, without the shell style of hell where the contents of your program wobbles around depending on context and quoting style. But most of the times when I’d want a sweet little embedded language (mostly gamedev) I already would be using Lua or Fennel…
I’ve never enjoyed writing Tcl. So when I learned about Lua, found it fit my brain better, and learned that it was just as easy (maybe easier!) to embed as Tcl, I started to prefer it for every scenario where I would have considered Tcl without Tk.
Leaving aside situations where you need/want Tk, do you see any particular functional advantage to Tcl over Lua?
lua is a pretty poor bash alternative. the standard library lacks basic functionality for manipulating directories, you have to shell out for basic stuff like mkdir, introducing security vulnerabilities.
Kind of. This is both a strength and weakness of Lua. The standard library is minimal, which is good because it means it’s easy to embed Lua (last time I did the comparison, the Lua interpreter was under half the size of bash, not including the readline dependency in bash). It’s bad because it means that common things are often missing. Lua is easy to extend, so if you have a use case that depends on shell-like functionality, it’s easy to add process manipulation things. This is good because you can easily create tailored Lua environments, it’s bad because code won’t be portable between them.
I would probably use Lua a lot less if Sol3 didn’t exist. It makes exposing C++ types into Lua trivial (it bridges shared pointers and so on automatically, so memory management just works), so creating something with a C++ core and all of the business logic in Lua is very easy.
Tcl also has an event-based concurrent IO system. A friend of mine used Tcl to write anti-spam Software Against UCE because it could fit in a single process, and thereby handle many tarpitted connections. (Well, two processes, he also wrote an asynchronous DNS resolver because Tcl lacks one.)
I had a somewhat similar experience with Tcl. It took me several attempts to grasp the semantics of the language, a classical example of “you’re holding it wrong”, as Tcl is more an offspring of old-school shell scripting than of the C-like languages like Lua is one, and you have to treat it as such. I would argue that both languages are easy to embed. I’ve written bindings to both and find them well implemented each in their own way: the elegant stack-oriented approach of Lua and the “fat” object-based API of Tcl that features such niceties like linked variables.
However, I can see the following advantages of Tcl:
For a larger engineering project, I’ve settled with Lua as the embedded scripting language in the end, due to Lua tables being the better data structure and because of the smaller interpreter size. Dictionaries came quite late to Tcl and are rather ugly inside a custom configuration file format (of course, I could have written something alike Lua tables in Tcl – everything is a command after all).
[Comment removed by author]
This certainly has me interested! I’m curious if anyone has recommendations for Tcl learning resources that are relatively up-to-date?
The Tcl Programming Language: a Comprehensive Guide is the most up-to-date book I’m aware of, and was a fun read for me a few years ago. It won’t cover the Tcl 9 stuff, but everything in the book I can remember ought to work as-is (and the Tcl 9 release notes will kind of just make sense afterwards, too).
This is also my favorite reference but it doesn’t cover Tk. The second edition of the original seminal text, Tcl and the Tk Toolkit (Ousterhout, et al) is an excellent reference.
For Tk specifically, my go to is https://tkdocs.com/
I second this. I used that book to learn it too and it is indeed a great resource. Once you get the concepts the official documentation is a pretty good reference.
Thank you so much!
It has been really long since I did any Tcl, but since the commands are… well, commands, I got really far with just reading the official docs for them. On top of that their Wiki has really nice examples.
It’s really fun especially in combination with Tk.
The manpages are excellent (
man n expretc)The website’s theme is very fitting to the article. Looks like a CDE inspired theme? But I could be wrong on that.
Looks like the about page confirms it:
Very nice video about Tcl from a core dev Karl Lehenbauer, who is one of the founders of FlightAware:
https://www.youtube.com/watch?v=3YwFHPFL20c&t=2317s
https://en.wikipedia.org/wiki/FlightAware
They use Tcl all over the place apparently. I think he mentioned that he just e-mailed Tcl creator Ousterhout right when it was created, started hacking on it, and kept using it ever since.
They started the company back in 2004 when Python was still pretty niche. My reading is that it is kind of a Unix-y data ingestion problem, which needs to integrate with C, and Tcl is very good for that.
Python used to be like that too – it was a language for C devs to write in 2 languages – with Ousterhout’s dichotomy!
But these days I think most Python programmers use package managers and don’t write their own C extensions (which is progress in some ways, but I recommend writing a C extension to see how much power and simplicity it opens up. Although integrating ASAN is non-trivial so there are
the usual C coding pitfallspitfalls related to using Python’s reference counting API correctly.)(This video happens to be from Houston Functional Programmers, which I spoke at, but that’s a coincidence. I just searched “Tcl” on YouTube, and this came up, and I ended up watching the whole video !! It is more for an “advanced” audience and thus very interesting/useful to me
https://www.oilshell.org/blog/2024/05/houston-fp.html
Thanks to Claude Jager-Rubinson for running this nice series)
Also I still have to publish a repo for “catbrain”, my Forth/Tcl/Lisp that can express Shell/Awk/Make … mentioned here http://www.oilshell.org/blog/2024/09/retrospective.html#help-wanted
I found that this is coherent language - Forth/Tcl/Lisp can be meaningfully integrated into a shell-like language :) Main difference from Tcl is that not everything is a string. Catbrain has 2 types – strings and lists. One primitive and one compound.
It’s even simpler just to plug Rust in and not worry about C coding pitfalls.
pyo3andmaturinmake it trivial, even.Sure, if you want to write Rust … but CPython is written in C, and you might want to wrap C or C++
There are many situations where that arises, e.g.
embedded systems, e.g. robotics, or custom sensors
proprietary systems like GPUs - they are very much in the C++ world
If you want to try out some newer kernel features - https://man7.org/linux/man-pages/man2/unshare.2.html - or say pledge() on OpenBSD
If your coworkers write a lot of C/C++, as is the case at FlightAware, Google, etc.
Being able to wrap these things yourself opens up a lot of power …
Also how do pyo3 and maturin handle the Python reference counting protocol? My experience with SWIG is that it works 98% of the time, with some good heuristics, and then there is a memory leak now and then
You’re supposed to “fix” the heuristics with your own annotations, and then you are never sure if there is a bug, because the mechanism is so large and opaque
In fact, the reason I learned the Python/C API is because SWIG generated a binding with a memory leak
I suspect you probably have to add a bunch of manual annotations even in Rust, which can be wrong. i.e. they need to be unit tested. Interfacing between languages is precisely where Rust helps the least, because CPython does not follow Rust’s type or ownership system
But yes, if you have a bunch of Rust code and want to wrap it, obviously those solutions become more appealing
Also, memory leaks are safe, according to Rust.
Correctly, probably? Like most FFI, it’s up to the side that allocates the memory to de-allocate it. I haven’t seen any issues with resource usage from using Polars, as an example. Nor from my other Rust-implemented, Python-binded tools I use in my everyday work.
According to everyone? What UB are you triggering by leaking memory?
in my experience, Tk is by far the easiest way to throw together a quick gui. even if you already know a different programming language, it’s often faster to just learn Tcl than to try and get something else working.
also, it can go from source code to an open window way faster than anything else.
it’s fairly niche, but for it’s niche, it is without competition.
I wonder why it isn’t used more. It’s been around since forever and people who know it seem to really love it, but it doesn’t seem very widely used. Is it like “the Lisp curse” where everyone rolls their own stuff, and it’s hard to standardize things and it doesn’t scale well to very large teams, so the corporate suits forbid it?
I know this is a very incomplete answer, but a large part of what’s been going on is that we’re spoilt for choice, and already were even when TCL was young!
Since this is the TCL thread: does anyone know how to do closures? Like the obvious way is something like
You can use
applyordict with:With a small convenience function you can avoid repeating variable names:
OK, but what if I want to pass some regular arguments to my function, and I want to make it look like a regular function instead of something that has to be
applyed. Then I have the same problem, since now my wrapper function has to close on the original closed arguments and function body. Maybe this could be automated…The wrapper gets a bit more complex though. See the bottom of https://wiki.tcl-lang.org/page/Closures
Challenge accepted!
Use {} to quote code.
Tcl doesn’t have lexical scope, so a code block does not close over captured variables.
Yes, I’m aware how quoting works. And I’m asking because want to have closures.
I liked Tcl very much when I was writing n2 scripts. Tk on the other hand, I could not get my head around
aaargh, ns-2