[ planet-factor ]

I had a lot of fun writing code to compute derangements recently. I thought I was done with that topic until I bumped into a question on StackOverflow asking how to generate a random derangement of a list. Being nerd sniped is a real thing, and so I started looking at solutions.

There’s a paper called “An analysis of a simple algorithm for random derangements” that has an, ahem, simple algorithm. The basic idea is to generate a random permutation of indices, breaking early if the random permutation is obviously not a derangement.

One way to take a random permutation would be to use our permutations virtual sequence:

IN: scratchpad "ABCDEF" <permutations> random .
"FCEBDA" ! is a derangement

IN: scratchpad "ABCDEF" <permutations> random .
"DFBCEA" ! is NOT a derangement

And so you could loop until a derangement of indices is found:

: random-derangement-indices ( n -- seq )
    f swap <iota> <permutations>
    '[ drop _ random dup derangement? not ] loop ;

But, since only 36% or so of permutations are derangements, perhaps it would be faster and better to implement the algorithm from that paper – making our own random permutation of indices and breaking early if obviously not a derangement:

:: random-derangement-indices ( n -- indices )
    n <iota> >array :> seq
    f [
        dup :> v
        n 1 (a..b] [| j |
            j 1 + random :> p
            p v nth j = [ t ] [ j p v exchange f ] if
        ] any? v first zero? or
    ] [ drop seq clone ] do while ;

We can use that to build a random-derangement word:

: random-derangement ( seq -- seq' )
    [ length random-derangement-indices ] [ nths ] bi ;

And then, for example, get a random derangement of the alphabet – of which there are one hundred and forty-eight septillion derangements, give or take – in under a millisecond:

IN: scratchpad "ABCDEFGHIJKLMNOPQRSTUVWXYZ" random-derangement .
"CZFABMSUXRQDEHGYJLTPVOIKWN"

We could check to make sure that we generate all derangments with equal possibility using a simple test case:

IN: scratchpad 1,000,000 [
                   "ABCD" random-derangement
               ] replicate histogram sort-keys .
{
    { "BADC" 111639 }
    { "BCDA" 110734 }
    { "BDAC" 110682 }
    { "CADB" 111123 }
    { "CDAB" 111447 }
    { "CDBA" 111147 }
    { "DABC" 111215 }
    { "DCAB" 111114 }
    { "DCBA" 110899 }
}

Looks good to me!

Derangements, also sometimes known as deranged permutations, are described as:

In combinatorial mathematics, a derangement is a permutation of the elements of a set in which no element appears in its original position. In other words, a derangement is a permutation that has no fixed points.

There is a fun online derangements generator tool that you can use to play with computing the derangements of a sequence as well as calculating the number of derangements for a given sequence size.

As an example, we can use the math.combinatorics vocabulary, to generate all the permutations of the sequence { 0 1 2 }:

IN: scratchpad { 0 1 2 } all-permutations .
{
    { 0 1 2 }
    { 0 2 1 }
    { 1 0 2 }
    { 1 2 0 }
    { 2 0 1 }
    { 2 1 0 }
}

Since a derangement is a permutation that requires each element to be in a different slot, we could write a word to check the permuted indices to see if that is true:

: derangement? ( indices -- ? )
    dup length <iota> [ = ] 2any? not ;

These would be the two derangements of the indices { 0 1 2 }:

IN: scratchpad { 0 1 2 } all-permutations [ derangement? ] filter .
{
    { 1 2 0 }
    { 2 0 1 }
}

The number of derangements is the subfactorial of the length of the sequence:

: subfactorial ( n -- ? )
    [ 1 ] [ factorial 1 + e /i ] if-zero ;

We can build a <derangement-iota> that is a sequence as long as that number:

: <derangement-iota> ( seq -- <iota> )
    length subfactorial <iota> ; inline

And we can build a next-derangement word that calculates the next permutation that is a derangement:

: next-derangement ( seq -- seq )
    [ dup derangement? ] [ next-permutation ] do until ;

We can then build upon some of the code for iterating permutations, designing an internal derangements-quot word that is similar in form to the existing permutations-quot word:

: derangements-quot ( seq quot -- seq quot' )
    [ [ <derangement-iota> ] [ length <iota> >array ] [ ] tri ] dip
    '[ drop _ next-derangement _ nths-unsafe @ ] ; inline

And then use it to build a series of words that can provide iteration across derangements:

: each-derangement ( ... seq quot: ( ... elt -- ... ) -- ... )
    derangements-quot each ; inline

: map-derangements ( ... seq quot: ( ... elt -- ... newelt ) -- ... newseq )
    derangements-quot map ; inline

: filter-derangements ( ... seq quot: ( ... elt -- ... ? ) -- ... newseq )
    selector [ each-derangement ] dip ; inline

: all-derangements ( seq -- seq' )
    [ ] map-derangements ;

: all-derangements? ( ... seq quot: ( ... elt -- ... ? ) -- ... ? )
    derangements-quot all? ; inline

: find-derangement ( ... seq quot: ( ... elt -- ... ? ) -- ... elt/f )
    '[ _ keep and ] derangements-quot map-find drop ; inline

: reduce-derangements ( ... seq identity quot: ( ... prev elt -- ... next ) -- ... result )
    swapd each-derangement ; inline

And, now we can use this to find the nine derangements for "ABCD":

IN: scratchpad "ABCD" all-derangements .
{
    "BADC"
    "BCDA"
    "BDAC"
    "CADB"
    "CDAB"
    "CDBA"
    "DABC"
    "DCAB"
    "DCBA"
}

This is available on my GitHub.

Years ago, I remember reading a blog post called “Why is Groovy is big?” by Slava Pestov, the original author of Factor. In it, he talks about lines of code and sets the stage for how concatenative thinking can lead to properties like conciseness and readability, ending with this:

I tend to think the majority of code people write is overly complicated, full of redundancy, and designed for such flexibility that in practice is not needed at all. I hope one day this trend reverses.

I’ve been thinking a lot recently about 20 years of Factor and I thought it would be fun to write a Zen of Factor. Perhaps inspired by the Zen of Programming book written in 1987 by Geoffrey James, there are a couple of examples I wanted to point to first.

Python

The Python programming language was one of the first languages that I am aware of to get a Zen, contributed at least as far back as February 2002 in a kind of easter egg fashion encrypted by ROT13:

$ python -c "import this" 
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Zig

Quite a bit more recently, the Zig programming language introduced a Zen through a series of commits starting in August 2017. You can see the current version by running zig zen:

$ zig zen

 * Communicate intent precisely.
 * Edge cases matter.
 * Favor reading code over writing code.
 * Only one obvious way to do things.
 * Runtime crashes are better than bugs.
 * Compile errors are better than runtime crashes.
 * Incremental improvements.
 * Avoid local maximums.
 * Reduce the amount one must remember.
 * Focus on code rather than style.
 * Resource allocation may fail; resource deallocation must succeed.
 * Memory is a resource.
 * Together we serve the users.

Factor

In that spirit, I thought it would be fun to put together some ideas for what a Zen of Factor might look like, incorporating aspects of the language that I enjoy, some thematic elements that might make you more successful learning and developing in it, as well as pointing out the strong community that we have and hope to grow.

Here is the current draft:

The Zen of Factor

The REPL is your playground.
Working code beats perfect theory.
Words are better than paragraphs.
Stack effects tell the story.
Any syntax you need, you can create.
Simple primitives yield powerful combinations.
First make it work, then make it beautiful.
Make it beautiful, then make it fast.
Quick hacks grow into robust solutions.
When in doubt, factor it out.
Every word should do one thing well.
Let the stack guide your logic.
Write less, compose more.
If it works, ship it.
If it doesn't work, fix it.
If you don't like it, change it.
Today's beginner is tomorrow's core developer.
Questions encouraged, PRs welcome.

I really appreciate hearing about programs and problems that developers work on, getting feedback that allows us to iteratively improve, and knowing that every commit leads us towards a better future.

Thank you!

Factor has a watching words feature that allows you to see the inputs and outputs of a word when it is called. I have wanted a way to define a watched code block that we could use to see the stack before and after some inner part of a word.

It turns out that it’s pretty simple to make this syntax using our existing watch implementation from the tools.annotations vocabulary:

DEFER: WATCH>

SYNTAX: <WATCH
    \ WATCH> parse-until >quotation dup (watch) append! ;

Using this, we can now watch the inner part of a word, for example this word that increments the input by 1:

: foo ( x -- y ) 1 <WATCH + WATCH> ;

And see it used:

IN: scratchpad 10 foo
--- Entering [ + ]
x 10
x 1
--- Leaving [ + ]
x 11

--- Data stack:
11

One thing that I’d like to improve on this someday, is making it so this watch syntax has a prettyprint implementation that allows it to be rendered as it is typed.

I have added this to the latest developer version if you’d like to update and give it a try!

Factor contains a REPL – called the Listener – available on the command-line and graphically as part of the UI developer tools. For many users, this is their main interface to programming in the Factor programming language.

We sometimes get requests to better support styling the user interface. This has led to improvements such as support for light and dark themes, adjustable font sizes, and other customizations. There have been a few existing ways to style the UI listener including support for keyboard commands to increase or decrease font sizes. But, until recently this only affected new output or new Listener sessions.

Today, I improved this to make adjusting the font size much more dynamic, using traditional keyboard shortcuts of Ctrl – or Cmd on macOS – combined with + or -:

Give it a try, and please let us know other ways we can make improvements!

Recently, I’ve been inspired by conversations taking place on our Factor Discord server. This sometimes reflects areas of interest from new contributors, curiousity exploring similarities and differences between Factor and other programming languages, or even early learning moments when exploring concatenative languages in general.

Today, someone asked about how to think about “accumulation of values in an array… to find all occurences (their position) of a subseq in a seq”. The solution to this might have this word name and stack effect:

: subseq-indices ( seq subseq -- indices ) ... ;

Before answering, I wanted to make sure they wanted to find overlapping indices vs. non-overlapping indices, and they clarified that they expect it to find this result – allowing overlapping subsequences:

IN: scratchpad "abcabcabc" "abcabc" subseq-indices .
{ 0 3 }

So, now that we have a reasonable specification, how do we think about solving this problem when we are at the same time learning to solve problems in stack languages and trying to see what features of Factor’s standard library would help.

There are a lot of ways to think about this, and I often recommend one of three approaches:

1. starting inside-out (working on the inner part of the loop)
2. starting outside-in (modeling the outer loop and then figuring out what’s inside it)
3. using local variables (helpful when coming from an applicative language background)

So let’s look at each approach in turn:

Inside Out

The inner logic is going to require something like “take an index to start from and find the next matching subseq index”, which looks an awful lot like subseq-index-from – except you also want to increment the found index afterwards to make sure you are progressing through the sequence.

: next-subseq-index ( index seq subseq -- next-index/f found-index/f )
    subseq-index-from [ [ 1 + ] keep ] [ f f ] if* ;

Then you could use it like so in a loop with an accumulator:

: subseq-indices ( seq subseq -- indices )
    [ V{ } clone 0 ] 2dip '[
        _ _ next-subseq-index dup [ [ pick push ] keep ] when
    ] loop drop ;

But that feels like we had to work hard to do that, directly using an accumulator, conditionals, and some stack shuffling. Luckily we have some higher level words that might help, for example the make vocabulary which has an implicit accumulator that we can use , or % to push into:

: subseq-indices ( seq subseq -- indices )
    [ 0 ] 2dip '[
        [ _ _ next-subseq-index dup [ , ] when* ] loop
    ] { } make nip ;

Or even using a while* loop, which is less code:

: subseq-indices ( seq subseq -- indices )
    [ 0 ] 2dip '[
        [ _ _ next-subseq-index ] [ , ] while*
    ] { } make nip ;

But that feels like a lot too, simpler might be produce:

: subseq-indices ( seq subseq -- indices )
    [ 0 ] 2dip '[ _ _ next-subseq-index dup ] [ ] produce 2nip ;

Or using follow, adjusting our start index and increment:

: subseq-indices ( seq subseq -- indices )
    [ -1 ] 2dip '[ 1 + _ _ subseq-index-from ] follow rest ;

Outside In

The outer logic approach would be something like “we need to loop from the start of the sequence, finding the next match, and accumulating it, until we hit some exit condition and then return a result” which you could write in a kind of non-functional stack pseudocode:

: subseq-indices ( seq subseq -- indices )
    0 [ find-next-match ] [ accumulate-match ] while ;

Then you have to kind of figure out what goes into those blocks:

: find-next-match ( seq subseq n -- found-index/f )
    -rot subseq-index-from ;

And also something like:

: accumulate-match ( accum found-index -- accum next-index )
    [ suffix! ] keep 1 + ;

Taking those, and maybe thinking about what items should be on the stack and in what order to reduce stack shuffling, becomes something like:

: subseq-indices ( seq subseq -- indices )
    [ V{ } clone 0 ] 2dip
    '[ _ _ subseq-index-from ] [ [ suffix! ] keep 1 + ] while* ;

It is true that [ suffix! ] keep 1 + is also [ suffix! ] [ 1 + ] bi, with varying aesthetics and ease of understanding, but sometimes when learning a new language especially a stack language with combinators, it is sometimes easy to start with stack shuffling and then learn about these forms later to see if they can improve your code.

Locals

Instead of those two stack approaches, we could instead use our local variables and write one big word in a manner similar to applicative languages, stepping back and focusing on the result we want:

:: subseq-indices ( seq subseq -- indices )
    V{ } clone :> accum
    0 :> i!

    [ i seq subseq subseq-index-from ]
    [ dup accum push 1 + i! ] while*

    accum ;

When working on this stuff, it’s nice to remember you can put a B to set a breakpoint in places to examine the stack at some inner point, or perhaps write a comment showing the incoming stack and optionally the outgoing stack that a piece of code is expected to have so that you understand what is happening in the next few lines:

! the next block of code finds the next index
! ( index seq subseq -- found-index )

! and pushes it into an accumulator
! ( accum found-index -- accum )

This was added to the developer branch in the sequences.extras vocabulary.

We love to hear questions and it’s even better when we can provide answers or guidance for learning and solving problems. Feel free to join our conversations and explore learning Factor!

There was an interesting question on the Unix & Linux StackExchange asking how to remove subdomains or existing domains. I thought it would be fun to show a few different approaches to solving this using Factor.

Our first step should be to understand what is a subdomain:

A subdomain is a prefix added to a domain name to separate a section of your website. Site owners primarily use subdomains to manage extensive sections that require their own content hierarchy, such as online stores, blogs, job boards or support platforms.

Common Subdomains

If we’re curious about what common subdomains are, we can turn to the SecLists project – described as a “security tester’s companion” – which maintains a list of common 5,000 subdomains, 20,000 subdomains, and 110,000 subdomains that were generated in 2015 as well as a combined subdomains list that has some additional ones added.

You can download the top 5,000 common subdomains using memoization to cache the result:

MEMO: top-5000-subdomains ( -- subdomains )
    "https://raw.githubusercontent.com/danielmiessler/SecLists/refs/heads/master/Discovery/DNS/subdomains-top1million-5000.txt"
    cache-directory download-once-into utf8 file-lines ;

And then see what the “top 10” are:

IN: scratchpad top-5000-subdomains 10 head .
{
    "www"
    "mail"
    "ftp"
    "localhost"
    "webmail"
    "smtp"
    "webdisk"
    "pop"
    "cpanel"
    "whm"
}

You could remove “common subdomains” – adding a dot to make sure we only strip a full subdomain – by recursively trying to clean the hostname until it stops changing.

: remove-common-subdomains ( host -- host' )
    top-5000-subdomains [ "." append ] map '[ _ [ ?head ] any? ] loop ;

And try it out:

IN: scratchpad "www.mail.ftp.localhost.factorcode.org"
               remove-common-subdomains .
"factorcode.org"

That works pretty well, but it’s reliant on a scraped list of subdomains that might not be exhaustive, and could become stale over time as the tools and techniques that developers use change.

Observed Subdomains

Similarly, another technique we could use would be to use our own observations about domains, and if we observe a domain being used and then subsequently see a subdomain of it, we can ignore the subdomain.

First, we write a word to remove any item that is prefixed by another, sorting to make sure we see the prefix before the item prefixed by it:

: remove-prefixed ( seq -- seq' )
    sort V{ } clone [
        dup '[
            [ _ [ head? ] with none? ] _ push-when
        ] each
    ] keep ;

Second, we can remove the subdomains by using a kind of Schwartzian transform:

1. reverse the domain names
2. remove the ones that are prefixed by another
3. un-reverse the domain names

: remove-observed-subdomains ( hosts -- hosts' )
    [ "." prepend reverse ] map remove-prefixed [ reverse rest ] map ;

And then see it work:

IN: scratchpad { "a.b.c" "b.c" "c.d.e" "e.f" }
               remove-observed-subdomains .
V{ "b.c" "c.d.e" "e.f" }

Resolving Domains

And, finally, another technique might be to use the Domain Name System to find the rootiest domain name.

First, we use our dns vocabulary to check that a host resolves to an IP address:

: valid-domain? ( host -- ? )
    {
        [ dns-A-query message>a-names empty? not ]
        [ dns-AAAA-query message>aaaa-names empty? not ]
    } 1|| ;

And try it out:

IN: scratchpad "re.factorcode.org" valid-domain? .
t

IN: scratchpad "not-valid.factorcode.org" valid-domain? .
f

Second, we write a word to split a domain into chunks to be tested:

: split-domain ( host -- hosts )
    "." split dup length 1 [-] <iota> [ tail "." join ] with map ;

And try it out:

IN: scratchpad "a.b.c.com" split-domain .
{ "a.b.c.com" "b.c.com" "c.com" }

Third, we find the rootiest domain that is valid:

: remove-subdomains ( host -- host' )
    split-domain [ valid-domain? ] find-last nip ;

And try it out:

IN: scratchpad "a.b.c.d.factorcode.org" remove-subdomains .
"factorcode.org"

IN: scratchpad "sorting.cr.yp.to" remove-subdomains .
"cr.yp.to"

This is available on my GitHub.

It’s fun to explore these kinds of problems!

Andrew Shitov recently published a book called “A Language A Day”, which is a collection of brief overviews to 21 programming languages – including Factor!

This book provides a concise overview of 21 different programming languages. Each language is introduced using the same approach: solving several programming problems to showcase its features and capabilities. Languages covered in the book: C++, Clojure, Crystal, D, Dart, Elixir, Factor, Go, Hack, Hy, Io, Julia, Kotlin, Lua, Mercury, Nim, OCaml, Raku, Rust, Scala, and TypeScript.

Each chapter covers the essentials of a different programming language. To make the content more consistent and comparable, I use the same structure for each language, focusing on the following mini projects:

1. Creating a ‘Hello, World!’ program.
2. Implementing a Factorial function using recursion or a functional-style approach.
3. Creating a polymorphic array of objects (a ‘zoo’ of cats and dogs) and calling methods on them.
4. Implementing the Sleep Sort algorithm—while impractical for real-word use, it’s a playful demonstration of language’s concurrency capabilities.

Each language description follows—where applicable—this pattern:

1. Installing a command-line compiler and running a program.
2. Creating and using variables.
3. Defining and using functions.
4. Exploring object-oriented features.
5. Handling exception.
6. Introducing basic concurrency and parallelism.

You can find all the code examples in this book on GitHub: https://github.com/ash/a-language-a-day.

You can buy it on Amazon or LeanPub as an electronic or Kindle edition, or as a paper hardcover or paperback version. More information with the links to the shops.

Check it out!