Data visualization is the discipline of trying to understand data information by placing it in a visual context so that patterns and trends that might not be detected easily will appear.

In this article I will describe step by step how to create a visualization using Roassal30 in a beginner friendly way. In addition we will show how we can turn a script in a visualization class with powerful customization hooks. Finally we will show how we can extend the object inspector with a new dedicated visualization. Doing so we show (1) how specific domains can be represented differently and specifically and (2) how domain objects can be navigated using diverse representations (lists, table, visualization).

Who is this article for?

The audience of this article:

• Data scientists – readers familiar with Pharo.
• Designers of visualization engines will find valuable resources regarding the design and implementation of a visualization engine.

Getting started with Roassal30

Roassal is a visualization engine developed in Pharo (http://pharo.org). Roassal offers a simple API. It shares similarities of other visualization engines like Mathplotlib and D3.js. Roassal3.0 produces naturally interactive visualizations to directly explore and operate on the represented domain object (https://link.springer.com/book/10.1007/978-1-4842-7161-2). Roassal 2’s documentation is available at (http://agilevisualization.com).

To use, it download the lasted stable version, currently Pharo10. Roassal3.0 is integrated in the Pharo environment, now if you want to use the last version of Roassal, select the menu item

Library >> Roassal3 >> Load full version

Loading Roassal should take a few seconds, depending on your Internet connection.

Visualizing a class hierarchy

Roassal provides examples as scripts directly executable in the Playground. We will start by designing a new visualization as a large script and in the following section we will turn into a class.

The following example, display the full hierarchy of a class with shapes whose size and colors depend on the class they represent. Evaluate the following example with Cmd+D

data := Collection withAllSubclasses.

boxes := data collect: [ :class |
	RSBox new
		model: class;
		popup;
		draggable;
		yourself ].
	
canvas := RSCanvas new.
canvas addAll: boxes.

RSNormalizer size
	from: 10;
	to: 100;
	shapes: boxes;
	normalize: #linesOfCode.
	
RSNormalizer color
	from: Color blue;
	to: Color red;
	shapes: boxes;
	normalize: #linesOfCode.
	
RSLineBuilder orthoVertical
	withVerticalAttachPoint;
	shapes: boxes;
	connectFrom: #superclass.
	
RSTreeLayout on: boxes.

canvas @ RSCanvasController.
canvas.

The execution of the previous script will produce the next image

This visualization shows a hierarchy of boxes, where each box is the graphical representation of a class. The width, height and color represent the size of the lines of code of the class. The lines show the connection or relationship between each class. In this case the connection is to the superclass of each class. The type of these lines is orthogonal vertical line. The position of each box shows a nice tree structure for the Collection with all subclasses.

Let’s analyse each part of the script:

Collecting data: getting a model for the visualization

data := Collection withAllSubclasses.

Usually when you are building a visualization, you use a group of objects, or collection of data. In this case our data is a collection of classes, but you can use CSV files, sql databases, or objects. In some cases you should preprocess your data before visualize it.

Generating shapes

Roassal uses several basic shapes to create the visualization, like RSBox, RSEllipse, RSLabel, RSLine, and RSComposite to combine them.

boxes := data collect: [ :class |
	RSBox new
		model: class;
		popup;
		draggable;
		yourself ].

This script creates a drawable box shape for each class. This box has a model (the class it represents). In addition this code puts a popup and a drag and drop interactions.

Creating the canvas

In Roassal, an instance of RSCanvas is the default container of shapes to show your shapes. When you want to show a shape, create it and then add it into the canvas.

canvas := RSCanvas new.
canvas addAll: boxes.

Normalization

To understand the relation between our data or to stress an aspect, sometimes it is necessary to apply a normalization or transformation for some properties of the visual elements, like the width, height, color, etc.

RSNormalizer size
	from: 10;
	to: 100;
	shapes: boxes;
	normalize: #linesOfCode.
	
RSNormalizer color
	from: Color blue;
	to: Color red;
	shapes: boxes;
	normalize: #linesOfCode.

In this example we are using lines of code for each class to represent the color and the size of each box.

• RSNormalizer size returns an instance of RSNormalizer, this object can modify the size(width and height) property for a group of shapes.

• The methods from: and to: define the values from the minimum and the maximum values from the collection of shapes. That means that for the class that have the minimum number of lines of code the size of that box will be 10. And for the class with the maximum number of lines of code the size will be 100.

• We use normalize: message to apply the transformation on each shape, using a block or method that takes as argument the model of each shape.

Lines

To show links between these boxes we can use lines.

RSLineBuilder orthoVertical
	withVerticalAttachPoint;
	shapes: boxes;
	connectFrom: #superclass.

A line builder has many parameters to create and add new lines to the canvas.

• RSLineBuilder orthoVertical returns an instance of RSLineBuilder, orthoVertical, is the type of line that we want to use, we can use line, or bezier or etc.
• withVerticalAttachPoint creates an attach point. This object knows the starting and ending point for the line.
• This builder interacts with a group of shapes so we set it using the message shapes:.
• connectFrom: is the method that will apply the builder and will generate new lines between the group of boxes, using #superclass for each box’s model.

If need it you can use directly the basic shapes RSLine or RSPolyline for some specific scenarios.

Layout

Finally a good visualization needs to put each element in the correct position by using a layout.

RSTreeLayout on: boxes.

canvas @ RSCanvasController.
canvas.

Roassal layout is an object that changes the position of shapes when the method on: is executed. You can use RSTreeLayout, RSVerticalLineLayout, RSHorizontalLineLayout, etc. There are many layouts in Roassal that you can use. You can also define your own layouts in case the default ones are not enough.

RSCanvasController puts basic interactions in the canvas, for zoom in or zoom out and navigate into the canvas. And the last line is the canvas itself to inspect it on the playground.

The next step: convert a script into a class

Using a script to prototype a visualization is handy and fast. However, it does not scale in terms of reusability and customization.

To reuse this code we need to create a class, there are many ways to do it. We suggest to create a subclass of RSAbstractContainerBuilder. Builders in Roassal, create a group of shapes set the interaction events and put each element in the correct position and they add the shapes into a canvas.

Create a Demo package in the system browser and then a new class TreeClassBuilder, this new class will create the same visualization as the script.

We should implement the method renderIn:. If we copy the previous script, but we omit the creation of the canvas, and then we declare the local variables and the we use the text code formatter, the method will be:

"hooks"
TreeClassBuilder >> renderIn: canvas

	| data boxes |
	data := Collection withAllSubclasses.

	boxes := data collect: [ :class | 
		         RSBox new
			         model: class;
			         popup;
			         draggable;
			         yourself ].

	canvas addAll: boxes.

	RSNormalizer size
		from: 10;
		to: 100;
		shapes: boxes;
		normalize: #linesOfCode.

	RSNormalizer color
		from: Color blue;
		to: Color red;
		shapes: boxes;
		normalize: #linesOfCode.

	RSLineBuilder orthoVertical
		withVerticalAttachPoint;
		color: Color black;
		shapes: boxes;
		connectFrom: #superclass.

	RSTreeLayout on: boxes.

	canvas @ RSCanvasController.
	canvas

To test it in a playground execute this line: TreeClassBuilder new open. It will open a window with the same visualization as before. That means that open method calls renderIn: method.

Now we would like to customise each part of the script, for example different domain model(data), different shapes, different transformations. or lines, or event layout, using OOP.

Test driven development

It is a good practice to use test cases, to create the test class, right-click over the class name and click over Jump to test class. Then we get the class TreeClassBuilderTest

We will need some tests, like the test that opens the window but this test needs to close/delete the opened window.

"tests"
TreeClassBuilderTest >> testOpen
	| builder window |
	builder := TreeClassBuilder new.
	window := builder open.
	window delete.

Creating new methods with Unit Tests

Create a new test to change the domain classes, because currently the visualization is using Collection withAllSubclasses

"tests"
TreeClassBuilderTest >> testClasses
	| builder classes |
	builder := TreeClassBuilder new.
	self assert: builder classes isEmpty.
	classes := { TestCase. Collection. Array }.
	builder classes: classes.
	self assert: builder classes equals: classes.

If we execute this test will fail. Because we do not have the accessor method for classes.

Using the debugger we can generate this this getter method, using the button Create, first we select the TreeClassBuilder, then we select/write the accessing protocol for the new method.

We return an instance variable classes,

"accessing"
TreeClassBuilder >> classes
	^ classes

Then click on Proceed button on the debugger. We will have a new error: classes is nil. To solve it we can define a default value in the initialize method. Browse the class TreeClassBuilder and create initialize method as follow:

"initialization"
TreeClassBuilder >> initialize
	super initialize.
	classes := #().

Then we define the setter method.

"accessing"
TreeClassBuilder >> classes: aCollectionOfClasses
	classes := aCollectionOfClasses

Finally our test is green, viva viva!

Now we need to change and separate the renderIn: method into different methods. Because:

• Big methods are hard to understand.
• Big methods are hard to customise.
• We can create test for each new method.
• We can reuse code creating small methods.

Overriding methods allow us to extent the class functionalities, adding new functionalities in the future with new classes.

Then renderIn: method would be:

"hooks"
TreeClassBuilder >> renderIn: canvas
	| boxes lines |
	boxes := self createBoxes.
	self normalize: boxes.
	lines := self createLinesFor: boxes.
	self layout: boxes.
	canvas
		addAll: lines;
		addAll: boxes

As you can see each part of the visualization is now in a method.

"hooks"
TreeClassBuilder >> createBoxes
	^ self classes collect: [ :class | self createBoxFor: class ]

"hooks"
TreeClassBuilder >> createBoxFor: aClass
	^ RSBox new
		model: aClass;
		popup;
		draggable;
		yourself

"hooks"
TreeClassBuilder >> normalize: aGroupOfBoxes
	RSNormalizer size
		from: 10;
		to: 100;
		shapes: aGroupOfBoxes;
		normalize: #linesOfCode.

	RSNormalizer color
		from: Color blue;
		to: Color red;
		shapes: aGroupOfBoxes;
		normalize: #linesOfCode.

"hooks"
TreeClassBuilder >> createLinesFor: boxes
	| lineBuilder |
	lineBuilder := RSLineBuilder orthoVertical.
	lineBuilder
		withVerticalAttachPoint;
		color: Color black;
		shapes: boxes.
	^ lineBuilder connectFrom: #superclass.

"hooks"
TreeClassBuilder >> layout: aGroupOfBoxes
	RSTreeLayout on: aGroupOfBoxes

"accessing - defaults"
TreeClassBuilder >> defaultContainer
	^ RSCanvas new @ RSCanvasController

Now lets try to use it in a playground with different data. Here we display two hierarchies one of ArrayedCollection and other of SliderMorph.

builder := TreeClassBuilder new.
builder classes: ArrayedCollection withAllSubclasses, SliderMorph withAllSubclasses.
builder asPresenter open

Inspector Integration

The inspector is a central tool in Pharo as it allow one to (i) see the internal representation of an object, (ii) intimately interact with an object through an evaluation panel, and (iii) get specific visual representations of an object. These three properties and the fact that developers can define their own visualizations or representations of a given object makes the inspector a really powerful tool. In addition the inspector navigation will let the user smoothly walk through the objects each of them via specific or generic representations.

In this case our data domain are classes, we can define an inspector pane to show the hierarchy. Open the class Class in the system browser, then add the following method:

"*Demo"
Class >> inspectorTreeBuilder
	<inspectorPresentationOrder: 1 title: 'Subclass Hierarchy'>

	^ TreeClassBuilder new
		classes: self withAllSubclasses;
		asPresenter

The previous method is an extension method, for the package Demo, this means that the source code of this method will be save in the Demo package. By using the pragma <inspectorPresentationOrder:title:>, Pharo developers can define a new view pane for that object in the inspector.

Also because our TreeClassBuilder is subclass of RSAbstractContainerBuilder, you can use the method asPresenter. Now execute: Collection inspect

If you click over one box you will see the subclass hierarchy for the selected box in a new inspector pane.

With the approach presented in this article, you can extent any of your domain objects or event the ones offered by the system and have different views of the same object.

The inspector and its specific panes let the user navigation from one visualization to another. Such interaction and navigation constitute a context to ease the manipulation and understanding of domain or objects.

Export the visualization to other formats

To export in other image formats like PNG, PDF, SVG, we need the project Roassal3Exporters, use the next incantation in a playground:

Metacello new
    baseline: 'Roassal3Exporters';
    repository: 'github://ObjectProfile/Roassal3Exporters';
    load.

After that we need to reopen the inspector. And we will find a new toolbar button

The button will open a file chooser window where we can use SVG, PNG or PDF file formats

The next pdf document was created by the exporter

You can use more file formats with different constraints. like html using https://aframe.io/ or mp4 files o video files with transparency using https://ffmpeg.org/.

Share the code in Github

Finally, to share our cool project with the people we will need to create a repository in a git server like Github.

Please visit this link: https://github.com/pharo-vcs/iceberg/wiki/Tutorial, to know how to publish use Iceberg.

In order to publish a project it is good idea to create a baseline check this link https://github.com/pharo-open-documentation/pharo-wiki/blob/master/General/Baselines.md

This project is in the Github repository https://github.com/akevalion/DemoClassBuilder, please follow the readme instructions to install it and used it.

Conclusion

We saw how to create a new visualization from scratch and how it can be scaled into a class. In addition, Pharo provides many tools in the development of basic and complex applications such as: the Inspector, the Debugger, Spec, Roassal, Iceberg, etc.

We show how we can extend the environment to be able to use our new visualization.

We saw that use can use Roassal visualizations to get a different perspective on domain objects. We show that a new visualization can be tested, shared and used by other developers. You also can use projects like: Athens-Cairo, Morphic or other backends such as GTK. But Roassal offers many tools to interact manipulate and talk with your object data.

In this post, I will explain what is a transmission, a transformation and how to have custom transmissions in the Pharo Inspector. We will show how transmissions are used in the inspector with a soccer database analysis tool that you can find at https://github.com/akevalion/Futbol/. We used this application because we needed a real domain to navigate within the inspector and to show some non-trivial situations where you want to abstract over the domain to get a smooth navigation flow.

In particular, the inspector is a special object presenter that control the navigation between spec presenters. And we will explain such a specific case.

So let us start by explaining transmissions.

What is a transmission?

Transmissions are a generic way to propagate information between Spec components. They are a way to connect presenters, thinking on the “flow” of information more than the way it is displayed.

For example, let us say that we want to have a presenter with a list and with a text inside. When one clicks on the list, the text inside the text presenter will update.

Imagine we defined the layout as follows:

layout := SpBoxLayout newHorizontal
     add: (list := self newList);
     add: (detail := self newText);
     yourself

But this does not say how list and detail are linked. We could do it by describing action to be raised on specific list events or via transmissions.

The transmission sub-framework solves this in an elegant way: Each presenter defines ”output ports” (ports to send information) and ”input ports” (ports to receive information). Each presenter defines also default input and output ports.

Transmitting

A transmission connects a presenter’s output port with a presenter’s input port using the message transmitTo: as follows:

list transmitTo: detail.

This will connect the list presenter default output port with the detail presenter default input port. This line is equivalent (but a lot simpler) to this one:

list defaultOutputPort transmitTo: detail defaultInputPort.

It is important to remark that a transmission does not connect two components, it connects two component ports. The distinction is important because there can be many ports!
Take for example the class SpListPresenter, it defines two output ports (selection and activation), this means it is possible to define also this transmission (note that we do not use the defaultInputPort but outputActivationPort instead):

list outputActivationPort transmitTo: detail defaultInputPort

Note that some presenters such as SpListPresenter offer selection and activation events (and ports). An activation event is a kind of more generic event. Now the selection event can be mapped to an activation event to provide a more generic layer.

Transforming values during a transmission

The object transmitted from a presenter output port can be inadequate for the input port of the target component. To solve this problem a transmission offers transformations.
This is as simple as using the transform: message as follows:

list
     transmitTo: detail
     transform: [ :aValue | aValue asString ]

list defaultOutputPort
     transmitTo: detail defaultInputPort
     transform: [ :aValue | aValue asString ]

Transforming to arbitrary input

We can also transmit from an output port to an arbitrary input receiver using the message transmitDo: and transmitDo:transform:.
It is possible that the user requires to listen an output port, but instead transmitting the value to another presenter, another operation is needed. The message transmitDo: message handles this situation:

list transmitDo: [ :aValue | aValue crTrace ]

Acting after a transmission

Sometimes after a transmission happens, the user needs to react to modify something given the new status achieved by the presenter such as pre-selecting an item, shading it…
The postTransmission: message handles that situation.

list
    transmitTo: detail
    postTransmission: [ :fromPresenter :toPresenter :value |
       "something to do here" ]

Transforming transmissions inside the Inspector

We will use as an example for this blogpost a football project. You can download it from herehttps://github.com/akevalion/Futbol/. There are instructions in the README of how to install it. The project allows one to visualise a european football database. Note that installing the project may take some minutes because it needs to install the database.

First, we can select the country league that we would like to see. In this case we choose France. Then we see a list with the seasons available. We select a season and we can see a table of points of that season of the French league.

The problem

For calculating the statistics of a team, we use a calculator class (TeamStatsCalculator) and not the real model. This is a common situation in which one needs to use a helper class not to pollute the model. So, if we click on a team of the table, instead of seing a soccer team object, which is what we would expect, we see a TeamStatsCalculator instance. In this example, we clicked on the team LOSC of Lille but we get a TemStatsCalculator instead of the LOSC team (yes our team is located at Lille so we took a local team).

What we would like to have is to transform the transmission. The class TeamStatsCalculator knows its real Team and it knows how to convert itself into an instance of Team with the message TeamStatsCalculator >> asTeam. So, we want to transform the object into an instance of Team and then transmit it.

Now there is a catch, as we are inside the inspector, doing the transformation is not the same as before because the inspector takes control of the presenter.

Using SpTActivable trait

SpTActivable is a trait that helps us to use transmission in the Inspector.

We need to use that trait in the class of our presenter. For this example we will use the trait in the class TablePointsPresenter, that is our presenter class that is used inside the inspector.

SpPresenter << #TablePointsPresenter
	traits: { SpTActivable };
	slots: { #table . #teamEvolutionData };
	tag: 'Core';
	package: 'Football-Spec'

Specifying the custom transmition

Once we user that trait, we need to define the method: outputActivationPort on our TablePointsPresenter class.

The method will say how we are going to transform the element. As we said previously, in this case, we only need to send the message asTeam to the object and it will return an instance of its real team when we click on an item of the table. To define the outputActivationPort method, we need to return an instance of SpActivationPort and specify the transform action.

For this example, we will return the output activation port of the table and to specify the transformation.

outputActivationPort

    ^ table outputActivationPort
        transform: [ :selection | selection selectedItem asTeam ];
        yourself

Now, we refresh the inspector, and when we click on a row and we see that now we have the real team object for the LOSC of Lille!

Conclusion

We saw that to transform value during a transmission in the Pharo Inspector, we only need to use the trait SpTActivable and to define the method outputActivationPort. Also, we saw how thanks to the transmissions sub-framework this work is easy to do as Spec takes care of it for us.

Sebastian Jordan-Montano

To work on Pharo’s virtual machine, you’ll need to set up both the virtual machine you’ll be modifying, as well as a Pharo image. Below are the instructions required to set up the environment. You will need both the VM and the image containing the VM compilation chain (VMMaker package).

Building the Pharo VM

• Clone the pharo-vm repo (https://github.com/pharo-project/pharo-vm) .
  • You may need to switch to a branch different from the default one, depending on top of which version you want to make your changes.
• Check you have all dependencies needed to build the pharo-vm
  • Tip: When working on macOs, you can install Command Line Tools for Xcode, that already contains many of the dependencies you’ll need.
• Build the virtual machine to check everything went correctly:
  • To do so, run the following command inside pharo-vm‘s root directory: cmake --build . --target install
  • If compilation completes without error, you’re good to go!

Configuring the Pharo Image with VMMaker

• Download PharoLauncher from http://pharo.org
• Create a new Pharo image (if in doubt, choose the latest stable version)
• Set-up pharo-vm in Pharo
  • Open Pharo image
  • Add the pharo-vm github repository to the Pharo image via Iceberg (Add > Import from existing clone)
  • Right-click repo -> Metacello > Install baseline of VMMaker (Default)
• Verify that the installation completed correctly
  • Run all VMMakerTests test suites
  • If all tests pass (except from the JIT-related ones), you are good to go!
• To run a Pharo image with a specific version of the VM, you can use the following command in the root of pharo-vm‘s root directory:
  • build % ./build/dist/Pharo.app/Contents/MacOS/Pharo ~/Documents/Pharo/images/Pharo\ 10.0\ -\ 64bit\ \(stable\)/Pharo\ 10.0\ -\ 64bit\ \(stable\).image --interactive
  • Note: the exact path may differ in your case, depending on your Pharo version and OS.

Now you are ready to go and change the VM code and generate a new image.

Imagine that you want to find a specific expression and that you want to find it in the complete system. How many classes would you have to look for? How can you be sure that you did not miss any class and being sure that you won’t be frustrated because of the number of issues thrown on compilation or execution? In addition imagine other scenario where you want to transform that expression into another one.

Changing code, removing, or replacing deprecated methods is costly for a developer by doing it manually instead of using an automated feature.

This blog post will explain how to find a specific piece of code we may look for inside a Pharo program, and make it easy for the developers to deal with pattern matching and RBParseTreeSearcher class.

Following work will be about how to replace code using RBParseTreeRewriter and doing the exact same thing automatically using the Rewrite tool (a tool built on top the RBParseTreeRewriter). 

For the moment, we will explain some fundamental definitions and for that the post is structured following below sections:

1. Pattern code description
2. RBParseTreeSearcher description
3. RBParseTreeSearcher examples with pattern code

1. Pattern code description

A pattern expression is very similar to an ordinary Smalltalk expression, but allows one to specify some “wildcards”. The purpose is simple. Imagine that you have a piece of code:

car isNil ifTrue: [ ^ self ].

You can of course compare it with the same piece of code for equality, but wouldn’t it be cool if you could compare something similar, but ignore the fact that the receiver is named car? With pattern rules you can do exactly that. Consider the following code and notice the back-tick before car:

`car isNil ifTrue: [ ^ self ].

Now this expression can match any other expression where isNil ifTrue: [^self] is sent to any variable (or literal). With such a power you can find all the usages of isNil ifTrue: and replace them with ifNil. So what are the “wildcards” that we are using?

(`)Basic pattern nodes

Code prefixed with a back-tick character (`) defines a pattern node. The table below is listing three simple patterns that can be declared with the back-tick:

Example with matches:

`receiver foo 

matches:

• self foo
• x foo
• OrderedCollection foo

(`#) Literal pattern nodes

A back-tick can be followed by the hash sign to ensure that matched receiver will be a literal:

Example:

 `#lit size

matches:

• 3 size
• 'foo' size
• #(a b c) size

(`@) List pattern nodes

To have complete flexibility, there is the possibility to use an at sign @ before the name of a pattern node which turns the node into a list pattern node, which can be empty, returns one or multiple values.

Disclaimer:

• You may write an expression with just args instead of `@args.
• The list patterns does not make any sense for literal nodes i.e. `#@literal.

Example 1:

`x := `@value

matches:

myVar := OrderedCollection new

Example 2:

`sel1 at: `@args1 `sel2: `@args2

matches:

self at: index putLink: (self linkOf: anObject ifAbsent: [anObject asLink])

Where:

• `args1 and `args2 have different values
• `sel1 matches self
• `@args1 matches index
• `sel2: matches putLink:
• `@args2 matches (self linkOf: anObject ifAbsent: [anObject asLink])

Example 3:

`@rcvr `@msg: `@args matches:

(self class deferUpdates: true) ifTrue: [^aBlock value].

Where:

• `@rcvr matches (self class deferUpdates: true)
• `@msg: matches ifTrue:
• `@args matches [^aBlock value]

Example 4:

|`@args1 `myArgument `@args2| matches:

| t1 t2 |

Here we need to have at least 1 argument myArgument , and the example is matching because `@args1 can be empty. So here we have:

• myArgument is matching with t1
• `@args2 is matching with t2

(`.) Statement pattern nodes

Back-tick can be followed by a period to match statements. For example:

Example1:

`.Statement1.

is matching:

• x := 1.
• myVal:= 'Hello World'.
• self assert: myVal size equals: 11.

Example2:

|`@temps|
`@.statements1.
`.duplicate.
`@.statements2

matches:

|x|
x := 1.
x := 2

Where:

• |`@temps| matches |x|
• `@.statements1. is nil
• `.duplicate. matches x := 1.
• `@.statements2

P.S. In the end it does not matter whether you will write `.@Statement or `@.Statement.

(`{ }) Block Pattern Nodes

These are the most exotic of all the nodes. They match any AST nodes like a list pattern and test it with a block. The syntax is similar to the Pharo block, but curly braces are used instead of square brackets and as always the whole expression begins with a back-tick.

Consider the following example:

`{ :node | node isVariable and: [ node isGlobal ] } become: nil

this pattern will match a message #become: with an attribute nil, where the receiver is a variable and it is a global variable. 

There is also a special case called wrapped block pattern node which has the same syntax and follows a normal pattern node. In this case first the node will be matched based on its pattern, and then passed to the block. For example:

`#arr `{ :node | node isLiteralArray } asArray

is a simple way to detect expression like #(1 2 3) asArray. In this case first #(1 2 3) will be matched by the node and then tested by the block.

Naming is Important

The pattern nodes are so that you can match anything in their place. But their naming is also important as the code gets mapped to them by name. For example:

`block value: `@expression value: `@expression

will match only those #value:value: messages that have exactly the same expressions as both arguments. It is like that because we used the same pattern variable name.

2. RBParseTreeSearcher description

So, after figuring out what are the patterns that can be used and what kind of matches they can perform, now we can move forward to discover how RBParseTreeSearcher class works in Pharo , in order to be able to understand in the last section how RBParseTreeSearcher and defined patterns work together to find the matches we are looking for.

RBParseTreeSearcher is supposed to look for a defined pattern using the ‘wildcards’ of a matcher defined as a Tree, and on success (when match is found) a block can be executed.

Basically, when a developper uses this class, the most used instance variables are:

• #matches:do: which a message that looks for patterns defined in matches: block using the wildcards, and if a match is found the do: block is executed.
  The do block takes two parameters: :aNode and :answer. The aNode refers to each node of the pattern defined, and the answer can be used for example to increment value on each node match.
  The blocks defined in #matches:do: are called rules, and they are stored only in success in instance searches of RBParseTreeSearcher defined below.
• searches which type is Ordered collection, contains all the successful rules applied whenever using: #matches:do:, #matchesMethod:do … to store rules of type Rule, MethodRule, ArgumentRule, TreeRule …
• context which type is dictionary: contains all the successfully matched patterns.
• executeTree: this method takes aParseTree as input parameter, which is the possible matching code that we are looking for, and starts the matching process using the defined pattern.
• messages of type OrderedCollection, and returns the list of messages found in a match.
• hasRules returns searches list

Consider the following example which is using the instance sides defined above:

|searcher dict|
searcher := RBParseTreeSearcher new.
searcher
    matches: '`@rcv at:`@arg `sel:`@arg1'
    do: [ :aNode :answer | dict := searcher context ].
searcher executeTree:
    (RBParser parseExpression: 'cache at: each ifAbsentPut: [ each ].').

The method #matches:do: is used to define the pattern that we are looking for, using the ‘wildcards’ defined in first section; In addition of that, the do is running only on match, and in our case it will fill the dictionary dict with the searcher context (which is the pattern defined in matches block).
This execution is fired on executeTree: which defines the matcher that is a String parsed as a Tree using parseExpression, then starts matching it with the pattern.

3. RBParseTreeSearcher examples with pattern code

Finally, in this section we use patterns with the RBParseTreeSearcher class and do some magic by finding some matches in Pharo code !

Consider the following example:

| dict searcher|
searcher := RBParseTreeSearcher new.

searcher  
   matches: '`@receiver assert: `@arg equals: true'
   do: [ :aNode :answer | dict := searcher context ].

searcher 
   executeTree: (RBParser parseExpression: 'self assert: reader storedSettings first realValue equals: true.').

dict 	
   collect: [ :each | each displayString ].

The example is matching successfully and the dictionary dict will return different values during the iteration:

Match 1: (key) `@receiver is matching with (value) self
Match 2: (key) `@arg is matching with (value) reader storedSettings first realValue

If we want to check all the messages in the matcher, we can use searcher messages which will return an array of one item containing message #assert:equals: as it is the only message available in the matched expression.

*********************************

In this post we will see how to use custom styles in Spec applications. We will start to present styles and then build a little editor as the one displayed hereafter.

We will show that an application in Spec manages styles and let you adapt the look of a presenter.

How do styles work?

Styles in Spec work like CSS. They are style sheets in which the properties for presenting a presenter are defined. Properties such as colors, width, height, font, and others. As a general principle it is better to use styles instead of fixed constraints, because your application will be more responsive.

For example, you want a button to have a specific width and height. You can do it using constraints with the method add:withConstraints: or using styles. In both cases the result will be this:

But, if you change the size of the fonts of the Pharo image using Settings/Appearance/Standard Fonts/Huge, using fixed constraints, you will obtain the following result. You will for example do not be able to see the icons because the size is not recomputed correctly.

If you use styles, the size of the button will also scale as shown below.

Style format

The styles in Spec format are similar to CSS. Style style sheets are written using STON as format. We need to write the styles as a string and then parse it as a STON file.

Here is an example that we will explain steps by steps below.

'.application [       
    .lightGreen [ Draw { #color: #B3E6B5 } ],          
    .lightBlue [ Draw { #color: #lightBlue } ] ]'

We will go by steps.

SpPropertyStyle has 5 subclasses: SpContainerStyle, SpDrawStyle, SpFontStyle, SpTextStyle, and SpGeometryStyle. These subclasses define the 5 types of properties that exist. On the class side, the method stonName that indicates the name that we must put in the STON file.

• SpDrawStyle modifies the properties related to the drawing of the presenter, such as the color and the background color.
• SpFontStyle manipulates all related to fonts.
• SpGeometryStyle is for sizes, like width, height, minimum height, etc.
• SpContainerStyle is for the alignment of the presenters, usually with property is changed on the main presenter, which is the one that contains and arranges the other ones.
• SpTextStyle controls the properties of the SpTextInputFieldPresenter.

If we want to change the color of a presenter, we need to create a string and use the SpDrawStyle property, which STON name is Draw as shown below. For setting the color, we can use either the hexadecimal code of the color or the sender of Color class.

'.application [       
    .lightGreen [ Draw { #color: #B3E6B5 } ],          
    .lightBlue [ Draw { #color: #lightBlue } ] ]'

Now we have two styles: lightGreen and lightBlue that can be applied to any presenter.

We can also use environmental variables to get the values of the predefined colors of the current theme, or the fonts. For example, we can create two styles for changing the fonts of the letters of a presenter:

'.application [
    .codeFont [ Font { #name: EnvironmentFont(#code) } ],
    .textFont [ Font { #name: EnvironmentFont(#default) } ]
]'

Also we can change the styles for all the presenters by default. We can put by default all the letters in bold.

'.application [
	Font { #bold: true }
]'

Defining an Application

To use styles we need to associate the main presenter with an application. The class SpApplication already has default styles. To not redefine all the properties for all the presenters, we can concatenate the default styles (SpStyle defaultStyleSheet) with our own. As said above, the styles are actually STON files that need to be parsed. To parse the string into a STON we can use the class SpStyleVariableSTONReader.

presenter := SpPresenter new.
presenter application: (app := SpApplication new).

styleSheet := SpStyle defaultStyleSheet, 
	(SpStyleVariableSTONReader fromString: 
	'.application [
	     Font { #bold: true },
            .lightGreen [ Draw { #color: #B3E6B5 } ],
            .bgBlack [ Draw { #backgroundColor: #black } ],
	    .blue [ Draw { #color: #blue } ]
]' ).

app styleSheet: styleSheet.

Now, can can add one or more styles to a presenter, like follows:

presenter layout: (SpBoxLayout newTopToBottom
	add: (label := presenter newLabel);
	yourself).

label label: 'I am a label'.
label addStyle: 'lightGreen'.
label addStyle: 'black'.
 
presenter openWithSpec.

Also we can remove and add styles at runtime.

label removeStyle: 'lightGreen'.
label removeStyle: 'bgBlack'.
label addStyle: 'blue'.

Using classes

To properly use styles, it is better to define a custom application as a subclass of SpApplication.

SpApplication << #CustomStylesApplication
	slots: {};
	package: 'Spec-workshop'

In the class we need to override the method styleSheet to return our custom style sheet concatenated with the default one.

CustomStylesApplication >> styleSheet

	^ SpStyle defaultStyleSheet, 
	(SpStyleVariableSTONReader fromString:
'.application [
	Font { #bold: true },

	.lightGreen [ Draw { #color: #B3E6B5 } ],
	.lightBlue [ Draw { #color: #lightBlue } ],
	.container [ Container { #padding: 4, #borderWidth: 2 } ],
	.bgOpaque [ Draw { #backgroundColor: EnvironmentColor(#base) } ],
	.codeFont [ Font { #name: EnvironmentFont(#code) } ],
	.textFont [ Font { #name: EnvironmentFont(#default) } ],
	.bigFontSize [ Font { #size: 20 } ],
	.smallFontSize [ Font { #size: 14 } ],
	.icon [ Geometry { #width: 30 } ],
	.buttonStyle [ Geometry { #width: 110 } ],
	.labelStyle [ 
		Geometry { #height: 25 },
		Font { #size: 12 }	]
]')

We can use different properties in the same style. For example, in labelStyle we are setting the height of the presenter to 25 scaled pixels and the font size to 12 scaled pixels. Also, we are using EnvironmentColor(#base)for obtaining the default background colour according to the current theme. Because the colour will change according to the theme that used in the image.

For the main presenter, we will build a mini-text-viewer in which we will be able to change the size and the font of the text that we are viewing.

SpPresenter << #CustomStylesPresenter
	slots: { #text . #label . #zoomOutButton . #textFontButton . #codeFontButton . #zoomInButton };
	package: 'Spec-workshop'

In the initializePresenters method we will first initialise the presenters, then set the styles for the presenters and finally initialise the layout.

CustomStylesPresenter >> initializePresenters

	self instantiatePresenters.
	self initializeStyles.
	self initializeLayout

CustomStylesPresenter >> instantiatePresenters

	zoomInButton := self newButton.
	zoomInButton icon: (self iconNamed: #glamorousZoomIn).
	zoomOutButton := self newButton.
	zoomOutButton icon: (self iconNamed: #glamorousZoomOut).

	codeFontButton := self newButton.
	codeFontButton
		icon: (self iconNamed: #smallObjects);
		label: 'Code font'.
	textFontButton := self newButton.
	textFontButton
		icon: (self iconNamed: #smallFonts);
		label: 'Text font'.

	text := self newText.
	text
		beNotEditable
		clearSelection;
		text: String loremIpsum.

	label := self newLabel.
	label label: 'Lorem ipsum'

CustomStylesPresenter >> initializeLayout
	
	self layout: (SpBoxLayout newTopToBottom
		add: label expand: false;
		add: (SpBoxLayout newLeftToRight
			add: textFontButton expand: false;
			add: codeFontButton expand: false;
			addLast: zoomOutButton expand: false;		
			addLast: zoomInButton expand: false;
			yourself)
		expand: false;
		add: text;
		yourself)

Finally, we change the window title and size:

CustomStylesPresenter>> initializeWindow: aWindowPresenter

	aWindowPresenter
		title: 'Using styles';
		initialExtent: 600 @ 400

Without setting the custom styles nor using our custom application in the presenter, we have:

We do not want the black background color for the text presenter. We will like to have a sort of muti-line label. We want the zoom buton to be smaller as they only have icons. We want to have the option to change the size and font of the text inside the text presenter. Finally, why not, we want to change the color of the label, change the height and make it a little more bigger.

CustomStylesPresenter >> initializeStyles

    "Change the height and size of the label."
    label addStyle: 'labelStyle'.
    "But the color as light green"
    label addStyle: 'lightGreen'.

    "The default font of the text will be the code font and the size size will be the small one."
    text addStyle: 'codeFont'.
    text addStyle: 'smallFontSize'.
    "Change the background color."
    
    text addStyle: 'bgOpaque'.

    "But a smaller width for the zoom buttons"
    zoomInButton addStyle: 'icon'.
    zoomOutButton addStyle: 'icon'.
	
    codeFontButton addStyle: 'buttonStyle'.
    textFontButton addStyle: 'buttonStyle'.

    "As this presenter is the container, set to self the container
    style to add a padding and border width."
	
    self addStyle: 'container'

Finally, we have to override the start method in the application. With this, we are going to set the application of the presenter and run the presenter from the application.

CustomStylesApplication >> start

	(self new: CustomStylesPresenter) openWithSpec

Now, if we run CustomStylesApplication new start we will have:

The only thing missing is to add the behaviour when pressing the buttons.

For example, if we click on the zoom in button we want to remove the smallFontStyle and add the bigFontSize. Or, if we click on the text font button, we want to remove the style codeFont and add the textFont style. So, in the connectPresenters method we have:

CustomStylesPresenter >> connectPresenters

	zoomInButton action: [
		text removeStyle: 'smallFontSize'.
		text addStyle: 'bigFontSize' ].
	zoomOutButton action: [ 
		text removeStyle: 'bigFontSize'.
		text addStyle: 'smallFontSize'].

	codeFontButton action: [
		text removeStyle: 'textFont'.
		text addStyle: 'codeFont' ].
	textFontButton action: [ 
		text removeStyle: 'codeFont'.
		text addStyle: 'textFont']

Now, if we click on zoom in we will have:

And if we click on text font:

Conclusion

Using styles in Spec is great. It make easier to have a consistent design as we can add the same style to several presenters. If we want to change some style, we only edit the styles sheet. Also, the styles automatically scale if we change the font size of all the image. They are one of the main reason why in Spec we have the notion of an application. We can dynamically change how a presenter looks.

Sebastian Jordan-Montano

In this post I am going to show you how to call external functions from a Pharo image. Here, we are going to use the LAPACK (Linear Algebra Package) library that is written in Fortran.

Why do we need this library?

In the Pharo AI project (https://github.com/pharo-ai), we are working on an implementation of linear regression. Currently, we are writing the logic completely in Pharo. But, linear regression can be formulated in terms of least squares and Lapack implements efficiently the minimum-norm solution to a linear least squares problem.

So we want to get the best of the two worlds: nice objects in Pharo and call some fast and low-level libraries for crunching numbers. We want to bind the routine dgelsd() that does exactly what we want.

Implementing the binding

We need to have the library already installed in our machines. As a first step, we create a subclass of FFILibrary called LapackLibrary and we need to override the methods: macLibraryName, win32LibraryName and unixLibraryName. In those methods we should return the path in which the library is installed. For MacOS, we override the macLibraryName method as follows:

LapackLibrary >> macLibraryName
	
	^ FFIMacLibraryFinder new 
		userPaths: { '/usr/local/opt/lapack/lib/' };
		findAnyLibrary: #('liblapack.dylib').

And for Windows:

LapackLibrary >> win32LibraryName

	^ FFIWindowsLibraryFinder new
		userPaths: { '/usr/local/opt/lapack/lib/' };
		findAnyLibrary: #( 'liblapack.dylib' )

For using this binding on Linux is only needed to override the remaining method. One can use the class FFIUnix32LibraryFinder or FFIUnix64LibraryFinder.

Now, we are going to create the class LapackDgelsd. We override the method ffiLibrary to return just the class LapackLibrary.

LapackDgelsd >> ffiLibrary

	^ LapackLibrary

Now we can implement the method which will eventually make the FFI call. We saw in the documentation that dgelsd receives 14 parameters, all pointers, that have different types. To make the FFI call we have to use self ffiCall: and inside put the signature of the foreign function.

The variables that are passed must be either local or instance variables. We can specify the type of each of the variables, or we can say that the type is void*. That means that the FFI library is not going to do the mapping to the correct type, but is a responsiblity of the programmer to instantiate the variables with the correct type. So, we get:

LapackDgelsd >> ffiDgelsdM: m n: n nrhs: nrhs a: a lda: lda b: b ldb: ldb s: s rcond: rcond rank: aRank work: work lwork: lwork iwork: iwork info: anInfo
 
	^ self ffiCall: #( void dgelsd_(
		void* m,
        void* n,
        void* nrhs,
    	void* a,
        void* lda,
    	void* b,
        void* ldb,
    	void* s,
    	void* rcond,
        void* aRank,
    	void* work,
        void* lwork,
    	void* iwork,
    	void* anInfo ) )

In the documentation of the routine , we see that we need integer pointers, double pointers, integer arrays, and double arrays. To use pointers in Pharo, we need to use the class FFIExternalValueHolder. It actually will create an anonymous class of the type that we need. To ease the work, we will create a helper class that will create the pointer for us.

We can name the class LapackPointerCreator and that class has to have two class variables: DoublePointerClass and IntegerPointerClass. In the initialize method of the class, we instantiate the value of the class variables to be:

LapackPointerCreator class >> initialize
	
	DoublePointerClass := FFIExternalValueHolder ofType: 'double'.
	IntegerPointerClass := FFIExternalValueHolder ofType: 'int'

And then we create the two helper methods:

LapackPointerCreator class >> doublePointer: aNumber

	^ DoublePointerClass new value: aNumber

LapackPointerCreator class >> integerPointer: aNumber

	^ IntegerPointerClass new value: aNumber

And we will also create an extension method of Collection to convert a collection into n FFI external array.

Collection >> asFFIExternalArrayOfType: aType
	"Converts collection to FFIExternalArray.
	
	Example:
	#(1 2 3) asFFIExternalArrayOfType: 'int'"
	
	| array |
	array := FFIExternalArray newType: aType size: self size.
	self withIndexDo: [ :each :index | array at: index put: each ].
	^ array

Now we are ready to make the call to the Lapack dgelsd() routine. We are going to follow this example of how to use dgelsd() with the same values. First, we need to create all the variables that we are going to pass as arguments.

numberOfRows := 4.
numberOfColumns := 5.
numberOfRightHandSides := 3.

matrixA := #( 0.12 -6.91 -3.33  3.97 -8.19
			  2.22 -8.94  3.33  7.69 -5.12
			 -6.72 -2.74 -2.26 -9.08 -4.40
		 	 -7.92 -4.71  9.96 -9.98 -3.20 ) asFFIExternalArrayOfType: 'double'.

matrixB := #( 7.30  1.33  2.68 -9.62 0.00
		  	  0.47  6.58 -1.71 -0.79 0.00
			 -6.28 -3.42  3.46  0.41 0.00 ) asFFIExternalArrayOfType: 'double'.

reciprocalConditionNumberPointer := LapackPointerCreator doublePointer: -1.

singularValuesArray := FFIExternalArray newType: 'double' size: numberOfRows.
iworkArray := FFIExternalArray newType: 'int' size: 11 * numberOfRows.

numberOfRowsPointer := LapackPointerCreator integerPointer: numberOfRows.
numberOfColumnsPointer := LapackPointerCreator integerPointer: numberOfColumns.

numberOfRightHandSidesPointer := LapackPointerCreator integerPointer:
    numberOfRightHandSides.

leadingDimensionAPointer := LapackPointerCreator integerPointer: numberOfRows.
leadingDimensionBPointer := LapackPointerCreator integerPointer: numberOfColumns.

rankPointer := LapackPointerCreator integerPointer: 0.
infoPointer := LapackPointerCreator integerPointer: 0.

The WORK array is a workspace that is used internally by the dgelsd() routine. We must allocate the memory for this array before passing it to the routine. To calculate the optimal size of the WORK array, we run the dgelsd() routine with LWORK value -1 and WORK as any pointer. After this first execution, the optimal workspace size will be written into the WORK pointer. Therefore, we will be making two FFI calls to the routine: first to calculate the optimal workspace and then to find the solution to the least squares problem.

lworkPointer := LapackPointerCreator integerPointer: -1.
workPointer := LapackPointerCreator doublePointer: 0.
	
dgelsd := LapackDgelsd new.

dgelsd
	ffiDgelsdM: numberOfRowsPointer 
	n: numberOfColumnsPointer 
	nrhs: numberOfRightHandSidesPointer
	a: matrixA 
	lda: leadingDimensionAPointer 
	b: matrixB 
	ldb: leadingDimensionBPointer 
	s: singularValuesArray
	rcond: reciprocalConditionNumberPointer 
	rank: rankPointer
	work: workPointer 
	lwork: lworkPointer
	iwork: iworkArray 
	info: infoPointer.

Now, the variable workPointer contains the value of the optimal workspace. With that information, we run the dgelsd() routine again to solve the problem for matrices A and B.

optimalWorkspace := workPointer value asInteger.
workArray := FFIExternalArray newType: 'double' size: optimalWorkspace.
workArraySizePointer := LapackPointerCreator integerPointer: optimalWorkspace.

dgelsd	
	ffiDgelsdM: numberOfRowsPointer 
	n: numberOfColumnsPointer 
	nrhs: numberOfRightHandSidesPointer
	a: matrixA 
	lda: leadingDimensionAPointer 
	b: matrixB 
	ldb: leadingDimensionBPointer 
	s: singularValuesArray
	rcond: reciprocalConditionNumberPointer 
	rank: rankPointer
	work: workArray 
	lwork: workArraySizePointer 
	iwork: iworkArray 
	info: infoPointer

The result of the computation is stored in several variables. For example, the values of matrixB have been replaced with the minimum norm solution. The effective rank is contained in the rankPointer and the S array contains the singular values of matrix A in decreasing order. The INFO variable contains an integer value that informs us whether the routine succeded or failed (0 = successful exit).

To see the value of a pointer, we can use the value message, for example, rankPointer value.

Improving the API

Make the call to the Fortran routine was a little tricky. We don’t want to give the user the responsibility of creating all of those pointers for using the not-so-nice method signature. So we will use Pharo in our favour.

Actually, we only need to set 5 values of the method, the rest can be calculated internally. What we will do is to have all the needed variables as instance variables of the class LapackDgelsd. Like:'matrixA matrixB numberOfRows numberOfColumns numberOfRightHandSides leadingDimensionA leadingDimensionB singularValues rank info reciprocalConditionNumber workArraySize minimumNormSolution iworkArray'.

Then, we will create the setters for the 5 values that we need the user to insert. Note that the user does not need to create any pointer. They pass only a Pharo Array.

LapackDgelsd >> numberOfColumns: aNumber

	numberOfColumns := aNumber

LapackDgelsd >> numberOfRows: aNumber

	numberOfRows := aNumber

LapackDgelsd >> matrixA: anArray

	matrixA := anArray asFFIExternalArrayOfType: 'double'

LapackDgelsd >> matrixB: anArray

	matrixB := anArray asFFIExternalArrayOfType: 'double'

LapackDgelsd >> numberOfRightHandSides: aNumber

	numberOfRightHandSides := aNumber

Now, as we do not want to give the responsiblity to the user to call the method twice, one for obtaining the optimal workspace and then for solving the actual equation. We will create a solving method that does all the work. We have:

LapackDgelsd >> solve

	| singularValuesArray numberOfRowsPointer numberOfColumnsPointer
      numberOfRightHandSidesPointer leadingDimensionAPointer leadingDimensionBPointer
      rankPointer infoPointer reciprocalConditionNumberPointer workArray
      workArraySizePointer |

	singularValuesArray := FFIExternalArray newType: 'double' size: numberOfRows.
	
	"iwork dimension should be at least 3*min(m,n)*nlvl + 11*min(m,n),
     where nlvl = max( 0, int( log_2( min(m,n)/(smlsiz+1) ) )+1 )
     and smlsiz = 25"
	iworkArray := FFIExternalArray newType: 'int' size: 11 * numberOfRows.
	
	numberOfRowsPointer := LapackPointerCreator integerPointer: numberOfRows.
	numberOfColumnsPointer := LapackPointerCreator integerPointer: numberOfColumns.
	numberOfRightHandSidesPointer := LapackPointerCreator 
        integerPointer: numberOfRightHandSides.
	leadingDimensionAPointer := LapackPointerCreator integerPointer: self leadingDimensionA.
	leadingDimensionBPointer := LapackPointerCreator integerPointer: self leadingDimensionB.
	rankPointer := LapackPointerCreator integerPointer: 0.
	infoPointer := LapackPointerCreator integerPointer: 0.
	reciprocalConditionNumberPointer := LapackPointerCreator 
        doublePointer: self reciprocalConditionNumber. 
	
	workArraySize := self
		findOptimalWorkspace: numberOfRowsPointer 
		n: numberOfColumnsPointer 
		nrhs: numberOfRightHandSidesPointer
		a: matrixA 
		lda: leadingDimensionAPointer 
		b: matrixB 
		ldb: leadingDimensionBPointer 
		s: singularValuesArray
		rcond: reciprocalConditionNumberPointer 
		rank: rankPointer 
		work: nil  
		iwork: iworkArray 
		info: infoPointer.

	workArray := FFIExternalArray newType: 'double' size: workArraySize.
	workArraySizePointer := LapackPointerCreator integerPointer: workArraySize.

	self
		ffiDgelsdM: numberOfRowsPointer 
		n: numberOfColumnsPointer 
		nrhs: numberOfRightHandSidesPointer
		a: matrixA 
		lda: leadingDimensionAPointer 
		b: matrixB 
		ldb: leadingDimensionBPointer 
		s: singularValuesArray
		rcond: reciprocalConditionNumberPointer 
		rank: rankPointer
		work: workArray 
		lwork: workArraySizePointer 
		iwork: iworkArray 
		info: infoPointer.

	minimumNormSolution := matrixB asArray.
	singularValues := singularValuesArray asArray.
	rank := rankPointer value.
	info := infoPointer value.

LapackDgelsd >> findOptimalWorkspace: m n: n nrhs: nrhs a: a lda: lda b: b ldb: ldb s: s rcond: rcond rank: aRank work: work iwork: iwork info: anInfo

	| lworkPtr workPtr |
	lworkPtr := LapackPointerCreator integerPointer:  -1.
	workPtr := LapackPointerCreator doublePointer:  0.

	self ffiDgelsdM: m n: n nrhs: nrhs a: a lda: lda b: b ldb: ldb s: s rcond: rcond rank: aRank work: workPtr lwork: lworkPtr iwork: iwork info: anInfo.
	
	^ workPtr value asInteger

Now, all that we have to do, as users, is:

numberOfRows := 4.
numberOfColumns := 5.
numberOfRightHandSides := 3.
matrixA := #( 0.12 -6.91 -3.33  3.97 -8.19
			  2.22 -8.94  3.33  7.69 -5.12
		     -6.72 -2.74 -2.26 -9.08 -4.40
		 	 -7.92 -4.71  9.96 -9.98 -3.20 ).
matrixB := #( 7.30  1.33  2.68 -9.62 0.00
		  	  0.47  6.58 -1.71 -0.79 0.00
		     -6.28 -3.42  3.46  0.41 0.00 ).

algorithm := LapackDgelsd new
    numberOfRows: numberOfRows;
    numberOfColumns: numberOfColumns;
    numberOfRightHandSides: numberOfRightHandSides;
    matrixA: matrixA;
    matrixB: matrixB;
    yourself.

algorithm solve.

This is definitively better, since it hides the implementation details of the underlying implementation. And for getting the solution, we only need to call these accessors:

"Info represents if the process completed with success"
algorithm info.

"The array with the solutions"
algorithm minimumNormSolution.

"The effective rank"
algorithm rank.

"And the singular values of matrix A"
algorithm singularValues.

Conclusion

As we saw, doing the binding for the first method was the hardest part. But all the next methods will be easier to implement because we can use the same infrastructure.

Here we showed a binding for a widely used linear algebra library. That will help us to speed up the mathematical computations of libraries such as PolyMath (https://github.com/PolyMathOrg/PolyMath/) and Pharo-AI (https://github.com/pharo-ai). Lapack is a huge library so we don’t want to bind all the methods, but only the ones that we need. If people from the community would like us to migrate other methods of the library, we will be happy to do it. We will work on demand. The code and instructions are available here.

Our next step is to use dgelsd() in our linear regression implementation in Pharo for benchmarking against Python and R. Because those languages use also Lapack fortran library and they are considered as industry standards.

Sebastian Jordan Montano

The other day we were working on the compiler detection of hotspots, originally implemented by Clément Béra during his PhD thesis. In Sista, hotspot detection is implemented as a countdown that looks like the following: the counter is loaded in a register, decremented and then a jump if carry detects if the substraction underflowed.

self MoveA32: counterAddress R: counterReg.
self SubCq: 16r10000 R: counterReg. "Count executed"
countTripped := self JumpCarry: 0.

We were building some unit tests for this functionality, and we were interested at first at seeing how counters increment/decrement when methods execute. We wrote a couple dozen of tests for different cases (because the code for the counters is a bit more complicated, but that’s for another day). The code of one of our tests looked like the following: we compile a method, we execute it on a machine code simulator, then we verify that the counter was effectively incremented (because the public API is in terms of positive counts and not count-downs):

testCounterOfMethodWithAndAndJumpIncrementsReachedOfJump
    | nativeMethod counterData jumpCounter |
    nativeMethod := self jitMethod: (self class>>#methodWithAndAndJump:).
    self
        callCogMethod: nativeMethod
        receiver: memory nilObject
        arguments: { memory trueObject }
        returnAddress: callerAddress.

    counterData := interpreter picDataFor: nativeMethod.
    jumpCounter := memory fetchPointer: 1 ofObject: counterData.
    self
        assert: (memory integerValueOf: (memory fetchPointer: 1 ofObject: jumpCounter))
        equals: 1

There was however something fishy about the ARM64 version. In addition of incrementing the counter, the code was taking the carry jump! Which lead our test to fail…

Doing some machine code debugging

So everything was working OK on intel (IA32, X64) but not on ARM (neither 32 or 64 bits). In both ARM versions the jump was __incorrectly__ taken. I first checked the instruction was being correctly assembled. And since that seemed ok, I went on digging in our machine code debugger. I found the corresponding instruction, set the instruction pointer in there and started playing with register values to see what was happening.

As you can see in the screenshot, the code is being compiled into subs x25, x25, x16, which you can read as x25 := x25 - x16. So I started playing with the values of those two registers and the carry flag, which is the flag that activates our jump carry. The first test I did was to check 2 - 1.

self carry: false.
self x25: 2.
self x16: 1.

Substraction was correct, leaving the correct result in x25, but the carry flag was set! That was odd. So I tested a second thing: 0 - 1.

self carry: false.
self x25: 0.
self x16: 1.

In this case, the carry flag was not set, but the negative was set. Which was even more odd. The case that should set carry was not setting it, and vice-versa. It seemed it was inverted! I did a final test just to confirm my assumption: 1-1 should set both the negative and carry flags if the carry flag was inverted.

self carry: false.
self x25: 0.
self x16: 1.

ARM Carry is indeed strange

I was puzzled for a moment, and then I got to look for a culprit: was our assembler that was doing something wrong? was it a bug in Unicorn, our machine code simulator? or was is something else?

After digging for some time I came to find something interesting in the ARM documentation:

For a subtraction, including the comparison instruction CMP and the negate instructions NEGS and NGCS, C is set to 0 if the subtraction produced a borrow (that is, an unsigned underflow), and to 1 otherwise.

https://developer.arm.com/documentation/dui0801/a/Condition-Codes/Carry-flag

And something similar in a stack overflow post:

ARM uses an inverted carry flag for borrow (i.e. subtraction). That’s why the carry is set whenever there is no borrow and clear whenever there is. This design decision makes building an ALU slightly simpler which is why some CPUs do it.

https://stackoverflow.com/questions/53065579/confusion-about-arm-documentation-on-carry-flag

It seems that the carry flag in ARM is set if there is no borrow, so it is indeed inverted! But it is only inverted for substractions!

Extending the Compiler with this

Since carry works different in different architectures, but only for substractions, I created a new instruction factory method JumpSubstractionCarry: that detects carry for substractions and is supposed to be platform specific. Then I replaced the code of the counter by the following:

self MoveA32: counterAddress R: counterReg.
self SubCq: 16r10000 R: counterReg.
countTripped := self JumpSubstractionCarry: 0.

JumpSubstractionCarry: is backend defined:

JumpSubstractionCarry: jumpTarget
    backEnd genJumpSubstractionCarry: jumpTarget

the default implementation just delegates to the original factory method:

genJumpSubstractionCarry: jumpTarget
    ^cogit JumpCarry: jumpTarget

and the ARM (both 64 and 32 bits) do use a jump if no carry instead!

genJumpSubstractionCarry: jumpTarget
    ^cogit JumpNoCarry: jumpTarget

With these, our tests went green for all platforms 🙂

If you want to see the entire related code, you can check the following WIP: https://github.com/pharo-project/opensmalltalk-vm/compare/headless…guillep:sista?expand=1