l'essentiel est invisible pour les yeux

Showing posts with label Erlang. Show all posts
Showing posts with label Erlang. Show all posts

Monday, July 14, 2008

[Erlang tips] Sometimes to call mnesia:wait_for_tables is required

Hmm..,
I spent three hours for a Mnesia's strange error message, so write tips about error I'm confused. The tables on Mnesia often need a time to prepare their tables, and if tables aren't available, then we'll see storange error message.

% db_test.erl


write_db_test() ->
start(), % Start the server, mnesia and create required tables.
?assertMatch({ok, _}, mnesia:transaction(fun() -> #person{name="rakuto"})).

You may see the error is as follows:

{badmatch,{aborted,{no_exists,ready_queue}}}


It seems it need to call mnesia:wait_for_tables.

% db_test.erl

write_db_test() ->
start(), % Start the server, mnesia and create required tables.
mnesia:wait_for_tables([person], 3000),
?assertMatch({ok, _}, mnesia:transaction(fun() -> #person{name="rakuto"})).

I think that error message "{no_exists, table_name}" is not human friendly.

Friday, July 04, 2008

[Erlang] How to package an application with erlware(faxien and sinan commands)

We usually need to package system and repositories in order to create a big application. Erlware is a repository for Erlang programs and it provides softaware to create a package and release it to the world. There are pretty cool, it is just thing I want to. This entry introduces that how to create an application as package and install to system.

I create a HAVAL bindings for Erlang in order to learn how to create linked-in driver.
See Tutorial for how to create Erlang linked-in driver.

In this entry, I introduce how to create an application provides API for HAVAL bindings for Erlang. We need to install Faxien and Sinan previously.

0. Dicied a names of application and start to development a pakcage.
Create required directories for the application.

% mkdir haval && cd haval && mkdir -p cmds doc bin lib
% ls .
bin/ cmds/ doc/ lib/ releases/
%
Create directories for HAVAL bindings.
% mkdir -p lib/haval && cd lib/hava && mkdir src ebin include priv/lib
Configuration file named "_build.cfg" for this application.
% cat > _build.cfg
project : {
name : haval
vsn : "1.0.0"
},

repositories : ["http://repo.erlware.org/pub", "http://repo.martinjlogan.com/pub"]
1. Create an application
Create a HAVAL bindings for Erlang in this entry. This includes following source codes:
HAVAL bindings is provides as linked-in driver, these source are in lib/haval/c_src directory.
  • Makefile
  • config.h
  • haval.c
  • haval.h
  • haval_drv.o
2. Build the shared library
HAVAL bindings requires shared library named "erl_drv.so", we may think want to integrate building task for linked-in driver to the package manager(Faxien). But unfortunately I don't know that, so I need to research for handling shared library.

All sources are here.

Do make task manually here.
% cd lib/haval/c_src
% make # Copy haval_drv.so haval/priv/lib
3. Build the application.
It 'sinan' command is used for building the application. We need to run background server with 'sinserv' command before compile them.
% sinserv # Placed in "/usr/local/erlware/release_packages/sinan-0.10.0.12/bin/sinserv" in my enviroment
Build the application with 'sinan' command. We can see help when '+help' command is passed.
% sinan +help
sinan [args] [task]
local args (+) and server args. local args may be any of the following
+url : The url to connect to and control
+help : This help message

Server args are much more complex. There are always sane defaults so
you shouldn't need them, but you may. To get information about server
args read the sinan documentation.
%
% sinan # Default task is build
starting run
[check_depends] start
[check_depends] stop
[build] start
[build] Building ~/haval-1.0.0/lib/haval/src/haval_server.erl
[build] Building ~/haval-1.0.0/lib/haval/src/haval_sup.erl
[build] Building ~/haval-1.0.0/lib/haval/src/haval_app.erl
[build] Building ~/haval-1.0.0/lib/haval/src/haval.erl
[build] stop
run complete
% ls _build/development/apps/haval-1.0.0 # Generated following sources
c_src/ ebin/ include/ priv/ src/
%
Create a package for release. This package is generated underneath "_build/development/apps/" directory.
% sinan release
starting run
[check_depends] start
[check_depends] stop
[build] start
[build] stop
[release] start
[release] stop
run complete
%
4. Install the application locally.
Some of tasks are required in order to install the application. At first, we need to prepare the directory for released package, copy "_build/development/release" to "releases/haval-1.0.0/" and copy "_build/development/haval-1.0.0" to "releases/haval-1.0.0".
% mkdir releases/haval-1.0.0/ && mkdir lib
% cp -r _build/development/release releases/haval-1.0.0/
% cp -r _build/development/apps/haval-1.0.0 releases/haval-1.0.0/lib

Install the application locally, there will be deployed on "/usr/local/erlware/release_packages/haval-1.0.0/". We can use 'faxien' command in order to handle the packages (local and remote). Please execute "faxien help commands" for more details.

% faxien install-release releases/haval-1.0.0
ok
% ls /usr/local/erlware/release_packages/haval-1.0.0/
LICENCE README lib/ release/
Run and test.
% erl
Erlang (BEAM) emulator version 5.6.2 [source] [smp:2] [async-threads:0] [kernel-poll:false]

Eshell V5.6.2 (abort with ^G)
1> haval:start().
ok
2> haval:haval_string("test").
"593C9AED973BB51A3C852FB4E051D7C26686B9468B4E405350CB6805DC1B99E6"
3> haval:haval_file("_build.cfg").
"5A114E356FBB0CCCED8C7574B7A71780C8F33D3A9B37B8642126B78429B2988F"
4>
5. Test the application
We can use EUnit for testing the application, and do test with following command:
% sinan test

Tuesday, July 01, 2008

Manipulate Erlang binary term format in C/C++ with ei library

Erlang includes ei library for handling Erlang binary term format. This entry introduces how to use this library.

1. Pass the one string to C driver from Erlang.
It's pretty simple case, and pass one string argument to C driver from Erlang. It should be passed port_control/3 or erlang:port_control/3 after all arguments should be encoded as binary. The "index" variable are used for pointing position in the buffer.

% echo.erl


-define(DRV_ECHO, 1).

echo(Str) ->
control(?DRV_ECHO, Str).

control(Cmd, Data) ->
[{port, Port}| _] = ets:lookup(echo_table, port),
Bin = term_to_binary(Data), % XXX: All arguments will be encoded as binary term format.
Res = port_control(Port, Cmd, Bin),
binary_to_term(Res).

% echo_drv.c

#define DRV_ECHO 1

static int control(ErlDrvData drv_data, unsigned int command, char *buf,
int len, char **buf, int rlen) {
int index, type, size, arity ,ver;
char *arg1;

switch(command) {
case DRV_ECHO:
// It must call ei_decode_version at the beginning when argument is Erlang binary term format.

// All ei_xxx functions return 0 if it will be success, else return -1
ei_decode_version(buf, &index, &ver); // Value of 'index' will become 1.

// Get size of string in order to allocate buffer for argument
ei_get_type(buf, &index, &type, &size); // value of 'type' equal ERL_STRING_EXT

// Allocate memory for buffer
arg1 = new char[size + 1]; // Extra bite is required for NULL string.
/* do something */
}
}


Type of Erlang term
It can obtain type of Erlang term with result of ei_get_type function. All value of types are defined in ei.h.

#define ERL_SMALL_INTEGER_EXT 'a'
#define ERL_INTEGER_EXT 'b'
#define ERL_FLOAT_EXT 'c'
#define ERL_ATOM_EXT 'd'
#define ERL_REFERENCE_EXT 'e'
#define ERL_NEW_REFERENCE_EXT 'r'
#define ERL_PORT_EXT 'f'
#define ERL_PID_EXT 'g'
#define ERL_SMALL_TUPLE_EXT 'h'
#define ERL_LARGE_TUPLE_EXT 'i'
#define ERL_NIL_EXT 'j'
#define ERL_STRING_EXT 'k'
#define ERL_LIST_EXT 'l'
#define ERL_BINARY_EXT 'm'
#define ERL_SMALL_BIG_EXT 'n'
#define ERL_LARGE_BIG_EXT 'o'
#define ERL_NEW_FUN_EXT 'p'
#define ERL_FUN_EXT 'u'

#define ERL_NEW_CACHE 'N' /* c nodes don't know these two */
#define ERL_CACHED_ATOM 'C'


2. Pass the List
It should call ei_decode_list_header function when parse the list term. It must be called when parse the list term.

% echo_list.erl

-define(DRV_ECHO_LIST, 1).

echo(List) when is_list(List) ->
control(?DRV_ECHO_LIST, List).

control(Cmd, Data) ->
[{port, Port}| _] = ets:lookup(echo_list_table, port),
Bin = term_to_binary(Data), % XXX: All arguments will be encoded as binary term format.
Res = port_control(Port, Cmd, Str),
binary_to_term(Res).


% echo_drv.c

#define DRV_ECHO 1

static int control(ErlDrvData drv_data, unsigned int command, char *buf,
int len, char **buf, int rlen) {
int index, type, size, arity ,ver;
char *arg1;

switch(command) {
case DRV_ECHO_LIST:
// It must call ei_decode_version at the beginning when argument is Erlang binary term format.

// All ei_xxx functions return 0 if it will be success, else return -1
ei_decode_version(buf, &index, &ver); // Value of 'index' will become 1.

// Get size of string in order to allocate buffer for argument
ei_get_type(buf, &index, &type, &size);

// Check the type
if(type == ERL_LIST_EXT) {
// It must be called at the begginning when hanle the list term.
ei_decode_list_header(buf, &index, &arity);

printf("number of elements: %d\n", arity);

// Check the type of first element in the list
ei_get_type(buf, &index, &type, &size);

swithc(type) {
case ERL_SMALL_INTEGER_EXT:
case ERL_INTEGER_EXT:
// do something
break;
case ERL_STRING_EXT:
// do something
break;
}
}
}
}

Monday, June 30, 2008

Tutorial for how to create Erlang linked-in driver.

Erlang provides mechanism to communicate between Erlang and other programs written by C/C++ or other. This entry introduce how to create simple linked-in driver. Create shared library and Erlang interface to use Erlang's linked-in driver.

Goal: To create HAVAL bindings for Erlang.
HAVAL is fast cryptographic hash algorithm is invented by Yuliang Zheng, Josef Pieprzyk, and Jennifer Seberry in 1992. This library is pretty simple, so this is good choice for this tutorial.

Step 0: Files


config.h (Generated by configure script in HAVAL package)
haval.h
haval.c
haval.erl
haval_drv.cc
Makefile


haval.c and haval.h are included haval-1.1.tar.gz (C source code).

You can checkout these sample sources with Git. see haval_erlang.

% git clone git://github.com/rakuto/erlang_haval.git erlang_haval


Step 1: Create Erlang's interface
Erlang provides port mechanism in order to communicate between Erlang and other loaded shared object. See also Writing an Erlang Port using OTP Principles


-module(haval).
-author('Rakuto Furutani <xri://=rakuto>').

-define(DRV_INFO, 1).
-define(DRV_HAVAL_STRING, 2).
-define(DRV_HAVAL_FILE, 3).

-export([start/0, stop/0]).
-export([info/0, haval_string/1, haval_file/1]).
-ifdef(debug).
-export([test/0, test/1]).
-endif.

%%
%% Public interface
%%
start() ->
start("haval_drv"),
ok.

start(SharedLib) ->
case erl_ddll:load_driver(".", SharedLib) of
ok -> ok;
{error, already_loaded} -> ok;
_ -> exit({error, could_not_load_driver})
end,
register_lib(SharedLib).

stop() ->
[{port, Port}| _] = ets:lookup(haval_table, port),
Port ! {close, self()},
ok.

%% TODO: Implement this function return the information for module
info() -> ok.

%% Caluculate a value of HAVAL from string
haval_string(Str) ->
binary_to_term(control(?DRV_HAVAL_STRING, Str)).

%% Caluculate a value of HAVAL from file
haval_file(FileName) ->
binary_to_term(control(?DRV_HAVAL_FILE, FileName)).

-ifdef(debug).
test() ->
haval:start(),
Str = "I love Erlang.",
FileName = "haval.erl",
Hash1 = haval:haval_string(Str),
Hash2 = haval:haval_file(FileName),
io:format("HAVAL(~p) = ~p ~n", [Str, Hash1]),
io:format("HAVAL(~p) = ~p ~n", [FileName, Hash2]),
haval:stop(),
halt().

test([Str|_]) ->
haval:start(),
Hash = haval:haval_string(Str),
io:format("HAVAL(~p) = ~p ~n", [Str, Hash]),
haval:stop(),
halt().
-endif.

%%
%% Internal functions
%%
register_lib(SharedLib) ->
Port = open_port({spawn, SharedLib}, []),
Tab = ets:new(haval_table, [set, protected, named_table]),
ets:insert(Tab, {port, Port}).

control(Cmd, Data) ->
[{port, Port}| _] = ets:lookup(haval_table, port),
erlang:port_control(Port, Cmd, Data).



Step 2: Create C/C++ shared library
Shared library writted by C++ is loaded by Erlang interface when call erl_ddll::load_driver/2. This driver uses ei library in order to manipulate Erlang binary term.


/**
* HAVAL (cryptographic hash function) bindings for Erlang
*
* @author Rakuto Furutani <xri://=rakuto>
* @date 2008/06/28
*/
#include <stdio.h>
#include <string.h>
#include <stdarg.h>
#include <new>
#include "ei.h"
#include "erl_driver.h"
#include "erl_interface.h"
extern "C" {
// Define constant for haval.h
#define PASS 3
#define NUMBER_OF_BLOCKS 5000
#define FPTLEN 256
#ifndef LITTLE_ENDIAN
#define LITTLE_ENDIAN 1
#endif

#include "haval.h"
}

// Functions are provided by this driver
#define DRV_INFO 1
#define DRV_HAVAL_STRING 2
#define DRV_HAVAL_FILE 3

/* Driver Interface Declarations */
static ErlDrvData start_haval_driver(ErlDrvPort port, char *command);
static void stop_haval_driver(ErlDrvData drv_data);
static int control(ErlDrvData drv_data, unsigned int command, char *buf,
int len, char **rbuf, int rlen);
static void haval_to_hex(unsigned char*, char*);
static ErlDrvBinary* ei_x_to_new_binary(const ei_x_buff*);

/* Driver Entry */
static ErlDrvEntry haval_driver_entry = {
NULL,
start_haval_driver,
stop_haval_driver,
NULL,
NULL,
NULL,
"haval_drv",
NULL,
NULL,
control,
NULL,
NULL
};

typedef struct _drv_data {
ErlDrvPort port;
} drv_data_t;

/* DRIVER INTERFACE */

static ErlDrvData start_haval_driver(ErlDrvPort port, char *command)
{
drv_data_t *data;

data = (drv_data_t*) driver_alloc(sizeof(drv_data_t));
data->port = port;
set_port_control_flags(port, PORT_CONTROL_FLAG_BINARY);
return (ErlDrvData) data;
}

static void stop_haval_driver(ErlDrvData drv_data)
{
driver_free((char*) drv_data);
}

static int control(ErlDrvData drv_data, unsigned int command, char *buf, int len, char **rbuf, int rlen)
{
int ret = -1;
char hex[(FPTLEN >> 3) << 1];
unsigned char fingerprint[FPTLEN >> 3];
char *arg1;
ei_x_buff x_buff;

try {
// argument
arg1 = new char[len + 1];
strncpy(arg1, buf, len);
strcat(arg1, "�0");

ei_x_new_with_version(&x_buff);
switch(command) {
case DRV_INFO:
ei_x_encode_string(&x_buff, "info");
ret = sizeof("info") * sizeof(char);
break;
case DRV_HAVAL_STRING:
haval_string(arg1, fingerprint);
haval_to_hex(&fingerprint[0], &hex[0]);
ret = sizeof(hex);
ei_x_encode_string(&x_buff, hex);
break;
case DRV_HAVAL_FILE:
if(!haval_file(arg1, fingerprint)) {
haval_to_hex(&fingerprint[0], &hex[0]);
ret = sizeof(hex);
ei_x_encode_string(&x_buff, hex);
} else {
erl_err_sys("haval_file");
}
break;
}
if(ret > 0) *rbuf = reinterpret_cast<char*>(ei_x_to_new_binary(&x_buff));
ei_x_free(&x_buff);
} catch(std::bad_alloc) {
erl_err_sys("can not allocate memory");
}

return ret;
}

static void haval_to_hex(unsigned char *fingerprint, char *hex)
{
for(int i=0; i < FPTLEN >> 3; ++i) {
sprintf(&hex[i << 1], "%02X", fingerprint[i]);
}
}

// Init the driver
extern "C" DRIVER_INIT(haval_drv)
{
return &haval_driver_entry;
}

// Utilities
static ErlDrvBinary* ei_x_to_new_binary(const ei_x_buff *x_buff)
{
ErlDrvBinary *bin = driver_alloc_binary(x_buff->index);
if(bin != NULL) {
memcpy(bin->orig_bytes, x_buff->buff, x_buff->index);
}
return bin;
}


Step 3: Compile all sources
Create Makefile for build.
FYI: If you want to build linked-in driveron Mac OS, see Compile Erlang linked-in driver on Mac OSX (Darwin)


.PHONY: clean
.SUFFIXES: .o .c .cc .erl .beam

OS= ${shell uname}
CC=gcc
CXX=g++
CXXFLAGS=-Wall -g

# Erlang
ERL_INCLUDE = -I/usr/local/lib/erlang/usr/include
ERL_LIBS = -L/usr/local/lib/erlang/usr/lib \
-lerts
EI_INCLUDE = -I/usr/local/lib/erlang/lib/erl_interface-3.5.6/include
EI_LIBS = -L/usr/local/lib/erlang/lib/erl_interface-3.5.6/lib \
-lei \
-lerl_interface

TARGET_LIB = haval_drv.so
ifeq ($(OS), Darwin)
EXTRA_OPTIONS = -fno-common -bundle -undefined suppress -flat_namespace
endif
ALL: $(TARGET_LIB) haval.beam

.erl.beam:
erlc -W -Ddebug $<

.c.o:
$(CC) $(CFLAGS) -c $<

.cc.o:
$(CXX) $(CXXFLAGS) $(ERL_INCLUDE) $(EI_INCLUDE) -c $<

haval_drv.so: haval.o haval_drv.o
$(CXX) -o $@ $^ $(ERL_LIBS) $(EI_LIBS) $(EXTRA_OPTIONS) -fpic -O2

clean:
rm -f *.beam *.o *.so



Step 4: Run and test

% erl -noshell -run haval test
HAVAL("I love Erlang.") = "3AECEDF5133ED147C704A25C2CCA9A994FBB984FBCB22C1A523F31963E418498"
HAVAL("haval.erl") = "E006098555DFF788E73C7A263615DF60CC82396B1BCE7AEB5FB7050C376F2B03"

Saturday, June 28, 2008

Compile Erlang linked-in driver on Mac OSX (Darwin)

In order to compile the Erlang linked-in driver on Darwin, we must specify some options when it will be linked. If you don't specify these options, you might see errors is as follows:

% gcc -o my_drv.dymlib my_drv.c -dynamiclib -fpic

/usr/bin/ld: Undefined symbols:
_driver_alloc
_driver_free
_driver_output

That's invalid, so it must specify some options below in order to create the shared object.

% gcc -o my_drv.so my_drv.c -bundle -flat_namespace -udefined suppress


see README in source code of Erlang/OTP for more details.

Friday, June 27, 2008

Asynchronous driver in Erlang using driver_async

Erlang provides some mechanism is to communicate other language, this can be used for to reuse module, speed things up (Erlang is ten times as slow as slow than C), to access OS resource and so on. The below mechanism are provided.

Pipe Drivers
The "pipe drivers" enables bi-directional communication. The external process can be implemented in C or any language, and it can use erl_interface and ei library in order to manipulate data of Erlang as ETERM.

Benefit:

  • Frexible, it can use some types data.
  • Any language

Demerit:
  • Communication overhead

Linked-in Drivers
A driver in Erlang is a library written in C/C++, that is linked to the Erlang emulator and called from erlang. A driver can be dynamically loaded, as a shared library (known as a DLL on windows), or statically loaded, linked with the emulator when it is compiled and linked.

Benefit:
* Faster than "Pipe Drivers"

Demerit:
* Not frexible, to pass the complex data between C/C++ and Erlang is difficult. (see also ei)


Asynchonous Linked-in Driver
The "Linked-in Drivers" is faster than "Pipe Drivers", but it can not scale with a lot of process, since it run synchronously. So If you'd like to take advantage of platforms with multiple CPUs, then you must run some Erlang virtual machine since it run on single thread.

It sounds like great!, but we should care the some problems, multi-thread problem and so on.

Resources:

Next, I'll write an entry about how to create asynchronous driver.