Ideas and essays on code development

JavaScript pattern matching

2018-10-22T08:58:00.000-07:00

If you like JavaScript (and I have been writing quite a bit again recently) then you probably also love Reg “raganwald” Braithwaite's mind stretching JavaScript articles and books.

I have just been enjoying a recent post on pattern matching that is elegant and demanding of the reader. Fabulous stuff which I enjoyed greatly but all the time I was reading the piece there was a niggle at the back of my head. The niggle kept trying to intrude with the message "matching parentheses is what stacks are for". OK, not strictly true - stacks are for a lot of things but the thought that JavaScript arrays act as an archetypal stack out of the box doubled down on the message.

I had to try "my way" to see how that went.

function balanced(str){
    let chrs = str.split("");
    let lefts = "({[";
    let rights = ")}]";
    let stack = [];
    for(let i = 0; i < chrs.length; i++){
        if(lefts.indexOf(chrs[i])>-1){
            stack.push(chrs[i]);
        } else {
            let idx = rights.indexOf(chrs[i]);
            if(idx >= 0 && idx <= 2){
                if(stack[stack.length-1] === lefts.substr(idx,1)){
                    stack.pop();
                } else {
                    stack.push(chrs[i]);
                }
            }
        }
    }
    return stack.length === 0;
}

The strings lefts and rights store the characters to be balanced in a nice extendable way. The first line that splits the function string argument uses the empty string as a delimiter so that the resulting array is constructed with each character in the input string becoming an individual array element.

The process is simple. If a left hand bracket character (lefts) is found then it is pushed onto the stack. If a right hand bracket character (rights) is found then the top element of the stack should be the matching left hand character. If the match is found then it is popped from the stack with any unmatched right hand items being added to the stack to force the correct result and to aid debugging. [If the test strings were likely to be long then you would probably force a return at that point.] The function returns true if the stack is empty after the process has run and otherwise false.

Works with raganwald's test strings such as ({}(()))(() => false and ({()[]})[[(){}]] => true.

Do we go with elegant or (brutal) pragmatic?

<Edit> Of course one of those lines of code should have been

if(idx >= 0){

</Edit>
<Addendum>

</Addendum>

Docker and .NET on a Windows machine

2017-01-15T09:09:00.000-08:00

It was time to take some first steps with Docker (way past time some might claim). I was moved to have a bash (pun – see later) on a Friday afternoon after seeing an intro article in issue 28 of the DNC “magazine” written by Daniel Jimenez Garcia. However I did hit the odd issue when following the article so this post follows the first steps but includes the odd tweak. I am writing this based upon a Windows 10 Pro, 64 bit host machine – valid alternate hosts exist but this is what I have and I suspect will be available to many who want to dip in a toe. The Hyper-V package must be enabled for Docker on Windows to work. The Docker for Windows installer will enable it for you, if needed. I think that Developer Mode must also be set for your Windows 10 installation in “Settings : Update and Security” but you probably already have that set.

First off, install Docker for Windows 10. Browse to https://www.docker.com/ click “Get Started” and then the offered download option. Run the install (.msi) and after the usual confirmations and a little set-up delay you will get confirmation of a successful install with the option to start Docker. Go for it.

Fire up a Command Prompt and then type the following

>docker run --rm –it microsoft/dotnet:latest

Docker then looks for the microsoft/dotnet image and if it is not found (as this is the first use) then it will be downloaded and installed. This will take a few short minutes during which there is plenty of feedback on progress.

N.B. If you follow the DNC article you will see that the image name has been capitalised there (Microsoft/dotnet) but this is a magazine editing error.

Docker will then start a new container for the downloaded image and execute the appropriate entry point with the container attached to your Command Prompt so you can interact with it.
You should see a Linux bash prompt – something like:

root@6077575d9218:/#

which is particularly cool if you have not previously fired up Linux on you Windows 10 installation.

Now we can create a new folder (don’t worry, as we typed –rm none of this will exist once you exit so will not clutter up your hard drive). Then create a new .NET program in that folder and run it.

root@6077575d9218:/# mkdir hellodocker
root@6077575d9218:/# cd hellodocker
root@6077575d9218:/hellodocker# dotnet new
Created new C# project in /hellodocker.
root@6077575d9218:/hellodocker# dotnet restore
log : Restoring packages for /hellodocker/project.json...
root@6077575d9218:/hellodocker# dotnet run
Project hellodocker (.NETCoreApp,Version=v1.1) will be compiled because expected outputs are missing
Compiling hellodocker for .NETCoreApp,Version=v1.1
Compilation succeeded.
0 Warning(s)
0 Error(s)
Time elapsed 00:00:05.7903169
Hello World!
root@6077575d9218:/hellodocker#

We have just run a .NET Core program in a Docker instance running on a Linux virtual machine hosted by our Windows 10 PC.

Type exit to terminate the container and return to the command prompt.

We are next going to create a .NET program instance in a Windows folder. This makes use of the .Net Command Line Interface (CLI) which may not yet be installed on your machine. The simplest way to check is to try and use it with the following sequence of commands:

>mkdir hellodocker
>cd hellodocker
>dotnet new

And if you get an error message from the last command then you will need to install the Command Line Interface. The latest version should be available here https://www.microsoft.com/net/core#windowsvs2015

The install presupposes that you have Visual Studio 2015 installed and that it is up to date (update 3 as at January 2017). Once installed you will need to close and re-open your command prompt window as the system path has now been updated to point to dotnet.

Now typing >dotnet new will result in the response that a new C# project has been created in your folder. The folder now contains two files – Program.cs and project.json
They look like this


using System;
namespace ConsoleApplication
{
    public class Program
    {
        public static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
        }
    }
}

and


{
  "version": "1.0.0-*",
  "buildOptions": {
    "debugType": "portable",
    "emitEntryPoint": true
  },
  "dependencies": {},
  "frameworks": {
    "netcoreapp1.0": {
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.0.1"
        }
      },
      "imports": "dnxcore50"
    }
  }
}

Now we can start a new Docker container mounting the Windows folder we just created as a volume.

You will need to substitute your user name into the Windows path below but on my machine I typed:

docker run --rm -it -v /c/Users/mike/hellodocker:/app microsoft/ dotnet:latest

which bumped nicely into the next problem as my C: drive was not shared.

There will be a Docker icon sitting somewhere at the bottom right of your screen (could be in the “hidden” group), so right click that and select “Settings”. Click the “Shared Drives” tab and check the relevant box before the “Apply” button.

You should be prompted for your password and then Docker can implement the share. Back in our Command Prompt window, repeating the last command should see that all is now well. We can check that the Windows folder is now mounted as /app by typing the following command:

ls /app

which will list the two existing files in the .NET Core project.

Now it should be straightforward to compile and run the program with the following:

root@4c4e67eedfdc:/# cd /app && dotnet restore && dotnet run

which in my case produced a lot of interesting output but failed to run. It looked like the two versions of .NET Core (Docker’s and the Windows CLI version just installed) were different.

The version specified in the json file I created in Windows was 1.0.1 but the Docker image was version 1.1.0. This was probably a timing issue and thus transient though I might have tried specifying a version instead of :latest in my Docker command.

The command cat project.json listed the json file and confirmed the project specified version number.

My work around was a simple text substitution in the json file:

sed -i -e 's/1.0.1/1.1.0/g' project.json

followed by a dotnet restore and dotnet run and all was well with the program compiling and executing. A quick look back at the Windows folder shows that bin and obj debug folders have been created and populated.

I can only recommend continuing with the DNC article and building some initial Docker experience – I will be next Friday. http://www.dotnetcurry.net/s/dnc-mag-28th-single

Links:

.NET Core CLI tools https://docs.microsoft.com/en-us/dotnet/articles/core/tools/

Get started with Docker for Windows https://docs.docker.com/docker-for-windows/

(Related follow up) Turning on the Windows subsystem for Linux and getting to know “Bash on Ubuntu” http://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/ with a helpful MS FAQ here https://msdn.microsoft.com/en-us/commandline/wsl/faq

Fuzzy Dates

2016-09-27T07:42:00.002-07:00

I can remember Fuzzy Logic being quite the thing in the nascent computer based AI projects in the late 70’s and early 80’s. The idea being that you might not have precise values for a given data or that a set of rules might be more or less applicable within a given process – things could be fuzzy but you could still hope to obtain a near optimal output for a given system. If I recall correctly then some of the Expert Systems development work relied upon classes of input that were often akin to broad estimates (at least ranges) rather than precise values.

Anyone who has developed software that deals with scheduling or calendars or some such has run into issues relating to the human cognitive imprecision with dates and times. Often there is an intention to get something done “that evening” or by “Thursday week” and mostly software ends up forcing the user to plump for some measure of precision and carry the fuzziness of that intention in their heads which is suboptimal.

Entering dates to a computer based system has always been a programming challenge. In far off pre-GUI days we built a callable routine for date entry on Vax/VMS that was pretty liberal in what it would accept and attempt to convert into a precise date. Among a number of date formats this also included short-hand like t (today) and t-1 (yesterday) etc.. GUIs have brought us clickable calendar like objects but these can be remarkably clumsy to use and do little to help with the issues around imprecision.

App outputs need to do more to be human sensible. “Tomorrow” or “next week” might communicate a date more effectively to a user than 02/11/2016 (which in any case might be in November or February – you choose). A while back we prototyped some software that converted a date of birth to an age and, within the context of young children, that needed to output values like “3 days old” or “4 weeks” or “9 months” or “4 years” as the date receded into the past and our human expression of age changed accordingly.

The iOS NSDateFormatter has a doesRelativeDateFormatting option that can manage yesterday, today and tomorrow (at least) and Rails has a more extensive distance_of_time_in_words function and I believe that John Resig wrote pretty.js for JavaScript. I do not doubt a great many programmers have knocked up something along these lines over the years. These are all fine but are based upon a specific date or date/time. Being a little less sure about when (“next week” is a nice example) makes things a little more interesting.

The Chrono project in GitHub is a nice introduction to the challenges of parsing a date time string as precise as Sat Aug 17 2013 18:40:39 GMT+0900 or as fuzzy as “last Friday” but that in turn raises the implementation issue of how precisely will a given user supply their date as well as reminding us that in many contexts the time zone might be extremely relevant. There is also some discussion with code samples on StackExchange

Storing an imprecise date or date time is probably best accomplished with a maximum and minimum value. One might consider storing a single value with a precision indicator but I suspect that code would inevitably be required to convert that precision into a range for comparison purposes and so you might as well settle for twin values from get go. There are some interesting StackExchange discussions on this topic as well.

A satisfactory solution on the input side would need to take account of semantics. The shorthand for today plus one week (T+1w perhaps) is not the same as “next week” which is also not the same as “in the next week”. A humanised output is probably simpler but it looks like this would have to be heavily influenced by context so a standardised class might need some alternate output vocabulary options (age and “elapsed time” being two obvious examples).

This is a DRAFT version of a class that will accept a range of strings and convert those in turn into a date range representing a fuzzy date. The class steals happily from some of the sources mentioned above. Input can include things like:

Today
Tomorrow
Yesterday
next or Last Month
Next or Last Year
In the Next n Months
In the next n Days
During the next week/month/year
T(oday) + or – n d(ays)/w(eeks)/m(onths/y(ears)
next/last (day of week)
next/last (named month)


class FuzzyDate
{
    #region private declarations
    #region static declarations
    private static readonly DateTime sqlMinDate = new DateTime(1753, 1, 1); // minimum SQLServer datetime
    static List<string> dayList = new List<string>() { "sun", "mon", "tue", "wed", "thu", "fri", "sat" };
    static List<string> monthList = new List<string>() { "jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep", "oct", "nov", "dec" };
    private static List<IDateTimeInput> parsers = new List<IDateTimeInput>()
    {
        new RegexDateParser(
            @"next +([2-9]\d*) +months",
            delegate (Match m) 
            {
                var val = int.Parse(m.Groups[1].Value);
                return new DateTime[] {DateTime.Now, DateTime.Now.AddMonths(val)};
            }),
        new RegexDateParser(
            @"next +([2-9]\d*) +days",
            delegate(Match m)
            {
                var val = int.Parse(m.Groups[1].Value);
                return new DateTime[] {DateTime.Now, DateTime.Now.AddDays(val)};
            }),
        new RegexDateParser(
            @"tomorrow",
            delegate(Match m)
            {
                var dt = DateTime.Today.AddDays(1);
                return new DateTime[] {dt,  new DateTime(dt.Year, dt.Month, dt.Day, 23, 59, 59, 999)};
            }),
        new RegexDateParser(
            @"today",
            delegate(Match m)
            {
                var dt = DateTime.Now;
                return new DateTime[] { dt, new DateTime(dt.Year, dt.Month, dt.Day, 23, 59, 59, 999)};
            }),
        new RegexDateParser(
            @"yesterday",
            delegate(Match m)
            {
                var dt = DateTime.Today.AddDays(-1);
                return new DateTime[] { dt, new DateTime(dt.Year, dt.Month, dt.Day, 23, 59, 59, 999)};
            }),
        new RegexDateParser(
            @"(in|during) * (last|next) * (year|month|week)",
            delegate(Match m)
            {
                if(m.Groups[2].Value == "last")
                {
                    switch(m.Groups[3].Value)
                    {
                        case "year":
                            var dt = DateTime.Today.AddYears(-1);
                            return new DateTime[] {dt, DateTime.Today};
                         case "month":
                            var dtm = DateTime.Today.AddMonths(-1);
                            return new DateTime[] {dtm, DateTime.Today};
                         case "week":
                             var dtl = DateTime.Today.AddDays(-7);
                            return new DateTime[] {dtl, DateTime.Today};
                         default:
                            return null;
                    }
                } else
                {
                    switch(m.Groups[3].Value)
                    {
                        case "year":
                            var dt = DateTime.Now.AddYears(1);
                            return new DateTime[] {DateTime.Now, dt};
                         case "month":
                            var dtm = DateTime.Now.AddMonths(1);
                            return new DateTime[] {DateTime.Now, dtm};
                         case "week":
                             var dtl = DateTime.Now;
                            return new DateTime[] {dtl, dtl.AddDays(7)};
                         default:
                            return null;
                    }
                }
            }),
        new RegexDateParser(
            @"(last|next) *(year|month|week)",
            delegate(Match m)
            {
                int val = (m.Groups[1].Value == "last")? -1 :1;
                switch(m.Groups[2].Value)
                {
                    case "year":
                        var dt = DateTime.Now.AddYears(val);
                        return new DateTime[] {new DateTime(dt.Year,1,1), new DateTime(dt.Year,12,31,23,59,59,999)};
                     case "month":
                        var dtm = DateTime.Now.AddMonths(val);
                        return new DateTime[] {startOfMonth(dtm), endOfMonth(dtm)};
                     case "week":
                        val = (val == 1) ? 7 - (int)DateTime.Today.DayOfWeek : -(7 + (int)DateTime.Today.DayOfWeek);
                         var dtl = DateTime.Today.AddDays(val);
                        return new DateTime[] {dtl, dtl.AddDays(7).AddSeconds(-1)};
                   default:
                        return null;
                }
            }),
        new RegexDateParser(
            String.Format(@"(last|next) *({0}).*", String.Join("|", dayList.ToArray())),
            delegate(Match m)
            {
                var day = m.Groups[2].Value;
                var val = dayList.IndexOf(day.Substring(0,3));
                var adj = (m.Groups[1].Value == "last") ? -1 : 1;
                if(val >= 0)
                {
                    val = adj * (val - (int)DateTime.Today.DayOfWeek);
                    if(val <= 0) {val += 7; }
                    var dt = DateTime.Today.AddDays(val * adj);
                    return new DateTime[] {dt, dt.AddDays(1).AddSeconds(-1)};
                } else { return null;}
            }),
        new RegexDateParser(
            String.Format(@"(last|next) *({0}).*", String.Join("|", monthList.ToArray())),
            delegate(Match m)
            {
                var month = m.Groups[2].Value;
                var val = monthList.IndexOf(month.Substring(0,3));
                var adj = (m.Groups[1].Value == "last") ? -1 : 1;
                if(val >= 0)
                {
                    val = adj * (val - ((int)DateTime.Today.Month-1));
                    if(val <= 0) {val += 12; }
                    var dt = DateTime.Today.AddMonths(val * adj).AddDays(-(DateTime.Today.Day -1));
                    return new DateTime[] {dt, dt.AddMonths(1).AddSeconds(-1)};
                } else { return null;}
            }),
        new RegexDateParser(
            @"t(\s)?(\-|\+)(\s)?([1-9]\d*)(\s)?(d|m|y|w)", // t \=/- n d,w,m,y format
            delegate(Match m)
            {
                string sign = m.Groups[2].Value;
                string mVal = m.Groups[4].Value;
                string mType = m.Groups[6].Value;
                var val = Int32.Parse(mVal);
                if(sign == "-") {val *= -1; }
                DateTime tDay = DateTime.Today;
                switch (mType)
                {
                    case "y":
                        tDay = tDay.AddYears(val);
                        break;
                    case "m":
                        tDay = tDay.AddMonths(val);
                        break;
                    case "w":
                        tDay = tDay.AddDays(7 * val);
                        break;
                    case "d":
                        tDay = tDay.AddDays(val);
                        break;
                    default:
                        return null;
                }
                return new DateTime[] {tDay, tDay.AddDays(1).AddSeconds(-1)};
            })
    };
    #endregion
    
    private DateTime minDate = sqlMinDate;
    private DateTime maxDate = DateTime.MaxValue;
    private bool timeSignificant = true; // or false - we shall see
    #endregion
    #region public constructors and methods
    public FuzzyDate()
    {
    }
    public FuzzyDate(DateTime setDate)
    {
        minDate = maxDate = setDate;
    }
    public FuzzyDate(string dateString)
    {
        setDate(dateString.ToLower());
    }
    public void SetDate(string dateString)
    {
        setDate(dateString.ToLower());
    }
    public void SetDate(DateTime dateTime)
    {
        minDate = maxDate = dateTime;
    }
    #endregion
    #region public properties
    public DateTime MinDate
    {
        get { return minDate; }
        set { minDate = value; }
    }
    public DateTime MaxDate
    {
        get { return maxDate; }
        set { maxDate = value; }
    }
    public bool TimeSignificant
    {
        get { return timeSignificant; }
        set { timeSignificant = value; }
    }
    public bool IsFuzzy
    {
        get { return minDate == maxDate; }
    }
    #endregion
    #region private methods
    private void setDate(string dateString)
    {
        DateTime[] dt;
        foreach(var parser in parsers)
        {
            dt = parser.Parse(dateString);
            if(dt != null)
            {
                minDate = dt[0];
                maxDate = dt[1];
                break;
            }
        }
    }
    private static DateTime startOfMonth(DateTime dayInMonth)
    {
        return new DateTime(dayInMonth.Year, dayInMonth.Month, 1);
    }
    private static DateTime endOfMonth(DateTime dayInMonth)
    {
        var rVal = dayInMonth.AddMonths(1);
        rVal = new DateTime(rVal.Year, rVal.Month, 1);
        return rVal.AddSeconds(-1);
    }
    #endregion
    #region local interface and class
    private interface IDateTimeInput
    {
        DateTime[] Parse(string dateString);
    }
    private class RegexDateParser : IDateTimeInput
    {
        public delegate DateTime[] Interpreter(Match m);
        protected Regex regEx;
        protected Interpreter interpreter;
        public RegexDateParser(string regexString, Interpreter interpreter)
        {
            regEx = new Regex(regexString);
            this.interpreter = interpreter;
        }
        public DateTime[] Parse(string dateString)
        {
            var match = regEx.Match(dateString);
            if (match.Success)
            {
                return interpreter(match);
            }
            return null;
        }
    }
    #endregion
}

The output of human sensible dates based upon a given DateTime is rather simpler with the details being rather dependent upon the context. In my working examples of “elapsed time” and “age” the output is dependent upon the difference between a given date and the current one. This is normally expressed in the .NET environment as a TimeSpan object. However the TimeSpan class lacks a few features that would make life much simpler when dealing with time spans that represent more than just a few days. I have therefore explored the potential for a custom DateTimeSpan class that could probably be collapsed and simplified into a set of extensions for the .NET supplied standard. So please treat this as another draft idea.


public class DateTimeSpan
{
    private const int WEEK = 7;
    private const int MONTH = 30;
    private const int YEAR = 365;
    private const int MONTHS_IN_YEAR = 12;
    private int years;
    private int months;
    private int weeks;
    private int days;
    private int hours;
    private Int64 minutes;
    private Int64 seconds;
    private Int64 milliseconds;
    public DateTimeSpan(TimeSpan timeSpan)
    {
        grabValues(timeSpan);
        estimateValues();
    }
    public DateTimeSpan(DateTime datePast)
    {
        grabValues(DateTime.Now.Subtract(datePast));
        calculateValues(datePast, DateTime.Now);
    }
    public DateTimeSpan(DateTime fromDate, DateTime toDate)
    {
        grabValues(toDate.Subtract(fromDate));
        calculateValues(fromDate, toDate);
    }
    public int TotalYears { get { return years; } }
    public int TotalMonths { get { return months; } }
    public int TotalWeeks { get { return weeks; } }
    public int TotalDays { get { return days; } }
    public int TotalHours { get { return hours; } }
    public Int64 TotalMinutes { get { return minutes; } }
    public Int64 TotalSeconds { get { return seconds; } }
    public Int64 TotalMilliseconds { get { return milliseconds; } }
    public bool IsNegative { get { return milliseconds < 0; } }
    private  void grabValues(TimeSpan timeSpan)
    {
        milliseconds = (Int64)timeSpan.TotalMilliseconds;
        seconds = (Int64)timeSpan.TotalSeconds;
        minutes = (Int64)timeSpan.TotalMinutes;
        hours = (int)timeSpan.TotalHours;
        days = (int)timeSpan.TotalDays;
        if (days >= WEEK)
        {
            weeks = days / WEEK;
        }
    }
    private void estimateValues()
    {
        if(days >= MONTH)
        {
            months = days / MONTH;
        }
        if(days >= YEAR)
        {
            years = days / YEAR;
            months = years * MONTHS_IN_YEAR + ((days - years * YEAR) / MONTH);
        }
    }
    private void calculateValues(DateTime fromDate, DateTime toDate)
    {
        int sign = 1;
        if(fromDate > toDate)
        {
            var hdate = toDate;
            toDate = fromDate;
            fromDate = hdate;
            sign = -1;
        }
        int monthCount = -1;
        while (fromDate <= toDate)
        {
            fromDate = fromDate.AddMonths(1);
            monthCount++;
        }
        months = monthCount * sign;
        years = monthCount / 12;
    }
}

Any given output requirement would probably only need one set of rules but just for fun here is a class that can manage two and could be easily expanded to handle more.


public static class PrettyDate
{
    public enum PrettyTypes
    {
        [Description("Returns date as an age string")]
        Age,
        [Description("Returns date/time as elapsed time")]
        Elapsed
    }
    public static string GetPrettyDate(DateTime theDate, PrettyTypes prettyType)
    {
        DateTimeSpan dSince = new DateTimeSpan(theDate);
        if(prettyType == PrettyTypes.Elapsed)
        {
            if(dSince.IsNegative) { return "not happened yet"; }
            if(dSince.TotalYears > 0)
            {
                // could be tweaked for a rounded number in a number of ways - this is one
                int months = dSince.TotalMonths - dSince.TotalYears * 12;
                int years = (months >= 10) ? dSince.TotalYears + 1 : dSince.TotalYears;
                return years + " year" + ((years > 1) ? "s ago" : " ago");
            } else if (dSince.TotalMonths > 0)
            {
                return dSince.TotalMonths + " month" + ((dSince.TotalMonths > 1) ? "s ago" : " ago");
            } else if(dSince.TotalWeeks > 0)
            {
                return dSince.TotalWeeks + " week" + ((dSince.TotalWeeks > 1) ? "s ago" : " ago");
            }
            else if(dSince.TotalDays > 0)
            {
                return dSince.TotalDays + " day" + ((dSince.TotalDays > 1) ? "s ago" : " ago");
            }
            else if (dSince.TotalHours > 0)
            {
                return dSince.TotalHours + " hour" + ((dSince.TotalHours > 1) ? "s ago" : " ago");
            }
            else if(dSince.TotalMinutes > 0)
            {
                return dSince.TotalMinutes + " minute" + ((dSince.TotalMinutes > 1) ? "s ago" : " ago");
            }
            else
            {
                return "now"; // adjust to taste
            }
        } else
        {
            if (dSince.IsNegative) { return "not born yet"; }
            if (dSince.TotalYears >= 5)
            {
                return dSince.TotalYears + " years";
            } else if(dSince.TotalYears >= 3)
            {
                int months = dSince.TotalMonths - dSince.TotalYears * 12;
                string fract = "";
                if(months >= 8)
                {
                    fract = String.Concat(" ", (char)190);
                } else if(months >= 5)
                {
                    fract = String.Concat(" ", (char)189);
                } else if(months >= 2)
                {
                    fract = String.Concat(" ", (char)188);
                } else
                {
                    fract = " years";
                }
                return dSince.TotalYears + fract;
            } else if(dSince.TotalMonths >= 6)
            {
                return dSince.TotalMonths + " months";
            } else if(dSince.TotalWeeks >= 2)
            {
                return dSince.TotalWeeks + " weeks";
            } else if(dSince.TotalDays >= 2)
            {
                return dSince.TotalDays + " days";
            } else
            {
                return "newborn";
            }
        }
    }
}

It is very likely that any application with a requirement to process dates in the manner suggested here would need substantial code changes to match the working context. I may well return to this topic to explore the usage of dates stored as a range rather than as a single value.

Edited Jan 2017
Added this link to a great piece by Peter-Paul Koch on "input type=date" and why that is (or is not) complicated even when user hostile. Well worth your time.

A Persistent Queue with C#

2016-08-05T09:07:00.000-07:00

This is just a bit of programming trivia.

I wanted to run a queue of tasks each of which could take a while to complete. Given that the program could be terminated before all tasks had been run then I needed to store any remaining tasks until a later opportunity to process them. Also I wanted to run the task queue on a background thread and be able to add to the queue of tasks from the UI thread.

Fortunately .NET has an inbuilt ConcurrentQueue<T> in System.Collections.Concurrent that can be accessed from multiple threads without issue. All that needed to be added was some code to persist uncompleted queue tasks. In the code below I am using the .NET port of SQLite but a flat file would work fine as would any other database.

As I was using a database I wanted to differentiate between queued tasks already in the relevant table and any added from the UI thread. [Not of course forgetting to delete tasks in the database that are completed.] So a little class to represent a task – the task itself being just a string in this instance.


class QueuedTask
{
    private string task;
    private bool fromDatabase = false;
    public QueuedTask(string task)
    {
        this.task = task;
    }
    public QueuedTask(string task, bool fromDatabase)
    {
        this.task = task;
        this.fromDatabase = fromDatabase;
    }
    public bool InDatabase
    {
        get { return fromDatabase; }
    }
    public string Task
    {
        get { return task; }
    }
}

Creating an instance of my PersistentQueue Class starts a timer that loads any saved tasks into the queue on another thread so as not to block the UI thread.. The main program form FormClosing() event calls the class SaveTasks() method that saves any remaining tasks in the queue to the database.


class PersistentQueue
{
    public ConcurrentQueue<QueuedTask> TaskQueue = new ConcurrentQueue<QueuedTask>();
    private string dbPath = "";
    private Timer readTimer = new Timer();
    private const int startDelay = 10000;
    public PersistentQueue(string dbPath)
    {
        this.dbPath = dbPath;
        readTimer.Interval = startDelay;
        readTimer.Elapsed += (sender, e) => loadSavedTasks();
        readTimer.Start();
    }
    public void SaveTasks()
    {
        readTimer.Close();
        if (!TaskQueue.IsEmpty)
        {
            try
            {
                SQLiteConnection newCon = new SQLiteConnection(dbPath);
                newCon.Open();
                SQLiteCommand mCommand = new SQLiteCommand("Insert Into Tasks (Task) Values(@Task)", newCon);
                mCommand.Parameters.Add(new SQLiteParameter("@Task"));
                QueuedTask mTask;
                while (TaskQueue.TryDequeue(out mTask))
                {
                    if (!mTask.InDatabase)
                    {
                        mCommand.Parameters["@Task"].Value = mTask.Task;
                        mCommand.ExecuteNonQuery();
                    }
                }
                mCommand.Dispose();
                newCon.Close();
            }
            catch (Exception ex)
            {
            }
        }
    }
    private async void loadSavedTasks()
    {
        readTimer.Stop();
        await Task.Run(() => loadTasks());
        // this is where you might start a new Timer to execute any tasks sitting
        // in the queue - again using await Task.Run() and to periodically check the
        // queue for new entries after the current queue has been emptied
    }
    private void loadTasks()
    {
        try
        {
            SQLiteConnection newCon = new SQLiteConnection(dbPath);
            newCon.Open();
            SQLiteCommand mCommand = new SQLiteCommand("Select Task From Tasks", newCon);
            SQLiteDataReader mReader = mCommand.ExecuteReader();
            while (mReader.Read())
            {
                TaskQueue.Enqueue(new QueuedTask(Convert.ToString(mReader[0]), true));
            }
            mReader.Close();
            mCommand.Dispose();
            newCon.Close();
        }
        catch (Exception ex)
        {
            LiteLog myerror = new LiteLog("PersistentQueue.cs", "loadTasks", ex.Message);
            myerror.UpdateClass();
        }
    }
}

Not a Porter Stemmer

2016-08-04T10:03:00.000-07:00

If you are considering indexing a sequence of text documents with a view to applying some sort of search facility then there is a lot to be said for stemming.

Stemming is the process of reducing words to their stem or root such that related words are reduced to the same stem. The result may not be the true etymological stem – all stemming algorithms look for is consistency. The benefit is that searches based upon such stems can find references for related words and this reduces the pressure on search users to guess the optimal words or phrases for a given requirement.

Some examples might be:

caresses => caress
ponies => poni
cats => cat
matting => mat
meetings => meet

Perhaps the best known stemming algorithm is the “Porter Stemmer” by Martin Porter. This algorithm was “frozen” in the past and now new developments in such techniques are based upon the newer Snowball stemmer.

C# Snowball* stemmers for a range of human languages can be found on Codeplex with the warning that the code has been ported from Java (a brief review more or less confirmed that). Martin Porter’s stemming site presents his algorithm in a range of languages but again it is clear from the structure that the C# versions are translated.

I took one of the C# Porter Stemmer versions and ran it against the 23,531 word test set and reassured by a perfect score decided to then “translate” the code into something just a little easier to follow and possibly tweak. Now my version (presented below) is not a Porter Stemmer as the output varies from the standard for a handful of words (I just have to mess with things).

I preferred:
advertise, advertised and advertising being stemmed to advert rather than advertis
merchandise becomes merchand and not merchandis
etc.

You probably get the picture – I have treated “ise” the same as “ize” (which works better in GB English I think) but it would not take anyone 5 minutes to undo if you wanted to be a purist but alternately you might fancy tweaking things further.

The code presented here is almost certainly much slower than the original but it is way easier to follow and debug. I have also added a cache so that words previously stemmed can be retrieved without re-processing although I will have to watch cache growth if the process is long running.


/// <summary>
/// AditStemmer is a stemmer algorithm based upon the Porter stemmer algorithm 
/// but has some slight but deliberate differences in output
/// see http://tartarus.org/~martin/PorterStemmer/ for more details and test data
/// This software is completely free for any purpose but is not warrented to be defect free
/// </summary>
class AditStemmer
{
    private Dictionary<string, string> cache = new Dictionary<string, string>(); // could initialise this to avoid known issues (eg universal, university, universe)
    public string stemWord(string word)
    {
        string stemWord;
        if (!cache.TryGetValue(word, out stemWord))
        {
            stemWord = tryStem(word);
            cache.Add(word, stemWord);
        }
        return stemWord;
    }
    private string tryStem(string word)
    {
        if (word.Length > 2)
        {
                // following the steps of giants - step 1 remove plurals, -ed, -ing
                word = step1(word);
                word = step2(word);
                word = step3(word);
                word = step4(word);
                word = step5(word);
                word = step6(word);
        }
        return word;
    }
    private string step1(string word)
    {
        int wordLen = word.Length;
        if (word.EndsWith("s"))
        {
            if (word.EndsWith("sses") || word.EndsWith("ies"))
            {
                wordLen -= 2;
            }
            else if (!word.EndsWith("ss"))
            {
                wordLen--;
            }
            word = word.Substring(0, wordLen);
        }
        if (word.EndsWith("eed"))
        {
            if (countConsonants(word, wordLen - 1) > 0)
            {
                wordLen--;
            }
        }
        else
        {
            if ((word.EndsWith("ed") && hasVowelInStem(word, wordLen - 2)) || (word.EndsWith("ing") && hasVowelInStem(word, wordLen - 3)))
            {
                int checkTo = wordLen - (word.EndsWith("ed") ? 2 : 3);
                string stem = word.Substring(0, checkTo);
                if (stem.EndsWith("at") || stem.EndsWith("bl") || stem.EndsWith("iz") || stem.EndsWith("is"))
                {
                    word = stem + "e";
                }
                else if (isDoubleCon(word, checkTo - 1))
                {
                    if (!(stem.EndsWith("l") || stem.EndsWith("s") || stem.EndsWith("z")))
                    {
                        word = stem.Substring(0, checkTo - 1);
                    }
                    else { word = stem; }
                }
                else if (countConsonants(word, checkTo) == 1 && isConVowCon(word, checkTo - 1))
                {
                    word = stem + "e";
                }
                else
                {
                    word = stem;
                }
                wordLen = word.Length;
            }
        }
        return word.Substring(0, wordLen);
    }
    private string step2(string word)
    {
        // not too exciting just changes y to i if another vowel in the stem - but that comes in useful later
        if (word.EndsWith("y") && hasVowelInStem(word, word.Length - 1))
        {
            word = word.Substring(0, word.Length - 1) + "i";
        }
        return word;
    }
    private string step3(string word)
    {
        if (word.EndsWith("ational")) { return conditionalReplace(word, "ational", "ate"); }
        if (word.EndsWith("tional")) { return conditionalReplace(word, "tional", "tion"); }
        if (word.EndsWith("enci")) { return conditionalReplace(word, "enci", "ence"); }
        if (word.EndsWith("anci")) { return conditionalReplace(word, "anci", "ance"); }
        if (word.EndsWith("izer")) { return conditionalReplace(word, "izer", "ize"); }
        if (word.EndsWith("bli")) { return conditionalReplace(word, "bli", "ble"); }
        if (word.EndsWith("alli")) { return conditionalReplace(word, "alli", "al"); }
        if (word.EndsWith("entli")) { return conditionalReplace(word, "entli", "ent"); }
        if (word.EndsWith("eli")) { return conditionalReplace(word, "eli", "e"); }
        if (word.EndsWith("ousli")) { return conditionalReplace(word, "ousli", "ous"); }
        if (word.EndsWith("ization")) { return conditionalReplace(word, "ization", "ize"); }
        if (word.EndsWith("isation")) { return conditionalReplace(word, "isation", "ize"); }
        if (word.EndsWith("ation")) { return conditionalReplace(word, "ation", "ate"); }
        if (word.EndsWith("ator")) { return conditionalReplace(word, "ator", "ate"); }
        if (word.EndsWith("alism")) { return conditionalReplace(word, "alism", "al"); }
        if (word.EndsWith("iveness")) { return conditionalReplace(word, "iveness", "ive"); }
        if (word.EndsWith("fulness")) { return conditionalReplace(word, "fulness", "ful"); }
        if (word.EndsWith("ousness")) { return conditionalReplace(word, "ousness", "ous"); }
        if (word.EndsWith("aliti")) { return conditionalReplace(word, "aliti", "al"); }
        if (word.EndsWith("iviti")) { return conditionalReplace(word, "iviti", "ive"); }
        if (word.EndsWith("biliti")) { return conditionalReplace(word, "biliti", "ble"); }
        if (word.EndsWith("logi")) { return conditionalReplace(word, "logi", "log"); }
        return word;
    }
    private string step4(string word)
    {
        if (word.EndsWith("icate")) { return conditionalReplace(word, "icate", "ic"); }
        if (word.EndsWith("ative")) { return conditionalReplace(word, "ative", ""); }
        if (word.EndsWith("alize")) { return conditionalReplace(word, "alize", "al"); }
        if (word.EndsWith("alise")) { return conditionalReplace(word, "alise", "al"); }
        if (word.EndsWith("iciti")) { return conditionalReplace(word, "iciti", "ic"); }
        if (word.EndsWith("ical")) { return conditionalReplace(word, "ical", "ic"); }
        if (word.EndsWith("ful")) { return conditionalReplace(word, "ful", ""); }
        if (word.EndsWith("ness")) { return conditionalReplace(word, "ness", ""); }
        return word;
    }
    private string step5(string word)
    {
        int truncAt = word.Length;
        if (word.EndsWith("al")) { truncAt -= 2; }
        if (word.EndsWith("ance") || word.EndsWith("ence")) { truncAt -= 4; }
        if (word.EndsWith("er")) { truncAt -= 2; }
        if (word.EndsWith("ic")) { truncAt -= 2; }
        if (word.EndsWith("able") || word.EndsWith("ible")) { truncAt -= 4; }
        if (word.EndsWith("ant")) { truncAt -= 3; }
        if (word.EndsWith("ement"))
        {
            truncAt -= 5;
        }
        else if (word.EndsWith("ment"))
        {
            truncAt -= 4;
        }
        else if (word.EndsWith("ent"))
        {
            truncAt -= 3;
        }
        if (word.EndsWith("ion") && word.Length > 4)
        {
            if (word.ElementAt(word.Length - 4) == 't' || word.ElementAt(word.Length - 4) == 's')
            {
                truncAt -= 3;
            }
        }
        if (word.EndsWith("ou")) { truncAt -= 2; }
        if (word.EndsWith("ism")) { truncAt -= 3; }
        if (word.EndsWith("ate")) { truncAt -= 3; }
        if (word.EndsWith("iti")) { truncAt -= 3; }
        if (word.EndsWith("ous")) { truncAt -= 3; }
        if (word.EndsWith("ive")) { truncAt -= 3; }
        if (word.EndsWith("ize")) { truncAt -= 3; }
        if (word.EndsWith("ise")) { truncAt -= 3; }
        if (countConsonants(word, truncAt) > 1)
        {
            word = word.Substring(0, truncAt);
        }
        return word;
    }
    private string step6(string word)
    {
        // mostly removes trailing e
        if (word.EndsWith("e"))
        {
            int cCount = countConsonants(word, word.Length - 1);
            if (cCount > 1 || cCount == 1 && !isConVowCon(word, word.Length - 2))
            {
                word = word.Substring(0, word.Length - 1);
            }
        }
        // reduces ll to l where appropriate
        if (word.EndsWith("ll") && countConsonants(word, word.Length - 1) > 1)
        {
            word = word.Substring(0, word.Length - 1);
        }
        return word;
    }
    private string conditionalReplace(string word, string replace, string with)
    {
        // the condition is that there is at least one preceeding consonant sequence
        int stemLength = word.Length - replace.Length;
        if (countConsonants(word, stemLength) > 0)
        {
            word = word.Substring(0, stemLength) + with;
        }
        return word;
    }
    private int countConsonants(string word, int to)
    {
        //counts sequences of consonants before to but after any initial consonant sequence
        int cCount = 0;
        bool wasVowel = false;
        for (var ci = 0; ci < to; ci++)
        {
            if (hasConsonantAt(word, ci))
            {
                if (wasVowel)
                {
                    wasVowel = false;
                    cCount++;
                }
            }
            else
            {
                wasVowel = true;
            }
        }
        return cCount;
    }
    private bool isDoubleCon(string word, int at)
    {
        // is double consonant
        if (at < 1) { return false; }
        if (word.ElementAt(at) != word.ElementAt(at - 1))
        {
            return false;
        }
        return hasConsonantAt(word, at);
    }
    private bool isConVowCon(string word, int to)
    {
        /* is true <=> i-2,i-1,i has the form consonant - vowel - consonant
       and also if the second c is not w,x or y. this is used when trying to
       restore an e at the end of a short word. e.g.
          cav(e), lov(e), hop(e), crim(e), but
          snow, box, tray.
        */
        if (to < 2 || !hasConsonantAt(word, to) || hasConsonantAt(word, to - 1) || !hasConsonantAt(word, to - 2))
        {
            return false;
        }
        if (word.ElementAt(to) == 'w' || word.ElementAt(to) == 'x' || word.ElementAt(to) == 'y')
        {
            return false;
        }
        return true;
    }
    private bool hasVowelInStem(string word, int to)
    {
        for (var vi = 0; vi < to; vi++)
        {
            if (!hasConsonantAt(word, vi))
            {
                return true;
            }
        }
        return false;
    }
    private bool hasConsonantAt(string word, int at)
    {
        switch (word.ElementAt(at))
        {
            case 'a':
            case 'e':
            case 'i':
            case 'o':
            case 'u':
                return false;
            case 'y':
                return (at == 0) ? true : !hasConsonantAt(word, at - 1); // y can act as consonant and vowel
            default:
                return true;
        }
    }
}

*More details of the Snowball algorithm can be found here http://snowball.tartarus.org/

<Edit>Fixed the colour errors in the demo code. The cause (something to do with text size) is under investigation</Edit>

.NET Reflection bug

2016-07-21T07:33:00.001-07:00

Had a fun bug with an app the other day. I had been leaving the app to run for a long time so I could monitor memory usage to check for any possible leaks. But the following bug was reported after a couple of hours:

“target invocation exception” in System.Reflection

at the following line

this.Icon = ((System.Drawing.Icon)(resources.GetObject("$this.Icon")));

but that line was in the form.designer.cs generated by Visual Studio and you would assume that this code was only executed when the form was first rendered.

I was potentially fighting an abstraction then – clearly one that had leaked

Then I remembered that the program updated the task bar icon to alert its user to potentially new and interesting occurrences. I had made sense at the time of writing to make use of the icon stored as a resource associated with the form. The image was retrieved using:

Bitmap b = this.Icon.ToBitmap();

then decorated and set as a revised taskbar icon – which is (of course) also the current form icon which just might give some insight into what was going wrong or at least a faint hint.

The icon file itself was properly formatted and contained 16x16 and 32x32 pixel bitmaps but it was clear that somewhere along the line and after several successful invocations in each instance something went wrong in mscorlib and the debug facility in Visual Studio pinned the problem back on the partial (designer) class for the form.

A couple of solutions came to mind but I went with embedding a 32x32 .png image as an additional assembly resource and retrieving that when it was required. So far, that has yet to fail but yes, there is a small memory leak somewhere so I am not finished yet. Testing and modifying programs designed to run continuously for extended periods within a Windows environment can be a challenge.

Drag and Drop

2016-05-12T06:20:00.000-07:00

Drag and Drop is one of the fundamental building blocks of a modern GUI. It is so fundamental it is easy to overlook the potential for enhancement. There are lots of code fragments about if you look for enhancing the basics but most explanations seem a bit partial – hence this tutorial that I hope covers the wider ground.

Starting with the basic default process and a simple (but perhaps unlikely) example.

Consider a Label with some text that you might want to drag into a text box to edit. The TextBox has the AllowDrop attribute set to true. The drag process starts with a mouse down event on the label so we need some code to start the DragDrop process.


private void label1_MouseDown(object sender, MouseEventArgs e)
{
    DoDragDrop(label1.Text, DragDropEffects.Copy);
}

The DoDragDrop method has been passed the label Text property as the data and the DragDropEffects value of Copy set to indicate the intention of the process. The main values are Copy, Link and Move with additional values of Scroll, None and All (multiple values can be combined with the Or operator) . The data can be supplied as an object and in this instance is a string.

Our target Textbox needs to handle two events, DragEnter and DragDrop.


private void textBox1_DragEnter(object sender, DragEventArgs e)
{
    if (e.Data.GetDataPresent(DataFormats.Text))
    {
        e.Effect = DragDropEffects.Copy;
    }
}
private void textBox1_DragDrop(object sender, DragEventArgs e)
{
    textBox1.Text += e.Data.GetData(DataFormats.Text);
}

That’s all that is required for this basic example. The drag drop process gets the default visual feedback effects and a drop onto the TextBox copies and adds the Label Text to the TextBox Text.

Now let us consider a more complex scenario. I might have multiple objects that could be dragged and dropped on the same page. It could also be true that things might be dragged with multiple potential purposes – I might be implicitly copying the content if I dropped it at one location but simply moving the content about within another screen area. The user needs feedback on the impact of a drop event at different locations – this makes a case for custom cursors. It is also important that the control receiving the “drop” knows the drag source as this might influence that drop action and certainly identifies the unique “content”.

As the DoDragDrop method accepts type “object” as the data source we could rewrite the code from above to use the Label (or any other) control itself:


private void label1_MouseDown(object sender, MouseEventArgs e)
{
    DoDragDrop(label1, DragDropEffects.Copy);
}
private void textBox1_DragEnter(object sender, DragEventArgs e)
{
    if (e.Data.GetDataPresent(label1.GetType()))
    {
        e.Effect = DragDropEffects.Copy;
    }
}
private void textBox1_DragDrop(object sender, DragEventArgs e)
{
    textBox1.Text += ((dynamic)e.Data.GetData(label1.GetType())).Text;
}

but this approach is somewhat limited as it would mean writing specific code for each individual data source.

The DataFormats class has a long list of predefined fields ranging from Bitmap through Text (as we have seen) and includes useful types like XAML. This list of DataFormats is the same list as is used by the Windows Copy/Paste functionality and it is perfectly straightforward to add additional types to that list.

The code below declares a new custom DataFormat and then uses that DataFormat identity to store an instance of a class in a DataObject and then passes that DataObject through the Drag and Drop process to the target control. The code can thus support multiple instances of said class generically.


private DataFormats.Format cardFormat = DataFormats.GetFormat("CCard");
private void label1_MouseDown(object sender, MouseEventArgs e)
{
    CCard mCard = new CCard(); // create a class instance
    mCard.CardTitle = "New Card Title"; // set a property value
    DataObject mDataObject = new DataObject(cardFormat.Name, mCard);
    DoDragDrop(mDataObject, DragDropEffects.Copy);
}
private void textBox1_DragEnter(object sender, DragEventArgs e)
{
    if (e.Data.GetDataPresent(cardFormat.Name))
    {
        e.Effect = DragDropEffects.Copy;
    }
}
private void textBox1_DragDrop(object sender, DragEventArgs e)
{
    CCard cCard = (CCard)e.Data.GetData(cardFormat.Name); // retrieve the class instance
    textBox1.Text += cCard.CardTitle; // and use one or more proprty values
}

In our more complex scenario, each data source type might be represented by a specific class and then each data source instance would simply supply an instance of the relevant class. The DragDrop targets can inspect the DataFormat to decide upon the relevant Effect for that DataFormat and the Drop event can process the given class instance where appropriate.

As that may not be the clearest explanation it would probably help to have some demo code. First though I want to look at custom Cursors.

My custom Cursor creation functions are encapsulated within a utility class


namespace Adit.Classes
{
    public struct IconInfo
    {
        public bool fIcon;
        public int xHotspot;
        public int yHotspot;
        public IntPtr hbmMask;
        public IntPtr hbmColor;
    }
    public static class CursorUtil
    {
        [DllImport("user32.dll")]
        public static extern IntPtr CreateIconIndirect(ref IconInfo icon);
        [DllImport("user32.dll")]
        [return: MarshalAs(UnmanagedType.Bool)]
        public static extern bool GetIconInfo(IntPtr hIcon, ref IconInfo pIconInfo);
        [DllImport("gdi32.dll")]
        public static extern bool DeleteObject(IntPtr handle);
        [DllImport("user32.dll", CharSet = CharSet.Auto)]
        extern static bool DestroyIcon(IntPtr handle);
        public static Cursor CreateCursor(Bitmap bm, int xHotspot, int yHotspot, bool resize = true)
        {
            IntPtr ptr = (resize) ? ((Bitmap)ResizeBitmap(bm, 32, 32)).GetHicon() : bm.GetHicon();
            IconInfo inf = new IconInfo();
            GetIconInfo(ptr, ref inf);
            inf.xHotspot = xHotspot;
            inf.yHotspot = yHotspot;
            inf.fIcon = false;
            IntPtr cursorPtr = CreateIconIndirect(ref inf);
            if (inf.hbmColor != IntPtr.Zero) { DeleteObject(inf.hbmColor); }
            if (inf.hbmMask != IntPtr.Zero) { DeleteObject(inf.hbmMask); }
            if (ptr != IntPtr.Zero) { DestroyIcon(ptr); }
            Cursor c = new Cursor(cursorPtr);
            c.Tag = (resize) ? new Size(32, 32) : bm.Size;
            return c;
        }
        public static Bitmap ResizeBitmap(Image image, int maxWidth, int maxHeight)
        {
            double ratio = System.Math.Min((double)maxHeight / image.Height, (double)maxWidth / image.Width);
            var propWidth = (int)(image.Width * ratio);
            var propHeight = (int)(image.Height * ratio);
            var newImage = new Bitmap(propWidth, propHeight);
            using (var g = Graphics.FromImage(newImage))
            {
                g.DrawImage(image, 0, 0, propWidth, propHeight);
            }
            return newImage;
        }
        public static Bitmap GetControlBitmap(Control c, Color transparent)
        {
            var bm = new Bitmap(c.Width, c.Height);
            c.DrawToBitmap(bm, new Rectangle(0, 0, c.Width, c.Height));
            if (transparent != null)
            {
                bm.MakeTransparent(transparent);
            }
            return bm;
        }
        public static Bitmap OverlayBitmap(Bitmap baseBitmap, Bitmap overlay, Point atPosition)
        {
            using (var g = Graphics.FromImage(baseBitmap))
            {
                g.DrawImage(overlay, new Rectangle(atPosition, overlay.Size));
            }
            return baseBitmap;
        }
    }
}

I have used these functions to create cursors from 96x96 pixel .png files (some originally sourced from the Material Icons download page). I have also used the GetControlBitmap function to create a Cursor from a Bitmap displaying a copy of a control captured at run-time.


private Cursor[] cursors = new Cursor[4];
        //later
cursors[0] = CursorUtil.CreateCursor((Bitmap)imageList1.Images[2], 0, 0);
        //and
cursors[2] = CursorUtil.CreateCursor(CursorUtil.GetControlBitmap(label1, SystemColors.Control), 0, 0, false);

Important thing is, we can easily create custom Cursors from bitmaps ready to apply during our Drag and Drop.

Just one quick aside though, the cursor "hot-spot" does not have to be at position 0,0 as in the above examples. Think of (say) the Paint.NET flood-fill or Color-Picker cursors where the hot-spot is marked by a cross or is at the tip of the "eye dropper". You might want to use something like my OverlayBitmap() function to add a suitable symbol to a control image and set the hot-spot to the relevant position in the final image.

OK - one other aside. My cursor building function adds the image size to the Tag property. You might wonder why. If you create a custom cursor (say) from a control image you might be surprised to find that the resulting cursor Size attribute will always return 32,32 (or just possibly 64,64) and not the actual size of the cursor. You can access (say) the true Width of a cursor via the Tag property with code something like:


int cWidth = ((dynamic)myCursor.Tag).Width;

We can apply our custom cursors in a GiveFeedback event handler. so first we might create custom Cursors for Copy and None and then apply them in a function


private Cursor myCopyCursor, myNoneCursor;
    myCopyCursor = CursorUtil.CreateCursor((Bitmap)imageList1.Images[2], 0, 0);
    myNoneCursor = CursorUtil.CreateCursor((Bitmap)imageList1.Images[3], 0, 0);
private void giveFeedback(object sender, GiveFeedbackEventArgs e)
{
    if(e.Effect == DragDropEffects.Copy)
    {
        e.UseDefaultCursors = false;
        Cursor.Current = myCopyCursor;
    }
    else if(e.Effect == DragDropEffects.None)
    {
        e.UseDefaultCursors = false;
        Cursor.Current = myNoneCursor;
    }
}

We add the giveFeedback() function to the GiveFeedback event of the Form - perhaps during the form Load()


this.GiveFeedback += giveFeedback;

and then slightly counter-intuitively set the AllowDrop property to true for the form and all controls over which you anticipate the user might drag. This does not allow a drop to occur at these sites (that requires code to handle DragEnter and DragDrop but it does allow any relevant custom Cursor to be displayed during any drag over the form and control surfaces.

A point worth noting is that DragDrop event arguments include X/Y coordinates and these are screen co-ordinates as drags are not restricted to the originating window. Drag/Drop is a key element of OLE (Object Linking and Embedding). This can make using those conveniently presented values slightly tricky when managing things like local scrolling under program control. Which brings me nicely to my demo application (zipped as a Visual Studio 2015 project ready to download) that manages just that.

First off, this is demo and not production code and thus likely to include bugs, ignore edge cases, certainly takes shortcuts, is unduly verbose in places and might totally fail to explain some key point you are interested in. However it might provide a platform that can be used to try out alternate scenarios and approaches without having to build the whole thing from scratch.

Here you can see a screen image from the demo program. Addresses can be entered in the mini form on the left of the window and dragged onto the FlowLayoutPanel on the right that is acting as an "address store". To save energy, the "Fill" button will populate the form with up to 50 addresses taken randomly from an embedded list.

Addresses in the right hand panel can be dragged back to the address entry form for editing. Addresses can also be dragged up and down the list to reorder them.

We thus have two sorts of object that can be dragged for (is it) three different purposes. Some simple custom icons are created and used during the drag events.

The demo features loading data from an embedded resource (.csv file) using the Visual Basic TextFieldParser class (don't panic this is C# demo code). There are two custom DataFormats declared and used in the Drag operations. The addresses in the notional "address store" are encapsulated within multiple UserControls and these controls use Delegates to communicate with the Form. Dragging an address within the FlowLayoutPanel will trigger an automatic scroll when the scroll bar is active and the drag nears the top or bottom of the panel..

Exercises for further study might include using an image of an AddressControl as a cursor and/or dragging new addresses to a specific location in the FlowLayoutPanel instead of just adding them at the bottom irrespective of the drop location.

An analysis of the code will note that the demo "cheats" when determining the target location of a move within the FlowLayoutPanel and that it is possible to dodge around one or more address and expose the bug.

Inspect or download the demo code at https://github.com/MikeGWem/DragDropDemo

If anyone knows or finds out what DragDropEffects.Scroll actually does then please let me know as so far the answer to that has eluded me.

Addendum:

The "bug fix" (to decide where a drop happens relative to the existing list of addresses in the "store" could be based upon a calculation performed at the same time as the test to see if the panel should be scrolled.


private void maybeScrollpanel(int dragY)
{
    if ((dragY - panelTop) <= flowLayoutPanel1.VerticalScroll.Value && flowLayoutPanel1.VerticalScroll.Value > 0)
    {
        flowLayoutPanel1.VerticalScroll.Value -= (flowLayoutPanel1.VerticalScroll.Value > scrollAt) ? scrollAt : flowLayoutPanel1.VerticalScroll.Value;
    }
    else if (dragY - panelTop >= flowLayoutPanel1.Height - scrollAt && flowLayoutPanel1.VerticalScroll.Maximum > 0)
    {
        flowLayoutPanel1.VerticalScroll.Value += (flowLayoutPanel1.VerticalScroll.Maximum - flowLayoutPanel1.VerticalScroll.Value > scrollAt) ? scrollAt : flowLayoutPanel1.VerticalScroll.Maximum - flowLayoutPanel1.VerticalScroll.Value;
    }
    calculatedPosition = calculatePosition(dragY - panelTop);
}
private int calculatePosition(int dragY)
{
    if (flowLayoutPanel1.Controls.Count == 1 || dragY < flowLayoutPanel1.Controls[1].Height) // || allows lazy evaluation
    {
         return 1;
    }
    Point pt = new Point(flowLayoutPanel1.Width / 2, dragY); // asumes single column of controls in panel
    Control cAt = flowLayoutPanel1.GetChildAtPoint(pt);
    while (cAt == null && pt.Y < (flowLayoutPanel1.Height + flowLayoutPanel1.VerticalScroll.Maximum))
    {
        pt.Y += 5; // arbitrary - adjust at will
        cAt = flowLayoutPanel1.GetChildAtPoint(pt);
    }
    return (cAt != null) ? ((AddressControl)cAt).Sequence : (flowLayoutPanel1.Controls.OfType<AddressControl>().Max(a => a.Sequence) + 1);
}

Then applied to the drop events like


private void flowLayoutPanel1_DragDrop(object sender, DragEventArgs e)
{
    // the two paths could share more of the code but...
    if (e.Data.GetDataPresent(dragAddress.Name))
    {
        //handles drop of new address onto the panel
        AddressControl newControl = new AddressControl((DragAddress)e.Data.GetData(dragAddress.Name), DoControlDrag, DraggedPast, flowLayoutPanel1.Controls.Count);
        foreach(AddressControl ac in flowLayoutPanel1.Controls.OfType<AddressControl>().Where(a => a.Sequence >= calculatedPosition))
        {
            ac.Sequence++;
        }
        newControl.Sequence = calculatedPosition;
        flowLayoutPanel1.Controls.Add(newControl);
        reorderAddresses();
        resetPnlEdit();
    } else if (e.Data.GetDataPresent(addressControl.Name))
    {
        // handles drop of an existing AddressControl into (presumably) a new position
        int newSeq = 1;
        foreach (AddressControl ac in flowLayoutPanel1.Controls.OfType<AddressControl>().OrderBy(a => a.Sequence))
        {
            if(ac.Sequence == dragItemSequence)
            {
                ac.Sequence = calculatedPosition;
            } else if(ac.Sequence == calculatedPosition)
            {
                ac.Sequence++;
                newSeq = ac.Sequence;
            } else
            {
                ac.Sequence = newSeq;
            }
            newSeq++;
        }
        newSeq = 1;
        // is redoing the sequence to avoid breaks a bit too anal?
        foreach (AddressControl ac in flowLayoutPanel1.Controls.OfType<AddressControl>().OrderBy(a => a.Sequence))
        {
            ac.Sequence = newSeq;
            newSeq++;
        }
        reorderAddresses();
    }
}

For the rest of the code check out GitHub at https://github.com/MikeGWem/DragDropDemo

End Note:
None of this looks like it will work with UWP and .NET Core so I will try and write up a few notes on that once I have something working that looks usable for others.

A textbox that auto-wraps to fit contents

2016-04-22T06:35:00.000-07:00

The Windows Forms Textbox control is a thin wrapper for the underlying Win32 control and there is no Paint event you can hang extra code on in the normal way. If you write something like mycontrol.Paint += myFunction; and mycontrol is a Textbox them myFunction() will never get called.

You can go the whole hog and


SetStyle(ControlStyles.UserPaint, true);

and provide a Paint method but you have to do everything as it is no good calling the base.OnPaint(). Give it a try, and you will end up with an astonishingly unresponsive textbox.

I wanted to create a version of a Textbox that would resize itself vertically to accommodate extended word wrapped text. That did not result in any issues – all it needed was a FitToContents() method that retained the set width and allowed the height to vary to fit the control content.


protected virtual void FitToContents()
{
    Size size = this.GetPreferredSize(new Size(this.Width, 0));
    if (multiLine) { this.Height = size.Height; }
}

However I also wanted to be able to switch between a label displaying a given string and a textbox capable of editing that string. Just to make that interesting I wanted a “fade in” and “fade out” animation to switch between the two. While I was at it, I also wanted to support a textbox “Placeholder” facility to keep the UI nice and clean. It was that final element that reminded me that it was possible to slide a limited paint facility into position to be used in place of the controls own paint when the control was relatively inactive. My own paint facility only needed to manage the fades and the placeholder text display – all the rest could be left to the underlying control.

Fading text in and out was just a matter of tweaking the Alpha component of the control ForeColor in steps over a defined time period. It was a good opportunity to use the recently introduced async/await functionality although a timer would have worked just fine. The control border is unreachable to all intents and purposes although if the control was set borderless one could be drawn around it and arrangements made to fade that rectangle in and out.

OK – I know - WPF and all that, but I also have this funny feeling that as soon as I invest any real time in WPF code Microsoft are going to announce a newer superer duperer common base for Windows apps and it will all be to no avail. Don’t take advice from me though – anything could happen.

The full code for this class can be found at the bottom of this post – usual caveats.

In case anyone fancies just using the placeholder functionality I have generously added an attribute (AutoMultiLine) that can be set false to stop the control re-sizing in response to text longer than the control width. Just set the PlaceHolderText attribute at design or run time. You can always ignore or remove the code associated with the fade.

The control based upon Label with a matching fade facility is very similar but simpler and again the code can be found below. This label also supports automatic height resizing to match the Text.

Please note that these controls are using a language feature from .NET v4.5 so you would need to switch to using a timer to make use of the fade functionality with earlier versions.

Combining these two custom controls into a UserControl made sense as they could then be addressed together as a single entity. This did have its challenges – partly because the TextBox BackColor property can’t be set Transparent and that makes the fade through a bit clunky in one direction – perhaps I should have gone for a “wipe” effect instead…


namespace Adit.Classes.Controls
{
    [ToolboxBitmap(typeof(TextBox))]
    class MultiLineTextBox : TextBox
    {
        #region Private Declarations
        private Color foreColour;
        private Color placeholderColour = Color.Gray;
        private int opacity = 256;
        private int fadeSteps = 10;
        private int fadeTime = 750;
        private string placeholderText = "Type here";
        private bool showPlaceholderText = false, multiLine = true;
        #endregion
        public MultiLineTextBox()
        {
            this.Multiline = multiLine;
            this.WordWrap = multiLine;
        }
        #region Design attributes
        [Description("Fade time in milliseconds"), Category("Behavior")]
        public int FadeTime
        {
            get { return fadeTime; }
            set { fadeTime = value; }
        }
        [Description("Number of steps from transparent 0 to opaque 256"), Category("Behavior")]
        public int FadeSteps
        {
            get { return fadeSteps; }
            set { fadeSteps = value; }
        }
        [Description("Placeholder Text"), Category("Appearance")]
        public string PlaceHolderText
        {
            get { return placeholderText; }
            set { placeholderText = value; Invalidate(); }
        }
        [Description("Placeholder Colour"), Category("Appearance")]
        public Color PlaceHolderColour
        {
            get { return placeholderColour; }
            set { placeholderColour = value; }
        }
        [Description("Automatic switch to multiline"), Category("Appearance")]
        public bool AutoMultiLine
        {
            get { return multiLine; }
            set {
                multiLine = value;
                Multiline = value;
                WordWrap = value;
            }
        }
        #endregion
        #region Public methods
        public async void FadeInOut()
        {
            bool saveUserPaint = GetStyle(ControlStyles.UserPaint);
            foreColour = ForeColor;
            int opStep = 256 / fadeSteps;
            if (Visible)
            {
                opacity = 256;
                opStep *= -1;
            } else
            {
                Visible = true;
                opacity = 0;
            }
            SetStyle(ControlStyles.UserPaint, true); // set after any Visibility change
            opacity = opacity + opStep;
            while(opacity > 0 && opacity < 256)
            {
                foreColour = fadeColour(opacity, foreColour);
                placeholderColour = fadeColour(opacity, placeholderColour);
                Invalidate();
                await Task.Delay(fadeTime / fadeSteps);
                opacity = opacity + opStep;
            }
            Visible = (opacity >= 255);
            SetStyle(ControlStyles.UserPaint, saveUserPaint);
            if (Visible)
            {
                Invalidate();
                Focus();
            }
        }
        #endregion
        #region Override control methods
        protected override void OnResize(EventArgs e)
        {
            base.OnResize(e);
            this.FitToContents();
        }
        protected override void OnKeyUp(KeyEventArgs e)
        {
            base.OnKeyUp(e);
            FitToContents();
        }
        protected override void OnTextChanged(EventArgs e)
        {
            base.OnTextChanged(e);
            placeholderToggle();
            FitToContents();
        }
        protected override void OnCreateControl()
        {
            base.OnCreateControl();
            placeholderToggle();
        }
        protected override void OnPaint(PaintEventArgs e)
        {
            using (var drawBrush = new SolidBrush((showPlaceholderText)? placeholderColour: foreColour))
            {
                e.Graphics.DrawString((showPlaceholderText) ? placeholderText : Text, Font, drawBrush, this.ClientRectangle);
                // The underlying control is probably using TextRenderer.DrawText (gdi not gdi+)
            }
        }
        protected virtual void FitToContents()
        {
            Size size = this.GetPreferredSize(new Size(this.Width, 0));
            if (multiLine) { this.Height = size.Height; }
        }
        #endregion
        #region Private methods
        private Color fadeColour(int opacity, Color argbColour)
        {
            return Color.FromArgb(opacity, argbColour);
        }
        private void placeholderToggle()
        {
            showPlaceholderText = (Text.Length > 0) ? false : true;
            SetStyle(ControlStyles.UserPaint, showPlaceholderText);
        }
        #endregion
    }
}

and


namespace CheckBuilder.Classes.Controls
{
    public partial class WrapLabel : Label
    {
        #region Private Declarations
        private Color foreColour;
        private int opacity = 256;
        private int fadeSteps = 10;
        private int fadeTime = 750;
        private bool fading = false;
        #endregion
        public WrapLabel()
        {
            base.AutoSize = false;
        }
        #region Public methods
        public async void FadeInOut()
        {
            fading = true;
            foreColour = ForeColor;
            int opStep = 256 / fadeSteps;
            if (Visible)
            {
                opacity = 256;
                opStep *= -1;
            }
            else
            {
                opacity = 0;
                Visible = true;
            }
            opacity = opacity + opStep;
            while (opacity > 0 && opacity < 256)
            {
                Invalidate();
                await Task.Delay(fadeTime / fadeSteps);
                opacity = opacity + opStep;
            }
            Visible = (opacity >= 255);
            fading = false;
        }
        #endregion
        #region Design attributes
        [Description("Fade time in milliseconds"), Category("Behavior")]
        public int FadeTime
        {
            get { return fadeTime; }
            set { fadeTime = value; }
        }
        [Description("Number of steps from transparent 0 to opaque 256"), Category("Behavior")]
        public int FadeSteps
        {
            get { return fadeSteps; }
            set { fadeSteps = value; }
        }
        #endregion
        #region Override control events
        protected override void OnPaint(PaintEventArgs pe)
        {
            if(fading)
            {
                using (var drawBrush = new SolidBrush(fadeColour(opacity, foreColour)))
                {
                    pe.Graphics.DrawString(Text, Font, drawBrush, ClientRectangle);
                }
            } else
            {
                base.OnPaint(pe);
            }
        }
        protected override void OnResize(EventArgs e)
        {
            base.OnResize(e);
            this.FitToContents();
        }
        protected override void OnTextChanged(EventArgs e)
        {
            base.OnTextChanged(e);
            this.FitToContents();
        }
        protected virtual void FitToContents()
        {
            Size size = this.GetPreferredSize(new Size(this.Width, 0));
            this.Height = size.Height;
        }
        protected override void OnCreateControl()
        {
            base.OnCreateControl();
            this.AutoSize = false;
        }
        #endregion
        #region Stomp on AutoSize
        [DefaultValue(false), Browsable(false), EditorBrowsable(EditorBrowsableState.Never), DesignerSerializationVisibility(DesignerSerializationVisibility.Hidden)]
        public override bool AutoSize
        {
            get { return base.AutoSize; }
            set { base.AutoSize = value; }
        }
        #endregion
        #region Private methods
        private Color fadeColour(int opacity, Color argbColour)
        {
            return Color.FromArgb(opacity, argbColour);
        }
        #endregion
    }
}

Structure C# like JavaScript

2016-04-19T12:00:00.000-07:00

Been building a “proof of concept” Windows program that ended up including 9 different user controls, 4 Custom controls and some (gasp) printed output.

My custom controls included:

A Textbox with the equivalent of the HTML Textbox “placeholder”.
A custom Checkbox to emulate a material Design checkbox.
A Label control to automatically support word wrapped text – like HTML.
A Panel with rounded corners (which also required a couple of extensions to System.Drawing.Graphics to support drawing and filling the bounding round cornered “box”**).

There was the makings of a “rant” here; taking Microsoft to task on the lack of updates to the basic controls to meet modern requirements. I stifled it when I realised that rather than providing solutions matching what would, after all, be ever shifting changes in design “fashion” they had simply provided the tools necessary to remedy the situation. If you count yourself as a programmer then you have to accept the challenge to one’s design skills – mine do lack I admit.

What is the true difference between a Custom Control and a User Control?

This is a question I have seen asked in many places and so far all the answers I have seen have been “lies for children*” (simplifications that attempt to help someone make the right choice).

The key difference is that a User Control inherits from UserControl and a Custom control inherits from Control. That’s about it. You can add other controls to either and both will turn up in the visual designer “toolbox” in Visual Studio after a “build”. They can both place custom attributes and events into the control properties window as well as take advantage of properties inherited from their base types (these can also be suppressed if required).

Generally, you would choose to create a Custom Control if it is intended as a single control rather than a defined collection. (However there is no reason why a custom control might not act as a custom container for other Windows controls added, perhaps, at design time – like my panel with rounded corners). You would also select this type if you intend to manage drawing the control and also if you intend to subclass an existing control type.

User controls are a good choice if you are going to build a control using two or more pre-existing controls. This approach effectively provides a local name space and would normally provide code to handle individual sub-component events. User controls are better used when adding additional visual attributes during the paint event rather than managing the whole drawing requirement.

I think that my custom built control base type selection would normally be founded upon usage. “User Controls” being particular to a given application while “Custom Controls” might well be ported to other projects. Which probably does not help at all.

Why all those User Controls?

Well the program required repetitive collections of controls arranged as cards, subsections of cards and subsections of those subsections – all at the whim of the user. In addition, the program provides a preview pane to show how the output would be presented with support for interactive testing on the fly. At the design stage I could see that this had quite a lot of potential to get messy built using a conventional Windows Forms approach. I knew that if I built something similar in JavaScript then I would end up with a set of objects that managed their portion of the UI and dealt with the relevant events. Any required communication would be through callbacks to functions. The C# equivalent then became clear – a set of UserControls communication between themselves using Delegates. In fact some of the presentation side of the program had already been built using JavaScript and it was interesting to note the strong similarities between some of the C# and JavaScript code blocks.

That was how it worked out – a varying number of custom UserControls communicating with each other in the main but with the owning form being called upon to manage “global” functions like database saves. I can only estimate the savings in code lines as “substantial”.

Printing controls

Providing a print output started out feeling a bit “retro” as the most obvious way to accomplish the task was to print what was effectively an image of the program output preview pane (or at least the contents). I could just about recall that the Visual Basic 3 manual(s)*** had included some functionality to print a window content but had never actually done any such thing in all my years of code.

Establishing the basic mechanism proved simple enough as any control has the capacity to draw itself to a bitmap and the resulting bitmaps can easily be drawn to the printed page (scaling as required).

Worth noting that when doing this it is easy to add a drop shadow by extending the size of the bitmap by a few pixels. You can then draw some lines in a sequence of grey shades to create the shadow effect.


private Bitmap drawControlsImage(Control control, bool addShadow = false)
{
    Bitmap bm = new Bitmap(control.Width, control.Height + ((addShadow) ? 3 : 0));
    control.DrawToBitmap(bm, new Rectangle(0, 0, control.Width, control.Height));
    if(dropShadow)
    {
        Point pt = new Point(0, bm.Height - 3);
        using (Graphics g = Graphics.FromImage(bm))
        {
            using (var pen = new Pen(shadow[0]))
            {
                for (var sp = 0; sp < 3; sp++)
                {
                    pen.Color = shadow[sp];
                    g.DrawLine(pen, pt.X, pt.Y, pt.X + bm.Width - 2, pt.Y);
                    pt.Y++;
                }
            }
        }
    }
    return bm;
}

having previously coded


private Color[] shadow = new Color[3];
        and
shadow[0] = Color.FromArgb(181, 181, 181);
shadow[1] = Color.FromArgb(195, 195, 195);
shadow[2] = Color.FromArgb(211, 211, 211);

My printer output took the form of one or more major components (User Controls) emulating a Material design “card”. Finding the best way to order the cards in columns to optimise the printed layout looked set to be an interesting problem. It was analogous to the 2D rectangle bin packing challenge but modified in that there was at least an implicit order in the cards plus the target rectangle could be proportionately increased in size (at least conceptually) by applying ScaleTransform() to the output graphics surface and thus (negative) zoom. While thinking of the best approach I also thought about the issue as a “flow layout” problem – with the option to resize the logical bounding rectangle until a fit was achieved.

After a happy hour reading around the potential algorithms I realised that my requirement was very much a special case – a simple stacking exercise.

Which at first sight produced an acceptable result – but a little thought turned up the most obvious fault. Early positioning of objects in a column could undermine later reductions in the overall width.

What was needed was a two pass process with the first pass constrained to not building a column “higher” than the measured page “vertical” space. Second pass could then work as before towards the smallest packed size. So far, testing shows that the two pass method produces a better result and (on most occasions) a nicely balanced layout.

It amused me to note that I was quite happy to write the first draft of my stacking algorithm based upon a two dimensional array of SizeF (SizeF[,]) but when I came to re-write it to include two passes I changed the structure to List<List<SizeF>> (effectively a sparse array with inbuilt equivalents of Push() and Pop()) and achieved some code reduction as a consequence.

* Lies-to-children have a scholarly root.

** Graphics extensions: You might like to take a look at this Code Project post by Arun Zaheeruddin which provides a fully overloaded set of rounded rectangle drawing functions that look enticing.

*** The VB3 manual set (included in the box) was probably the pinnacle of written language documentation. It was clearly intended to empower a generation of new Windows programmers – as indeed it probably did. We have come to accept that Google searches and Stack Overflow now provide detailed technical documentation for most things but this well written, detailed and accurate set in three (as I recall) volumes filled in the gap for a vast army until the World Wide Web was invented.

Dynamic C#

2016-04-11T05:39:00.001-07:00

Just used the dynamic key word for the first time in C# - to get the compiler to relax a bit and act more like VB which allows late binding.

I was using a control Tag to store a couple of data items and as Tag accepts an object decided to use an Anonymous Object rather than define another class or whatnot. Something like:

aControl.Tag = new { Sequence = ItemSequence, id = ItemID};

But how to consume those object attributes in another function?

The IDE and compiler are not going to take kindly to anything like aControl.Tag.Sequence as that can only make sense at runtime.

The dynamic keyword comes to the rescue.

Here I am looping through a control collection filtered by my specific control type and in the order of the Sequence attribute in the anonymous object stored in the control Tag:

foreach(MyCheckbox mb in this.Controls.OfType().OrderBy(p => ((dynamic)p.Tag).Sequence))
            { … }

Also (after prompting by Visual Studio) used the following syntax to check if a reference to a delegate is null and execute it if it is valid:

notifyChange?.Invoke(ids);

You can add a conditional call to a delegate to a standard control event. For example:

myControl.SizeChanged += (object s, EventArgs e) => { checkSize?.Invoke(); };

The “is” keyword also came in handy with the ability to say something like:

if (myobj is CardSubsection){}

and do something if myobj can be safely cast to the relevant class. “is” makes it way simpler (and produces clearer code) when passing one of a range of classes into a function via an object reference.

On the subject of clarity, I came across a little bit of code like this in a blog:

if (pa.TicksToGo-- > 0) {}

which is shorter than


pa.TicksToGo--; // or maybe pa.TicksToGo -=1;

if (pa.TicksToGo > 0) {}

but less easy to maintain at some future date. I know that JSLint would like to persuade you that -– and ++ are evil in and of themselves but I am not convinced – safe to use with due caution I think.

Looks like C# is going to “keep on giving” with the news that the dev teams for C# and VB are going to let the two languages drift further apart. Seems that developers using VB favour stability (presumably finding multiple language versions bothersome within their organisations) and the C# types are gluttons for novelty. Should be interesting.

Read more here: http://www.infoworld.com/article/3051066/application-development/microsoft-c-visual-basic-are-now-set-to-diverge.html

Material Design Lite

2016-03-20T09:55:00.004-07:00

A short and incomplete review

Material Design Lite (MDL) is a Google project intended to deliver a “Material design” look and feel to websites. This HTML, CSS and JavaScript project is a work in progress so shortcomings experienced might well be short lived so please bear in mind that this limited review was based upon version 1.1.2.

The project FAQ makes the following point. “We optimise for websites heavy on content, such as marketing pages, articles, blogs and general web content that isn’t particularly app-y.” However the supplied layout components respond well to variations in screen size and thus provide a solid basis for web pages targeting smartphones alongside desktop browsers. Where things are a bit clunky at the moment is in the area of dynamic page changes after the initial load and it may be that this will prove to be an enduring issue as (after all) that could be considered somewhat app-y.

The layout components provide a good range of page formats that can include navigation, a drawer, tabs, a straightforward and responsive grid and a footer. There is inbuilt JavaScript support for most of the layout options – you just set out your mark-up as directed and you are good to go. Colour choice is deliberately restrained but you can select from a range of themes to build a ‘custom’ CSS file.

Other components include a good variety of buttons, excellent cards, lists, menus, sliders, toggles and text fields. There is a dialog component but this is limited by only partial browser support although there is a polyfill available. The only obviously lacking component is a <select> but I found a CSS only solution that filled the gap nicely.

There is full support for the Google Material Design Icons although for MDL I only used the icon font having previously used the SVG and CSS sprite versions on other projects. As expected the font more than met my requirements.

I have a fondness for the consistency and flexibility of jQuery when it comes to DOM manipulation (however untrendy) and can confirm that jQuery works happily with MDL as long as you remember that jQuery objects are a collection of elements and that when calling an MDL function you may need to pass the first (or selected) element of the relevant collection:

Which brings me nicely to the app-y bits.

I first drafted out a reasonably complex static page to get a feel for the components and how they are laid out. The results were pleasing with just enough animation to give the design some life. Next I set up a skeleton MDL layout and attempted to build the same page content from JSON using JavaScript in the browser. Once the DOM has been updated with the new elements then you must call the componentHandler.upgradeElement() utility to apply the MDL layout magic to the new DOM elements. This worked very well for the set of cards I inserted together with their <div>s, text and checkboxes.

I then started to build the layout for a “visual” form constructor and started to run into some limitations. I envisaged my form constructor being based upon multiple tabs with the user being able to add a new tab to start a new ‘form’. I was able to inject the same structures into the DOM as would be required to initialise an MDL tabbed page but calling the upgradeElement() utility only gave me a partial result lacking in any MDL script support.

Currently most of the MDL documentation consists of asking questions on Stack Overflow.

There I learned about the downgradeElements() utility (although it does get a mention on the MDL web pages) that should be applied to a container that has already been sprinkled with the MDL fairy dust prior to calling the upgradeElement() function to rebuild things with any new inclusions. Indeed user HR_117 has kindly supplied a CodePen demo of this being applied to a set of MDL tabs. Trouble is, I found this failed when the tab container was itself wrapped within an MDL header (reported as an improbable bug so the true cause was masked in some way). Could very well be that I missed a trick here but how many hours can you spend on just one issue? My workaround (horror) was to create a bunch of tabs up front and to hide them on page load until they were required (a remarkably low overhead in reality as I was able to recycle previously used and subsequently closed tabs).

I hit similar problems with the clever “Text with Floating Label” component when injected into a pre-existing layout. After a short losing battle (quite possibly my misunderstanding or whatever) I switched to using this http://clubdesign.github.io/floatlabels.js/ alternative jQuery plugin that was a very nice substitute well within the style and spirit of Material Design.

In the end I got a functional page although many interactive elements of the layout were untidy. I knew I was in for a long round of CSS tweaking to get things polished but then again I have hit similar walls when working with Bootstrap. It was one of the reasons I had my first dalliance with Polymer where, by default, you can avoid the intrusion of inherited CSS – and the overhead of tracking down and fixing localised layout glitches. So maybe I should have a bash at mixing Polymer components into an MDL ‘frame’ – now that sounds like a winner and indeed should be very doable. A definite if I get to re-visit this specific task.

I am convinced enough by MDL as it stands to be sure of using it for aspects a commercial project currently in development. If you can avoid bumping into the limitations (and they will fast retreat I am sure – or maybe future documentation will show me, at least, the error of my ways) then it is very effective and can support a responsive web page that is bang up to date in design terms.

The experiment with MDL was a nice opportunity to exercise my JavaScript muscles after a bit of a lengthy break where C#, Java and even a touch of VB.NET had been the required tools. JavaScript is always a delight to get back to, as I find it a very satisfying language to work with. You can take liberties that other languages could not support but in the end I always feel a certain obligation to refactor code until it is fit for public display – even though I then feed it through Closure * to optimise browser download and execution. I found the (reasonably) recent addition of the Array.forEach() functionality (Mozilla polyfill available) very effective in reducing code complexity and enhancing readability – looks like ECMA 5 is creeping into my code. Array.find() is also another great – er – find.

* I do hope that Closure is extended to be a compiler for WebAssembly when that gets support across the browser range (looking good so far).

Addendum:

In many ways it makes more sense to implement the (rather restricted) forms designer app-y page that highlighted the MDL issues above as an app. Mobile development platforms are obviously going to be happy with the design concepts but how about good old Windows?

There is a Material Design skin project on GitHub addressing Windows forms projects although it has been a while since there were any updates. There is also a WPF focused project that looks a lot more complete and is certainly active - maybe attention has drifted from the former to the latter.

A little experiment showed that some Material Design styling can be added to a pretty basic Windows Forms app by overriding the Paint() events on some of the key components. Turning a panel into a card, for instance, does not require a lot of effort.

So, for Windows, a bit of mix and match should do the trick...

Kickstart a Graphics Book

2016-01-06T02:28:00.000-08:00

Why you should fund this graphics Book

https://www.kickstarter.com/projects/2002981747/net-graphics-programming-omnibus/description

You are primarily a .NET developer and/or a regular user of C#, VB or even F# but do not encounter graphics as a regular requirement within your projects. You are now wondering why I am promoting a book about .NET graphics to you in particular. There are two reasons.

The first is just that when you do need to include some graphical techniques into some development task then you will need a book that can jumpstart your skills acquisition and/or provide just the code you need for an unexpected task.

The second is that keeping up with developments in User Interface (UI) techniques is going to make increasing demands upon graphical skills. The UI will advance faster than the Windows API (or under surrogate APIs like WPF) and users, product managers and designers are going to be making some tough demands upon all developers with code running in the “human domain” (shall we call it?).

Now you could prepare for this upcoming eventuality by brushing up on the maths involved - maybe starting here http://codeofthedamned.com/index.php/introduction-to-the-math-of - but it would be way easier to let someone else tease out the crucial bits and present the results alongside their immediate sample application.

Why this specific Book? The answer to that is very straightforward. The last time Rod wrote a book about graphics it was brilliant - and I have some authority here. When I first started Windows programming I chose a development area that was entirely graphical (an early mapping application). This was before the Internet and the vast resources that has exposed - then if you wanted to know how something was done then you had to sit and work it out for yourself. I had an early copy of Charles Petzold’s master work (Programming Windows) to lookup the (Windows 3.1.) API calls and, in extremis, access to Microsoft developer support via CompuServe [https://en.m.wikipedia.org/wiki/CompuServe ] (yup you could email the guys that actually wrote the stuff in those dim and distant days and if your enquiery was interesting enough they would reply). Rod had not yet written his book and I struggled to learn and to develop the techniques required to produce even moderately efficient graphics code.

Some time later Rod’s book was published and I grabbed a copy - partly as personal affirmation but also to learn just how often I had done it all wrong. Here was a definitive work that you could dip into as required and I found that thereafter I did, saving time in recalling (or discovering) the optimal path every time I needed to. As .NET took over as the best development framework then I found I was using the book rather less. I knew that Rod had an updated opus in mind but sadly he could not convince a publisher that a market of the scale they required was waiting for a new book to be written.

Here is our opportunity to fund a book of undoubted value for all - a definite plus on anyone's electronic bookshelf. Please join me in funding this Kickstarter. It will benefit the community of .NET developers as well as each individual “investor”.

Start here https://www.kickstarter.com/projects/2002981747/net-graphics-programming-omnibus/description and then please encourage others to join in and make sure this book becomes a reality.

Addendum:

Sadly this Kickstarter has been withdrawn.

Bare Minimum Software

2015-12-07T09:19:00.002-08:00

With the financial year end fast approaching it is time to consider any kit purchases in anticipation of activities during 2016 (and I don’t get that list right very often in truth). A laptop upgrade to something smaller and lighter looks a good idea – particularly with one of the teenagers having a dead machine and thus more than grateful for my cast off (actually not a bad spec - certainly going to outperform most current low end offerings).

Buying hardware is just a few clicks in a web browser these days but the software component list promises a long session just running installs.

What would be on your list for a basic set-up?

I started with:

Chrome browser
FireFox browser
Visual Studio 2015
Android Studio
SQL Server
Paint.NET
Notepad++
SourceGear diff/merge tool
SQLiteBrowser
Bruce Schneier ‘s PasswordSafe
Hex Editor
IPScanner
Dropbox
LinQPad
Putty

Plus some homemade tools and utilities.

Then there is the question of MS Office – will Gmail’s ability to display (say) Word documents fill in enough on incoming attachments to get by?

Something to read PDF’s I suppose.

Any suggested additions?

Factor in a few system updates to add to the confusion and slow package download speeds and this already looks like a day’s work

Edit:

Cracked and added a copy of MS Office. With the release of Office 2016 there are a lot of bargains around - particularly with earlier versions. In fact by a series of stock shortages I ended up with a copy of 2016 Pro for the price of a 2013 Home & Office on the residual market.

Raspberry Pi music

2015-12-01T10:00:00.001-08:00

Somewhat off topic.

I am the very happy owner of a five year old Brennan JB7 digital music system which includes a couple of their speakers sitting on some carefully positioned stands. Recently Brennan sent me an email to introduce the new “B2” model – and very nice it looks but I wondered if the price at £579.00 for the top model was strictly warranted (with all due respect for the undoubted product quality).

As the new device seemed to be based upon a Raspberry Pi I ran through a few numbers for building something equivalent:

1tb USB drive – retails at around £40
Raspberry Pi 2 – around £30 (often less)
Micro sd card - £5
Amplifier - £50
Power supplies- £20
CD drive - £8
Sundries (cables, case etc.) - £25

The Brennan box also has digital SPDIF and Line Out plus input options (all of which add a little to their costs) but I can’t see me making use of those in these minimalist times. I would just like to add digital radio and streaming to a disk based music playback capability.

In going through my shopping list I started to get enthusiastic and wondered just how good a device I could put together. Now I am a software man – so what am I doing with what looks like a hardware project? Well not much in the way of electronics – I just need to hook the main components together – so perhaps just a little soldering. The nitty gritty all looks to be in the software.

First task was effectively a side project. I felt that testing with speakers would be clumsy within the relatively confined area of my personal workspace – but I could manage headphones. So I decided to build a (near) empty box to hook some speaker terminals up to a headphone jack socket. With this I could provide an amplifier with something to drive and use the headphones to check the output without disturbing the peace.

The hard drive is one recovered from a retired laptop sitting in a box that adapts the SATA interface and makes it a USB drive – cost less than £3 on Amazon if I recall. The Pi can’t supply enough power for the drive through the USB port so there is an additional power supply that takes the 12 volts arriving at my “box” and converts it to the 5 volts required delivered through a USB connection and a standard “Y” USB cable.

The main power supply is a butchered mains/car adapter but an old laptop power supply would probably do – depending upon the voltage limits of any amplifier used. The HifiBerry Amp+ used here can happily operate between 12 volts and 18 volts and in turn powers the Raspberry Pi which helps reduce the complexity of the wiring rat’s nest you can see below. The idea was to start with longish hook-ups and then shorten them as required when the components get fixed into the case.

The case already has a power lead entry and an on/off rocker switch drilled and fitted. I have also drilled the back plate for the speaker connections and super-glued the main part of the loudspeaker terminal block to the outer face.

I will have to decide if I am going to go with wifi or add an RJ45 fitting to the back plate for a wired Internet connection. The box will certainly need a ‘power on’ led indicator and I would like to add an infra-red remote control for volume and track skipping at least (although most control will probably be via a web browser interface). Given that the pi can join the network and expose the music storage location (so I can add albums) the inclusion of a CD drive is still a choice to be made (one that might tax my ability to neatly tackle the case modifications).

I first started the Raspberry Pi with a copy of the standard Raspbian Linux variant. This booted just fine but (of course) was not aware of the amplifier so a rather quiet initial test. I then downloaded and installed the distro supplied by HiFiBerry and tried that. Now I quickly ran into the limits of my Linux knowledge and had some difficulty in getting sound to the amplifier. In the end, I pointed the web browser at Amazon, found the latest Enya album page and clicked on the track samples. That initially got some loud clicks from the headphones but after adjusting the (xwindows) volume control I heard some very nicely re-produced music. That at least confirmed that the amplifier worked and that my speaker/headphone adapter also functioned as expected.

It was now time to widen the testing and this meant entering (at least initially) the sometimes murky world of “open source” software. I might have the odd moan in this section but I would not want to imply any criticism of any of the projects mentioned or the teams that labour on them. Without open source and free projects we would all be the poorer (financially and culturally). In the past I have published open source software myself and that’s all I am saying (I have shared the pain guys).

A few minutes Googling showed that the Pi has inherited a solid core of key music related components from Linux and that there were a number of live projects looking to deliver the ‘ultimate’ solution. Two projects (RuneAudio and Volumio) seemed to be forks of the much respected RaspyFi project that itself now looks defunct. These projects currently sport PHP web interfaces that in turn make use of the MPD (Music Player Daemon) music playing server. A few posts suggest that the RuneAudio/Volumio divide was a little bitter.

I tried the RuneAudio project first. This was partly because the main internet page showed a picture of the HiFiBerry amp I had selected with the implicit promise that this and other selected Raspberry Pi (HiFi) add-ons were supported. I was also slightly put off by the way the Volumio project announcement was ‘camped out’ on top of the RaspyFi site. First I had to download the RuneAudio distribution (based I think on ArchLinux) and pop it on the micro SD card. It booted and I quickly located the web interface from the browser on my PC. It was clear the RuneAudio had located the albums stored on the hard drive and so I had a stab at selecting one to be played. Not a lot could be heard. A Google search found a commitment to support my amplifier from early 2014 and some suggestions on how to manage the trick short of that support eventually being delivered. This involved installing some software and settings. I logged into the distro and started trying to follow the instructions. The first irritation was that my keyboard was not supported (symbols all over the place and inexplicably the <y> and <z> keys swapped) but I struggled on. Then the installations failed with 404 errors and so I decided to try my luck elsewhere.

Next up was Pi MusicBox which has a web site like something from the early 90’s but – it’s the software that counts. Unfortunately (for me at least) the software sort of booted and then hung. To be fair, this might have been because my Pi was the recent Pi2. So I tried again.

I tried Volumio after all. Another distro (based on Debian this time I think) to install on the old SD card. This distro booted and the web app quickly became available from my PC’s web browser. The web page presented was nearly identical to the RuneAudio one which probably means both are inherited in turn. A quick trip to the settings page located the HiFiBerry amp and well - it just worked, playing albums and tracks from the hard drive. I turned to the Web radio options and soon had the BBC World Service coming through load and clear (sitting there with my headphones it could have been a clip from an old 1960’s spy film – although there was no short-wave crackle and I was not wearing a hat).

So with a working sound source it was time to stop and take stock (actually I should try Mopidy soon as that is Python based and thus potentially susceptible to some constructive hacks).

On the face of it Volumio might look like a good start point for further development. I am no fan of PHP but how bad could it be? However, the Volumio project has now started a complete re-write using Node.js. Now that is probably the best current start point for a new project of this nature but this does imply that future releases might be “feature incomplete” and that not much attention is going to be paid to moving the current release forward (this is not a criticism – just a comment).

Everything tried thus far has been based upon a custom distro which presumably greatly simplifies the task of distributing software updates (as well as integrating any developments with core OS changes). This does make the process of adding custom additions (like a CD “ripper” and IR interface) a teensy bit problematic. I am assuming that the additional power of the Pi2 will compensate for any extra “drag” from any software mods and additions I make – but that we will have to see.

I will next dig out a pair of old Mission monitor speakers I have in the garage and give them a try – cranking up the output to see just what this little rig can deliver.

One sad note – I had hoped that this little project could deliver BBC radio output (2, 4 and 5 anyway) and thus preclude the medium term purchase of a DAB radio for my office space. Turns out that earlier in the year the Beeb retreated into their iPlayer and websites so they could control (read stop) who got to hear the output based upon a proxy for location (IP address). I am not at all sure how to view this – particularly as I though the BBC had a mission to communicate with the world and not just those who pay the UK license fee. The continued availability of the World Service channel just sort of underlines this parochialism.

Speaker Update:

The amp drove the speakers very well indeed. Loads of power - going to be as loud as anyone would want even in a large room. Not set up for a more discerning listening test yet but the "sound stage" was clearly evident and I could not detect any obvious amplifier induced distortion.

And This Week’s Language is F#

2015-11-16T08:20:00.000-08:00

What? Why? Are you going to endlessly post indecipherable and incomprehensible snippets and rants on functional programming like those Haskell types? (Just joking guys – honest.) Well at least I can’t bang on about Monads as F# does not have them (‘cos the close equivalent are called “Computation Expressions”)

Continuing on my “meta” travels I started to look at what was available to the .NET developer in the general field of “numerics”. Linear Algebra, Regression, Optimization, Time Series Analysis and beyond – hopefully with a liberal license. There are a lot out there but an interesting subset were clearly developed in F# - time to clone the odd GitHub repository (and also see what Nuget could provide as packages) and see if I could make sense of anything.

You might be interested by Microsoft Research’s “Tabular” http://research.microsoft.com/en-us/projects/tabular/

Or Deedle from BlueMountain Capital http://blogs.msdn.com/b/fsharpteam/archive/2013/11/13/announcing-deedle-data-frame-and-time-series-package-for-exploratory-data-programming-with-f-and-c.aspx

Or indeed

Cronos http://cronos.codeplex.com/

Math.NET http://numerics.mathdotnet.com/

Meta Numerics http://metanumerics.codeplex.com/

Of course R.NET http://nugetmusthaves.com/Package/R.NET.Community to “wrap” the R statistical package https://www.r-project.org/

And lots more…

Please don’t get the impression these are all F# projects but it does look like F# has a distinct appeal to the people that build them and those that use them.

You may already have F# as an optional programming language available within your copy of Visual Studio but VS2015 installed the option “on the fly” for me after I cloned “Tabular” from GitHub. After that I did a quick search on my C: drive for fsi.exe (the F# interactive console), located it among the Microsoft SDKs and then created a desktop shortcut. Once double clicked, this allowed me to try out some of the tutorial stuff on the web – just to get familiar with the basics of the language. Yes, in that respect at least it does feel a bit like Python – but with the comfortable option to compile code and include it in a C# app. Did I say comfortable? F# is distinctive – say that for it.

At first glance the variable assignment statements above look like those of many other languages – the double semi-colons by the way are a feature of the interactive interpreter as they indicate that a statement is complete – which is a marker for some of what is to come.

First off, variables are not variables but immutable values and are thus referred to as values. Well except when they are not (because sometimes you might need to switch to an imperative style or do stuff like populate and save data records) and that is when you use the mutable keyword:

> Let mutable x = 8;;

With a special mutable assignment operator:

> X <- 23;;

> x;;

Val it : int = 10 \\the fsi output

But I am wandering from the point – stick with values are immutable.

Functions are simply defined:

> let addfunction x y = x + y;;

Which defines a function that takes two integers and returns their integer sum. Did I mention that F# is (very) strongly typed? Probably not. I did not define any types in the code statements above but F# inferred a type – that defaults to integer. I could have written

> let addfunction (x : float) (y : float) = x + y;;

But what I could not have done (unlike most languages) is write

> let addfunction (x : float) ( y : int) = x + y;;

error the type ‘int’ does not match the type ‘float’

F# does not support implicit casts even safe ones like int to int64.

> let addfunction (x : float) ( y : int) = x + (float y);; // works though, with the explicit cast.

Here is a longer function – our old friend Fibonacci – and no I have not yet figured out how to memoize it:

let rec fib = function
                | 0 -> 0
                | 1 -> 1
                | n -> fib (n – 1) + fib (n – 2)

The “rec” keyword indicates the function is recursive (despite that being obvious) and you can clearly see something similar to a switch statement and “arrow notation” indicating the values to be returned.

Going back to a simpler form – say:

let addString x y = (x + y).ToString();;

the ToString() bit looks .NET normal shall we call it but if you type it into fsi.exe you get the following response:

val addString : x:int -> y:int -> string

and this needs exploring. The implication is that the second parameter is passed to a function that returns a string and the first integer parameter is passed to a function that returns an integer. This would (broadly) be correct as F# functions only take a single argument and only return a single type. So how is that? F# automatically curries functions. See https://en.wikipedia.org/wiki/Currying for a quick dip into currying if you have not played with the technique. Then pause and thing about how that might impact upon a language where functions are not just first class citizens but rather the whole point.

To quote the Wikipedia article “F# is a strongly typed, multi-paradigm programming language that encompasses functional, imperative, and object-oriented programming techniques.” Which maybe understates things a bit as there does not seem much at the cutting edge of language development left out – and all encapsulated within a functional programming paradigm.

For the moment I will be content when I have taken on board enough of the language syntax to be able to follow the relevant math library code but I will be coming back to try and write something non trivial in F#. The syntax is superficially daunting but starts to make sense as the underlying functionality becomes a little clearer. F# feels like it could give you (like Oddball's Sherman) a very nice “edge” (in some of life's more complex coding challenges).

Your browser does not support this audio

Addendum:

If you are a user of Visual Studio it is easier to run Fsi.exe withing that environment by pressing <ctrl><alt><f> with an F# project open.. The text is clearer as well.

F# is more that somewhat absorbing - although it does suffer from the "this is the only way" crowd banging on a bit. I can see my next side project might be C# for the UI and F# for all the functionality which would be a neat solution solving some of the issues raised by Python - and hey guess what? F# borrowed the concept of "significant white-space" from that very source (plus a few other bits I think).

Constructing and compiling C# using Roslyn

2015-10-16T04:20:00.000-07:00

The optimal path for my current project requires the ability to create and use new classes at runtime. The alternatives all look rather clumsy while this approach looked elegant.

.NET has had some capacity to compile arbitrary code since the beginning but a lot of effort has recently gone in to a new compiler – known as Roslyn. On the face of it there are two ways towards my goal – use the CodeDom which is language agnostic or try out the newer (Microsoft) open source Roslyn technology that is designed to compile C# and VB.NET.

CodeDom and Roslyn are only partly related and after some research it looked like Roslyn was the way ahead as it provides a feature complete compiler for my language of choice C#. The other major plus is that Roslyn runs “in process” while CodeDom runs compiles through another executable with (perhaps) a rather clunky API.

If you fancy going the well tried CodeDom way then MSDN has a good code example towards the bottom of this article.

To add the Roslyn tooling to a project use the NuGet Package Manager option on the Tools menu and Choose “Manage NuGet Packages for Solution…” Select nuget.org as the package source and then search for “Microsoft.CodeAnalysis”. Click on Microsoft.CodeAnalysis.CSharp , click the “Install” button and when offered, accept the license.

The actual development task turned out to be pretty steady.

I started with a class to define the properties of the class to be constructed


internal class ClassProperty
{
    public string PropertyName { get; set; } = "";
    public string DataType { get; set; } = "";
    public ClassProperty() { }
    public ClassProperty(string propertyName, string dataType)
    {
        PropertyName = propertyName;
        DataType = dataType;
    }
}

And supplied a populated List<ClassProperty> to a new class that I was going to use to construct and compile the code. In fact I created a little method to loop through the fields of a database table and populate the list (as that was the direction I was going in and I am lazy) but a List can be quickly populated with just a few lines of code.

The new class also had properties exposed for a namespace, class name, using directives, inheritance and interfaces. (The latter two are part implemented in the following code in that they are added to the class declaration but any related methods remain unimplemented at this stage.)

I backed up the property creation by creating a Dictionary of default values and you can see how it is applied after this.


private Dictionary<string, string> defaultValues = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
{
    ["string"] = " = String.Empty;",
    ["Int64"] = " = 0;",
    ["Int32"] = " = 0;",
    ["Int16"] = " = 0;",
    ["DateTime"] = " = new DateTime(1970,1,1);",
    ["double"] = " = 0;",
    ["single"] = " = 0;",
    ["Decimal"] = " = 0;",
    ["bool"] = " = false;",
    ["Boolean"] = " = false;",
    ["int"] = " = 0;",
    ["long"] = " = 0;"
};

The method to create the code is pretty short.


private string buildClassCode()
{
    StringBuilder sb = new StringBuilder();
    foreach(string uses in usings)
    {
        sb.AppendLine("using " + uses + ";");
    }
    sb.AppendLine("namespace " + nameSpace);
    sb.AppendLine("{"); // start namespace
    string classInherits = (inherits.Length > 0 || interfaces.Count > 0) ? " : " + inherits : "";
    foreach(string inface in interfaces)
    {
        classInherits += (classInherits.Length > 3) ? ", " + inface : inface;
    }
    sb.AppendLine($"public class {tableName}{classInherits}" );
    sb.AppendLine("{"); // start class
    sb.AppendLine($"public {tableName}()"); // default constructor
    sb.AppendLine("{}");
    foreach (ClassProperty newProperty in classProperties)
    {
        sb.AppendLine($"public {newProperty.DataType} {newProperty.PropertyName} {"{ get; set;}"}{defaultValues[newProperty.DataType]}");
    }
    sb.AppendLine("}"); // end class
    sb.AppendLine("}"); // end namespace
    return sb.ToString();
}

I tested this by popping the output into a multi-line textbox. while the code lacks indentation it is perfectly clear as the image below shows

Now to get that code compiled and run an initial test using VS Debug mode to step through the code and watch the results.


private void buildinstance()
{
    string instance = buildClassCode();
    SyntaxTree syntaxTree = CSharpSyntaxTree.ParseText(instance);
    // inspect the tree
    //SyntaxNode root = syntaxTree.GetRoot();
    //foreach (var node in root.DescendantNodes())
    //{
    //    var x = node.GetText();
    //}
    string assemblyName = tableName; //Path.GetRandomFileName();
    MetadataReference[] references = new MetadataReference[]
    {
        MetadataReference.CreateFromFile(typeof(object).Assembly.Location),
        MetadataReference.CreateFromFile(typeof(Enumerable).Assembly.Location)
    };
    CSharpCompilation compilation = CSharpCompilation.Create(
        assemblyName,
        syntaxTrees: new[] { syntaxTree },
        references: references,
        options: new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary)); // you can also build as a console app, windows.exe etc
    Assembly assembly = null;
    var ms = new MemoryStream();
    EmitResult eResult = compilation.Emit(ms);
    if (eResult.Success)
    {
        
        ms.Seek(0, SeekOrigin.Begin);
        assembly = Assembly.Load(ms.ToArray());
        Type type = assembly.GetType(nameSpace + "." + tableName);
        object newObject = Activator.CreateInstance(type);
        // now we can prove that worked by stepping through the code while iterating over the properties
        foreach (PropertyInfo propertyInfo in newObject.GetType().GetProperties())
        {
            string pName = propertyInfo.Name;
            string dataType = propertyInfo.PropertyType.Name;
            // and test we can assign a value
            if (dataType.Equals("String", StringComparison.OrdinalIgnoreCase))
            {
                propertyInfo.SetValue(newObject, "foo", null);
            }
        }
    }
    else
    {
        IEnumerable<Diagnostic> failures = eResult.Diagnostics.Where(diagnostic =>
            diagnostic.IsWarningAsError || diagnostic.Severity == DiagnosticSeverity.Error);
        string msg = "";
        foreach (Diagnostic diagnostic in failures)
        {
            msg+= $"{diagnostic.Id}: {diagnostic.GetMessage()}" + "\r\n";
        }
        // do something useful with the message
    }
    ms.Close();
    ms.Dispose();
}

Ran first time, although I went back and added the diagnostics you can see just to step throght things as they happened.

The SyntaxTree built a structure for the code elements from the string. The CSharpCompilation object took that syntax tree and compiled the code as C#. The compiled assembly was saved into a memory stream and the compilation results checked for errors. Assuming no errors, the code was loaded into an Assembly object and a new instance of the compiled object (class in this instance) created and inspected.

Next I checked that I could execute a method on the compiled class. I added the following line to the class code creation method:


sb.AppendLine("public string getBar(string foo) {return foo + \" bar\";}");

and a code line to call the method after the object is created in the buildinstance() method tested above


string res = (string)type.InvokeMember("getBar", BindingFlags.Default | BindingFlags.InvokeMethod,
    null, newObject, new object[] { "foo" });

which returned the string “foo bar” to the variable res.

Job done.

Addendum

Thought I would check that the default class constructor was executed when the compiled object instance was created. Thought it must be but…

Adjusted the class constructor to set one of the properties and then read that back when looping over the properties. Worked as expected.

The Activator.CreateInstance() method can be called with a parameter list to actuate alternate constructors with their parameters thus:

public static object CreateInstance(Type type, params object[] args)

which probably covers that issue as an Interface can’t contain a constructor but can contain any other methods I want to expose. Clearly, there is a wee bit more code to write to get a practical implementation but it is also clear that the overall approach is very practical.

And Then The Real World

In the real world your compiled class will probably reference one or more System/.NET dll, a custom library or your new class may inherit from another class already defined in the main program assembly.

You can add such references but you need to supply the path to the relevant dll or exe file. Apparently this changed from early Roslyn releases so (like me) you might find many obsolete methods if you Google around the subject. You also need to include the relevant "using" statements in the code itself of course (see above).

In my first stab at this the CSharpCompilation constructor was supplied with a default array of MetadataReference objects. This array can be extended to include whatever is required.

The default location for the main System dlls can be found with this line of code:

var assemblyPath = Path.GetDirectoryName(typeof(object).Assembly.Location);

You can then construct a path to (say) System.dll and Syatem.Data.dll like so:


string sysTm = Path.Combine(assemblyPath, "System.dll");
string sysData = Path.Combine(assemblyPath, "System.Data.dll");

You can similarly construct the path to the classes in your own program as well (assuming they are public):

string mePath = Path.Combine(Application.StartupPath, "MyProg.exe");

The paths can then be used to add the references like so:


MetadataReference[] references = new MetadataReference[]
{
    MetadataReference.CreateFromFile(typeof(object).Assembly.Location),
    MetadataReference.CreateFromFile(typeof(Enumerable).Assembly.Location),
    MetadataReference.CreateFromFile(sysTm),
    MetadataReference.CreateFromFile(sysData),
    MetadataReference.CreateFromFile(mePath)
};

And that is how to add those vital references to your Roslyn compiled assembly.

Interfaces and Unknown Classes

2015-10-13T04:49:00.000-07:00

I wanted to make use of the ExcelReader class I created as a source of data to create class instances.

Generally the classes in question inherit from a base class that also knows how to save itself to a database. I could, of course maintain a list of such classes within my Excel reading code and call an appropriate routine as required. However I had in mind to create something that was generic enough to handle a whole raft of classes – including some created and compiled “on the fly” and thus unknowable to the source code.

The normal method for dealing with a range of classes through a common function would be to specify an interface common to all and then to interact through that interface.

Interfaces (if this is an unfamiliar term)

An interface is a list of public properties and methods (also indexers and events) that ~~should~~ must be exposed by a class that implements the interface. Interfaces themselves can be inherited but contain no actual code – just the signatures (if you like) of the components being specified.

One of the simplest ways of creating an interface is to (VS2015) right click on the class name in the code window and select “Extract Interface” from the “Quick Actions” context menu. Earlier versions of Visual Studio offer a “Refactor” menu option that includes extracting an interface.

Requiring a specific Interface is an excellent way of ensuring that objects calling the functionality of a class you provide supply full support. You can rely upon a method or event handler being available – even if it just a “stub”. IComparable would be a good example of just such an interface.

I have used interfaces in the past for testing alternate algorithms. An interface allowed me to construct a standardised software socket into which I could plug different classes at will. I could thus test performance and function of alternate approaches without having to change the code “consuming” the service/process.

If I come across a great in depth web page on Interfaces I will post a link here but for a couple of helpful examples you can do a lot worse than this C# Station post

Just about the only thing my target classes have in common is an Update method. This limits the scope of any Interface a bit. The properties could be addressed through reflection and it was a reasonable bet that a class Clone() method could be used to create new class instances. So that was beginning to sound like a workable strategy.

So my interface ended up being defined like this:


interface ObjectClone
 {
     object CloneAsObject();
     void UpdateClass();
 }

Each class implementing the interface needs then to add a CloneAsObject method alongside an UpdateClass method. The CloneAsObject() method is very simple:


public object CloneAsObject()
{
    return (MyClassName)this.MemberwiseClone();
}

There is an assumption here that all such classes expose public properties for all of the key data elements.

I defined an ExcelDataLoader class that will use an instance of the ExcelReaderclass to gather data from selected columns in a spreadsheet and use this data to create instances of a class that implements that very simple ObjectClone Interface.

The ExcelDataLoader class needs a local definition and a public property:


private ObjectClone targetClass = null;
public ObjectClone TargetClass
{
    set { targetClass = value; }
}

To pass in a new instance of the required class.

Plus I also needed a way of passing the required instructions to the ExcelDataLoader. A simple class to define the required columns and the associated data name and type seemed the thing and that could be as simple as:


public class ExcelColumnData : IComparable
{
    public string ColumnName { get; set; } = "";
    public string DataName { get; set; } = "";
    public ExcelColumnData()
    {
    }
    public ExcelColumnData(string ColumnName, string DataName)
    {
        this.ColumnName = ColumnName;
        this.DataName = DataName;
    }
    public int CompareTo(object objx)
    {
        ExcelColumnData x = (ExcelColumnData)objx;
        int res = this.ColumnName.Length.CompareTo(x.ColumnName.Length);
        return (res == 0) ? this.ColumnName.CompareTo(x.ColumnName) : res; // AA after Z
    }
}

Although you will note that this class implements an interface – Icomparable so that List<T>Sort() can be used to ensure the columns are presented in order.

These column definitions could then be passed as a List<ExcelColumndData> to the ExcelDataLoader class.

The ExcelDataLoader class loadData() method uses an instance of the ExcelReader class to extract some columns of data from a specified sheet and then creates an instance of the class to be created. It then uses reflection to loop through the class properties and if one of these matches a specified data name associated with an Excel data column then the expected data type is checked and an attempt made to cast (or parse) the data extracted from Excel to the required type. If the parse works then the property is set to the excel value.


private int loadData()
{
    int recCount = 0;
    excelReader = new ExcelReader(excelFilepath, sheetName);
    List<string> colNames = new List<string>();
    if(excelColumns.Count > 1) { excelColumns.Sort(); }
    foreach (ExcelColumnData eColData in excelColumns)
    {
        colNames.Add(eColData.ColumnName);
    }
    List<List<string>> selCols = excelReader.GetSelectedColumns(colNames, skipFirstRow);
    foreach(List<string> excelRow in selCols)
    {
        ObjectClone newObject = (ObjectClone)targetClass.CloneAsObject();
        foreach (PropertyInfo propertyInfo in newObject.GetType().GetProperties())
        {
            if (propertyInfo.CanWrite)
            {
                string propertyName = propertyInfo.Name;
                string dataType = propertyInfo.PropertyType.Name;
                for(var eci=0; eci< excelColumns.Count; eci++)
                {
                    if (excelColumns[eci].DataName.Equals(propertyName, StringComparison.OrdinalIgnoreCase))
                    {
                        // the strategy is to set the Property value if the incoming data is valid for the Property type
                        // otherwise the value is left as the class default setting
                        if (dataType.Equals("String", StringComparison.OrdinalIgnoreCase))
                        {
                            propertyInfo.SetValue(newObject, excelRow[eci], null);
                        } else if (dataType.Equals("Int64", StringComparison.OrdinalIgnoreCase))
                        {
                            long intVal = 0;
                            if (long.TryParse(excelRow[eci], out intVal))
                            {
                                propertyInfo.SetValue(newObject, intVal, null);
                            }
                        } else if (dataType.Equals("Int32", StringComparison.OrdinalIgnoreCase))
                        {
                            int intVal = 0;
                            if (int.TryParse(excelRow[eci], out intVal))
                            {
                                propertyInfo.SetValue(newObject, intVal, null);
                            }
                        }  else if (dataType.Equals("Int16", StringComparison.OrdinalIgnoreCase))
                        {
                            Int16 intVal = 0;
                            if (Int16.TryParse(excelRow[eci], out intVal))
                            {
                                propertyInfo.SetValue(newObject, intVal, null);
                            }
                        } ... etc
                          ''' 
                          ...
                }
            }
        }
        newObject.UpdateClass();
        recCount++;
    }
    return recCount;
}

And you can pump data from Excel (or anywhere else) to create classes that are anonymous as far as the process is concerned and (in this instance) call a method to retain the data in a database for future use.

Usual caveats as this is prototype (although working) code and my production version will inevitably grow "horns" over time as I deal with edge cases and extra features.

Reading Excel with the Open XML SDK

2015-10-09T03:28:00.000-07:00

A while back I posted Look Mum no Office on the subject of the Microsoft Open XML SDK that I had used for writing an Excel file without the need for a copy of Office on the machine running the code. This proved to be very straightforward and there was some excellent samplecode available

Yesterday I needed to read data from an Excel workbook and decided to write a class that I could use in multiple scenarios to grab data from this source. Using the Open XML SDK looked the way to go as this should work with all Excel files produced by versions after Office 2003.

First use the Package Manager to grab the support libraries

PM> Install-Package DocumentFormat.OpenXml

And then add WindowsBase.dll to the project references (it is in the Framework Assembly list so easy to spot using the “Add Reference” dialog.

My first Excel reading class function was to read the list of Sheet Names in a workbook.


private List<string> getSheetNames()
{
    List<string> sheetNames = new List<string>();
    try
    {
        var document = SpreadsheetDocument.Open(excelFilePath, false);
        var workbookPart = document.WorkbookPart;
        var sheets = workbookPart.Workbook.Descendants<Sheet>();
        foreach(Sheet sheet in sheets)
        {
            sheetNames.Add(sheet.Name);
        }
        document.Close();
    }
    catch (Exception ex)
    {
        ...
    }
    return sheetNames;
}

This was a good indicator that this process was going to be pretty steady although there were a couple of twists coming up (with all making sense when thought about).

I wanted my class to have two further main features. One to take a sample of rows from a specified sheet and another to extract defined columns from a specified sheet. As both cover similar ground, let’s look at the development of the sample row extract.

The process is 1. Identify the worksheet, 2. Grab the sheet, 3. Extract the sheet data and then 4 Loop through the Rows in the sheet data. Within each row we can loop through the cells and read the string content.


private List<List<string>> initialGetSampleRows()
{
    List<List<string>> samples = new List<List<string>>();
    try
    {
        int selCount = 0;
        var document = SpreadsheetDocument.Open(excelFilePath, false);
        var workbookPart = document.WorkbookPart;
        var sheets = workbookPart.Workbook.Descendants<Sheet>();
        foreach (Sheet sheet in sheets)
        {
            if (sheet.Name == sheetName)
            {
                var workSheet = ((WorksheetPart)workbookPart.GetPartById(sheet.Id)).Worksheet;
                var sheetData = workSheet.Elements<SheetData>().First();
                List<Row> rows = sheetData.Elements<Row>().ToList();
                foreach (Row row in rows)
                {
                    var thisRow = new List<string>();
                    foreach (Cell cell in row.Descendants<Cell>())
                    {
                        var cellContent = cell.CellValue;
                        thisRow.Add((cellContent == null) ? cell.InnerText : cellContent.Text);
                    }
                        samples.Add(thisRow);
                    selCount++;
                    if (selCount >= sampleRowCount)
                    {
                        break;
                    }
                }
                break;
            }
        }
        document.Close();
    }
    catch (Exception ex)
    {
        ...
    }
    return samples;
}

Does it work? Well there are a few snags. Dates are (perhaps not unexpectedly) showing as a number, numbers themselves are fine but cells with a string content are being read as an integer of some sort.

Turns out that strings are held in a shared string table so we need to change the code to read that table when required based on the cell DataType ("s").


foreach (Cell cell in row.Descendants<Cell>())
{
    var cellContent = cell.CellValue;
    var cellText = (cellContent == null) ? cell.InnerText : cellContent.Text;
    var dataType = cell.DataType ?? CellValues.Error;
    if (dataType == CellValues.SharedString)
    {
        cellText = workbookPart.SharedStringTablePart.SharedStringTable.Elements<SharedStringItem>().ElementAt(Convert.ToInt32(cell.CellValue.Text)).InnerText;
    }
    thisRow.Add(cellText);
}

Excel dates are stored as an integer number of days since 31^st December 1899 – so the Excel date for 1^st January 1900 is 1. There is a snag inherited for compatibility reasons from Lotus – where the year 1900 was assumed not to be a leap year when in fact (unlike the year 2000) it was. So the numbers are almost always out by 1. Dates are represented as human sensible dates on an Excel spreadsheet by using a “style”. It is possible to detect a style value for a cell although this is normally null.

Something like:


uint styleIndex = cell.StyleIndex ?? uint.MinValue;
if (styleIndex == 3)
{
    Double serialDate;
    if (Double.TryParse(cellText, out serialDate))
    {
        DateTime cellDate = DateTime.FromOADate(serialDate);
        cellText = cellDate.ToString(dateFormat);
    }
}

Where we can use the inbuilt DateTime.FromOADate() function to handle the conversion, applying string formatting to suite.

There is another snag. It would not make any sense for an XML based Excel file to include empty cells. This means that when a row of cells is read the row only contains cells with content. This would present a problem where empty cells appear (perhaps now and again) before columns of cells of interest. Fortunately each cell object contains a CellReference that can be used to extract the Column+Row identity we are familiar with on Excel sheets. This can be used to detect a cell being presented earlier than expected in a row.

One solution (now deprecated - see below) is to create an Enumerator that can manage the detection of missing cells and supply empty ones as required. The class code was re-structured a bit at this point in time as well – partly to make the code units available to functionality to be added to the class later.

The Enumerator looks like this:


private IEnumerator<Cell> excelCellEnumerator(Row row)
{
    int currentCount = 0;
    foreach (Cell cell in row.Descendants<Cell>())
    {
        string columnName = columnNameFromReference(cell.CellReference);
        int thisColIndex = convertColumnNameToNumber(columnName);
        while(currentCount < thisColIndex)
        {
            var emptyCell = new Cell()
            {
                DataType = null,
                CellValue = new CellValue(string.Empty)
            };
            yield return emptyCell;
            currentCount++;
        }
        yield return cell;
        currentCount++;
    }
}

With support from:


private string columnNameFromReference(string cellReference)
{
    var regex = new Regex("[A-Za-z]+");
    var match = regex.Match(cellReference);
    return match.Value;
}
private int convertColumnNameToNumber(string columnName)
{
    char[] colAlphas = columnName.ToCharArray();
    Array.Reverse(colAlphas);
    int colNumber = 0;
    for (int i = 0; i < colAlphas.Length; i++)
    {
        char alpha = colAlphas[i];
        int tAlpha = i == 0 ? alpha - 65 : alpha - 64;
        colNumber += tAlpha * (int)Math.Pow(26, i);
    }
    return colNumber;
}

And with the nitty gritty of the sample extract function being changed to use the Enumerator thus:


foreach(Row row in rows)
{
    var thisRow = new List<string>();
    var cellEnumerator = excelCellEnumerator(row);
    while (cellEnumerator.MoveNext())
    {
        var cell = cellEnumerator.Current;
        var contentString = readCellContent(cell, workbookPart);
        thisRow.Add(contentString);
    }
    samples.Add(thisRow);
}

And the reading of the cell content (all code earlier in the post) being moved to a separate function.

Addendum

A little experimentation indicated that it might also be a good idea to make provision for Boolean values in an Excel sheet. Where Booleans exist they are presented in Excel as TRUE and FALSE but the underlying values are a (reasonable) 1 and 0 respectively. You can detect Boolean values using the cell DataType which (in this instance) would be set to CellValues.Boolean.

There is also a CellValues enum CellValues.Date but strangely this is not set in any of my sample data so the original technique of inspecting the StyleIndex still stands.

You might however wish to add the following code snippet to deal with Booleans:


if(dataType == CellValues.Boolean)
{
    int bVal;
    if(int.TryParse(cellText, out bVal))
    {
        if(bVal == 0)
        {
            cellText = "false";
        } else
        {
            cellText = "true";
        }
    }
}

Parsing the value rather than just casting it may just be paranoia but…

Further Thought

Of course Yield return is sheer flummery in this context – although the Iterator was lots of fun.

Yield should be reserved for computationally expensive processes where you want to spread the load a bit. In this instance most or all cells are going to be available in the Row object and so we would be better writing a function to return the whole list of cells in one go.


private List<Cell> getExcelRowCells(Row row)
 {
     List<Cell> theCells = new List<Cell>();
     int currentCount = 0;
     foreach (Cell cell in row.Descendants<Cell>())
     {
         string columnName = columnNameFromReference(cell.CellReference);
         int thisColIndex = convertColumnNameToNumber(columnName);
         while (currentCount < thisColIndex)
         {
             var emptyCell = new Cell()
             {
                 DataType = null,
                 CellValue = new CellValue(string.Empty)
             };
             theCells.Add(emptyCell);
             currentCount++;
         }
         theCells.Add(cell);
         currentCount++;
     }
     return theCells;
 }

And then consume the list as before but with conventional List<T> iteration:


foreach (Row row in rows)
{
    var thisRow = new List<string>();
    List<Cell> rowCells = getExcelRowCells(row);
    foreach(Cell cell in rowCells)
    {
        thisRow.Add(readCellContent(cell, workbookPart));
    }
    samples.Add(thisRow);
    selCount++;
    if (selCount >= sampleRowCount)
    {
        break;
    }
}

C# Memoization

2015-10-05T10:21:00.000-07:00

Memoization is a program optimisation technique that stores the results of a computationally expensive function in a cache and then returns values from that cache if the function is called again with the same input(s).

Wikipedia has the term coined by (a personal hero) Donald Michie in 1968 although I can’t find the word in the original paper (dated 1967) [link: https://saltworks.stanford.edu/assets/kr445kk4694.pdf ] so it probably came into being as the idea was applied.

I had previously used the technique in JavaScript but that was a while ago and it took me a few moments to even remember the precise term – let alone work out how to do the trick in C#.

The archetypal example of memoization in JavaScript is a function to calculate Fibonacci numbers and I found this one on http://stackoverflow.com/


var fibonacci = (function () {
    var memo = [0, 1];
    var fib = function (n) {
        var result = memo[n];
        if (typeof result !== 'number') {
            result = fib(n - 1) + fib(n - 2);
            memo[n] = result;
        }
        return result;
    };
    return fib;
}());

The var fibonacci contains a function that can be used to calculate a result. It will return a cached value for any previously requested calculation and will recursively call itself to calculate a new value – which will often be returned from the cache (well certainly if a sequence of values are requested).

The cache takes the form of an array (memo[]) and this is wrapped in a “closure” so that it remains accessible to the inner function fib().

Closure, recursion and memorization in one function – has all the dangerous signs of an interview question.

I wanted a function in C# to look-up a subset of values previously stored in a database (in this instance identity integers) with a high likelihood that individual values would be repeatedly requested over the run of the process but without being able to sort the requests in any meaningful way. So assuming that database reads are relatively slow, memorization looked like the technique to try.

My requirement is a simple case – I have a function into which I pass a string as a key and the function does a database lookup and retunes a long integer. So let us start there


private long lookUpDimension(string dkey)
{
    long idVal = 0;
    try
    {
        // code to access the databse and do the lookup
    }
    return idVal;
}

Just a simple function (well after all the database access code is removed for clarity). Point being – it is just an ordinary function – takes a single argument and returns the output.

The class containing that look-up function also has two static functions to work some magic.


private static Func<TArg, TResult> Memoize<TArg, TResult>(Func<TArg, TResult> function)
{
    return Memoize(function, new Dictionary<TArg, TResult>());
}
private static Func<TArg, TResult> Memoize<TArg, TResult>(Func<TArg, TResult> function, IDictionary<TArg, TResult> cache)
{
    return key => cache.ContainsKey(key) ? cache[key] : (cache[key] = function(key));
}

So – what is this Func<TArg, TResult> stuff?

Func is a predefined Delegate, this one takes a single argument in and outputs a single result. There are more defined (up to 8 inputs) and we could express the second one (2 inputs) thus:

public delegate TResult Func<in T1, in T2, out TResult>(

        T1 arg1,

        T2 arg2

But Func<T1, T2, TResult> is syntactic sugar of the right sort.

My code needed to declare something like this


private Func<string, long> memLookup = null;

which declares a delegate instance with a string input and a long integer output

and then the following code needs to be executed:


memLookup = Memoize<string, long>(lookUpDimension);

sometime when the class is being created perhaps or certainly before the memorized function is required. The class then exposes a method like:


public long LookupDimValue(string dkey)
{
    return memLookup(dkey);
}

Because memLookup() now “contains” a memoized version of the original lookUpDimension() function taking a string input and returning a long integer.

One note of caution about the sort of memorized function I have used in this example. If the function fails to locate a database record corresponding to the given key then it returns a value indicating that (a zero). One option might be to then insert a new record but the memorized function will “remember” that the key is not valid and continue to return the original response. The insert would therefore have to be managed from within the memorized function and the relevant value for the new record returned.

Just a thought (added 11/10/15)
What is the simplest C# code to match the JavaScript function we started with?


Dictionary<int, long> memo = new Dictionary<int, long>() { { 0, 0 }, { 1, 1 } };


private long fib(int n)
{
    long result;
    if (!memo.TryGetValue(n, out result))
    {
        result = fib(n - 1) + fib(n - 2);
        memo.Add(n, result);
    }
    return result;
}

Fewer lines but less "functional" as it makes use of a Dictionary object that is external to the function. Memoization would often be a better approach.

If you are wondering if memoization (at least in some form) is an effective technique then try writing a fibonacci function without and give it a run - as the input n increases the response time can be considerable.

And more (added 13/10/2015)

C#6 now allows a different form for initialising a Dictionary


Dictionary<int, long> memo = new Dictionary<int, long>
{
    [0] = 0,
    [1] = 1
};

which is bound to be useful when the assignment can add value (just a thought).

More .NET monospace font nonsense

2015-09-25T07:52:00.000-07:00

Just to make the point. Wikipedia defines a monospaced font as follows:

A monospaced font, also called a fixed-pitch, fixed-width, or non-proportional font, is a font whose letters and characters each occupy the same amount of horizontal space.

Now not everything you read on Wikipedia is necessarily true but I am confident that a technical reader would be happy with that definition.

You might be less happy to find that the .NET Graphics.DrawString() method does not adhere strictly to this approach. My earlier experience while trying to measure a monospaced font character width was a clue that issues might arise in actual usage. Arise they did.

You can’t see it at first but as printed lines become longer then it becomes clear that the line length is less than it should be. Pixels in the X axis are being “lost” along the way. My test data was printed at the correct register for the first 7 characters but the 8^th was one pixel to the left of where it should have been. Somewhere around 90 plus characters into a line the positioning is out by 10 pixels (a single character width for the font in question).

Each character in every line is correctly positioned relative to the other lines but incorrectly positioned based upon any rational definition of a monospaced font.

The image below shows the 97^th character (H) in the first line of text showing correctly in position while the second line rendered by the DrawString() method being passed the whole line of text shows the “H” well to the left of the correct location.

In case you wondered the blue “divider” is placed by an algorithm that looks for probable field positions by analysing transitions between character types (in this case a white space prior to a “letter”)

So if (like me) you are working in the .NET environment and you want to know where the nth character is on a “painted” surface of a window/control using (for crying out loud) a monospaced font – then you need to print each character individually at the correct location. On that basis, you could do just as well with a nominally proportional font of course.

At least this showed that, at longer string lengths, there was some consistency between MeasureString() and DrawString() even if both (in this instance) are wrong.

Programming in .NET is sometimes all about fighting .NET (or GDI+ or whatever is actually responsible for the rendering under the covers). On reflection, this is true of every software platform I have ever worked on.

Final word:
While subsequently working on something else completely, I found out that MeasureString() uses GDI+ while by default .DrawString() uses GDI and their rendering is not the same. Well we live and learn I suppose but you would have expected consistency within the .NET defaults wouldn't you?

Accurately measure monospace font widths

2015-09-21T04:42:00.000-07:00

I was building a custom control in C# to allow a user to identify the positions of data fields in a flat text file. So one of the things the control needs to know is where a given character position is in a line of text.

Easy peasy you may say – make the font Courier New or GenericMonospace (of course) and use the Graphics.MeasureString() function.

Here is some test code:


private void Dummy2_Load(object sender, EventArgs e)
{
    Font mFont = new Font(FontFamily.GenericMonospace, (float)12);
    this.Font = mFont;
    Graphics g = this.CreateGraphics();
    label1.Text += Convert.ToString(g.MeasureString("A", mFont));
    label2.Text += Convert.ToString(g.MeasureString("AA", mFont));
    label3.Text += Convert.ToString(g.MeasureString("AbCdEfGhIj", mFont));
    g.Dispose();
    mFont.Dispose();
}

Which gives a result like:

So is a character 15.2 pixels wide or 12.5 pixels wide or 10.4 pixels wide or...?

If I grab a screen shot and count the pixels I can see ALL of the characters are drawn with a width of 10 pixels. [I am currently ignoring the supposed character height as I assume that is a notional line height (but w.t.f?).]

So if you or I want to determine the character width on an unknown device with unknown settings we have a bit of a problem.

One might wonder if the function returns a notional “overhead” for any given string. But if we subtract the width returned for the single character from the width of 10 characters and then divide by 9 – we get 9.89 pixels. So any overhead is not consistent.

The reducing “error” returned by MeasureString() as the length of the initial test strings increased might make you wonder if a very long string would give an accurate result. Sadly a string of 100 characters measures at 9.94 pixels per character which at least could “round up” to the correct value but raises new questions about just what is being measured.

Perhaps there is an optimal string length that could be applied?

A bit more code came up with a value of 48 characters for an optimal result on my development system – but I have no idea if that is fully “portable” for the full range of font sizes and screen resolutions.

N.B. Just in case you were wondering, and even though you know the code is using a monospaced font – you get identical results substituting lower case i's into any or all of the test strings.

Addendum
After posting this I built a little class to measure the true font size. The class creates a bitmap and then draws the characters q and j onto the bitmap and scans the pixels to find the first and last rows of pixels that contain part of one of the characters - this gives the true height. Then the bitmap is cleared and two j's drawn next to each other. The class then measures the number of pixels between the first column containing a pixel for the first character and the first column of pixels used to draw the second. This gives the true width.

The code is more than a bit scrappy so feel free to correct/optimise. Please treat it as a prototype (which it is) and not production quality code.


class MeasureMonoFont : IDisposable
{
    private Graphics g;
    private Bitmap bm;
    private SolidBrush br;
    StringFormat drawFormat;
    private const int defBitmapSize = 100;
    public MeasureMonoFont()
    {
        bm = new Bitmap(defBitmapSize, defBitmapSize);
        g = Graphics.FromImage(bm);
        br = new SolidBrush(Color.Black);
        drawFormat = new StringFormat();
    }
    public SizeF GetCharSize(Font withFont)
    {
        SizeF cSize = new SizeF();
        g.Clear(Color.White);
        PointF cPoint = new PointF(0, 0);
        g.DrawString("qj", withFont, br, cPoint, drawFormat);
        Color tPixel;
        int firstRow = -1;
        int lastRow = -1;
        for (var yp = 0; yp < defBitmapSize; yp++)
        {
            for (var xp = 0; xp < defBitmapSize; xp++)
            {
                tPixel = bm.GetPixel(xp, yp);
                if (!(tPixel.ToArgb() == Color.White.ToArgb()))
                {
                    if(firstRow == -1) { firstRow = yp; }
                    if(yp > lastRow) { lastRow = yp; }
                }
            }
        }
        cSize.Height = ++lastRow - firstRow;
        g.Clear(Color.White);
        g.DrawString("jj", withFont, br, cPoint, drawFormat);
        firstRow = -1;
        lastRow = -1;
        bool secondChar = false;
        bool firstChar = false;
        bool emptyColumn = false;
        for (var xp = 0; xp < defBitmapSize && lastRow == -1; xp++)
        {
            emptyColumn = true;
            for (var yp = 0; yp < defBitmapSize && lastRow == -1; yp++)
            {
                tPixel = bm.GetPixel(xp, yp);
                if (!(tPixel.ToArgb() == Color.White.ToArgb()))
                {
                    if (!firstChar)
                    {
                        firstChar = true;
                        firstRow = xp;
                    }
                    if (secondChar) { lastRow = xp; }
                    emptyColumn = false;
                }
            }
            if(firstChar && emptyColumn) { secondChar = true; }
        }
        cSize.Width = lastRow - firstRow;
        return cSize;
    }
    public void Dispose()
    {
        g.Dispose();
        bm.Dispose();
        br.Dispose();
        drawFormat.Dispose();
    }
}

Reading a delimited text file in C#

2015-09-16T07:41:00.000-07:00

You can of course write your own string parsing code or you can take the easy way and add a reference to Microsoft.VisualBasic within your Visual Studio project and make use of the TextFieldParser

This little snippet loads the content of a delimited text file into a ListView


using Microsoft.VisualBasic.FileIO;
        
private void loadCSVFile(string filePath, string delimiter)
{
    TextFieldParser tfParser = new TextFieldParser(filePath);
    tfParser.TextFieldType = FieldType.Delimited;
    tfParser.SetDelimiters(delimiter);
    string[] curRow;
    ListViewItem lvi;
    while (!tfParser.EndOfData)
    {
        curRow = tfParser.ReadFields();
        lvi = listView1.Items.Add(curRow[0]);
        for (var ci = 1; ci < curRow.Length; ci++)
        {
            lvi.SubItems.Add(curRow[ci]);
        }
    }
    tfParser.Close();
    tfParser.Dispose();
}

Sometimes VB is your friend.

Usual caveats about production code (try/catch etc.)

Wot no Python Executable?

2015-09-14T07:22:00.000-07:00

So how did my first foray into Python go?

Well partly I cheated and wrote Iron Python code in Visual Studio – so obviously I write a lot of .NET code but using Python syntax. The idea was to get a running start on how to structure my Python and I wrote a bunch of classes with methods, adopting the MVC pattern to push that beyond where I would normally go. So far, I am still enthusiastic about the language. There are odd puzzles to do with variable scope but I am confident those will work out nicely as I write more code. The short and snappy syntax bodes well for productivity although debugging minor spelling mistakes (character case and all that) counteracts that a bit.

You might think that building a .NET Python app was a quick route to having something I could distribute as an executable to others but (and this is a big BUT) – it is possible to encapsulate an Iron Python program as an executable but it is a big hill to climb.

This post covers the general Python app distribution problem domain way better than I can.

Python looks like a programming language for YOU to use but the infrastructure lacks more than a little when it comes to sharing. A given program can (perhaps must) be built on a plethora of pre-existing libraries that may not be apparent to your neighbour when it comes to sharing. Worse for me ‘cos I generally think command line programs stink (and I do understand why you might think otherwise) – perhaps I had better try and explain where I am coming from. Some (or maybe all) of it is a history.

My first programs were COBOL encoded in EBCDIC and punched onto 80 column cards. These cards were fed into a card reader’s “hopper”, compiled and (if all went well) the program ran. The program caused other punched cards containing data to be read and the resulting output was printed (astonishingly quickly by today’s standards) on a line printer – or just possibly output as newly punched cards ready for another process. If I got my program and data into the computer room “in-tray” by 17:00 I usually got the output the following day. The ultimate “command line”.

Then I started in a new job and was assigned to try out something called BASIC on a DEC PDP 11/70 as this was a platform that was seen as suitable for “end user” computing and someone was needed to be able to help out with development work within various departments of the business. I was handed the language specification for DEC’s Basic+ and left to my own devices for a bit. It was a pivotal moment because BASIC contains two very important keywords – INPUT and PRINT. I realised that my future programs could interact with the user while they were running. The other “tools” were a bit primitive – VT52VDUs and printer terminals hooked up to the computer room via 300 baud modems. Clunky kit but the software (in comparison to what had gone before) could soar.

At about the same time the Apple ll and Commodore PET micro computers were also introducing a new and interactive model to a wider audience (here too BASIC was a key component). A common anti-pattern of that era was to get the user to type in some data, validate it and then display the result. The “anti” bit came when the validation failed – typically this resulted in user punishment with a GOTO command taking the program back to the beginning with all data previously entered lost. This was lazy programming and (despite the miniscule available RAM) it was a personal fundamental that user data was corrected with minimal effort on the part of the human running the program.

As clerical tasks were “computerised” it was also crucial that the user remained in control. We needed to preserve the knowledge of how things were done – as otherwise the next edge case could become a major problem. Software has a duty to educate as well as undertake the drudgery of so many tasks. Fortunately terminals (and micro-computers) evolved and improved. Every advance was applied to the task of improving the quality of the interaction between user and software.

To cut to the chase, it became second nature for me to ensure that the UI (User Interface) was a critical part of every bit of software. So much so that the tools I write for myself typically have a strong UI even as this adds perceptibly to the development time.

So even if I had to explain to you why (say) I have a utility that takes an SQL table definition and outputs boilerplate class code (in C#, VB.NET or Java at choice) together with support stored procedure definitions (with variants for SQLite if required) you could copy the executable to your machine and just run it. You might not appreciate the “why” but the “how” should be obvious.

So given my fervour for strong user interfaces I have some difficulties with the current Python infrastructure. I have in mind a project that would make very effective use of all that Python list handling goodness. My problem would be in sharing any resulting software with others who might make use of it – particularly if it became a commercial proposition.

For the moment Python has to be just for me and is not for sharing. Probably confined to the Raspberry Pi – actually it will be interesting to try out the Kivy library to interact via a touch screen on that platform. So I am not finished yet that's for sure.

This week’s language is Python

2015-09-08T09:45:00.001-07:00

Warning – these notes on Python are based on very short acquaintance and may well be misleading and/or factually incorrect.

OK – coming late to the table most people would think – but then again. The last time I took a look at Python was around the time Python 3 came out and the status and functionality of the mass of C support libraries was in some doubt. Turns out that is still true in many instances with a lot of effort still seemingly going into Python 2.n and it is not clear that the world at large has collectively taken the big step to Unicode.

Anyway, I wanted to do some stuff with one of those new Raspberry Pi 2 machines and Python looks to be the lingua Franca for that platform when running Linux. So I had another look. **

The specific feature that figures in any introduction to Python is that white space (specifically line indentation) is significant and part of the language syntax. Most introductions wax on about there being no need for {curly braces} or semicolons but then again you do have to frown a bit at those unmentioned colons at the end of significant lines

>>> def myfun(arg):

. . .

>>> for w in something:

. . .

Then my reading got to len(something).

What? I thought. Surely the length of something (number of characters in a string or the count of entries in a list) would be an attribute of the relevant object. I even found Guido van Rossum’s explanation in an old Python FAQ but also found it less than convincing. I looked further and found there were at least 60 language features implemented as methods. Wow! I thought this is BASIC born again. There are even Input and Print methods. Then I looked at how such methods as len() were implemented – and then I think I reached an understanding.

Each object type (tricky to get the terminology right) where a length measurement makes sense implements a method called __len__() that will be used automatically by the language len() method. Programmers can implement the __len__() method on their custom objects confident that the language’s len() method will act as a standard interface to that functionality. In a language envisioned for code sharing (and multiple standard libraries) this is actually a powerful concept. You don’t have to know how (say) an unknown object implements a readable string representation of itself – you can just use str(object) with confidence.

So it looks like the BASIC style language methods can be extended to include new types of object which is a massive step forwards.

Python is also big on Lists. I suspect that languages influential in the design of Python included Prolog, Smalltalk and Lisp (I might have said a Lispy JavaScript but the languages while not quite contemporaneous were both born around the same time). No surprise then that Python is a very popular language in AI research. There is nothing in Python Lists that you can’t do with (say) C# but the language is rich in list manipulation functionality “out of the box”.

I love the fact that you can use expressions to create and populate a list:

>>> squares = [x**2 for x in range(10)]
>>> squares
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

And range() itself is a fascinating and flexible tool for controlling loops.

Tuples look interesting as immutable collections of heterogeneous elements although I have yet to work out how a mutable element (like a list) might fare inside a tuple.

Sets are another specialised form of list with support for set operators like union and intersection. Takes me back to college maths. Can’t wait to see what I can do with these. Curley braces do turn up in the language here and then also with dictionaries.

Dictionaries are available and look well supported and I have always liked the “associative array” concept

Functions are first class objects and I greeted the huge range of options for passing arguments into a function with a lot of interest. There are potential traps for the unwary here I think but also a flexibility not available anywhere else. Lambda’s as well of course.

Built in JSON serialisation and de-serialisation supports the storage and transfer of objects and the capacity to execute strings as code opens up the opportunity to implement a DSL (Domain Specific Language) among other opportunities. Then there are Generators (see http://stackoverflow.com/questions/102535/what-can-you-use-python-generator-functions-for) to help scale data presentation and high data volumes.

Classes have similarities to the equivalent JavaScript objects. They can contain the equivalent of static variables, instance variables, attributes and methods. Instance variables can be created and manipulated on individual instances of a class. Scope and the order that objects are defined or assigned can impact upon the meaning of the code.

Classes support inheritance (including multiple inheritance) with the ability to override base class methods. Operator overloading is also supported.

The construction and importing of code modules (which then act as name spaces) is an effective mechanism for code structure and code sharing.

Python runs into a bit of a steep incline (not a wall) when it comes to complex interactions with the user. It is fundamentally a scripting language and is not primarily intended to maintain a GUI (Graphical User Interface). KiVy is going to be bigger on the Raspberry Pi with the newly announced touch display. For PCs Iron Python is an open source community project that integrates the Python language with .NET might well be one workaround for more complex projects and with Visual Studio 2015 (community edition) being free for a lot of projects and developers it is a viable option.

When you have been working in JavaScript for a few days and the code is really starting to flow then this is partly because you have erected a good mental model of the variable and method scope in play. I strongly suspect that constructing and maintaining a similar mental model for a large Python codebase would be essential – there are some similarities but also some key differences. Getting “in the groove” might take a little effort.

None of this is intended to act as any sort of Python introduction – rather the intention is to communicate a little of the enthusiasm I am feeling for the language now I have taken a proper look. Python is different (a big bonus), clever and very capable. Now all I have to do is learn how to write effective code in this language – a big step on from just assembling an overview of the functionality after all. So the next side-project is going to be in Python.

** plan is to have two SD cards so I can “dual boot” one of the Linux variants available for the Pi and Windows 10 – should be fun. I also see Python runs on Windows OIT Core as well as on the usual Linux variations.

Eating the Browser cake

2015-09-01T07:18:00.000-07:00

That side-project again. One of the primary outputs of the app is data and text collected from web sites in (mostly) HTML format. This can most simply be wrapped to create an HTML document and displayed in the WebBrowser control made available by Visual Studio. This works well up to a point – the WebBrowser control is a “wrapper” for the Internet Explorer COM object but only exposes a limited set of events and properties. There is also an issue with multi-threading – using the WebBrowser control makes it very difficult to update the UI thread from (say) a BackgroundWorker.

I was happy with the HTML presentation within the WebBrowser control but wanted to handle any user clicks on links by pushing the link off to the default browser on the relevant machine. An external link could require a whole host of support facilities not enabled by the WebBrowser control so this made sense as well as (probably) reflecting the expectations of any user.

So, how to push responsibility to those external links off to the user’s preferred browser. Turns out to be easy to handle in the Navigating event:


private void webViewer_Navigating(object sender, WebBrowserNavigatingEventArgs e)
{
    if (!hasStarted)
    {
        hasStarted = true;
        return;
    }
    e.Cancel = true;
    var startInfo = new ProcessStartInfo
    {
        FileName = e.Url.ToString()
    };
    Process.Start(startInfo);
}

The bool pageShown is set false before loading the original page in the WebBrowser control and this is set true by the first Navigating event resulting from that page load. Subsequent Navigating events before the pageShown value is reset are pushed off to the default application (a web browser certainly) that handles URLs.

This worked fine up until the HTML content being displayed contained an <iframe> tag. While loading graphics and scripts from remote servers caused no issues filling the content of an <iframe> triggered a Navigating event. So I needed to know if a Navigating event was being triggered by an <iframe> requesting content as this may happen after the main content had loaded and the DocumentCompleted event fired (so I could not reliably re-set the pageShown Boolean within that event handler.

StackOverflow is your friend here of course. Questions like “possible to detect whether an iframe is loaded in the navigating event” hit the nail on the head. This starts a trail that initially leads to a March 2006 Code Project post by Jeroen Landheer http://www.codeproject.com/Articles/13598/Extended-NET-2-0-WebBrowser-Control#CreateIWebBrowser2 but also mentions the BeforeNavigate2 event thus exposed and the opportunity to insert a test into the BrowserExtendedNavigatingEventArgs class.

There is also this http://www.codeproject.com/Articles/18935/The-most-complete-C-Webbrowser-wrapper-control CodeProject article from May 2007 which is worthy of investigation if you need to follow a similar path.

It looks like the Windows System SHDocVw.dll exposes a whole range of additional interfaces and events for IE not exposed by the WebBrowser control but getting a handle on them takes a little effort to say the least – hence the projects to sub-class the WebBrowser in the two CodeProject articles mentioned.

Anyway – the only snag remaining is that the standard WebBrowser Navigating event fires before the BeforeNavigate2 event which is slightly counter-intuitive. So the trick is to move the functionality from the Navigating event into the WebBrowserExtendedEvents class BeforeNavigate2 event by adding a Boolean property to that class for the main form to notify the start (and end) of the main document loading.

As an aside, I did wonder if there were alternatives to the provided WebBrowser control. Turns out there have been projects to “wrap” WebKit and FireFox browsers but what looks like the most able and up-to-date project provides a wrapper for the Chrome browser and the GitHub repository is here https://github.com/cefsharp/CefSharp/wiki . However if you were expecting a nice Visual Studio control to “drop onto” a form then you might be disappointed. I think the idea is that you will want to wrap the functionality in a custom control of your own perhaps supporting tabs and other browser flummery. The simplest way to add the CefSharp control to your form is programmatically in the form class constructor thus:


private CefSharp.WinForms.ChromiumWebBrowser mBrowser;
public Form1()
{
    InitializeComponent();
    mBrowser = new CefSharp.WinForms.ChromiumWebBrowser("http://www.hanselman.com/blog/")
    {
        Dock = DockStyle.Fill,
    };
    this.Controls.Add(mBrowser);
}

It works and seems very fast and responsive. The documentation currently takes the form of “read the code” so it was not immediately obvious I could solve my problems by switching the underlying browser but at first glance it did look possible. What stopped me spending too much time here was the requirement to build your project against the 32bit or 64 bit version. While there are fewer 32bit PCs out there it still represents an additional issue should this project ever be shared with others even informally.