Dev Jar

After a long hiatus from writing on this blog, I’ve decided to start writing again; about tips I come across as I work on my new startup, Leven Labs.

Normally, when using proxy_pass in Nginx you define a single upstream or maybe a hardcoded list of upstream’s. What if you use SkyDNS, follow my tips for effectively running a microservice architecture, and don’t have a hard-coded list of upstream endpoints? That’s not quite as easy.

At first, I thought I could provide a hostname and it would look it up for each request:

proxy_pass dank.services.admiral

But instead, that resolves the hostname once at startup. In order to resolve the hostname for each request, you need to use set.

set $backend_upstream "http://dank.services.admiral:8333";
resolver 10.0.0.5 10.0.0.6;
proxy_pass $backend_upstream;

Note you’ll also need to provide a list of resolvers in order to look up the hostname. You should set these to your SkyDNS addresses.

Having random file access problems with your CIFS share? The error I was specifically seeing was from Apache: “Cannot allocate memory: file permissions deny server access”. Luckily the solution to this is easy and doesn’t even require a restart!

If you’re not comfortable modifying the registry, this isn’t for you.

Open up the registry and find “HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\LanmanServer\ Parameters\Size”. Change the value to 3 (Decimal). This allows for maximum throughput for file shares.

Then open up an Administrator command window and type “net stop srv”. Hit “Y” at the prompts to stop other services and then type “net start srv”. You will have to unmount and remount your Linux share, but otherwise, everything should be fine!

Ivan Pepelnjak has posted a few blog posts throughout the years about IPv6 multihoming: Small-site Multihoming in IPv6: Mission Impossible? and IPv6 Multihoming Without NAT: The Problem. I have 2 upstream links (ISPs) at home and I’m currently using stateful tagging to keep traffic flowing through the correct link.

Ivan has been talking a solution, not using NAT, for state-less multihoming. The solution he proposes is assigning multiple IPv6 addresses for each host, which seemingly can be achieved by assigning 2+ prefixes for a given interface in radvd, for example. The only problem with this is, when a link goes down, you need to have everyone behind the router know to remove that one prefix from their connections. You could limit the duration of the downtime by setting your lifetime to something small (say 5 minutes), but clients still will be partially down until they sync up. Also, you need to have all the hosts modify their system to actually send traffic over both of the connections. If you have another connection just as a backup, then you don’t need to worry about that, but my goal is to use both upstream links at all time and load-balance traffic on both.

I’ll be playing around with this at home over the next few weeks and I’ll post updates with the results. If anyone has any suggestions, let me know in the comments!

CloudFlare recently posted a blog post about how they do load balancing inside the data center.

It makes sense to use BGP for server fail-over, since you can configure a server to stop publishing it’s routes when it dies. But I fail to see how it’s effective at load balancing. Let’s say you have each external IP/range associated with a set of servers. Since your primary router is going to choose the “shortest path”, that means that all traffic to 1.1.1.1 will always go to Server A, until another server has a higher weight/priority or if Server A stops publishing its route. If 1.1.1.1 starts to get a lot of traffic that particular server will get hammered with traffic and it won’t be “load balanced”. When this happens there are a few things that could happen. Server A could realize that it has a large load and lower it’s weight, which would cause another server to start getting ALL the traffic for 1.1.1.1. If Server B could realize that Server A is getting hammered it could raise it’s weight because it has no traffic, but once again, it will start to receive ALL traffic for 1.1.1.1.  Finally, if the DNS server knew that Server A is getting hammered at 1.1.1.1 and specifically, domain xyz.com, it could change the address for xyz.com to 1.1.1.2, effectively moving the traffic to a better server. If that traffic is too insane to be handled by any single server/endpoint, then you’re out of luck.

However, there are possible solutions. You could use BGP to point IPs at different hardware/software load balancers to distribute the load further for an IP, which seems like the most desirable solution. Or you could rely on DNS round-robin, which can be used, but notably isn’t as effective, on domain-based setups. See updates below for other solutions brought to my attention.

If you have a lot of domains/IPs and you’re running a proxy service like CloudFlare, then you might be able to pull off internal BGP-only balancing. But for the rest of the Internet, it doesn’t make a lot of sense. I’d suggest using load balancers, which are easier to setup, anyways.

(Disclosure: I have been using CloudFlare for over a year now.)

Update: It’s been brought up that you could use ECMP but there can be issues with connection-based protocols, like TCP, if you’re not using flows. If you’re just serving UDP content, then that would be a good solution for you.

I was trying to change my virtual machine to use a “Bridged Adapter”, but quickly realized that I could not under my Windows 7. Other Windows 7 installs show the host adapters to bridge, but on this particular setup, there were none listed.

The solution involves editing the registry in Windows 7 to raise the maximum number of device network filters. I’m assuming you already know how to open the registry editor and if you don’t, you probably shouldn’t be editing it. Find the key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Network\MaxNumFilters

and raise it to something higher than 8 (max of 24). Make sure you choose “Decimal” before closing and then uninstall VirtualBox, restart your computer, re-install VirtualBox and enjoy!

I was compiling Node on a new CentOS server and ran into an issue when compiling:

Build failed:  -> task failed (err #2): 

    {task: libv8.a SConstruct -> libv8.a}

Thanks to this bug report, I was able to solve the problem relatively easy. Assuming that gcc and g++ are in the /usr/bin directory, I ran:

make distclean

export CC=/usr/bin/gcc

export CXX=/usr/bin/g++

./configure

make

Viola! Everything worked from there.

I see a lot of PHP code that uses regex’s to validate emails. PHP has a built in function, filter_var, that validates a variable with your choice of a bunch of different filters. So instead of a regex, you could use:

filter_var($email, FILTER_VALIDATE_EMAIL)

There are known issues with this method if you’re using PHP 5.2.x version less than 5.2.14 or a 5.3.x version less than 5.3.3, but as long as you have something greater than that (which hopefully most people do by now) you’re good!

Node.js allows you to manually run the Garbage Collector when you think it is necessary, instead of doing it on idle ticks. This is directed at more advanced JavaScript programmers and should only be used when the normal algorithm causes issues.

Start Node with the flags –nouse_idle_notification and –expose_gc, and then when you want to run the GC, just call global.gc().

If you have large objects in memory (2+ million hash entries, etc), the GC can be a huge CPU hog because the current version of V8 does not continue the GC where the previous GC cycle left off. Fortunately this was recently fixed (V8 Bug #1458) in V8, but it will take a while for this to make its way into Node.js. Ben has also stated that they’re “revisiting the [‘GC on idle’] algorithm in an upcoming release” (source).

If you frequently connect to a particular host via SSH, you might find it easier to use an alias for that host. This can easily be achieved by modifying ~/.ssh/config (on Windows its %USERPROFILE%.ssh\config) and adding the host configuration like so:

Host alias1

        Hostname myhost.com

        Port 22

        User james.hartig

        IdentityFile ~/.ssh/keys/id.rsa

The Port and IdentityFile params are optional but can be used to change the default port and add a SSH key if you don’t use Pageant

Pageant saves you a lot of hassle with SSH keys by running in the background and hooking into ssh sessions run by PuTTY’s ssh client. After installing git, all you have to do is add a environment variable pointing to the plink.exe file in the PuTTY directory and restart. If you don’t have Putty, download it here.

To add the environment variable, you need to open System Properties and then choose “Advanced system settings”. Then click “Environment Variables” at the bottom. In the “System variables” section, hit “New…”. 

Name: GIT_SSH

Value: C:\Program Files (x86)\PuTTY\plink.exe

Change to location of where you installed PuTTY.

Unfortunately, at this point I had to restart my computer for git to start using plink. Before you can actually connect to your remote server, you must connect to it in PuTTY and accept the signature. Open PuTTY type in the hostname (github.com) and leave everything else at the defaults. Then click “Open” and when prompted click “Yes”. Then you can close the new window it opened and you’re good.

To actually use Pageant. Launch it via the Start Menu (Putty -> Pageant) and then in the taskbar find the little computer icon and hit “Add key”. Then add your PuTTY private key. If your private key is not a PuTTY private key, open it in PuTTYGen (Conversions -> Import) and then save it as a private key (with extension .ppk). That should be it!

If you’re server uses a port besides 22, then you need to configure it in %USERPROFILE%.ssh\config. Add something like:

Host myownhost.com

        User git

        Port 220

Having an issue getting the built-in PPTP client working on your DD-WRT router? It is because the build script generates a syntax error in the config. Put the following code in your Startup script under Commands:

fork_pptp_change() {
#wait until pptp client is initiated
pptpfile=/tmp/pptpd_client/options.vpn
while [[ ! -f $pptpfile ]]
do
  sleep 2
done
sleep 10
#fix options vpn file
sed -i '/logfd2/d' /tmp/pptpd_client/options.vpn
sed -i '/debug/d' /tmp/pptpd_client/options.vpn
sed -i '/dump/d' /tmp/pptpd_client/options.vpn
#stop the client
/tmp/pptpd_client/vpn stop
sleep 5
#restart it
/tmp/pptpd_client/vpn start &
}

#this needs to be forked or we would never have the pptp client initiated
fork_pptp_change &

Save Startup and then reboot your router. VPN should be working.

If you wish to add a route to your Windows comp for the VPN, then run the following from an admin console:

route -p add (network) MASK (subnetMask) (routerIP)

Now you’re all set!

Still having issues? Try setting your encryption options to: “mppe required,stateless,no40,no56” and possibly setting your hostname to the IP address not a host.

Yes, you read that correctly. After my previous post discussing how Grooveshark’s new HTML5 site would not work on Windows Phone 7 devices, Ben Riga, Technical Evangelist on the Windows Phone team, contacted me and offered to help. He discussed the issues with Eric Lawrence, Program Manager at Microsoft, who sent me an example of how to use postMessage to create a “proxy” for the HTTPS requests the site made. I had previously tried doing this but I was using document.write in the callstack of getting a message and it was throwing an error saying “HTTPS security is compromised”. Removing this call and using Eric’s site as a guide, I successfully made a proxy for Grooveshark’s site. Windows Phone 7 users running Mango can now use the site to listen to their favorite tunes on the go!

Are you trying to talk to Javascript on a page from a Chrome Extension? You can interact with the DOM so what is stopping you from doing:

var script = document.createElement(“script”); script.appendChild(document.createTextNode(’(’+ code +’)();’)); document.head.appendChild(script);

Does anyone have any better solutions? Obviously the above method allows for some pretty nasty XSS but it is just a starting point and I would hope that you perform some sanity checks on the code before just placing it on the page.

Only a few years ago, if you had to make an AJAX request across domains or ports, you had to use Flash or do it server-side. Since then, all major browsers (except Opera <12) have added support for CORS (Cross-Origin Resource Sharing). CORS allows you to make AJAX requests provided the server you’re requesting allows such requests. This is done through the Access-Allow-Control-Origin header. According to the spec, you should be able to provide a comma-delimited list of domains that can access your resource, but this does not work everywhere so you end up having to get the Origin header from the incoming request, validate it, and send it back the same exact way. Optionally, you can send * to allow all requests.

The browser checks to see if it can talk to your domain by making a preflight request. These should be avoided, if possible. A preflight request must be sent if you set a content-type of something other than text/plain (certain browsers might allow other ones to be set without preflight). Also, if you are not doing a GET/POST request or are setting custom headers (X-SomethingCoolHere). You should set your content-type as text/plain explicitly and make sure to do a GET or POST. Also, make sure that your library (read: Zepto) does not try to set custom headers, like X-Requested-With, as then a preflight request will be made and you must return a Allow-Access-Control-Headers header with the allowed headers.

Finally, if you’re going to use CORS in IE, make sure to use XDomainRequest, not XMLHttpRequest. It works just as well except for the few caveats I blogged about earlier.

Grooveshark’s new HTML5 web app was released with official support for Android and iPhone. Instantly, Windows Phone 7 owners complained that their platform was not supported. Last weekend, I began to look into WP7 and what changes were necessary to support the platform. Grooveshark doesn’t have any WP7 phones that I could use to test, so I used IE9 as the test platform. We were using the Zepto Javascript library for the mobile site. Zepto does not support IE9 because it uses __proto__ to extend objects, which is a hack especially for Webkit browsers, so I first attacked that and got IE9 working with the object extensions. Since the site appeared to work fine in IE9 so we announced that we now supported WP7 on Friday.

Oddly enough, I still read reports and emails of people saying that the site did not stream on their phone. I then began to debug more on the actual streaming. It seems that the AJAX calls we were making to our API were failing when they were requested over HTTPS. The Microsoft team states that it is a bug with IE8 that you cannot do AJAX calls through XDomainRequest to HTTPS pages from HTTP pages. They never bothered to fix it for IE9. The simple solution would be to just load the site in HTTPS and make all calls in HTTPS, however that is not possible right now, so I began to look for hacks to get around it. I tried making a Silverlight proxy but sadly, mobile Silverlight doesn’t support the HTML bridge needed for this. We only have 1 call to HTTPS on the site right now, but in the future, users will need to login and perform actions that require HTTPS, so this issue is quite the show-stopper.

The above issue aside, the audio tag on WP7 seemed to be making 2 requests to the audio URL and then wouldn’t play the song because the second request failed. Grooveshark does not allow a stream key to be used twice, so the second request will always fail. I spent 12 hours figuring out how to force WP7 to only make one request and in the end it came down to changing our stream endpoint from stream.php to stream.mp3. This is an easy change as we can just use a server-side rewrite to point stream.mp3 to stream.php.

In the end, we will not be supporting Windows Phone 7 until there is a solution for the HTTP<->HTTPS AJAX problem with IE9. Maybe with a future Windows Phone update the Internet Explorer team will finally fix the bug they acknowledged 2 years ago.