Justin's Ramblings

erlang hadoop ikiwiki life meta pylons python tech xen

Welcome to my blog. Have a look at the most recent posts below, or browse the tag cloud on the right. An archive of all posts is also available.

RSS Atom

I while back I got fed up with trying to find a decent spinner gif for some AJAX pages. I gave up and just wrote one using ascii. This has the benefit of always fitting in with the page style and users font settings.

do_dotspinner = function (scope)
{
    var spin_element = function (el)
    {
        max = 5;
        var text = el.innerHTML
        if (text.length==max)
            text="";
        text += ".";
        el.innerHTML = text;
    }

    var els = $(".dotspinner",scope);

    if(!els.length) return;

    els.each(function(){
        spin_element(this);
    });

    var repeat = function(){
        do_dotspinner(scope);
    }

    setTimeout(repeat, 200);
}

download file "/ramblings/files/js/dotspinner.js"

And since I doubt it will work if I inline it into a blog post, here is a demo.

I've been trying to make it work like an actual plugin, but I haven't been able to get that to work yet.

I should be able to make something like this work:

$(".dotspinner").dotspinner();

But I need to figure out how to tell when an element goes away

For example, if I have

p=$("#foo");
//and do this:
$("body").html("Hi");
// then these still works like nothing happend:
p.text();
$(p).text();
//even though if I do this again, it is gone:
p=$("#foo");
//The one thing I know will work, but is probably wrong, is this:
p = $('#' + p.attr("id"));

Tags: jquery js tech

web development would be a lot easier if

Posted Thu 01 Jan 2009 09:27:43 PM EST

Web development would be a lot easier if the W3 would standardize some new Widgets that browsers have to support. I've noticed that many Javascript frameworks these days exist primarily to implement standard GUI widgets for the web. Boring GUI widgets like:

Combo Boxes (see example)
Tabs
Trees
Grids
Dialogs
Menus
Tool tips
etc etc

My problem isn't that these widgets aren't availabile in web applications, it is that a hundred different projects are all re-implementing the same thing. At what point should something so basic like a combo box just be added to HTML?

I guess there is always HTML6.

Tags: tech

tech vision

Posted Thu 04 Dec 2008 08:14:46 PM EST

One thing I've noticed recently is a complete lack of vision when it comes to technology in the future. This is really obvious when you read those "What the world will be like in 2000" articles from 1950.

One common theme is simply making things bigger or smaller.. Bigger when it comes to machines, buildings, airplanes. Smaller when it comes to computers and other gadgets.

I remember one that talked about a revolutionary mail/news delivery system.. something like pneumatic tubes. What I've noticed is that when people have tried to predict the future, they tend to come up with new ways for using existing technology, but not new technology. Another example: every future prediction story from 100 years ago always talks about flying cars, but never something like teleportation. A flying car is something we might see in the future, but to me, that is not futuristic. A system that delivers a newspaper to your door using pneumatic tubes might have been cool in 1950, but today I read the news from around the world on my cell phone.

Instead of thinking of new ways to solve the same problems(delivering a physical newspaper to someone) we should be taking one more step back and thinking of new problems to solve. The problem shouldn't be "How to deliver a newspaper to someones door", it should be "How to deliver the news to someone". If you think about it, this is also the same problem the recording industry ran into. Too much time spent thinking about ways to sell people shiny disks, not enough time spent thinking about how to sell people music.

Not to pick on Russel Coker, but I think he falls into the same trap:

In 2020 a device the size of an iPaQ H39xx with USB (for keyboard and mouse) and the new DisplayPort [5] digital display interface (that is used in recent Lenovo laptops [6]) would make a great PDA/desktop

While I think such a device would be cool, I hope that in 2020 we are not still using keyboards and mice and hardwired displays.

Here are some ideas for future portable computers(if such things even exist, we may just wirelessly tap into a global grid.. who knows)

There have been a lot of attempts at voice input, and I don't think it will ever work. I can type a lot faster than I can talk, and I can think even faster. I think the future in input devices is going to involve some kind of biofeedback input.. at least to replace mice.

We are already starting to see the development of tiny embedded projectors. I think all portable devices are going to have them within a few years. Even more futuristic would be something like a wireless high resolution heads up display embedded in a contact lense. Actually, that isn't nearly futuristic enough.. I'm sure someone will invent a device that meshes directly with your brain so you do not even have to read a display.

Tags: tech

odd nmap timings

Posted Fri 22 Aug 2008 10:02:33 PM EDT

Back story

A section on a web application I have pings (using a background AJAX request) a list of IP addresses. Most of the time all of these adresses are up, sometimes one or two of them are down. One day I noticed that if all of them were down, nmap would take much longer to ping them all.

The odd part

Lets ping 19 addresses on my home network, none of which exist.

justin@latitude:~$ time nmap  -sP 192.168.5.2-20 

Starting Nmap 4.53 ( http://insecure.org ) at 2008-08-22 21:56 EDT
Nmap done: 19 IP addresses (0 hosts up) scanned in 4.072 seconds

real    0m4.081s
user    0m0.068s
sys     0m0.004s

Ok... now lets add the routers address, which is pingable.

justin@latitude:~$ time nmap  -sP 192.168.5.1-19

Starting Nmap 4.53 ( http://insecure.org ) at 2008-08-22 21:58 EDT
Host router (192.168.5.1) appears to be up.
Nmap done: 19 IP addresses (1 host up) scanned in 2.258 seconds

real    0m2.259s
user    0m0.048s
sys     0m0.008s

Notice anything odd?

I have experimented with the usual host timeout and max rtt time options, but I am not sure what the problem is. As soon as I get a chance I will look into the code. I am not sure if it is a BUG or just user error. A simple strace of the two commands show much different 'select' behaviour.

Tags: tech

Python Evolution: From Script To Program

Posted Sat 21 Jun 2008 11:18:12 PM EDT

The Evolution of a Python Programmer is funny, but it only covers one aspect of programming. Many times I will see code that is fine from a CS point of view, but absolutely horrible when it comes to program structure and module organization.

You often see people saying things like "Hello World in python is just 'print "Hello World"'", and that is true. It is very easy to get started writing python, but if you don't structure your modules correctly, you will be in a world of pain later on. It is something that can be hard to explain, since the results in the short term are the same, and it may not be clear at first why one way of doing things is better than the other.

Instead of Hello World, let's take the example of a program to get stock quotes. The actual implementation here is not relevant, pretend it contacts a web service or database or something.

A common case is the "python script". I HATE python scripts. "script" almost always ends up being a single file with no entry points, no main function, and mixes IO with logic.

s = raw_input("symbol:")
if s == 'MSFT':
    print 'price=', 28.23
elif s == 'GOOG':
    print 'price=', 546.43

The first step in fixing this is to define an actual function. Now you can import the module and run get_price().

def get_price():
    s = raw_input("symbol:")
    if s == 'MSFT':
        print 'price=', 28.23
    elif s == 'GOOG':
        print 'price=', 546.43

The (hopefully) obvious problem with this is that the IO is mixed in with the logic. What if you wanted to get the stock price for 1000 stocks and output a nice summary? This next version is slightly better, here the input is a proper parameter, but you still have no control over the output. You could get your 1000 quotes, but you would have no way to report on the output. Again, this should be obvious, but I come across code that does this way too often.

def get_price(s):
    if s == 'MSFT':
        print 'price=', 28.23
    elif s == 'GOOG':
        print 'price=', 546.43
###
if __name__ == "__main__":
    s = raw_input("symbol:")
    get_price(s)

The first respectable version adds a main() function that handles the input and output. The main function should also get the stock from the command line arguments, rather than interactively. I think you tend to see things like this more often from windows users, who like to double click on things rather than run them from a shell. You could probably write a whole book on this subject though :-)

def get_price(s):
    if s == 'MSFT':
        return 28.23
    elif s == 'GOOG':
        return 546.43
###
def main():
    s = raw_input("symbol:")
    print 'price=', get_price(s)

if __name__ == "__main__":
    main()

The final steps are to make a proper python package out of this module, but I'll save that for a later post.

Tags: python tech

erlang basic distributed application

Posted Mon 09 Jun 2008 06:15:31 PM EDT

Erlang with OTP is a fairly powerful framework for creating distributed redundant applications. The basic gen_server behavior can easily extended to create a redundant server with built in failover. With Mnesia you also get a replicated Database.

I've been trying to figure out how exactly this is supposed to work, so I've been working on a quick application to demonstrate this. It's nothing fancy, just a simple set(k,v) and get(k) API.

The files are available as a tarball from here ddict.tgz

-module(ddict).
-behaviour(gen_server).
-export([start/0,stop/0,terminate/2]).
-export([init/1, handle_call/3, handle_cast/2,handle_info/2]).
-export([create_schema/0]).
-export([get/1,set/2]).


-define(GD,{global, ddict}).

-include_lib("stdlib/include/qlc.hrl").
-record(rec, {key, value}).


init_mnesia() ->
    mnesia:start(),
    ok = mnesia:wait_for_tables([rec], 2000).

init(_Arg) ->
    process_flag(trap_exit, true),
    io:format("dict server starting~n"),
    init_mnesia(),
    {ok, []}.

start() ->
    gen_server:start_link(?GD, ddict, [], []).

stop() ->
    gen_server:cast(?GD, stop).

terminate(Reason, State) ->
   io:format("dict server terminating~n").

%"model" methods
do_get(Key) ->
    Res = mnesia:dirty_read({rec, Key}),
    case Res of 
        [] -> undefined;
        [Rec] -> Rec#rec.value
    end.

do_set(Key, Value) ->
    F = fun() ->
            Row = #rec{key=Key, value=Value},
            mnesia:write(Row)
        end,
    {atomic, ok} = mnesia:transaction(F),
    ok.

%"controller" methods
handle_call({get, Key}, From, State) ->
    Rec = do_get(Key),
    {reply, Rec, State};

handle_call({set, Key, Value}, From, State) ->
    Rec = do_set(Key, Value),
    {reply, Rec, State}.

handle_cast(stop, State) ->
    io:format("ddict server stopping~n"),
    {stop, normal, State}.

handle_info(Info, State) ->
    {noreply, State}.


%"client api" methods
get(Key) ->
   gen_server:call(?GD, {get, Key}).

set(Key,Value) ->
   gen_server:call(?GD, {set, Key, Value}).


create_schema() ->
    mnesia:create_schema([node()|nodes()]),
    mnesia:start(),
    %this is defnitely wrong
    lists:foreach(fun(N) ->
        io:format("starting mnesia on ~w~n", [N]),
        rpc:call(N, mnesia, start, [])
    end, nodes()),
    mnesia:create_table(rec, [
        {disc_copies, [node()|nodes()]},
        {attributes, record_info(fields, rec)}
    ]).

download file "main gen_server file"

The key to this server being distributed is the use of {global, ddict} as the server name, instead of {local, ddict}. This enables other nodes in the cluster to see this server.

do_get() and do_set() are the "model" like methods that deal with mnesia. handle_call defines the gen_server api. get() and set() are helper functions that call the remote gen server. If there was more to this module, it would be a good idea to put these methods in separate modules.

The one thing I am not sure about is the create_schema() method. I'm sure there is a propper way to initalize mnesia on a cluster, I just have no idea what it is yet :-)

To make this into a propper gen server the supervisor and application needs to be defined with the following three files:

-module(ddict_sup).
-behaviour(supervisor).

-export([start_link/0]).
-export([init/1]).

start_link() ->
    supervisor:start_link(ddict_sup, []).

init(_Args) ->
    {ok, {{one_for_one, 10, 60},
          [{ddict, {ddict, start, []},
            permanent, brutal_kill, worker, [ddict]}]}}.

download file "/ramblings/files/erlang/ddict/ddict_sup.erl"

-module(ddict_app).
-behaviour(application).

-export([start/2, stop/1,go/0]).

start(_Type, _Args) ->
    ddict_sup:start_link().

stop(_State) ->
    io:format("ddict server terminating~n"),
    ok.

go() ->
    application:start(ddict).

download file "/ramblings/files/erlang/ddict/ddict_app.erl"

{application, ddict,
[
    {mod, {ddict_app,[]}}
]}.

download file "/ramblings/files/erlang/ddict/ddict.app"

To get erlang to start this application on boot, a config file for each node needs to be written:

[{kernel,
  [{distributed, [{ddict, 3000, [one@media, {two@media}]}]},
   {sync_nodes_optional, [two@media]},
   {sync_nodes_timeout, 5000}
  ]
 }
].

download file "/ramblings/files/erlang/ddict/one.config"

[{kernel,
  [{distributed, [{ddict, 3000, [one@media, {two@media}]}]},
   {sync_nodes_optional, [one@media]},
   {sync_nodes_timeout, 5000}
  ]
 }
].

download file "/ramblings/files/erlang/ddict/two.config"

To create the initial database I ran the ddict:create_schema method, which I'm sure is completely incorrect, but it works:

erl -sname one -config one.config
erl -sname two -config two.config

(one@media)1> ddict:create_schema().
starting mnesia on two@media
{atomic,ok}
(one@media)2> mnesia:info().
...
running db nodes   = [two@media,one@media]
disc_copies        = [rec,schema]
[{one@media,disc_copies},{two@media,disc_copies}] = [schema,rec]
...
ok

download file "/ramblings/files/erlang/ddict/create_db.txt"

Once that is done, the application can be started with

erl  -pa . -sname one -config one.config -s ddict_app go
erl  -pa . -sname two -config two.config -s mnesia start -s ddict_app go

I have to start mnesia separately on the second VM because I haven't yet figured out how mnesia should be started when dealing with distributed applications. mnesia needs to be running on both nodes, but not the ddict application itself.

Once it is running, you can call ddict:set("Foo","bar") and ddict:get("Foo"). You can also kill either VM, and it will restart the server after 3 seconds on the other node.

Comments here

Tags: erlang tech

using nmap for network monitoring

Posted Fri 30 May 2008 10:34:40 PM EDT

The problem

You need to know if any of 900 IP addresses are unreachable. You also need to know this within about a minute of any outages. Nmap is primary a security tool, but it can be very helpful when it comes to monitoring as well.

fping

For years I used fping for this, here is an example of what it can do:

$ wc -l ips.txt 
900 ips.txt
$ time fping < ips.txt 
...
real    0m41.347s
user    0m0.028s
sys     0m0.248s

Not too bad.. 41 seconds to poll 900 devices. It actually seems to finish at around 35 seconds, and then sits there for a bit before exiting.

nmap

Now lets try with nmap. Nmap needs to be ran as root to allow it to send icmp packets, otherwise it will use connect(). In my tests it is actually faster when running in tcp mode, but some devices only respond to ICMP. (It would be best for security to put this into a nmap_ping helper script and put that in sudoers instead of allowing all nmap commands to be ran as root. It is probably also possible to use the capabilities system to just allow a normal user to send ICMP packets.)

$ time sudo nmap -n -sP -PE -iL ips.txt
...
real    0m3.961s
user    0m1.072s
sys     0m1.780s

Not bad at all, about 10 times faster than using fping!

Note in these examples, all of the addresses are pingable, so timeouts and retry times do not come into play. My monitoring system maintains separate lists of the reachable and unreachable devices, and pings them from different processes. This prevents unreachable devices from slowing down the normal process of making sure everything else is working. Currently the time between pings to a single device is about 8 seconds.

Tags: tech

building pig on debian

Posted Thu 15 May 2008 11:52:12 PM EDT

I've been playing with Hadoop and Pig. It is some really neat technology.

I had a bunch of trouble getting pig to build though, it seems that this error from ant:

Could not create task or type of type: jjtree

Is caused by missing the 'ant-optional' package.

Tags: hadoop tech

nin theslip

Posted Fri 09 May 2008 01:34:51 AM EDT

This is so beyond awesome.

01:31:48 (1.13 MB/s) - `nintheslipmp3.zip saved [90209737/90209737]

BB.Net / Justin's Ramblings

setInterval and setTimeout

JQuery Dotspinner

web development would be a lot easier if

tech vision

odd nmap timings

Back story

The odd part

Python Evolution: From Script To Program

erlang basic distributed application

using nmap for network monitoring

The problem

fping

nmap

building pig on debian

nin theslip