Kevin's Weblog

Just to see if I can make my head explode, I’m also learning a bit of Erlang at the moment. In fact, I just finished writing my first Erlang program. (Or, at least, the first program I started. I did write a Hello World in the middle, just to make sure I wasn’t totally confused about some issues with the compiler and run time.) It’s a simple URL checker, that checks the status of any url you send it. My predominant positive first impression is that it’s pretty cool that you can write something like this so easily upon just learning the language, and that the finished product is short, simple, and readable. Here it is:

-module (urlchecker).
-export ([main_loop/2, fetch/2]).
-import (ibrowse).

main_loop (Seen, Pids) ->
    receive
        {url, Url} ->
            case sets:is_element(Url, Seen) of
                true ->
                    io:fwrite ("~s seen previously.~n", [Url]),
                    main_loop(Seen, Pids);
                false ->
                    Pid = spawn(urlchecker, fetch, [self(), Url]),
                    main_loop(sets:add_element(Url, Seen), [Pid|Pids])
            end;
        {reply, Pid, Url, RC} ->
            io:fwrite ("~s : ~s~n", [Url, RC]),
            main_loop(Seen, lists:delete(Pid, Pids));
        done ->
            case Pids of
                [] ->
                    io:fwrite ("Exiting~n", []);
                _ ->
                    io:fwrite ("Waiting for children to finish~n", []),
                    timer:send_after (1000, done),
                    main_loop(Seen, Pids)
            end
    end.

fetch(Pid, Url) ->
    case ibrowse:send_req(Url, [], get, [], [], 30000) of
        {ok, Status, _, _} ->
            Pid ! {reply, self(), Url, Status};
        {error, Reason} ->
            io:fwrite ("Error fetching ~s : ~w~n", [Url, Reason])
    end.

That’s a total of 36 lines, including copious debugging output, graceful shutdown, and rudimentary politeness. And, of course, concurrency. So, this is a pretty high-level language.

(Looking over the code, you might wonder why I use sets for the seen URLs and a list for the pids. The answer is, I wrote the pid handling first and figured that the list would probably stay smallish, so it didn’t need to be particularly efficient. Only when I added politeness did I go to look for an efficient data structure. Of course, for real use, you’d want something like a timeout cache, not a set. I also suspect the child pid tracking might be better done with process groups.)

So, that’s the good. The bad is that the documentation is fairly minimal, and the error messages make Haskell look good. Some examples I encountered:

{"init terminating in do_boot",{undef,[{set,new,[]},{urlchecker,main,0},{init,start_it,1},{init,start_em,1}]}} — That’s a fancy way of saying “you made a typo!”, i.e., the module is ‘sets’, not ‘set’.

Error in process <0.75.0> with exit value: {undef,[{urlchecker,fetch,[<0.73.0>,"http://kevin.scaldeferri.com/adsf"]}]} — That means, “you spawned a process using a function you didn’t export”. I find this particularly annoying because I don’t want to export “fetch”, since it’s an implementation detail.

** exited: {{badmatch,<0.123.0>},[{erl_eval,expr,3}]} ** — That means, “you tried to redefine a variable you already assigned to”. I ran into this in the normal cycle of making a change in the editor, then reloading it in the REPL and running some test code. It took me a while to realize that the problem wasn’t the code in my module, it was that I was reusing the same variable in the REPL.

Finally, when I downloaded ibrowse, I just got a directory of files. There was no configure script, no install script, no install directions, and no top-level Makefile. I found a Makefile in the src directory, and used it to compile the code, but it didn’t have an install target. It turns out that you can just copy the src and ebin directories into /usr/local/lib/erlang/lib by hand, but I had to go to #erlang to find that out.

I’m hoping that I’m past the nasty part of the learning curve now. We’ll see. I do really like the conciseness of the language, and how quickly you can prototype something moderately sophisticated. And, in theory, I like the idea that you can trivially distribute something like this across multiple nodes, although I haven’t gotten to seeing how that works in practice. I just wish it was a little more helpful when you mess up.