Planet Perl

The Perl 6 design team met by phone on 20 May 2009. Larry, Allison, Patrick, Jerry, and chromatic attended.

Larry:

• changed the time function to return a Rat
• thinking about the traits that have been bothering Jonathan and others
• have some changes to check into the spec when I'm happy with them
• thinking about the primitives we use to define use
• breaks down into load and import
• thinking of establishing compile-time keywords for both concepts
• intended so that I can import from anything acting like a module -- an inlined role, for example
• otherwise trying to keep up with the flow of IRC

Allison:

• working on the Parrot book
• changed its focus to a small, 100-page PIR book from a monolithic Parrot book
• the intent is to get something out for YAPC and OSCON
• will send out a draft for review
• will merge my changes into the repo later this week

Patrick:

• released Rakudo #17 last week
• was easy again
• 875 more tests since #16, so we pass 68% of the spectest suite
• finished implementing the root_new opcode in Parrot
• cleans up a lot of the PMCProxy issues from moving Rakudo to its own HLL
• gained half of the speed we lost from the migration
• we'll get more back as we update more places that need it
• NQP never expected anything like that
• I have to rework some it and PCT
• haven't quite figured out how to do that
• refactoring use and import in Rakudo
• the current implementation doesn't work
• will hopefully match with what Larry's putting in the spec
• it seems like the logical way to do things
• updated Rakudo's ROADMAP in docs/ROADMAP
• gives us an idea of dependencies and next tasks
• may also help people understand what blocks features they want

Jerry:

• the bonding period has ended for GSoC
• time for students to start coding
• everyone on the Perl 6 and Parrot projects is ready

c:

• fixed some memory leaks in Parrot and Rakudo
• there are still some in Rakudo, but the web examples should be able to live longer
• did more profiling
• think NFG is important for Parrot in the near term
• have some documentation to write
• have been editing the Parrot book

Patrick:

• how's the command line for Rakudo coming?

Jerry:

• I expect to get back to that

Patrick:

• the "parens build captures" decision surprised me
• what's the rationale?
• I really liked "parens mean grouping"
• maybe I haven't reconfigured my worldview yet, but it feels messy

Larry:

• when used in an argument list, it has the same effect as a capture

Patrick:

• it even works when they're used as a term

Larry:

• they still mean that you have to look at what you're binding to and decide
• am I binding this to a scalar or to an array?
• (1, 2, 3) bound to an array...

Patrick:

• I'm going to have to think about that
• the zip operator in slice context....
• is this three or one positional arguments? zip($a,$b,$c)
• how many positional arguments are in this case? zip($a,$b,$c;$d)

Larry:

• one slice
• you wouldn't want to write that

Patrick:

• what in the arg list distinguishes the use of the semicolon versus the comma
• inside of an argument list we have to recognize a variety of syntactic things
• comma, semicolon, colon, array or hash sigil, named parameters
• seems like captures need more information than just positional
• they need to store metadata about positional arguments
• I like the syntactic stuff showing up in the argument list
• but I don't want to handle them in three different ways

Larry:

• I'll have to think about that

Patrick:

• haven't figured out how to deal with slice context either

Larry:

• might say that the presence of a semicolon implies the presence of other parens
• the comma implies...
• that might be more consistent binding for a top-level list

Patrick:

• I half expected that answer
• I can see the semicolon as just a lower precedence grouping operator

Larry:

• otherwise you have a semicolon that's just not there in every other argument list

Patrick:

• assuming that, the other commas form an argument list through the infix semicolon
• an array in there means Capture of Capture of Capture
• we were about to refactor List and Array in Rakudo anyway
• the question is "Do we really have a List type now?"
• Rakudo assumes that

Larry:

• if we can unify args list with List, that's probably healthy

Patrick:

• I'd really like that
• that makes things a lot cleaner
• infix comma and infix semis now just create arglists

Larry:

• or Lists
• if you define List as "something that has out of band metadata"

Patrick:

• any more decisions that you can make about that will help our implementation
• I probably won't get around to that this week

Larry:

• we make syntactic distinctions
• we know that this is an arg list
• we treat pairs as named arguments
• we don't do that if we know it's not an argument list
• it stays positional
• that's the only distinction between an arg list and a List
• purely syntactic

Patrick:

• to summarize
• zip($a, $b, $c) has three positional arguments
• zip($a, $b, $c; $d) has two, the first of which is itself a list/capture

Yesterday I had to write some code that goes through ~700.000 datasets (seperated into 6 tables) and denormalise them (see the yet unwritten Part II). As we're using DBIx::Class, I first used it. Even though I avoided some in/deflators and used columns to only get the stuff I needed, the process took ages (~50 items per second or aprox 4 hours for the whole job). Well, 4 hours might be bearable, but this was only the Swiss dataset. The German one, which we have to tackle soon, is at least 10 times as big, and 40 hours is just a way too long runtime.

So I rewrote the core of the programm using raw DBI calls (I had several flashbacks to the 90's :-) The results were quite astounding, as I was now getting 500 items a second. 10 times faster!

And that was before koki told me to set work_mem to a bigger value. I choose 30MB, and got another speedup to ~900 items per second. Sweet!

Of course, it's completely unfair to compare DBI with DBIx::Class performance wise, because DBIx::Class is so much nicer to work with. But if speed is an issue, dropping back to raw DBI is the way to go.

A recent thread on the Mason users list reminded me of the problems I had grokking Catalyst when I first looked at it. Raymond Wan wrote "I'm skimming over the MVC part and as my system doesn't use an SQL database, I'm wondering if Catalyst is overkill??"

Those of us who know Catalyst know that this question is based on some wrong assumptions, but it's easy to see why Raymond has those assumptions. Look at the Catalyst tutorial. In the very first chapter, it's already talking about databases and DBIx::Class.

It's easy to look at this and assume that Catalyst is somehow tightly bound to DBIx::Class or SQL databases.

The problem is that the tutorial docs really need to serve two different audiences, though both audiences are Catalyst newbies.

On the one hand, you have people with relatively little web app experience. Presumably, they know some Perl, and they do a web search for "perl web application framework". Eventually, they'll get to Catalyst and start reading about it. For those people, being given a set of standards and a full walk through of Model and View (in addition to Controller) is very valuable. It gives them all the tools they need to get started on simple web apps without having to make too many choices.

The other audience is people who have some real web app development experience with something more primitive than Catalyst. These could be people coming from a Mason-based site without any real controllers, or people who used something like Apache::Template with mod_perl, or maybe they wrote their own controllers "by hand" using the mod_perl API.

Many of those folks will already have some experience with models. To refer back to Raymond, presumably his system already has some sort of model code, it just doesn't use a SQL DBMS. Those people will look at the existing Catalyst tutorial and get confused. Isn't Catalyst flexible? Then why does it look like RoR, with all my tool choices made for me?

It took me a while to realize that Catalyst, at its core, is smaller than you might think, and you can use just the pieces you like.

The very core of Catalyst is its dispatching system. Given a URI, it selects a piece of code to run. Its dispatcher is very powerful (Chained methods are great!), and with plugins like Catalyst::Action::REST, it's even better.

Along with the dispatching system, Catalyst also provides an abstraction over the typical web app request/response cycle. The Request makes it easy to look at incoming query arguments, POST data, file uploads, and headers. The Response lets you set headers and return output to the client, whether that be HTML or a downloadable file.

Catalyst also includes engines (think "environment adapters") for a number of common web application environments, including vanilla CGI, mod_perl, FastCGI, and more. These engines make sure that the Request/Response API works exactly the same in any environment where Catalyst can be deployed.

This is a huge win, since you can write your app without worrying about the deployment environment. If you're writing an app for public distribution, it gives the installing users a choice of how to deploy.

These core pieces are really the only parts of Catalyst you have to use when you use Catalyst. If you don't want dispatch and a request/response API, you don't want Catalyst.

Catalyst (really Catalyst-Devel) also includes a fantastic single-process development server. This server can be started straight from an application's checkout directory with one command. Even better, this dev server can be told to monitor all relevant files and restart itself when any of them change. Note that this is a proper restart, which avoids all the myriad problems that afflict Apache::Reload and its ilk, which attempt to reload modules in the same Perl interpreter.

Just these things - controllers, a request/response abstraction, deployment agnosticism, and a great dev environment - are enough to make Catalyst a great choice. Ignore everything else it does and you'll still have improved your development process and improved your code.

Catalyst also does some other things ...

It has a component system which has allowed people to release a whole host of useful plugins. If you look on CPAN, you'll find things like sessions, powerful authentication, dumb authentication, I18N, and much more. If a plugin does what you need, it'll save you a lot of development time.

Note that the old "Catalyst::Plugin" system is in the process of being deprecated, but the concept of pluggable components is still core to what Catalyst is. All that's changed is the way pluggability works.

Catalyst lets you have multiple views. While many apps will just output HTML via a templating system, this flexibility is great for RESTful apps that may want to output XML, JSON, and still fall back to HTML for browsers (see my REST-ForBrowsers for some help with that).

Catalyst also has configuration file handling built-in. Personally, I avoid it, because it only works within the context of the whole "Catalyst environment". That means it's awkward at best to load the configuration outside of the web environment. I always make sure that application wide setting are available for things like cron jobs. This falls into a category of Catalyst features which are not best practices, but are probably useful for people writing their first web app.

Catalyst gives you hooks for models. Again, this is something I never use, but it's another "useful for web app n00bs" feature.

There's probably many other things I've left out. The point is that Catalyst provides a very powerful set of core web app tools, and that core is actually small.

Relative to a "my way or the highway" type framework (RoR and Jifty, I'm looking at you), it's easy to port an existing application to Catalyst. In fact, using Catalyst::Controller::WrapCGI, you can wrap an existing CGI application with Catalyst, and then convert slowly over to "native" controllers.

And most importantly, you can to Catalyst without touching your model at all! Since many applications have the bulk of their code in the models (at least, they do if you're doing it right), this is a huge win.

Next step is to turn some of this rambling into doc patches. I think a section of the Catalyst tutorial aimed at folks coming from an "old school" web app background would be great, and would really help people like Raymond (and would've helped me a few years back).

Inheritance, as most of us know, is rather problematic. Specifically, it's been around since 1967 and for the past 42 years, people have been arguing about how to do it right. Some OO languages don't have inheritance (Self, Javascript). Some languages don't have multiple inheritance (C#, Ruby). Some languages have multiple inheritance with many safeguards (Eiffel) or none (Perl). Some languages just do strange things with inheritance (BETA). Even very experienced OO developers focusing on a single language will argue vociferously about whether or not a design has been done correctly (do you really give a damn about strict equivalence in overridden methods?)

The problem is ultimately that classes fulfill conflicting needs. As an agent of responsibility, a class needs to do everything a class needs to do. That sounds like a stupid tautology, but what it means is that as systems grow, classes grow. Thus, classes tend to get larger. Unfortunately, as agents of code reuse -- via inheritance or delegation -- classes should be smaller. How many times have you seen (or written) code which inherits from something because it needs one or two methods but you're pulling in a lot of extra behavior which you don't need?

This responsibility/reuse tension which leads to classes wanting to be both larger and smaller at the same time is much of the reason why inheritance has proven so problematic. Interfaces in Java and C#, along with mixins in Ruby and other languages were an attempt to decouple the responsibility and reuse needs of classes, but they've all had their problems. Roles seem to handle the decoupling of responsilibity and reuse quite nicely, but some of you may have noticed some of the disagreements that I and others have had about the proper use of roles. Why is that?

Ultimately, the disagreement boiled down to the fact that -- like inheritance -- roles serve more than one master. Roles provide behavior and roles provide an interface. Are we faced with another 40 years of arguing because we've again tried to shoehorn too much into one thing? I don't think this need be the case as, unlike classes, the different things the roles provide aren't necessarily antagonistic, but it would be nice if the different parties sat down and tried to come up with a strategy (syntax?) for implementing roles which cleanly addresses these disparate needs. Otherwise, there really will another 40 years of arguing.

It is probably not that interesting for most of the people but I have my excuses. What is more interesting - at least to me - is why others are involved.

After all there were already more than 40 people who made some contribution to Padre.

So I asked them to start writing down why are they involved. Some of them have already added themselves to the developers page and we started a separate page for translators though that one is new and has no been filled yet.

So why do I write Padre?

There is a short explanation on that page but let me bore you with a longer one here.

I have been teaching Perl 5 for almost 10 years now. Both beginner level and advanced courses. In the beginner courses the majority of the people use Windows with about 20-30 percent using Linux. Most of the Windows users use Notepad++ or a similar editor. Some of them can configure their editor to run the perl script right from the editor, others don't even know how to enable syntax highlighting for Perl 5. Some of the people on Windows don't know what the command line is but even those who know how to open the command prompt are either afraid of it or just dislike it.

Mind you these are not stupid people or anything like that. Some of them have many years of hardware design behind them. They are just used to some kind of IDEs.

Others might use Linux/Unix but many not by choice. In many cases the company they are working for gives them telnet access to an oldish Unix machine and tell them to code in Perl. They hardly know vi, they don't know how to configure syntax highlighting and they don't know how to install any other editor.

I have been using vim for many years and I can testify that it is a superb editor. Emacs is similary strong but I have not used it since university. The problem with both of them is that they are totally different from the editors most people are used to and their learning curve is long and steep. It takes several months or even years to become familiar with them. Most of the people don't want to invest that energy and I certainly don't have the time for that in a 4 days long Perl 5 course.

So I usually tell them to use whatever they are familiar with. I think I never had a student who picked Eclipse or Emacs. There were a few using vi/vim but most of the people used Notepad++ or a similar Windows based editor.

I don't think my students are too different from the average people learning and using Perl. Most of them will never write a full blown web application. Heck most of them won't have a need for object oriented coding as they write only 100-200 line long scripts in Perl.

They never get really familiar with Perl and they will always have to deal with strange code written by others. With all the other tasks they are required to do in their primary language or tool they will keep wondering what is $_ when they see it and will be surprised when they don't.

They will wonder what is the relationship between $_ and $_[0].

So my first objective was to provide a simple Notepad++ like editor with strong support for beginners or people with relatively little experience with Perl.

For example I'd like to make sure that Padre users will be able to highlight $_, press F1 and get an explanation. In case they highlight $_ from a $_[0] expression Padre should be clever enough to point this out and give an explanation on @_ and @_[0].

I knew it is a big task though I did not think it is soo big. In any case I knew I cannot handle this alone, especially as I am not such a good programmer. So from the very beginning I tried to make it interesting, accessible to experienced Perl 5 developers and in turn they made it fun and hackable.

Unfortunatelly most of the experienced Perl developers are also hard-core vim or emacs users and it is nearly impossible to move them away from their editor. Luckily there were a few who got interested by the idea of having a lot of control over their editor and that made the difference between a failing one man project to a project with a lot of potential.

Then at one point came the idea to integrate Parrot and Rakudo and try to provide a platform for Perl 6 development as well. With the awesome work of Ahmad M. Zawawi Padre has a lot to offer to anyone who wants to write Perl 6 code. With some more work we'll be able to integrate Parrot even better and then Padre will be able to handle any language running on top of Parrot.

While there are still lots of problems with Padre and installation can still be difficult it already more or less has the same power as Notepad++ on both Windows and Linux and probably on Mac too and it has some extra features that even vim people started to steal.

adam@svn:~/svn.ali.as/db$ ls -l
total 30884
-rw-r--r-- 1 adam adam 9558294 Jul  2 03:45 cpandb.gz
-rw-r--r-- 1 adam adam 8538979 Jul  2 03:45 cpandb.bz2
-rw-r--r-- 1 adam adam 5960155 Jul  2 03:45 cpandb.lz
-rw-r--r-- 1 adam adam 3014480 Jun 30 06:46 cpanmeta.gz
-rw-r--r-- 1 adam adam 2658756 Jun 30 06:46 cpanmeta.bz2
-rw-r--r-- 1 adam adam 1825600 Jun 30 06:46 cpanmeta.lz

The good people at GitHub have implemented a JSONP api for Gists, so I have updated my graceful degarding script accordingly (see previous post).

Now the code no longer needs to hijack document.write and has better load time characteristics as well. If you use traditional <script> tags then each gist will block the loading of the page at that point, causing jittery page renderings, and preventing the rest of the content from appearing. This script styles the <pre> tags when the DOM becomes ready, and only fetches the syntax highlighted HTML after the onload event, when all other page elements have finished loading.

The second new bit is the Perl scripts I've added to streamline working with these degradable Gists.

The first script is very simple, given a Gist ID it prints out a <pre> tag suitable for inclusion into your HTML document. For instance make_pre_tag.pl 115368 outputs:

<pre id="fake-gist-115368" class="fake-gist">use Moose;
has fun =&gt; ( isa =&gt; &quot;Constant&quot; );</pre>

The second script, update_gists.pl is much more interesting. It uses XML::LibXML to find all the pre tags with the class fake-gist in a document. If these tags have no ID then App::Nopaste is used to create new Gists. Otherwise AnyEvent::HTTP is used to concurrently fetch the updated versions from http://gist.github.com/.

I start by editing a plain XHTML document in my editor, adding code in the <pre> tags, and then I post the code blocks to github all at once. I use CDATA sections to escape the source code in the XHTML document.

<pre lang="perl" class="fake-gist"><![CDATA[
$object->method(
    foo => $bar, 
);
]]></pre>

The script re-escapes that using HTML entities so that it is valid HTML (which doesn't support CDATA). This way the <pre> tags render fine under quirks mode, allowing JQuery to append HTML strings to the document.

I'm going to try to do a series of posts about learning Perl internals. I am still a beginner, I have trouble remembering the many macros, or keeping everything in my head all at once, so hopefully I will be able to make some sense of this stuff in a way that is accessible to other beginners.

While I'm definitely diving right in to the deep end, I think PL_runops is a good place to start as any, there's not a lot you need to learn to see how it works. I don't think Perl has a shallow end.

PL_runops is a variable in the interpreter containing a function pointer, which in most cases will be Perl_runops_standard.

Perl_runops_standard is the function that executes opcodes in a loop. Here's how it's defined:

int
Perl_runops_standard(pTHX)
{
    dVAR;
    while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) {
        PERL_ASYNC_CHECK();
    }

    TAINT_NOT;
    return 0;
}

Perl makes extensive use of macros, which can sometimes be confusing, but in this instance it's not too daunting. This loop will essentially keep calling PL_op->op_ppaddr, assigning the result to PL_op. As long as a valid value is returned it will keep executing code.

Just to get it out of the way, PERL_ASYNC_CHEK is a macro that checks to see if any signals were delivered to the process, and invokes the handlers in %SIG if necessary.

So what's PL_op? It's essentially Perl's instruction pointer, it refers to the opcode currently being executed. When Perl compiles source code it produces an optree. This is probably one of the more complicated structure in Perl internals, but right now we're only concerned with one small part, the op_ppaddr field of a single node in the tree. The op_ppaddr field contains a pointer to the function that implements the op.

PP stands for push/pop, which means that it's a function that operates in the context of the Perl stack, pushing and popping items as necessary to do its work, and returns the next opcode to execute.

pp.h defines a PP macro, which sets up a signature for a function that returns an op pointer. Let's have a look at two simple PP functions. First is pp_const:

PP(pp_const)
{
    dVAR;
    dSP;
    XPUSHs(cSVOP_sv);
    RETURN;
}

This is an implementation of the const op, which pushes the value of a literal constant to the stack. The cSVOP_sv macro is used to get the actual SV (scalar value structure) of the constant's value from the optree. SVOP is an OP structure that contains an SV value. The c stands for "current".

Let's rewrite the body of the macro using some temporary values:

/* the current opcode is op_const, so the op structure is an SVOP
 * Instead of using the long chain of macros we'll put it in a variable */
SVOP *svop = (SVOP *)PL_op; 

/* and then we can simply use the op_sv field, which
 * contains a pointer to a scalar value structure */
SV *sv = svop->op_sv;

This SV is then pushed onto the stack using the XPUSHs macro. The X in XPUSHs means that the stack will be extended if necessary, and the s denotes SV. To read the documentation of these macros, refer to perlapi.

The next thing that executes is the RETURN macro. This macro is defined in pp.h, along with a few others:

#define RETURN          return (PUTBACK, NORMAL)

#define PUTBACK         PL_stack_sp = sp

/* normal means no special control flow */
#define NORMAL          PL_op->op_next

/* we'll use this one later: */
#define RETURNOP(o)     return (PUTBACK, o)

Recall that there are no lists in C. The comma operator in C executes its left side, then its right, and returns that value (unfortunately we have that in Perl, too). The RETURN macro therefore desugars to something like:

PL_stack_sp = sp;
return(PL_op->op_next);

The XPUSHs macro manipulated the local copy of the pointer to the stack, sp as it was adding our SV. This change is not immediately written to the actual stack pointer, PL_stack_sp . The PUTBACK macro sets the "real" stack to the version we've manipulated in the body of our opcode.

Then the opcode simply returns PL_op->op_next. The op_next field in the op contains a pointer to the next op that should be executed. In this case if the code being executed was:

my $x = 42;

then the const op compiled to handle the 42 literal would have pushed an SV containing the integer 42 onto the stack, and the op_next in this case is the assignment operator, which will actually use the value.

So, to recap, when PL_op contains a pointer to this const op, PL_op->op_ppaddr will contain a pointer to pp_const. PL_runops will call that function, which in turn will push ((SVOP *)PL_op)->op_sv onto the stack, update PL_stack_sp, and return PL_op->op_next.

At this point runops_standard will assign that value to PL_op, and then invoke the op_ppaddr of the next opcode (the assignment op).

So far so good?

To spice things up a bit, here's the implementation of logical or (||):

PP(pp_or)
{
    dVAR; dSP;
    if (SvTRUE(TOPs))
        RETURN;
    else {
        if (PL_op->op_type == OP_OR)
            --SP;
        RETURNOP(cLOGOP->op_other);
    }
}

The or op is of a different type than the const op. Instead of SVOP it's a LOGOP, and it doesn't have an op_sv but instead it has an op_other which contains a pointer to a different branch in the optree.

When pp_or is executed it will look at the value at the top of the stack using the TOPs macro, and check if it evaluates to a true value using the SvTRUE macro.

If that's the case it short circuits to op_next using the RETURN macro, but if it's false it needs to evaluate its right argument.

--SP is used to throw away the argument (so that $a || $b doesn't end up returning both $a and $b).

Then the RETURNOP macro is used to call PUTBACK, and to return PL_op's op_other. RETURN is essentialy the same as RETURNOP(NORMAL). op_other contains a pointer to the op implementing the right branch of the ||, whereas op_next is the op that will use the value of the || expression.

This is one of the most basic parts of the Perl 5 virtual machine. It's a stack based machine that roughly follows the threaded code model for its intermediate code structures.

The data types mostly revolve around the SV data structure, and moving pointers to SVs from op to op using the stack.

For me the hardest part to learn in Perl is definitely the rich set of macros, which are almost a language in their own right.

If you search for runops on the CPAN you will find a number of interesting modules that assign to PL_runops at compile time, overriding the way opcodes are dispatched.

For more information on Perl internals, the best place to start is perlguts, and the wonderful perlguts illustrated.

At the Perl QA hackathon, Andreas Koenig, the maintainer of PAUSE, demonstrated a way to decrease the lag between new distribution uploads to PAUSE and syndication to CPAN mirrors.  (See File::Rsync::Mirror::Recent on CPAN for details).   Running minicpan against a one of the new, fast updating mirrors results a very fresh minicpan, but, until now, only for regular releases, not for alpha development releases.

Last year, I wrote CPAN::Mini::Devel to include development releases in a minicpan, but it depends on an obscure, large, infrequently updated index.¹ Now, with smaller, fast-updating index files available for rsync, I released CPAN::Mini::Devel::Recent to take advantage of them for minicpan as well.

This is all still experimental, but if you do Perl programming and want bleeding-edge development releases within minutes of their upload to PAUSE, then find a fast mirror, try out CPAN::Mini::Devel::Recent and let me know how it works for you.  After installation, you can configure it to be used automatically by adding “class: CPAN::Mini::Devel::Recent” to your .minicpanrc file.

1. indices/find-ls.gz

On the back my my improved and high-coverage CPAN::Mini::Visit and Archive::* fixes, I've finally managed to build a complete-coverage version of CPANDB.

CPANDB is a merged and cleaned up schema that combines the CPAN index, the "CPAN Uploads" database (for PAUSE upload dates), and both class and distribution level dependency information held in META.yml files (replacing the CPANTS dependency graph).

To take a look at it, you can grab a copy of the SQLite database directly from the following URL.

http://svn.ali.as/db/cpandb.gz.

The data sources used to generate it are not perfectly time-synced yet, so I expect to see a few minor flaws for another release or two. But compared to everything else available (from pretty much everybody) this should be a significant improvement.

As well as clearing up the last tiny data quality issues, I'm also yet to merge in the rt.cpan.org database (which is almost ready) and the CPAN Ratings database (which is a text file I really don't want to have to parse).

But don't let this stop you trying it out now (I've appended the schema to the bottom of this post so you can get a clearer idea of what's in there).

As usual, feedback is welcome.

CREATE TABLE author (
        author TEXT NOT NULL PRIMARY KEY,
        name TEXT NOT NULL
);

CREATE TABLE distribution (
        distribution TEXT NOT NULL PRIMARY KEY,
        version TEXT NULL,
        author TEXT NOT NULL,
        release TEXT NOT NULL,
        uploaded TEXT NOT NULL,
        FOREIGN KEY ( author ) REFERENCES author ( author )
);

CREATE TABLE module (
        module TEXT NOT NULL PRIMARY KEY,
        version TEXT NULL,
        distribution TEXT NOT NULL,
        FOREIGN KEY ( distribution ) REFERENCES distribution ( distribution )
);

CREATE TABLE requires (
        distribution TEXT NOT NULL,
        module TEXT NOT NULL,
        version TEXT NULL,
        phase TEXT NOT NULL,
        PRIMARY KEY ( distribution, module, phase ),
        FOREIGN KEY ( distribution ) REFERENCES distribution ( distribution ),
        FOREIGN KEY ( module ) REFERENCES module ( module )
);

CREATE TABLE dependency (
        distribution TEXT NOT NULL,
        dependency TEXT NOT NULL,
        phase TEXT NOT NULL,
        PRIMARY KEY ( distribution, dependency, phase ),
        FOREIGN KEY ( distribution ) REFERENCES distribition ( distribution ),
        FOREIGN KEY ( dependency ) REFERENCES distribution ( distribution )
);

I'd like to have a system that can hold all the artifacts of a test execution and that also knows a lot about TAP. Smolder is a good start as it knows about TAP but there are tons of other things I'd like to have in it.

Some of them are hard to add, others are harder.

Anyway, earlier I wrote about Reporting Test Results. Now let me write about my wish-list:

1) I'd like Smolder to collect all the artifacts of a test run. Look at the data Buildbot collects or what is collected by the PostgreSQL test farm! Each run has several stages and the actual test run is only one of them. In a Test Reporting system I'd like to have access to all the output and error files of all the stages.

2) The tests might create some additional log files, I'd like to be able to include those too.

3) One major issue we have is test counting. TAP encourages you to have a plan - the number of test units you are going to execute in a test run - so that it can check the number of actually executed test to that number. Keeping the plan up to date is annoying, people don't like it so many use no plan. Some TAP producers don't even let you plan ahead (e.g. the ones I saw in PHP) but I think generally those that are more xUnit like won't have a test plan.

In larger settings this plan does not even help as it does not provide any tools to avoid reducing the test plan accidentally along with a shortcut in the test script: A developer can comment out several tests temporarily and adjust the plan for the time while he is trying to fix a bug forgetting to revert these changes after he is done.

Smolder could help a bit here: On the first result report the administrator of the project should look at the reports and mark them as having the correct number of tests. Then if Smolder notices a test report with a different number of test for the same test file it should raise an alert. The project administrator can examine the change and decided if that is the new correct number of tests setting the new expected number to this or if it was changed by mistake.

4) History of unit tests. If we would like to be able to provide a history based on the actual test units (calls to an ok() function) then we need to have some unique identifier for each test as we cannot assume that the test number or the test name will remain the same. After all people sometimes insert new tests or remove old test at the beginning of the test scripts and the names can change.

Smolder could also help here too. It could track the test numbers and test names. If it sees a change it can alert the project administrator who can then use some GUI to connect the old test units to the new ones.

Both involves some manual work but I think large organizations will be ready to invest that work in order to get the benefit of greater control and better reports.

What would you like to have in your test reporting system?

Autodie 2.00 released
This weekend the long awaited autodie 2.00 for Perl was released to the CPAN, which was almost immediately replaced by 2.02, which fixes some oopsed tests and which adds a couple more features to give us a really sweet experience. This blog entry assumes you're using 2.02.

Observant viewers will notice that the major version number has changed. I've taken the great leap from 1.999 to 2.00. Clearly, something is different, and you might be wondering what.

Well, autodie 2.0 now supports a hinting interface for user-defined subroutines. Put simply, if you have a user-defined subroutine that does something funny to signify failure, you can now tell autodie about that. Once it knows, it can Do The Right Thing when checking your subroutine. You can even put the hints into the same file as those subs, and if someone is using autodie 2.00, it will find the hints and use them.

This may not sound very exciting, but it is. It means that a lot of really ugly error-checking code, both on the CPAN and the DarkPAN, can go away. Lexically. Still not convinced this will change your life? Let's look a little more closely; trust me, you'll like it.

Let's pretend you're working on a piece of legacy code. For some reason, the people who wrote this code decided the best way to signal errors is by returning the list (undef, "Error message"). I don't know why, but I've seen this anti-pattern emerge independently in three 100k+ line projects I've been involved in.

sub some_sub {
    if ( not batteries_full() ) {
        return ( undef, "insufficient energy" );
    }

    if ( not coin_inserted() ) {
        return ( undef, "insufficient credit" );
    }

    my @results = some_calculation();

    return @results;
}

If you want to check to see if some_sub() returns an error, you need to capture its return values, look at the first one to see if it's undefined, and if it's not, use the second one as your error. At least, that's what you're supposed to do.

What actually happens is most developers decide that's way too hard, and don't bother checking for errors. Then one day, the batteries on your doomsday-asteroid-destroying-satellite go flat, nobody notices, and through an ironic twist of fate you're left as the last known human survivor, and there are zombie hordes and walking killer plants outside.

So, how can autodie help us? Well, before version 2.00, it couldn't. But now, with autodie::hints, it can! We can give autodie hints about how the return values are checked. They look like this:

use autodie::hints;

autodie::hints->set_hints_for(
    'Some::Package::some_sub' => {
        scalar => sub { 1 },
        list   => sub { @_ == 2 and not defined $_[0] },
    },
);

Our hints here are simple subroutines. If they return true, our subroutine has failed. If they return false, it's executed successfully. Notice that our scalar hint always returns true. That's because we consider any call of our subroutine in scalar context to be a mistake. It's returning a list of values, and you should be checking that list.

Once we've set our hints, we can then use autodie to automatically check if we're successful:

use Some::Module qw(some_sub);

sub target_asteroid {

    use autodie qw( ! some_sub );

    # autodie has lexical scope, so only calls to some_sub inside
    # the target_asteroid subroutine are affected.

    my @results = some_sub();     # Succeeds or dies
}

sub target_ufo {
    my @results = some_sub();

    # autodie is out of lexical scope, so we have to manually
    # process @results here.
}

If you're wondering what that exclamation mark means, it means "insist on hints", and is a new piece of syntax with autodie 2.00. If for any reason autodie can't find the hints for some_sub, our code won't compile. That's a very good thing, and avoids us having a false sense of security if we use autodie on an unhinted sub.

However the error messages from autodie aren't really that useful. They're going to be things like "Can't some_sub() at space_defense.pl line 53". There's a noticable lack of explanation as to why some_sub() failed.

Luckily, since the way early versions of autodie, we've been able to register message handlers. And with the new features in autodie 2.02, we can produce very rich messages. Let's see how!

use autodie::exception;

autodie::exception->register(
    'Some::Module::some_sub' => sub {
        my ($error) = @_;

        if ($error->context eq "scalar") {
             return "some_sub() can't be called in a scalar context";
        }

        # $error->return gives a list of everything our failed sub
        # returned.  We know this particular sub puts the error
        # message the second argument (index 1).

        my $error_msg = $error->return->[1];

        return "some_sub() failed: $error_msg";
    }
);

Now, whenever some_sub() fails, it'll print a genuinely useful message, like "some_sub() failed: Insufficient energy at space_defense.pl line 53". Yes, autodie automatically adds the file and line number for you. Nice!

But wait, there's more! We don't want to see this sort of code floating around in your programs. You may be dealing with other people's modules that you can't modify, so we can't hide all this configuration in there. So, we can write our own pragma that contains all this info. Here's the full code for a theoretical my::autodie pragma, and is the exact same code used by the t/blog_hints.t file in autodie's test suite.

package my::autodie;
use strict;
use warnings;

use base qw(autodie);
use autodie::exception;
use autodie::hints;

autodie::hints->set_hints_for(
    'Some::Module::some_sub' => {
        scalar => sub { 1 },
        list   => sub { @_ == 2 and not defined $_[0] }
    },
);

autodie::exception->register(
    'Some::Module::some_sub' => sub {
        my ($E) = @_;

        if ($E->context eq "scalar") {
            return "some_sub() can't be called in scalar context";
        }

        my $error = $E->return->[1];

        return "some_sub() failed: $error";
    }
);

1;

It works exactly the same as regular autodie, except it also knows how to handle some_sub(), and display good looking error messages. Here's how we'd use it:

use Some::Module qw(some_sub);
use my::autodie qw( ! some_sub );

my @results = some_sub();  # Succeeds or dies with a useful error!

There's a lot more you can do with autodie, and if you want to learn more, I'd suggest coming to my talk at OSCON or YAPC::EU, where I'll be covering all this and more, with a distinctive Star Trek twist. ;)

Posted: 1st July 2009.

Tags: autodie hints perl

Bookmark:

The latest developer release of Test::More allows subtests. Subtests are great in that they solve a lot of problems in advanced Perl testing, but they have required a change in Test::Builder. Previously you could do stuff like this:

package Test::StringReverse;

use base 'Test::Builder::Module';
our @EXPORT = qw(is_reversed);

my $BUILDER = Test::Builder->new;

sub is_reversed ($$;$) {
    my ( $have, $want, $name ) = @_;

    my $passed = $want eq scalar reverse $name;

    $BUILDER->ok($passed, $name);
    $BUILDER->diag(<<"    END_DIAG") if not $passed;
    have: $have
    want: $want
    END_DIAG

    return $passed;
}

1;

And you've have a simple (untested ;) test for whether or not strings are reversed.

The reason that worked is that Test::Builder->new used to return a singleton. This is no longer true. If someone uses your test library in a subtest, the above code would break. Instead, you want to do this:

sub is_reversed ($$;$) {
    my ( $have, $want, $name ) = @_;

    my $passed  = $want eq scalar reverse $name;
    my $builder = __PACKAGE__->builder;

    $builder->ok($passed, $name);
    $builder->diag(<<"    END_DIAG") if not $passed;
    have: $have
    want: $want
    END_DIAG

    return $passed;
}

It's a minor change, it's completely backwards-compatible and it supports subtests. There's a work-around being planned, but it's not out there yet.

An imaginary conversation synthesized from past discussions and the responses I wish I made.

• Customer: We need read-only access to your database.
• Ovid: No.
• Customer: Please?
• Ovid: No.
• Customer: But I need ad-hoc queries.
• Ovid: Your ad-hoc cartesian join returning 12 billion rows was real fun.
• Customer: I promise I won't do it again.
• Ovid: That's what you said about the ad-hoc cartesian join returning 10 billion rows.
• Customer: But this time I mean it.
• Ovid: So do I. We guarantee backwards-compatibility in our API, not our database. If we move a field from one table to another, your queries will break.
• Customer: Then tell us when you do that.
• Ovid: We did that with another team and had to keep delaying releases while they updated their system.
• Customer: Then you can provide views to maintain backwards-compatibility.
• Ovid: We do that already. "View" as in "Model-View-Controller". It's part of our REST API; you should check it out.
• Customer: But your REST API doesn't provide all the information I need.
• Ovid: It provides more than the information you need because much of it represents knowledge not stored in the database. If you need more information, let's see what we can do to add this to the API.
• Customer: Why are you being so difficult?
• Ovid: Because your temporary convenience is not more important than my long-term pain.

Don't let external customers read directly from your database. Just don't. The usual justification is the need to support ad-hoc queries. Get a few samples and try to figure out a general mechanism to support their actual business needs. If you let them read from your database, they will become dependent on this and beg you to hold off database changes or complain if you don't. As your project grows larger, the pain grows more severe. They will have the best of intentions, but good intentions mean nothing when you need to coordinate your internals with people who should know better than to violate encapsulation.

As a side note, ad-hoc queries, even if not causing performance issues, could potentially be dangerous if the people making them aren't really thinking them through. The problem is two-fold. One, they might not be really paying attention to their core business needs (this is subtle and hard to explain, but common). The other problem is that they might very well be making a query that your API already supports, but because they don't rely as much on your API, they don't know it.

After my little research into the Eclipse foundation (Comparing the Eclipse Foundation with The Perl Foundation and EPO) I was told to take a look at the success of Ubuntu. While I know there is a huge difference as Ubuntu is an operating system while Perl is just a language but I got interested and here are the results.

Ubuntu does not have a foundation. It has a strong and very enthusiastic community and it is backed by Canonical, the company of Mark Shuttleworth.

That means the Perl community probably cannot learn much from the way Ubuntu and Canonical are connected but we can still look at the larger picture of how Ubuntu succeeded so far?

First I'll look at the business model of Canonical and later on the elements that helped Ubuntu become one of the leading GNU/Linux distributions in only a few years.

The Ubuntu Business model

So if we look closer I think we can safely say that Ubuntu is a product Canonical is sponsoring in order to make money somehow. I would not say by selling added value service as I am not exactly sure what is their business model. I noticed a few areas where Canonical might be making money:

• Selling wearables and accessories.
• Providing support contracts at nice sums.
• Providing Training and Certification.

Distribution mechanism for software.

Selling T-shirts can bring in nice revenue to a small company but for Canonical it is more part of the marketing mix than the revenue mix.

Providing support can be an interesting business model, IBM makes a lot of money on it. I guess Canonical does too. This will just increase as Ubuntu invades the corporate infrastructure.

Training and Certification can be a lucrative business if executed well and I trust them to do it well. If they can convince companies that they need Ubuntu Certified Engineers to run Ubuntu smoothly then will see a huge number of people attending their classes and taking their certificate exams.

Still the biggest business opportunity I think is the one that is not even listed in the Canonical Store but only under Services. That is, they offer to be a distribution channel for ISVs.

In the Microsoft world people are used to download software and click-click-click install it. In the GNU/Linux world almost all the important software is packaged and distributed by the Linux distributor. It is especially true for Debian/Ubuntu but I think it is increasingly so with Fedora and Mandriva as well. That means the most common installation mode in these versions of Linux is via a mechanism similar to the "Add/Remove Software" on Windows.

So if you have a software which you'd like to distribute to Linux users, one of the best ways is to make sure each one of the Linux distributors will package your software. Otherwise you'll need to build .deb and .rpm files for each distribution or a stand-alone executable which is frowned upon by most of the Linux system administrators.

If your software is not open source you don't have much chance that any of the Linux distributors will add that package to their repository making it almost impossible to penetrate the Linux Desktop market.

If Canonical manages to get a large chunk of the market share in the Linux desktop market it effectively can open or close the distribution channel of ISVs making them pay well for the right to use Ubuntu as their distribution platform.

So what will happen to Ubuntu in 5 years?

I don't know.

If you look at the background of Mark Shuttleworth, he built Thawte and more or less at the peak of the dot.com bubble he sold it to VeriSign the only competitor in the digital certificates market. So he is a clever guy who won't be afraid to sell Canonical to the market leader with a near monopoly in the desktop business or to the competitor with the most money making a few billions of dollars. The sky isn't the limit

The Perl aspect

Perl is not an operating system and it is not the open source arm of a company. It cannot be sold but it can provide nice revenue streams to individuals or companies offering service with Perl. There might be other revenue sources as well but lets look at the 4 revenue streams that Canonical has:

• Selling wearables - on Perl conferences some stuff is auctioned which generates some revenue to organize the next conference. This could be further expanded but the income is very limited.
• Providing support: ActiveState does it as do a number of other small companies mostly in the local markets or in specialized fields (e.g. Shadowcat providing web development services using Catalyst or Summersault providing web development services using Titanium.) This could be exploited further providing similar services in other fields as well, not only web application development.
• Offering Perl training provides nice income to several people in the Perl community including myself. This does not seem to turn any of us rich but provides us with a reasonable income stream to support ourselves. Certification is a sour point in the Perl community. Some people say that they don't want to have any certification systems for Perl others see it as the Holy Grail to make Perl more accepted by corporations. In any case this could be a revenue stream but I don't think it will happen any time soon.
• Unlike an Operating system Perl does not control any distribution channel but it could be used in a related way. Actually it could be used similar to how the JVM provides a platform of application development. I am sure one day we will have a binary package of Perl with tons of CPAN modules already installed. Something like Strawberry Perl but with many more modules that can be installed on all major OS-es. That thing could be used as a development and distribution platform for all kinds of software.

I am not sure this can directly generate revenue but it will help enlarge the Perl user base and it could provide more opportunities for development, support and training services for more people in the Perl community.

I'll write about the success of Ubuntu in a separate post. for the meantime you can read the Ubuntu Story between 2004 and 2006. They don't keep their history up to date either.

I mean, they're not one-liners. They're full programs. They just do something really really simple.

This one blows away all remotes (left over from, say, GitHub) and adds all the remotes for users on our git.example.com git-on-ssh hosting box.

#!/usr/bin/perl
use strict;
use warnings;

{
  package Git::Our::Remotes;
  use base 'App::Cmd::Simple';

  use autodie qw(:default :system);
  use Cwd;
  use Getopt::Long::Descriptive;
  use Term::ReadKey;

  my $HOST = q{git.example.com};

  sub opt_spec {
    return (
      [ 'dry-run', "be like quality department; don't actually do anything" ],
    );
  }

  sub _runcmd {
    my ($self, $opt, $cmd) = @_;

    if ($opt->{dry_run}) {
      print "running: $cmd\n";
    } else {
      system($cmd);
    }
  }

  sub validate_args {
    my ($self, $opt, $args) = @_;

    my $user = $ENV{USER};

    unless (defined $args->[0]) {
      my $cwd = getcwd;
      $cwd =~ s{.*?/?([^/]+)\z}{$1};
      $args->[0] = $cwd;
    }

    my $repo = $args->[0];

    $self->usage_error("too many args given") if @$args > 1;
    $self->usage_error("do not run as root")  if $user eq 'root';
    $self->usage_error("illegal repo name")   if $repo !~ /\A[-_a-z0-9]+\z/;
    $self->usage_error("you should run this in a repo") if ! -d '.git';
  }

  sub run {
    my ($self, $opt, $args) = @_;

    my $repo = $args->[0];

    # should be using a library!
    print "press any key to reset all remotes for repo '$repo'";
    Term::ReadKey::ReadMode 'cbreak';
    Term::ReadKey::ReadKey(0);
    Term::ReadKey::ReadMode 'normal';
    print "\n";

    my $user    = $ENV{USER};
    my @others  = grep { $_ ne $user }split / /, (getgrnam('staff'))[3];

    my @remotes = `git remote`;
    chomp @remotes;

    $self->_runcmd($opt, "git remote rm $_") for @remotes;

    $self->_runcmd($opt, "git remote add deploy git\@$HOST:$repo.git");
    $self->_runcmd($opt, "git remote add origin $user\@$HOST:git/$repo.git");

    for my $o (@others) {
      $self->_runcmd($opt, "git remote add $o $user\@$HOST:~$o/git/$repo.git");
    }

    print "remotes created!\n";
  }
}

Git::Our::Remotes->import;
Git::Our::Remotes->run;

...and that's it. Anyone can run git ourremotes to fix remotes setup. If the user hasn't set up his own hosted ssh remote, he can just run git hubclone --bare-only using the script I blogged about on Saturday. Done!

After another year of doing basically nothing with CPANTS, I think it is finally time to look for somebody to take over maintainership. I don't see any tuits to emerge in the next 3 months (thanks to two big, cool, all-perl jobs) to fix all the bugs that become more an more apparent, as CPANTS data is used by more and more third parties. Plus there are lots of interesting new projects (mostly Metabase) that could and should be considered.

As a first step, I moved the repo from google code (and svn) to github for easier collaboration. But to really move CPANTS forward, the switch to git is by far not enough...

So if you're interested in taking a very close look at the ugly parts of CPAN, or want to tweak the algorithm of the CPANTS game to make you finally show up on top, drop me a line and I'll either add you as co-maint (or even as full maintainer, if somebody wants to take over completely (which I'd prefer)).

I guess that we can keep the site hosted on the Vienna.pm server, unless somebody wants to move the site to their own machine. I could also give an inofficial short/long intro/tutorial at YAPC::Europe to help the new maintainers find their way round the code.

CPANTS consists of three components, two of which (M:C:Analyse & M:C:Site) are rather easy to understand, while M:C:ProcessCPAN is a bit more complex. It has no real tests and takes ages for a complete run - so it's quite a pain in the ass to fix bugs and implement new features (hm, bad advertising...).

So if you're interested in taking over (parts) of CPANTS, please comment below and/or drop me a line (domm AT cpan.org)

We have just released the last big milestone on the way towards Mojo 1.0, everything from now on will be just documentation, artwork and other polishing tasks.

As usual there are awesome new features such as HTTP 1.1 pipelining, format handling, layout support and a brand new routes syntax.

Thanks to our GSOC Student Pascal Gaudette who squashed many RFC related bugs we are now even further ahead of other HTTP implementations on CPAN. And the best of all, after some refactoring the whole HTTP 1.1 client/server state machine is now completely reusable and independent from the IO layer. This will make modules such as AnyEvent::Mojo absolutely trivial in the future.

And while we are at it, here's a preview of the new "fluffy cloud" Mojo logo!

The complete changelog.

- Rewrote MojoX::Renderer, it is not backward compatible and templates need to be renamed in the following 3 part format "index.html.tt"!
- Added exception support to MojoX::Dispatcher::Routes, this change is not backward compatible and "dispatch" calls now return exception objects for errors and false otherwise.
- Changed routes syntax, this change is not backward compatible and you need to change all your existing routes.
"/:foo" becomes "/(foo)"
"/^foo" becomes "/((foo))"
"/*foo" becomes "/(((foo)))"
- Added full HTTP 1.1 pipelining support to all Mojo layers.
- Added layout support to MojoX::Renderer.
- Made render call optional.
- Added format support to MojoX::Routes.
- Added Mojo::Loader::Exception.
- Added wildcard symbol support to MojoX::Routes and rewrote many routes internals.
- Added Makefile.PL generator.
- Added HttpOnly support to Mojo::Cookie. (burak)
- Support more CGI implementations.
- Added support for namespaces only dispatching in MojoX::Dispatcher::Routes.
- Added encoding support to Mojo::Template and made "utf8" the default.
- Added HEAD support to Mojo::Server::Daemon. (acajou)
- Added new relaxed placeholder to MojoX::Routes::Pattern.
- Added Mojo::Template::Exception.
- Added HEAD support to the Mojo::Transaction state machine and related modules. (acajou)
- Added safe_post option to Mojo::Pipeline. (acajou)
- Made chained => 1 the default in Mojo::Base.
- Fixed compiler bug in Mojo::Template that prevented more advanced control structures, you might have to add additional semicolons to some of your templates.
- Fixed Mojo::Date to not crash on invalid dates. (vti)
- Fixed chunked support in Mojo::Server::Daemon and Mojo::Client.
- Fixed tokenizer in MojoX::Routes::Pattern to support "0" values. (Anatoly Sharifulin)
- Fixed parsing of "0" in Mojo::Path. (charsbar)
- Fix server tests on win32. (charsbar)
- Fixed leading whitespace problem in the request parser. (acajou)
- Fixed broken pipe problem in Mojo::Server::CGI. (vti)
- Added more diagnostics options to Mojo::HelloWorld. (uwe)
- Fixed empty cookie parsing. (vti)
- Fixed a case where child processes migth hang in Mojo::Server::Daemon::Prefork. (gbarr)
- Fixed a bug in MojoX::Dispatcher::Routes where the renderer would be called with an empty stack. (melo)
- Fixed a escaping problem in Mojo::Parameters. (vti)
- Updated Mojo::URL to be more template friendly.
- Improved Solaris compatibility.

P.S. In case you were wondering, the version number is the date of the first futurama episode. ;)

I've found that one of the best ways to try new testing ideas is to run them against our test suite at work. We've over 30,000 tests at this point, with several test harnesses integrating together, along with two fundamentally different test systems. This means that when I throw something at this test suite, I often shake out bugs. My new Test::Aggregate::Nested combined with our test suite managed to find a rather serious issue with the new subtest function in Test::More. The following, for example, fails:

    use Test::More tests => 2;
    use Test::XML;

    ok 1;
    subtest 'FAIL!' => sub {
        plan tests => 1;
        is_xml '<foo/>', '<foo/>', 'Singleton fail';
    };
    __END__
    xml.t ..
    1..2
    ok 1
        1..1
    Cannot run test (Singleton fail) with active children at /home/ovid/pips_dev/work/Pips3/branches/rights_modeling/deps/lib/perl5/Test/XML. pm line 57.
        # Child (FAIL!) exited without calling finalize()

There's nothing wrong with the code as it's written, but Test::XML knows that the Test::Builder object is a singleton, so there's a false optimization in the code. Near the top of the package, you see this line, defined outside of all functions:

    my $Test = Test::Builder->new;

If every Test::XML function simply had that line in the function rather than trying to share this across all test functions, the subtest code would work fine. Instead, the author knew he had a singleton and there's no point in re-instantiating, is there?

To be fair, I've done the same thing before (see Test::JSON), even though I knew it might be a bad idea. Heck, lots and lots of testing libraries have this issue. Now we need to figure out how to deal with this problem or else subtests aren't going to play well with a lot of code. Damn.

Running a standard Perl module's test suite is very easy using prove. If the module needs compilation, just run make before hand and pass the -b flag to prove, otherwise use -l to add lib to @INC.

If you pass --state=save then information about the last run will be saved in a .prove file in the current directory. This data can be used later to run only failed tests, or to reorder tests.

My testing workflow was greatly improved using this feature. I use two shell aliases to run prove:

alias ts="prove --state=slow,save -j3 -r"
alias tf="prove --state=failed,save -v -r"

I start by running ts -l t (or ts -b t if this module requires make to run before testing). The -r flag tells it search for .t files recursively in t. The -j option enables 3 concurrent jobs, and the --state flag orders tests from slowest to fastest, and causes the .prove file to be updated at the end of the run.

When I get failing tests I switch to using tf -l or tf -b. Since this has --state=failed it only runs tests that failed in the previous run. The -v causes the TAP output to be printed (with coloring).

I fix my tests or code one by one, and keep running tf. When tf has nothing left to run everything is passing, so I switch back to ts to run the full test suite again.

These two simple aliases let me run the test suite much more easily and effectively than make test, so a great many thanks to Ovid and Andy Armstrong [Update: and Andy Lester] for writing and maintaining these tools.

I thought about this on and off for some time but have not done anything with it yet. Maybe someone else with better ideas can do something like this.

I was sondering if it would be interesting to create a web application that on one hand would allow people using Perl to get some personalized services on the other hand it would allow the ongoing collection of some statistical data about the Perl community and about the usage of Perl.

The services that would be offered could include alert when one of the modules used by the person gets a new version. It could include a Google-alert like service when the modules are mentioned in one of the perl related sources.

It could allow the notification about Perl related events in their city/country/continent.

It could allow people to mark their favorite modules. It could allow to follow other people whose judgement they trust for modules which in turn would provide a better way to search for modules on CPAN for everyone.

It would allow an on-going survey similar to the one that was a year or two ago but that would allow people to change the data they provide and allow the site maintainers to provide information regarding the changes as well.

Boy oh boy, does nested TAP make Test::Aggregate much cleaner. It's not uploaded yet -- still documenting and working out corner cases for the new Test::Aggregate::Nexted -- but here's what aggregated test output used to look like. Remember, this is five separate test programs. Can you tell where each starts and ends?

Test-Aggregate  $ prove -lv t/pre_post.t
t/pre_post.t .. # ******** running tests for aggtests/check_plan.t ********

ok 1 - aggtests/check_plan.t ***** 1
ok 2 - aggtests/check_plan.t ***** 2
ok 3 # skip checking plan (aggtests/check_plan.t ***** 3)
ok 4 - env variables should not hang around
ok 5 - aggtests/check_plan.t ***** 4
ok 6 - findbin is reinitialized for every test
ok 7 # skip Testing skip all
#     ok - aggtests/check_plan.t (1 out of 5)
# ******** running tests for aggtests/findbin.t ********
#     ok - aggtests/findbin.t (2 out of 5)
# ******** running tests for aggtests/skip_all.t ********
#     ok - aggtests/skip_all.t (3 out of 5)
# ******** running tests for aggtests/slow_load.t ********
ok 8 - slow loading module loaded
ok 9 - env variables should not hang around
ok 10 - subs work!
ok 11 - Startup should be called once
ok 12 - ... as should shutdown
ok 13 - Setup should be called once for each test program
ok 14 - ... as should teardown
1..14
#     ok - aggtests/slow_load.t (4 out of 5)
# ******** running tests for aggtests/subs.t ********
#     ok - aggtests/subs.t (5 out of 5)
ok
All tests successful.
Files=1, Tests=14,  1 wallclock secs ( 0.02 usr  0.01 sys +  0.11 cusr  0.01 csys =  0.15 CPU)
Result: PASS

Now let's run that with Test::Aggregate::Nested (requires the development version of Test::More):

Test-Aggregate  $ prove -lv t/pre_post_nested.t
t/pre_post_nested.t ..
1..5
    1..5
        1..0 # SKIP Testing skip all
    ok 1 # skip Testing skip all
        1..1
        ok 1 - findbin is reinitialized for every test
    ok 2 - aggtests/findbin.t
        1..1
        ok 1 - subs work!
    ok 3 - aggtests/subs.t
        1..2
        ok 1 - slow loading module loaded
        ok 2 - env variables should not hang around
    ok 4 - aggtests/slow_load.t
        1..5
        ok 1 - aggtests/check_plan.t ***** 1
        ok 2 - aggtests/check_plan.t ***** 2
        ok 3 # skip checking plan (aggtests/check_plan.t ***** 3)
        ok 4 - env variables should not hang around
        ok 5 - aggtests/check_plan.t ***** 4
    ok 5 - aggtests/check_plan.t
ok 1 - nested tests
ok 2 - Startup should be called once
ok 3 - ... as should shutdown
ok 4 - Setup should be called once for each test program
ok 5 - ... as should teardown
ok
All tests successful.
Files=1, Tests=5,  2 wallclock secs ( 0.03 usr  0.01 sys +  0.11 cusr  0.02 csys =  0.17 CPU)
Result: PASS

Much, much nicer. As an added bonus, plans can now be cleanly asserted. I hope to have it on github soon, and later on the CPAN.

...not really. I still really like using GitHub, and (obviously) they do much more than I could churn out in a weekend, let alone an hour. Also, I used Perl.

That said, we've recently stopped using them for work code. Things just didn't work out. I hope we can migrate back to them someday, but for now we needed to make a change.

We're using gitosis, now. I've used it before, and it's a really fantastic little tool. As with most good tools, it does one job and does it very well. That means that it doesn't get in the way of the workflow you want to use.

That also means it doesn't go out of its way to make any particular workflow easy. That's fine!

I had originally thought we'd be fine having users share their repos from their home directories, as everyone has read access to every other user's homedir. This ended up being tedious and stupid, for a number of reasons, including that each user puts his code in a different subdirectory under home. (Each choice other than my own just seems so unnatural!)

When we were using GitHub, we had a program that did something like this:

• ensure that master-user/repository existed
• fork it to you/repository if needed
• clone you/repository to cwd

I updated this program to work on our gitosis setup with uniform per-user repository URLs. After all, a GitHub fork is just a clone with a tiny smidge of added metadata, near as I can tell.

The master user in our scenario is gitosis@git.example.com, and every user also has a login on that box. Here's the program:

#!/usr/bin/perl
use strict;
use warnings;

{
  package Git::Hubclone;
  use base 'App::Cmd::Simple';

  use autodie qw(:default :system);
  use Getopt::Long::Descriptive;
  use Sys::Hostname::Long;

  my $HOST = q{git.example.com};

  sub usage_desc { '%c %o <repo>' }
  sub opt_spec {
    return (
      [ 'bare-only', "do not create a local clone, only hub copy" ],
    );
  }

  sub validate_args {
    my ($self, $opt, $args) = @_;

    my $user = $ENV{USER};
    my $repo = $args->[0];

    $self->usage_error("no repo name given") unless $repo;
    $self->usage_error("do not run as root") if $user eq 'root';
    $self->usage_error("illegal repo name")  if $repo !~ /\A[-_a-z0-9]+\z/;

    if (! $opt->{bare_only}) {
      $self->usage_error("already in a git repository!")       if -d '.git';
      $self->usage_error("directory './$repo' already exists") if -d $repo;
    }
  }

  sub run {
    my ($self, $opt, $args) = @_;

    my $user = $ENV{USER};
    my $repo = $args->[0];

    if (hostname_long eq $HOST) {
      require File::HomeDir;
      require File::Path;
      require File::Spec;

      my $git_dir  = File::Spec->catfile( File::HomeDir->my_home, 'git' );
      my $repo_dir = File::Spec->catfile( $git_dir, "$repo.git" );

      unless (-d $repo_dir) {
        File::Path::mkpath($git_dir);
        system qq{git clone --bare gitosis\@$HOST:$repo.git $repo_dir};
      }
    } else {
      system "ssh $HOST git hubclone --bare-only $repo";
    }

    unless ($opt->{bare_only}) {
      my $clone_cmd = qq{git clone $user\@$HOST:git/$repo.git};
      system $clone_cmd;
    }
  }
}

Git::Hubclone->import;
Git::Hubclone->run;

I should really fix up App::Cmd::Simple to not require that import for package-in-program applications. Still, I'm pretty happy with how easy this was! Special thanks go out to Paul Fenwick for autodie, which saved me having to write loads of error checking.

First off, my slides. Whenever I give a presentation, someone will ask me whether the slides will go online. Sometimes they do, and sometimes they don't. This year, I gave three presentations: one on Git, one on some new email libraries, and one on Rx. The email slides will probably go online soon. I'm not sure the other two will.

It's great to get feedback on my slides, and of course praise is nice too, and I'd like everyone who can to be able to benefit from whatever explanation I can provide, but sometimes slides just don't make sense as a PDF online. Very often, the slides are merely visual aids for the spoken content. Without narration, the slides are often less than useful, or even counter-productive.

I would like to get my Git slides online, but it will require that I first record the audio and sync it with presentation. That will require a good bit of work, and I'm not sure when I'll make the time to do it. (After all, presenting for YAPC earns me free admission and interesting questions. Presenting for SlideShare mostly just gets me notices that someone else has posted spam to the comments.)

So, on to the YAPC recap.

I arrived Saturday with plans to get some beer at the Hofbrauhaus, but it fell through. Instead, we went to Claddagh. I was irritated that I had to order four beers before they actually had any left, and that it was Smithwick's instead of something local. Still, the service was great and the food was really good. The price was right, too!

Sunday I mostly holed up in my room and worked on slides. In the evening, Dieter and I headed out to the arrival dinner. As usual, it was underwhelming, but it was still nice to see people and have a beer. (Strangely enough, the only local beer I saw at the beer-o-centric Sharp Edge was Hop Devil, which is all I drink at home.)

During the conference proper, I kept busy polishing slides until just before each presentation. I also managed to see a good number of talks, many of which were very good. Josh ben Jore had a good talk on debugging, Jim Brandt spoke about Workflow.pm, Jonathan Swartz spoke about CHI, and there were lots of others. It was also fun to see Dieter's talk on Dist::Zilla, as it was the first time anyone else gave a talk on something I wrote. (The next day, Chris N. included a lot of App::Cmd in a talk, too.)

The conference dinner at Heinz Field was really good, and speakers got a VIP tour of the stadium, including the locker rooms, control rooms, and so on. I got lots of photos and called my brother to make him jealous. He's a big Steelers fan.

It was also great to see so many people I hadn't seen in years. Sungo, especially, had been MIA since Toronto. Lots of people have changed jobs, which was fun to hear about, or lost jobs, which was not.

It was great to see Dieter again and rehash long-dormant in-jokes. (At some point someone asked, "So... I guess you guys know each other?") Sadly, we did not produce our discussed Perl Filk Concert lightning talk.

Finally, on Wednesday evening, David Golden was in town and stopped by CMU. We had some pizza and beer, and we worked on the Metabase a bit. Only a few code changes were made, but the whole thing was documented some more and is now on the CPAN. Hopefully this will help get momentum back up.

That might be all I have to say about YAPC. It was loads of fun, and I'm looking forward to next year -- but I'm definitely going to enjoy having a few months with no slides to think about, too.

It’s at times like these, when I ponder my choice to give up life as a code grunt to be a community manager, that I’m tempted to kick myself. Sure, you feel bad if you miss a release deadline, but at least you don’t feel like a kid who invited everyone to her birthday party and then nobody showed up, you know?

We’re two weeks out from the second Freebase Hack Day, and I’m running round trying to get people to register. You know how it is: you contact everyone you know, nag them incessantly, tweet til everyone unfollows you, put it on your personal blog…

Our previous Hack Day. Photo credit: Greg Elin under CC-BY

But seriously, the last one was a blast, and I love the unconference format for how little organisation it actually takes to get heaps of intense, fascinating conversations going. Plus this year our hosted app development platform is ready for prime time, and I think that’s going to really spur a lot of rapid development. And the Semantic Web stuff has really picked up steam this year, so that’s a whole stream of stuff that didn’t really get much play last time round. And some of the Freebase Experts team are coming to town, and I’m just dying to meet them.

Photo credit: Greg Elin under CC-BY

There’s always a burst of registrations right at the end. I know this. I really do. But last night I had an anxiety dream that nobody showed up at all, and I was sitting in the empty office waiting for people who never arrived.

Don’t let it be real. Come to Hack Day. July 11th, San Francisco. It’ll be awesome.

Perl 6 is hurtling toward completion. The specification is nearly complete, and Rakudo now passes 68% of the specification tests. Applications like November are being written in Perl 6. Perl 6 is no vaporware, and the day when Perl 6 is ready for widespread use is coming quickly.

Perl 6 has been in development since 2000, and in that time many people may have forgotten about the plans we've had for Perl 6. There may be those who have never even heard about the plans for Perl 6. Those of us who live in the Perl world are aware of the great changes afoot, but there are plenty of people who are not.

I think that the time is right to help make those people aware of Perl 6, and to remind them constantly of what's coming. My proposed technique is simple and it takes advantage of the key elements of time and repetition to help remind everyone about Perl 6.

We need to stop referring to Perl 5 as "Perl" and start calling it "Perl 5."

Specifying the "5" in "Perl 5" calls attention to the fact that there is more than one Perl. It makes the listener or reader who is unaware of Perl 6 wonder why the 5 is specified. For the reader who knows about Perl 6, hearing "Perl 5" reminds her that Perl 6 also exists.

I don't think it will be too tough. All I ask is that, at the very least, when writing about Perl 5 in your blogs or mailing lists that you specify the version you're talking about. It doesn't even need to be every instance. I'm guessing we'll find that repeatedly saying "Perl 5" in a long message will get tedious both for writer and reader.

I think the way to look at it is that "Perl 5" is the formal name for the language, and later references can refer to it as "Perl," almost like a nickname. Just that first reminder of "Perl 5" will be enough to help lodge in the reader's brain.

With enough time & repetition, it will get to be habit in our minds. With enough time & repetition, the computing world will be reminded of Perl 6 coming soon.

With the perldoc-search program, you can easily search all your perl documentation to find where that elusive "add_build_element" method is documented.When I say this: perldoc-search add_build_element I get this: perltoc - perl documentation table of contentsModule::Build::API - API Reference for Module AuthorsModule::Build::Cookbook - Examples of Module::Build Usage

Read more of this story at use Perl.

I would like to announce the pilot Perl event in Baltic States: Baltic Perl Workshop 2009.It happens on 21st of November, 2009 in Riga, the capital of Latvia.Venue is located in the beautiful center of Old Town. Konventa hotel offers special price for attendees of Baltic Perl Workshop: 50 € per night (single or double room).Please consider attending first ever Baltic Perl Workshop (there will be no more first one). Talk submissions are open. The workshop is free.

Read more of this story at use Perl.

Whether you're talking about git, Mercurial, SVK or some other distributed version control system, it's very important to realize that one of its strongest benefits is "distributed". Many people don't care about this, but it can tremendously boost productivity when combined as a part of overall development strategy.

Recently I got so fed up with our dev box that I install Ubuntu on my work computer so I could work locally. This was because our dev box was routinely running with loads around 20, and on one occasion hit a load of 57. Seems the admins thought it would be a good idea to let a bunch of other teams develop on that server but not tell us about it. I just couldn't work like that. Instead of using my computer as a dumb workstation with Windows, I now develop locally and am (usually) not dependent on our dev box being available.

That's also the reason why I have MySQL server installed locally, rather than using our test database. Sometimes that test database goes down, so I don't want to depend on that, either. And the admins thought it would be a good idea to let a bunch of other teams develop on that server but not tell us about it.

In fact, sometimes I break down. With a recent back injury, I was working from home for a week. It was slow, frustrating and painful because I wasn't really set up to work from home and this caused me to lose even more productivity. Had I been able to work from my laptop, there would have been no performance hit. Of course, even without a back injury, we had thousands of employees unable to work full hours because of a Tube strike.

Which brings me back to version control. More than once I've been unable to check in code because the subversion server has been down (never mind when the repository simply freaks out and tells me I can't commit). With something like git, I could do all of my work locally and if I can't push to the central repository, oh well.

So yeah, you know all this, but seriously, do the math. Let's be really generous and assume that our dev box, subversion server, mysql server and ability to be physically present at work all run at 99% availability. This means that your chance being "available" to work is .99^4, or about 96%. If the average availability of all of those things seems around 95%, then your "availability" drops to about 81%. Before I started working so hard to "disconnect" from the network at work, my availability was often lower than 81%. It was awful.

Now imagine sitting in a job interview and telling them that you're only going to work 81% of the time. You're not going to get the job. But the reality is, many corporate environments enforce 81% availability. This is insane, but common. And of course, we also know that those four items aren't the only things which can disrupt your productivity, but those are the four which most impact mine, so that's why I chose them.

When you're setting up a dev environment for programmers and you want it to be as productive as possible, don't force them to depend on things they can't control, such as network availability. Of course, as an employee, it's always fun when the network goes down and you see everyone wandering around with a coffee cup in hand, having been electronically muzzled. What on earth could convince people that this is a sane development environment?

I finally got to my first YAPC this year - coincidences and conflicts kept me away in previous years. So excuse me for getting all dewy eyed, but it was a great experience! Haven’t felt that close to a development community since the early Oreilly Perl conferences. I feel all energized about Perl again and it makes me want to host a winter workshop in the bay area someday.

Here are some cool things I learned about:

• distroprefs - A way to improve the automatability of CPAN installs; namely, dealing with those pesky distributions that insist on asking you questions. We spent a bunch of time at Hearst writing expect scripts to deal with these so that we could automatically install our large CPAN base on different servers. Wish I’d known about distroprefs!
• Dist::Zilla - I’ve always been vaguely bothered by the amount of boilerplate in my CPAN distributions (copyright, author, version, etc.) but Dist::Zilla has finally provided the means and motivation to do something about it.
• KiokuDB - Stevan Little’s talk on KiokuDB got me excited about leaving behind RDBMSs and the Impedence Mismatch on my next web project. Just store Moose objects! Of course people have been getting excited about object databases since the 80s, and there’s probably a reason that Oracle is still around. But allow me to enjoy my dream before it smashes into reality.
• Memory persistence - As Steven Lembark gleefully hammered home, Perl never shrinks a variable. Once $foo has had a million bytes or @foo has a million entries, it’ll reserve that space forever, even if you empty it. Makes me want to look real close at any globals in persistent web environments.
• CGI::Inspect - Brock Wilcox wins the award for most satisfying feature demo. Insert CGI::Inspect’s inspect() somewhere in a web template. When it runs, a dashboard launches in a new browser window, with a stack trace and the ability to inspect and change lexical state before the page resumes its render. Of course, other environments like JSP/Eclipse have had live debugging of web pages for years, but in Perl this still seems groundbreaking :), and the possibilities for extending it are intriguing.

Here are the pages for the talks referenced above:

CPAN - A big enough lever to install the world

Dist::Zilla - Automating quality since 2008

KiokuDB - A Real World Introduction

Memory Manglement

Drop-In Web-Based REPL for CGI Applications

Explaining Metabase

At YAPC::NA, Ricardo and I finally released some early alpha Metabase modules to CPAN. Metabase is a project we started in 2008 at the Oslo QA Hackathon to provide a new transport and storage infrastructure for CPAN Testers and other CPAN distribution metadata. We’ve been working out of github until now, but think it’s time to encourage others in the Perl programming community to take a look.

One of the challenges we’ve had is just explaining Metabase to people. Here’s what I’ve come up with: Metabase is a database framework and web API to store and search opinions from anyone about anything. In the terminology of Metabase, Users store Facts about Resources. Facts are really opinions or claims; there is no inherent truth in them. Resources are logical identifiers, generally in the form of a URI, since the things we’re most interested in describing are online.

Since CPAN Testers is our first use case, I’ll use that to illustrate the concept. Currently, CPAN Testers email test reports about distributions on CPAN. In the Metabase world, each CPAN tester is a User. The Resource is a CPAN distribution file – a ‘distfile’ URI that describes a distribution tarball on CPAN such as cpan://distfile/DAGOLDEN/Capture-Tiny-0.06.tar.gz. The Fact is the test report. Today that’s just the text of the email message, but in the future it will be structured data. ¹

It’s easy to see how this can be generalized to an entire ecosystem of facts and opinions about CPAN distributions, such as CPANTS, ratings, annotations, tags, etc. Today, every CPAN website has its own database and its own API so mashups and remixes require building an interface to each. Metabase offers the possibility of a unified way for all these tools to store, share and cross-reference information about CPAN.

Design considerations

Metabase has some similarities to other, general ontology frameworks, but we decided to focus our efforts around some pragmatic tradeoffs using CPAN Testers to guide the design. We think these decisions generalize well to other uses, but we’re trying to follow a YAGNI strategy and only implement what CPAN Testers will need before worrying about more general cases.

(1) High fact volume

There are currently about 18,000 “latest” distributions on CPAN, and the entire history of CPAN is a bit over 100,000 distributions. By contrast, there are over 4 million CPAN Testers reports already and the total is growing by about 250,000 per month. Reports are usually consumed as a batch process to create summary statistics on the CPAN Testers website and relatively few reports are ever reviewed in detail. Therefore, the Metabase design to date prioritizes Fact submission over search and retrieval.

(2) Minimalist Perl for client libraries

Many CPAN Testers want to run smoke tests using a Perl installation that has as few extra modules installed as possible. Therefore, any Metabase components used by clients such as Metabase::Fact or Metabase::Client::Simple need minimal non-core dependencies. On the other hand, anything that will run on the server side can take full advantage of Modern Perl tools like Moose, DBIC and Catalyst.

(3) No barriers to contribution

Today, new CPAN Testers don’t have to register or get authorized to start contributing, they just start sending emails to a mailing list that collects reports. For Metabase, users generate a user profile fact and submit it as a credential with their facts. Metabase manages user identities within the Metabase itself – adding new users automatically – allowing new contributors to join without any external pre-authorization hurdles that might discourage participation.²

(4) Open to extension and evolution

We want to make CPAN Testers tools available for corporations or perl5-porters or anyone with customized testing needs. We want people to come up with new facts and new resources. That means defining interfaces well and leaving implementation details open to change. Metabase allows lots of choices about things like underlying data storage technologies or permissible fact types.

Metabase specifies data storage capabilities, but the actual database storage is pluggable, from flat files to relational databases to cloud services. Metabase defines how Fact classes can provide validation and custom index data on top of a very simple data model. Moreover, Metabase is a fairly dumb repository; any intelligent analysis must be done by third parties extracting and transforming data.

Where do we go from here?

I’ve successfully submitted test reports to a local Metabase, but for Metabase to become what we need for CPAN Testers (and more), there are still quite a number of things to do:

• Improve documentation
• Test and refine Metabase components
• Design and implement search capabilities
• Establish one or more ‘development server’ Metabases for testing robustness and throughput
• Migrate the 4 million CPAN Testers reports from NNTP to Metabase
• Migrate CPAN Testers downstream analytics to use Metabase as a source
• Have some CPAN Testers pilot submitting reports to Metabase
• Develop and release new CPAN Testers clients that can submit structured Metabase reports

If you are interested in following or participating in Metabase, there is a (still low-volume) mailing list you can subscribe to and several of the key developers can be found on #toolchain on irc.perl.org.

1. Technically, a CPAN Testers report will be a collection of more granular facts like prerequisites, test output, environment and so on.
2. Verification or authorization is optional

Reposted from a blog entry by Mark Stosberg

Today Melody was announced as a fork of the Perl-based Movable Type platform. I helped the Melody project as it prepared to launch, in part advising on how to best to relate to the Perl community.  One of the stated interests of Melody is to refactor the project to use CGI::Application, which I maintain. Tim Appnel has already spelled out a vision of what a "CPANization" of Movable Type might look like, and I've looked in depth at what the initial steps towards using CGI::Application could be.

My own vision for Melody is a code base that's very focused on publishing and content management, with all the infrastructure outsourced to CPAN modules that are well-written, well-documented, and well-tested.  The collaboration between Melody and CPAN would be a two-way code flow. While there are more CPAN modules that Melody could make use of, there are number of pieces of Melody which should be packaged as independent modules on their own and released to CPAN.

One example is the great "dirification" that already exists in Movable Type. This is the functionality that turns any given string of words into a reasonable representation in URLs. It seems like an easy problem on the surface, but Movable Type has a sophisticated solution that takes into account what it means to do this well across many different languages. I also couldn't find any existing CPAN module which already takes on this problem space, so I started to extract this out of Movable Type myself and published a draft of String::Dirify. For that initial release, I ripped out all the fancy multi-language support, and there is still more significant work to be done to untangle this layer from from Movable Type. (If you want to pick up that project and work on it, there's also some discussion of testing String::Dirify).

While Movable Type already had an open source release, I expect Melody to have  a more adventurous evolution, and I look forward to it becoming a shining star in the Perl community, not just for the exterior functionality, but also because internals have an opportunity to become an example of best practices.

Mark Stosberg has been using programming Perl for the web for over a decade. He is the Principal Developer at Summersault and maintains several CPAN modules including Titanium and Data::FormValidator.

Without fanfare, v0.37 of Padre has been released.

The list of changes based on the Changes file:

• "Introduce Temporary Variable" refactoring tool (SMUELLER)
• Added a friendly icon on the toolbar to toggle comments (GARU)
• Crazy Win32::API AllowSetForegroundWindow hack to allow the Single Instance Server to correctly foreground itself (ADAMK)
• Added Padre::Search search and replace API (ADAMK)
• Switching to last edited file is now Ctrl-Shift-p (SZABGAB)
• Be compatible with older version of File::Path (RYAN52)
• Links to Mibbit were replaced by links to our irc.html (SZABGAB)
• Merged the code of Padre::Plugin::Encode into Padre (SZABGAB)
• Update directory only when switching to new editor (SZABGAB)

Soon it should arrive to a CPAN mirror near you.

Enjoy and thanks to all the people who put effort in Padre!

For the upcoming release of Apache SpamAssassin, we’re considering dropping support for perl 5.6.x interpreters. Perl 5.6.0 is 9 years old, and the most recent maintainance release, 5.6.2, dates back to November 2003. The current 5.x release branch is 5.10, so we’re still sticking with a “support the release branch before the current one” policy this way.

If you’re still using one of the 5.6.x versions, or know of a (relatively recent) distro that does, please reply to highlight this….

Because I've been distracted writing my new CPAN dependency graph generator (to repair the main flaw in the CPAN Top 100 website) I haven't been paying the website a whole lot of attention in the last week or so.

I just run the website data generator once or twice a week, and other than that, I'm focusing on creating the next iteration of the support software.

After updating the data file today, imagine my surprise when a new module absolutely roared ahead of the previous #1 highest score, posting a new record of over 500,000 FAILure points.

It would appear that the June 18th release of DBD::mysql was something of a disaster, and the high number of modules that depend on MySQL means that the impact on end users is very high.

It's good to see that when something like this happens, it is highlighted on the list very quickly and very obviously.

However, in retrospect I'm somewhat disappointed that it took this long to highlight it (because I don't run the updater often enough). What if it just took a while because it took a few weeks to gradually climb up to the top of the list, fighting against high-dependency modules with one or two rare failures, that aren't growing so fast.

I'm wonder what might be done to spot these dramatic failures faster, within a few days of their release, rather than having to wait a week or so.

Even starting to do something like that will probably require a serious improvement to the update date of the data that feeds the website, and changes to teach the analyser about the concept of time.

Food for thought, certainly.

The question now of course, is how high a really big OMGIBROKECPAN event might score. Something in the vicinity of several million failure points certainly seems possible, especially if the number of CPAN Testers installs grows.

On #padre the other night, a discussion broke out on the subject of collaborative editing (ala MoonEdit, SubEthaEdit, etc). A number of people had never even heard of it before.

Collaborative Editing, as it is typically implemented, involves "connecting" your editor to someone else's editor. Instead of opening the files for local editing, your typing events and such are transmitted to the original host in real-time.

The effect is that two or more people can edit the same document in real-time, everyone with their own individual syntax highlighting, key bindings, preferences and plugins active.

After establishing that all the standalone libraries for doing collab (like libgobby) weren't going to be appropriate, we started pondering how we might implement it ourself, and what features OTHER than pure collaborative editing might be handy.

Some of the sorts of things you could probably do quite easily is to "steal" a copy of an open unsaved file from someone else's Padre, open a file for remote viewing only (think live remote code reviews), or allow remote users to take actions for code that can only be run on one physical system (due to exotic hardware etc).

Then there's the whole social aspect to explore. Putting your Padre into "Conference Mode" might enable all sorts of curious effects (like automatically notifying you if someone else on the local is editing a CPAN distribution that you have a checkout of as well) and let you do different things compared to collaboration with someone that is truly physically remote. Perhaps it might allow other conference attendees to hijack your editor so they can run module test suites on platforms they don't have... the possibilities for sharing development effort in a more intimate environment are quite extensive.

There's all sorts of different things that you might do, and all sorts of different methodologies you could apply to make them happen. Multicast, broadcast, UDP, things like Bonjour.

And just as useful for Padre is that doing collaborative editing would require significant improvements in the cleanliness of Padre's internals. Before we could make half this stuff work, Padre would need to have a much more robust and sophisticated understanding of the ways in which concepts align and mix together across the Main Window, Editor Panels, Documents and Projects.

Having permanently running clients and servers would drive improvements in Padre's Task API and background processing capabilities, and will require improvements in threading and IPC.

Until this notional day in the future when we actually know how to implement all of this in Padre safely and sanely, I've created a new Padre::Plugin::Swarm plugin to serve as a vessel for all the insane test cases.

"Padre Swarm" is intended to be a container for experimental work and shiny demonstrations. It will intentionally ignore stuff like robustness, scalability and security.

Swarm gives people a chance for people to show off the kinds of shiny toys that might be possible when you don't have to care that implementing it means allowing arbitrary remote code execution by the entire world, and that it only works on multicast-capable networks, and that in the process the Single Instance server doesn't work any more and Padre crashes if you try to refresh the Outline panel.

As usual, if you find this interesting or exciting, I strongly encourage you to drop in at #padre and have a chat. If you like what is happening, we can get you commit and everything else from there.

(As an aside, Padre is now sitting at around 20 active contributors. We now only need another 50% increase in contributors to catch and exceed the number of active contributors on Emacs) :)

Note: Pictures of my trip to the Middle East are now available.
The events in this entry happened on 12-13th June

Saudi Arabian Adventures - Day 1
My first true day in Saudi Arabia was one of rest. After 30+ hours of travel, being able to sleep in a bed was truly lovely. I woke up fairly late in the day, caught up on e-mail, and then decided to go exploring. There was a food hall just around the corner from the hotel, and while I gawked at the sign displaying opening times, (I was concerned I had missed lunch), and worried about the wording that said the hall was for Saudi Aramco staff only, one of the staff members waved me in with a smile.

What I discovered is that the food hall has both very good and very affordable food. Cans of drink, bottles of water, and cups of coffee were almost universally 1 riyal (about $0.30 AUD). Some basic meals were available for about 4-5 riyals (under $2.00 AUD), with the best value for money undoubtedly being the "budget meal", which provided soup, vegetables, rice/potatoes, a generous main meal serving, and a dessert, all for 11.5 riyals (about $4 AUD).

I got myself a salad (which were excellent), some soup, and some coffee, and felt very thankful that I had changed myself some 300 riyals (about $100 AUD) while in Dubai, since I hadn't spotted any way of doing money exchange at the Dammam airport.

The rest of my travels included a walk around the camp, the most interesting part of which was a fenced area that seemed to include a library and a bowling alley. Here also a large sign proclaimed the facilities were for Aramco employees only, and passes will be checked on entry. I was getting the feeling that having an employee pass would be a good idea.

Saudi Arabian Adventures - Day 2
Today was my first day of teaching. Fuad, my contact at Aramco, had arranged to meet me at 7am at the hotel, and I had noted from my trip to the food hall yesterday that it opened for breakfast at 5am. Collectively, these were hints that Saudis were morning people. I set my alarm for 5am, which in my jet-lagged state would feel like waking up at noon in Australia.

Going to the food hall for breakfast once again had me feel like I was in America; the most popular food that people ordered were waffles! Of course, I didn't fly half-way around the world just to have waffles for breakfast, so I ordered a generous bowl of ful medames... and waffles.

Of the two breakfast foods, ful was the clear winner. It tasted good, and unlike the waffles, it had actual nutritional content. I finished breakfast with plenty of time to meet Fuad back at the hotel at 7am.

Fuad chuckled at my attempts at an Arabic greeting, and asked in perfect English "where are you learning this stuff, a phrase book?". I think that means my Arabic has some ways to go. ;)

Fuad asked about my flight here, explained that the Aramco camp was generally divided into zones, the general business hours (7:30am-3:30pm), and helped me get set up with my lab. I was also glad to hear he would be attending my first course.

At this point, I should explain a little about what I was teaching. My class consisted primarily of people with job titles that started with "Geo-", and who regularly had to deal with data that was in formats different to what they would prefer. Unsurprisingly, they'd like to use Perl to help solve these problems. That's an increasingly common story with the courses I teach.

What made this interesting is that rather than having a good four or five days (which is what I'd prefer), I was teaching two classes, a dozen students in each, with only 2.5 days per class. To top that off, I was in a different country, and I had no idea how comprehensible my Australian accent would be. I'd put together a plan that covered the basics (syntax, variables, and control structures) a tour of the CPAN, and as much regexp content as I could squeeze in.

When we were doing introductions, it became clear that my class had a good sense of my humour. My favourite introduction was from one student who said "my name is very hard to remember... it's Mohammed".

The first day of teaching went well, and from the bountiful, well-thought out questions, it was clear that I was being understood.

Posted: 25th June 2009.

Tags: adventure aramco dhahran perl saudi travel

Bookmark:

So YAPC::NA is over, or at least the conference proper is (there are still a few days of hacking to look forward to). It was a pretty bad YAPC for me, I was sick the whole time and didn't really see any talks and barely socialized. I do feel better now so hopefully I can still salvage some fun.

I am also a little disappointed in my own talk as well. I was hoping to be a bit more practical, but every time I tried to write slides with that in mind I got lost in endless digressions. In the end I think it was a little too shallow and ranty, not exactly what I had hoped for. It did seem to be rather well received so at least I'm happy about that.

I also don't think I presented well, but given that I feeling very bad I guess I can't complain. By the end I was having tunnel vision and my ears felt really pressurized. Someone said I sounded like I was bored with my talk, so I guess it didn't go as bad as I felt ;-)

Anyway, here's hoping for a productive hackathon, and a quick recovery.

These links are collected from the Perlbuzz Twitter feed. If you have suggestions for news bits, please mail me at andy@perlbuzz.com.

• How Perl 6 just sells itself
• Converting an A4 PDF to Letter with Margins
• Read-optimize your source code
• Optimizing life for reduced frustration
• Request for assistance: The most important toolchain bug
• Stealing from Padre for Vim
• Allison Randal in Linux Magazine: "Why Parrot is Important"
• Anyone want to adopt Getopt::Auto?
• Rakudo Perl 6 dev release #18: Pittsburgh
• DBIx::Class vs Class::DBI vs Rose::DB::Object vs Fey::ORM
• 3D graphics with Perl 6
• A gentle introduction to Moose
• "Radical Career Success in a Down Economy": A free webcast on July 1st
• Gray coding at the pediatrician's office
• What I got out of the Parrot Workshop
• Rakudo gets mainstream press attention in SD Times
• Movable Type fork is an opportunity to harness CPAN
• YAPC::NA 2010 will be in Columbus, Ohio
• Scaling out is frictionless when you use open source software

Schwern has released Test::Simple 0.89_01. Nested TAP is now available. In case you don't recall (or haven't heard about this), you might write a subtest like this:

  use Test::More tests => 3;

  pass("First test");

  subtest 'An example subtest' => sub {
      plan tests => 2;

      pass("This is a subtest");
      pass("So is this");
  };

  pass("Third test");

And get the following nested TAP:

  1..3
  ok 1 - First test
      1..2
      ok 1 - This is a subtest
      ok 2 - So is this
  ok 2 - An example subtest
  ok 3 - Third test

(Adrian, I believe you have a branch of Test::Class which uses this? :)

The main objective of the Iron Man Blogging Challenge is to encourage more people within the Perl community to promote the language to the world outside the Perl echo-chamber

There are nice things in Perl but I think it is hard to promote the language itself. Indeed, most posts were not about how awesome you can write a foreach loop in Perl...

There were posts about Moose which is ok as it shows to the outside world that Perl has a nice object system - even if it is an add-on but not many people will start using Perl because it has an nice object system.

There were posts about various little modules that can do interesting things. Things that can be interesting to people not yet using Perl. E.g. about Curses::Toolkit and Politics::AU::Geo ). The former while not on CPAN yet had nice demos, the latter pointed to the actual result page.

There were also lots of posts about Catalyst that can be interesting as a reason to learn Perl.

So while I enjoy reading the posts what I am missing is more posts on subjects that can be intersting for people who do not yet know Perl. I am missing both things that need some perl knowledge to actually use (such as the above 3 examples) but it would be nice to see posts about projects/products that do not require knowledge of Perl to start using. Projects that are written in Perl and that for advanced usage might require Perl knowledge. These would be reasons to start learning Perl.

So for example I'd like to see posts about

Bioperl, POE, PDL, Bricolage, Krang, SpamAssassin, Ack, WebGUI, RT, Gantry, Webmin, and even Bugzilla

just to name a few.

What are you missing?

Which of these two events sounds more interesting?

Joe Celko is going to be giving talks at YAPC this summer.

Or, in an announcement entitled "Learn Mad Database Skillz at YAPC::NA 2009"

It really, truly pays to learn the ins and outs of SQL, just like any other language. And if you’re a Perl hacker, you have a great opportunity to do just that at YAPC::NA 10 this summer. Famed SQL expert Joe Celko, author of numerous volumes on SQL syntax and techniques, will be offering two classes on SQL at YAPC:

• Introduction to RDBMS and SQL for the Totally Ignorant. Well, okay, the name of the course is a bit unfortunate, but the material covered is not. If you know little or nothing about SQL, this course should be a terrific introduction.
• The New Stuff in SQL You Don’t Know About. So much great stuff has been added to SQL over the years, and ORMs know virtually none of it. Learn how to put that stuff to work in your apps!

That first announcement is usually what I get when people ask me to announce something in Perlbuzz. "Hey, we're having the East Podunk Perlapalooza next week." Yeah, so? Who cares? Why will Perlbuzz readers care?

David Wheeler, author of the latter text, understands what most geeks seem not to grasp: The mere existence of your Foo is not enough for people to be interested. Look at all the topics that David covers, to encourage interest from as wide a range of people as possible.

• What YAPC is.
• Who Joe Celko is.
• Joe Celko's body of work.
• Why we still need to know SQL in the age of ORMs.

I was reminded of this while going through Chad Fowler's excellent The Passionate Programmer in preparation for our webcast "Radical Career Success in a Down Economy" next week. One of Chad's big points is "Your skills are not self-evident." It's not enough to do great work at work, but you must also let people know about what you've done, specifically your boss. The same is true of your open source projects.

Why do people use ack? It's useful, but how do people know about it? I talk about it, and I tell people why they should use it on its home page, in a section called "Top 10 reasons you should use ack."

If someone asks you about your project, can you explain its awesomeness, and why he should use it? If not, why are you bothering? And if you can, are telling everyone you can about it? If not, why are you bothering?

For more on writing interesting announcements, please see the Perlbuzz How to contribute page.

In an open source project - especially one that involves Perl - most of the people just want to write code. Nearly no one wants to do PR. It even has a bad reputation among many open source developers.

In Padre I constantly struggle if I should spend my time adding a new feature or trying to recruite more developers. The former works better in the short term, the latter works better in the long term. Eventually there is a ballance I try to keep.

One of the things I really miss from the perl 5 porters - the people who maintain and improve perl itself - is the communication.

In the last couple of weeks we saw lots of blog-posts and mailing lists posts - sometimes not in the nicest way - why there is no regular release of perl 5 and how a speed related bug that was fixed 18 months ago was not released in an intermediate version. In these discussions several of the p5p people got involved but unfortunatelly this is mostly deffensive talk. Not good PR.

I also got fed-up with the lack of release and asked on the p5p list what is stopping them from releasing the next version of perl and what is so hard in the release process that they cannot do it more regulary?

I got several related replies. One of the them on the mailing list others on IRC.

According to those the next perl 5 release is within a small number of weeks. That is certainly good news. I would prefer to see a Release Candidate already that people can take for a ride but in the meantime I can point you at the three major issues around the release that need to be done:

1. Fix a bunch of outstanding issues.
2. The release must be stable.
3. The actual release process should be automatic.

For item no. 1 you need to talk to the perl porters or look at the topic on the #p5p IRC channel where they point you at the list of outstanding issues

Item no. 2 was addressed in an e-mail of David Golden and the thread following it. That should be summarized in a document and the test automation system should be implemented.

For item no. 3 we first need to have a written document of what needs to be done and then we can attempt to automate it. The document can be based on the e-mail of Nicholas Clark. Actually I already created a pod version of that file that I hope soon will be added to the source tree of perl and then it can be fine tuned and implemented.

The interesting issue is that 2. and 3. need to be implemented only once and they can be reused for every future release of perl 5. So if you'd like to see more frequent releases that's where you need to help.

Luckily those, especially no. 2, are issues that any perl hacker can get involved easily. They only need perl coding.

I know I am not a good PR person and I hardly know anything about what's going on in the perl 5 porters but I hope this post helped a bit. At least it might trigger someone to write a better explanation of the situation.

Oh and a last note, if you'd like to get involved, the description on how to deal with the Git repository can be found here

I’m at CMU (my alma mater) for YAPC. I haven’t been back here since just after graduating in ‘91. It’s a trip seeing it after nearly twenty years, with a mix of the completely familiar and unfamiliar.

I wandered around campus for a while this morning. Each place jogs random memories from my head.

• Learning Unix through Andrew
• Working on the Oz interactive fiction project, still THE most fun software job I’ve ever had (if anyone wants to pay me to write text adventure games today, I’m there)
• Trying desperately (and failing) to understand Hidden Markov Models for a summer research job on Kai-Fu Lee’s Sphinx project
• Watching the “Dragon Lady” whip through dozens of programming languages in as many days, feeling like Neo in the Matrix (”I know Prolog!”)
• Working on a LISP machine
• Getting pathetically addicted to Moria
• Working as an esteemed Ccon (”Computer consultant”), helping users log in and sorting printouts for minimum wage
• Eating pizza on the floor of Wean hall at 3:30 AM during many all-night homework hacking sessions

I feel old.

At the BBC, we have some mandated technology stacks. While this is almost galling to some people's interpretation of TIMTOWTDI, the reality is that in a large-scale environment, it's very helpful to admins to not have to worry about which VCS, Wiki or other tools teams are using when needing different teams to work in shared environments. Of course, when you get down to lower level stuff, you really don't want to tell a team they can't upgrade DBD::mysql because another team requires an older version, but compromises have to be made. Today, that compromise is becoming painful.

I recently tried to commit some changes to our architecture dependent CPAN modules (for example,modules in deps/lib/perl5/i486-linux-gnu-thread-multi/). However, subversion kept insisting that a bunch of files I was adding already existed in the repository, even though svn ls svn://path/to/archdeps did not list those files. After I and another developer kept digging around, we finally gave up and came up with a work-around. First, I did this:

find . -type d -name '.svn' --exec rm -fr {} \;

I then tarred up the deps directory, made a fresh checkout and untarred the files over the fresh checkout. The commit then worked just fine. Unfortunately, I then found out that somehow this corrupted several XS extensions, forcing me to reinstall them. Luckily this was trivial, but the last time this happened (yes, this has happened before!), I couldn't even run the cpan shell due to corrupted XS dependencies.

Having run into numerous issues with Subversion and having found git to be so pleasant to work with, I thought it was time for us to give up and switch to git, but we can't. Even if we get developer buy-in, we use Trac. Trac is tightly integrated with subversion but not with git. There are git plugins for Trac, but they're not ready for production use. Even if they were, it shouldn't be the case that two developers dealing with a frustration invest a lot of time and money into solving a problem whose solution potentially impacts a number of other teams.

The fact that Trac is so tightly integrated with our subversion use of Subversion is a huge frustration (I don't know enough about the internals of Trac to know if this is our fault or theirs). I'd love to find a better way of dealing with this and I'd be ecstatic to see some strategy for decoupling the two. Unfortunately, it's not going to happen.

There are some great benefits of working with larger organizations, but this ain't one of them.

A few weeks ago Bruno Vecchi submitted failing tests that serialize code references to KiokuDB. I finally got around to trying to make these tests pass on my flight to YAPC|10. Long story short, KiokuDB now serializes closures, including managing shared closure variables between several code references.

Serialization is relatively trivial. If you're not familiar with PadWalker, it's a module that lets you introspect the lexical pads of subroutines, including extracting references to all the closed over variables using the closed_over function.

The code itself is deparsed to text using B::Deparse, and the hash returned by closed_over is serialized along side it, as normal making sure that shared references are handled consistently. Any closure data that can be handled by KiokuDB will work normally.

Deserialization was slightly trickier, PadWalker needed additional functionality to actually assign to the subroutines' pads.

First KiokuDB compiles the body of the code while creating an empty lexical environment:

my $vars = join(", ", keys %$pad);

eval qq{
    my ( $vars );
    sub $body;
};

Then the new set_closed_over function added to PadWalker goes through the pad and aliases any values from the provided hash into the lexical scope of this subroutine.

This function goes through the CV's PADLIST field, and simply aliases the referents into the appropriate slots in the pad array.

On the one hand it's very reassuring that when you want to do this sort of thing in Perl you almost always can. What's slightly disappointing is that you do have to drop to C for these stuff.

My first version used the B module to introspect the structures, but this API is simply not powerful enough to allow modification of the data with aliasing semantics.

Thankfully the incredible amount of work in recent years on Perl 5 seems to be inspiring some new directions in Perl 5's core, and while the C API is pretty cumbersome, it does let you do almost anything you want if you want it hard enough.

Dubai Adventures - Day 3
Today I decided to travel to Jumeirah beach and surrounds, a decision I made purely based upon the number of tags and photographs in the area for Google Earth. Since I'd be doing this during the day, I decided to take a practical approach. I left the laptop and most electronics back in the hotel, and packed lots of water and sunscreen. I wore my most breathable cotton top, a linux.con f.au hat, and what Jacinta calls my "pirate pants". These pants are very loose, and very breezy, and very comfortable. But as a result, I didn't look like a local, and I didn't look like a typical tourist, either. I was, however, protected from the sun and reasonably cool.

I caught a taxi to Jumeirah mosque, for just a little over 10 AED (about $3.50). I didn't expect to find the mosque open (and indeed, it wasn't), but I got some nice photographs. The main reason I wanted to visit the mosque is that it was walking distance to the beach, and (if you're me) Sheik Zayed Road. While two days ago I had discovered that walking around the old souks resulted in someone trying to sell me a fake rolex every few minutes, today I discovered that walking around Jumeirah had a taxi beep at me every few minutes, hoping that I would need a lift somewhere.

Jumeirah beach is very pretty, and very hot. There weren't many people out, and a lifeguard watched lazily from a tower. The beach came with lots of rules, one of which was "strictly no cameras", so I had to be a little more discreet with my photographs. The water looked extremely inviting, and I relished the idea of hopping in for a swim. Unfortunately (and accidentally), I had left my swimming gear back in the hotel, and my mask and snorkel back in Australia.

At the beach I got some great photographs of the skyline of Sheik Zayed Road, especially of the Burj Dubai, which distinctly reminds me of the Combine Citadel in Half-Life 2. Having looked around the beach, I decided to head off toward Sheik Zayed road, which I was certainly would be filled with marvels.

The walk was a lot harder than my previous wanderings around Bur Dubai, even though the distance was shorter. I was walking through a residential distract, and as such there was a lot less cover, and fewer amenitities. It was clear that Jumeirah was home to the rich, with magnificent mansions and expensive cars.

After about 1.5km of walking from the beach, I was finding that the heat and exposure was making me a little uncomfortable, and that while I had packed a lot of water, I'd probably need to think about looking for more. Luckily, I had just chanced upon a park. It was beautiful, green, lush, and immaculately maintained. For some reason, the gate was shut, with a large padlock, and it looked as if this gate had been shut for some time. Perplexed, I went to a second gate, and it was also closed. I noticed a person in the park, but he appeared to be a groundskeeper, who was distracted by talking on his mobile phone. This had me even more perplexed; why spend water, and money, and high-value property on creating a park that nobody can get into? This wasn't marked as a private park, and it even had a playground in the middle of it. Maybe it closed during the middle of the day, and re-opened later on?

A climbing check, and two stealth checks later, I found myself a section of soft grass and a shady tree under which to snooze. I woke after about half an hour, a little surprised that the groundskeeper hadn't woken me. Refreshed, I continued on my way, and discovered much to my delight that Dubai has public refrigerated drinking founains. With my water bottles refilled, and my face and arms splashed with blissfully cold water, I continued onwards.

The region I was walking through seemed to be filled with houses that were universally big. However one house in particular stood out. It was on the corner of two streets and was unfinished. That in itself is nothing special; Dubai seems to be in a constant state of construction, but it was clear this building had been unfinished for quite some time. The shell was made, but that's all which was there. Amusingly, a sign on the side said "For Sale or Rent".

After walking past another park (also beautiful, closed, and deserted), I eventually reached Sheik Zayed Road. This is the home to many of the tallest buildings in Dubai, and is a spectacular sight from the air, but amazingly dull on the ground. These were hotels, and office blocks, and the odd food outlet here and there. It was also completely impossible to cross on foot. It looked like some footbridges were under construction, but they were nowhere near finished, and they were so big I suspect they were for yet another road link, and not meant for pedestrian traffic at all.

I walked down the street, and got some decent photographs. My plan was to walk across to the Dubai Mall, take a look around, and then walk up the other side of Sheik Zayed Road to the Emerites Towers. As it happened, try as I might, I couldn't find a way to get across the street, which has twelve lanes, and cars travelling at considerable speed.

Hot, disappointed, and tired, I decided to try and find someplace cool, and preferably with restrooms. Being Dubai, there was a small mall not too far away, and I made my way toward it. It had a supermarket, and an electronics superstore, and in a rare moment of impulse purchasing, I grabbed a copy of EA Sports Active (which apparently sucks less than Wii Fit), after carefully checking it for any signs that indicated that perhaps it would only work in certain regions.

If I had my laptop with me, I would have pulled out my maps and looked for interesting activities, but since it was not available, and since I had gone all this way to discover that Sheik Zayed road sucked, I hailed a taxi to get back to town. This was one of the best parts of the day, since the driver was very talkative. He was from Pakistan, to which he'd love to return, but apparently there are no jobs there. He had worked previously in the Ukraine, and had moved to Dubai eight months ago.

My driver commented that Dubai is an artificial city. It's got artificial islands, artificial buildings, artificial snow, and artificial parks. People are here because Dubai has done an amazing job of marketing itself to the rich, but due to the global financial crisis, the rich aren't rich anymore, and they're flying back overseas. Tourism is down, jobs are more scarce, and so many of the not-so-rich who have come here for work are also going back overseas. Apparently all this was great reducing traffic congestion and getting me back to my hotel, but it sucked for the city as a whole.

Back at the hotel I had a bite to eat, and wrote this blog. My plane leaves at 9:35am, which means I want to be at the airport at 7:35am, which means I want to be leaving the hotel at 7am, which means waking before that. If I want to avoid too much jet-lag I should be going to bed now, and waking super-early in the morning. Instead I'm here posting blog entries, and discovering that the the UAE censors flickr, so there won't be any photographs until I get home.

Posted: 22nd June 2009.

Tags: dubai perl travel

Bookmark:

Dubai Adventures - Day 2
I'm never going to be awake early in the morning so I can visit the museum if I keep getting back to my hotel at 12:30am, and blogging about my day. And yet that's what I'm doing right now.

Today was my second day in Dubai, and I discovered that Dubai has westerners, women, and arabs; they all hang out in the malls. After sleeping in again (goodness, I was tired), I decided to go to the Mall of the Emirates (the one with indoor skiing in middle of the desert), and then make my way by foot to the Burj Al Arab area, which I estimated was a 2.5 km walk, and hence pretty easy.

I caught a taxi, as the distance was much too far to walk, and I had no idea which bus to catch. I had a great chat with the taxi driver; he was from Nepal, had moved to Saudi to work (not far from where I was working), and was now in Dubai. I asked if he had any family here, and he laughed. Apparently Dubai is so expensive, especially with rent, there's no way he could support a second person; the plan is to work here as long as he can stand it, and take his savings back home.

It seems that with the global financial crisis (GFC), while Dubai still has lots of taxis, it's lacking the tourists who normally use those taxis. The GFC and the drop in tourism is something I'd heard from a few sources now, and it's clear that a lot of Dubai depends upon the tourist trade.

When I stepped into the Mall, it felt like stepping into America. It had American shops, American food, and American people. Actually, as I was to discover after talking to a few of them; the majority of the westerners were from the UK. Almost everything in the mall had prices similar to what I'd pay back home, and like most malls, most of it were things I had absolutely no interest in at all.

Luckily, I found one of my objectives for the day, and that was a free wireless hotspot. I eventually found the access point, it's just outside the cinema, and the SSID is "yournetworkname". I used it to call home (again, hurray for VoIP), and after a good chat ran down my laptop battery. Unfortunately, I didn't bring my external battery (it's heavy!), so any remaining communication had to be done via a hand-held.

The other discovery I found in the mall was an arabic gift shop, which was filled with local wares and tourists. Amazingly, I actually enjoyed browsing here; this certainly wasn't stuff I'd see in Australia (or anywhere else), and the prices were extremely affordable. There was also a hilarious collection of art-work out the front of the door. Words can't do these justice, so you'll just have to look at the photos (coming soon) and see.

I purchased a selection of gifts and while often it was easy for me to say "$x will like this", I discovered that was very hard when $x = 'jarich'. Eventually I went for the shotgun approach, and got her one of everything.

Somehow, after two calls home, some IM chats with friends, a selection of gifts, some photographs of the ski slopes, and some chats with the locals, it had got rather late, so I decided to set out on my walk. This was made more challenging by the fact that my laptop had a flat battery, and that's where my maps were located, but since the Burj Al Arab is the world's tallest building, it's not hard to spot and walk toward it.

I should correct myself there, it's not hard to spot. Walking toward it was hard, since there were huge multi-lane highways, and construction work, and no obvious way to actually walk there. I could (and in hindsight, should) have got a taxi there. It would have cost AED 10 (about $3.50), which would have been a bargain for the XP I would have gained visiting it and the nearby Souq Madinat.

As it was, I ended up walking fruitlessly in what was essentially the wrong direction. Eventually I stopped for some food outside the Lulu Hypermarket, where the serving staff of the fast-food place I frequented were delighted that I was from Australia, and gave me detailed information about the busses in the area, and were generally awesome. They were from the Phillipines, and collectively were the nicest people I've met all trip. It's just a shame the food wasn't.

Having found the bus (AED 2 rather than AED 45 for a taxi), I looked forward to being able to see all the huge towers along Sheikh Zayed Road, which is the home of some of the most opulent hotels and establishments. Consequently, I was dreadfully disappointed when it decided to go down Al Wasl Rd instead, which doesn't have much sightseeing at all. Eventually the bus pulled into the Bur Dubai bus station, which I had never been to before, but which I knew was walking distance from my hotel.

Unfortunately for me, I didn't know in which direction my hotel lay, and due to an inefficient hashing algorithm being used in my brain, I couldn't recall the name of the street with my hotel, either. I knew I was staying in the Ramee Apartments, but apparently Ramee is a chain with a number of Dubai hotels. Plus, my laptop battery was dead, so no checking maps for me.

Luckily for me, I was in Dubai, so I just walked across the road to the mall (Dubai has plenty), plugged in my laptop, and fired up Google Earth with all my cached maps. The bus stop was only 600m from my hotel as the crow flies.

On walk to my residence, down a surprisingly deserted street, a man approached me from an alleyway. From experience I figured this probably meant he wants to sell me a 100% genuine fake rolex, although for a moment where I wondered if people get mugged in these parts, and there was a reason nobody else was walking down this street.

"Excuse me, but do you speak French?" Okay, I really didn't expect that. I admitted I didn't, and wondered where this conversation would go. "Oh. Then do you have a hotel?" I started to wonder if French-speaking, homeless tourists were common in these parts. "Yes, I have a hotel. I'm walking there now." "Oh. Are you sure? Because if you need a room, I have a room you can rent. It's very nice." "No, really, I do have a hotel." "Oh, okay, where are you from?" "Me? Australia. How about you?" "I'm from France."

Suddently, the conversation made sense. My new acquaintance was over here working, and with rent prices being so high, he was looking for a room-mate. He would have been most happy with someone who could speak French, but I'd do. If I was unhappy with my hotel, or sticking around in Dubai, he'd be happy to split the room costs with me, 50-50.

We had an interesting chat, and I pointed that I could see my hotel and was about to turn down one of the streets toward it. We wished each other a good night, and I dropped past the 24 hour Super Happy Mart to replenish some supplies for breakfast.

Tomorrow I should try to wake up early and visit the museum, but based on prior experience I'm not sure if I will. Failing that, my plan is to catch a bus or taxi to Jumeirah Mosque, check out the beach, and then walk through the back-streets of Al Bada to Sheikh Zayed Road, which should put me in the heart of the "down-town district". I'm sure I can find myself some trouble from there. ;)

Posted: 22nd June 2009.

Tags: dubai perl travel

Bookmark:

Last week we took Iris to the pediatrician for a checkup, during which they weighed, measured, and inoculated her. The measuring device, which I later learned is called a stadiometer, had a bracket on a slider that went up and down on a post. Iris stood against the post and the nurse adjusted the bracket to exactly the top of her head. Then she read off Iris's height from an attached display.

How did the bracket know exactly what height to report? This was done in a way I hadn't seen before. It had a photosensor looking at the post, which was printed with this pattern:

(Click to view the other pictures I took of the post.)

The pattern is binary numerals. Each numeral is a certain fraction of a centimeter high, say 1/4 centimeter. If the sensor reads the number 433, that means that the bracket is 433/4 = 108.25 cm off the ground, and so that Iris is 107.75 cm tall.

The patterned strip in the left margin of this article is a straightforward translation of binary numerals to black and white boxes, with black representing 1 and white representing 0:

0000000000
0000000001
0000000010
0000000011
0000000100
0000000101
0000000101
...
1111101000
1111101001
...
1111111111

If you are paying attention, you will notice that although the strip at left is similar to the pattern in the doctor's office, it is not the same. That is because the numbers on the post are Gray-coded.

Gray codes solve the following problem with raw binary numbers. Suppose Iris is close to 104 = 416/4 cm tall, so that the photosensor is in the following region of the post:

...
0110100001 (417)
0110100000 (416)
0110011111 (415)
0110011110 (414)
...

But suppose that the sensor (or the post) is slightly mis-aligned, so that instead of properly reading the (416) row, it reads the first half of the (416) row and last half of the (415) row. That makes 0110111111, which is 447 = 111.75 cm, an error of almost 7.5%. (That's three inches, for my American and Burmese readers.) Or the error could go the other way: if the sensor reads the first half of the (415) and the second half of the (416) row, it will see 0110000000 = 384 = 96 cm.

Gray code is a method for encoding numbers in binary so that each numeral differs from the adjacent ones in only one position:

0000000000
0000000001
0000000011
0000000010
0000000110
0000000111
0000000101
0000000100
0000001100
...
1000011100
1000011101
...
1000000000

This is the pattern from the post, which you can also see at the right of this article.

Now suppose that the mis-aligned sensor reads part of the (416) line and part of the (417) line. With ordinary binary coding, this could result in an error of up to 7.75 cm. (And worse errors for children of other heights.) But with Gray coding no error results from the misreading:

...
0101110000 (417)
0101010000 (416)
0101010001 (415)
0101010011 (414)
...

No matter what parts of 0101110000 and 0101110001 are stitched together, the result is always either 416 or 417.

Converting from Gray code to standard binary is easy: take the binary expansion, and invert every bit that is immediately to the right of a 1 bit. For example, in 1111101000, each red bit is to the right of a 1, and so is inverted to obtain the Gray code 1000011100.

Converting back is also easy: of the Gray code. Replace every sequence of the form 1000...01 with 1111...10; also replace 1000... with 1111... if it appears at the end of the code. For example, Gray code 1000011100 contains two such sequences, 100001 and 11, which are replaced with 111110 and 10, to give 1111101000.

The Perl 6 design team met by phone on 06 May 2009. Larry, Allison, Patrick, Nicholas, and chromatic attended.

Read more of this story at use Perl.

On behalf of the Rakudo development team, I'm pleased to announce the June 2009 development release of Rakudo Perl #18 "Pittsburgh". Rakudo is an implementation of Perl 6 on the Parrot Virtual Machine. The tarball for the June 2009 release is available from http://github.com/rakudo/rakudo/downloads . Due to the continued rapid pace of Rakudo development and the frequent addition of new Perl 6 features and bugfixes, we continue to recommend that people wanting to use or work with Rakudo obtain the latest source directly from the main repository at github. More details are available at http://rakudo.org/how-to-get-rakudo . Rakudo Perl follows a monthly release cycle, with each release code named after a Perl Mongers group. This release is named "Pittsburgh", which is the host for YAPC|10 (YAPC::NA 2009) and the Parrot Virtual Machine Workshop. Pittsburgh.pm has also sponsored hackathons for Rakudo Perl as part of the 2008 Pittsburgh Perl Workshop. In this release of Rakudo Perl, we've focused our efforts on refactoring many of Rakudo's internals; these refactors improve performance, bring us closer to the Perl 6 specification, operate more cleanly with Parrot, and provide a stronger foundation for features to be implemented in the near future. Some of the specific major changes and improvements in this release include: Rakudo is now passing 11,536 spectests, an increase of 194 passing tests since the May 2009 release. With this release Rakudo is now passing 68% of the available spectest suite. Method dispatch has been substantially refactored; the new dispatcher is significantly faster and follows the Perl 6 specification more closely. Object initialization via the BUILD and CREATE (sub)methods is substantially improved. All return values are now type checked (previously only explicit 'return' statements would perform type checking). String handling is significantly improved: fewer Unicode-related bugs exist, and parsing speed is greatly improved for some programs containing characters in the Latin-1 set. The IO .lines and .get methods now follow the specification more closely. User-defined operators now also receive some of their associated meta variants. The 'is export' trait has been improved; more builtin functions and methods can be written in Perl 6 instead of PIR. Many Parrot changes have improved performance and reduced overall memory leaks (although there's still much more improvement needed). The development team thanks all of our contributors and sponsors for making Rakudo Perl possible. If you would like to contribute, see http://rakudo.org/how-to-help , ask on the perl6-compiler@perl.org mailing list, or ask on IRC #perl6 on freenode. The next release of Rakudo (#19) is scheduled for July 23, 2009. A list of the other planned release dates and codenames for 2009 is available in the "docs/release_guide.pod" file. In general, Rakudo development releases are scheduled to occur two days after each Parrot monthly release. Parrot releases the third Tuesday of each month. Have fun! References: [1] Parrot, http://parrot.org/ [2] YAPC|10 http://yapc10.org/yn2009/ [3] Parrot Virtual Machine Workshop, http://yapc10.org/yn2009/talk/2045 [4] Pittsburgh Perl Workshop, http://pghpw.org/ppw2008/

Read more of this story at use Perl.

Putting the finishing touches on my YAPC::EU 2009 paper and at the last moment, decided to run a code snippet since I was trying to compare it to Ruby mixins:

package PracticalJoke;
use Moose;
with 'Bomb'   => { exclude => 'explode' },
     'Spouse' => { exclude => 'fuse' };

I should have written excludes and not exclude. That's a very easy mistake to make, but given that people might want to run my examples, it's embarrassing :)