Planet Perl

Just a quick reminder, the YAPC::NA 2009 Call for Venue will be closing in less than a month. If you plan on trying to host YAPC::NA 2009 and you haven't started on your bid, you'd better get to it! The deadline for submissions this year is June 1st (as we would like to announce the 2009 venue at this year's YAPC::NA). You can get all of the details and see the Call for Venue posting on The Perl Foundation's blog. If you have any questions, please feel free to contact me.

Read more of this story at use Perl.

I've been playing around with MAD this morning [...] The current state of the tests for MAD is a bit sad, though I hope to have something that will make interested porters glad in the near future. -- Jesse Vincent, not a bad lad to spend a tad on MAD. (gad, Dad! it's not a fad).

Read more of this story at use Perl.

For a party game, needed to repeatedly pick random letters from the (ASCII) alphabet. I had my laptop, so I just did it. A friend, a Java programmer, was amazed. My favorite part was when he said "you're going to type out all 26 letters, aren't you?" Um, no I'm not.

perl -le "print ['a' .. 'z']->[rand(26)] while <STDIN>"

For those not quite getting it, here's the shortest I could think of in Java (without getting to the point of obfuscation and with the caveat that they're functionally equivalent, though internally they handle things in a moderately different way):

import java.io.*;

class Pick {
    public static void main(String[] args) {
        String letters    = "abcdefghijklmnopqrstuvwxyz";
        BufferedReader br = new BufferedReader(new InputStreamReader(System.in));

        while (true) {
            try {
                br.readLine();
            }
            catch (IOException e) {
                System.out.println("Caught exception trying to readline: " + e);
                System.exit(1);
            }
            int rand = (int)(Math.random() * 26);
            System.out.println(letters.charAt(rand));
        }
    }
}

Of course, then you have to compile it and run it as separate steps.

Can anyone post any common code snippets which express themselves more naturally in Java than Perl? (Libraries which can be readily duplicated in Perl don't count)

The Perl 6 design team met by phone on 07 May 2008. Larry, Allison, Patrick, Will, Jerry, Nicholas, Jesse, and chromatic attended.

Read more of this story at use Perl.

brian d foy asks why people are passionate about Perl. So, brian, here are a pumpking's answers to your questions.

The person who introduced me to Perl showed me that I could throw away all those fragile sed and awk scripts and get the job done much quickier. So that gets us to the second question:

I first starting using Perl to process very large data files. Or at least they were considered very large at the time: less than 2 Go, I think. But I quickly grasped the thing and began to wrote small scripts to automate various things on my computer (and others').

I kept using Perl because it was fun. And it still is.

I can't stop thinking about Perl because... they won't let me! If I stop thinking about Perl, I know they're out there to get me, lurking in the dark corners of the intarweb, with their bug reports, their suggestions and their patches.

I'm still using Perl because, well, basically, because I'm paid for it. But on the other hand, I chose a job where I was getting paid to use Perl. I could have chosen a Java, a C, or (my goodness) a PHP job. But with a proper Perl job you have the CPAN at hand, ready to be used. So you can concentrate with the problem solving and the business side of your applications, because almost all the libraries you'll need are already there.

I get other people to use Perl by hitting them on the head with a -- no wait. Actually that's the most interesting question. I get other people to use Perl by trying to make Perl a less annoying language, in my own scale.

I also program in C, but mostly when I have to patch perl itself -- the only place where you can't use Perl.

The Perl 6 design team met by phone on 07 May 2008. Larry, Allison, Patrick, Will, Jerry, Nicholas, Jesse, and chromatic attended.

Allison:

• spent my time this week slicing and dicing the exceptions implementation
• replaced the old internals with the new system
• checked that in yesterday
• still a few failing tests in edge cases on the branch
• did more work on the Parrot Foundation

c:

• I own an acre of Mars, we could incorporate there

Allison:

• don't you own a cow in the Philippines?

c:

• yes, but that doesn't give me any governmental powers

Patrick:

• isn't that worth a lot?

c:

• the peso is improving against the dollar

Jesse:

• moving on...

Larry:

• clear bill of health from my medical reports
• hacking a lot against the standard grammar in my STD5 implementation
• lots of refactoring
• all of the various parameters that used to go through separately now go through as part of the Match object
• including the "fate" and whether we're peeking at the longest token set
• the longest token matcher works now
• I threw out my old mechanism for gathering Match objects
• it now creates the more-or-less correctly
• lots of grammar tweaks, as suggested by Mitchell Charity
• lots of refactoring of how logging works so that it doesn't always spew enormous quantities of information to the screen
• I can actually run the parser quite quickly now, for some definition of quickly that approximates 2000 characters per second
• matches symbols directly, rather than calling a rule, which is faster
• does the backoff now on longest token matching
• started refactoring the grammar on the assumption that I cna trust the longest token matcher
• no longer any nofat rule
• the longest token should match the fat arrow, if there is one
• started refactoring the quoting rules to parse as if they were sublanguages
• getting rid of extra rigamarole to recreate the other mechanism we already use for other languages
• working out the linkage for switching in and out of sublanguages
• how to get to the outer language from the inner language
• calling into pure Perl from closures in a regex
• or the host language if you're calling the regex from another language
• nailed down the available methods for Match objects in the specs
• giving a talk in Seattle on Friday at SPU
• flying to Japan on Saturday

Patrick:

• spent a lot of time teaching this past week
• cleared up now
• mostly I've answered questions on mailing lists and IRC
• I'm not always sure that I'm helpful, but I'm there
• yesterday I worked on trying to get a bunch of little small things here and there
• fixed up a few things in PCT internals
• today I'm bringing PGE up to date with some of the latest changes in S05
• these all help Rakudo and other languages in small ways
• trying to clean out my backlog and clean up a bunch of RT tickets
• I'll continue over the next couple of days
• and blogging about it as I go

Jerry:

• things are busy, mostly non-Parrot related stuff
• submitted a ticket that I hope Patrick can close today

Patrick:

• many languages depend on the old behavior, including Plumhead
• I'm not certain about some of them

Jerry:

• mostly otherwise answering questions on #parrot
• making sure that things are set up for the real work phase of GSoC
• making sure that students have their CLAs, if not commit bits
• astonished to see how much work Jonathan is getting done in just two funded days
• it's amazing to see how much a motivator money can be
• I'd like to see more of it, hint hint

c:

• working on closing as many open Parrot bugs as possible
• applying as many open patches as possible
• should be able to help on the concurrency branch soon
• otherwise preparing for the release
• going to check on received CLAs this week

Nicholas:

• found it curious that Perl 5.10 has the best state implementation of any language
• wanted to steal tests from another implementation
• had a discussion with Leon about SMOP
• there's no real description of how all of these implementations fit together
• Rakudo plus Parrot is a complete implementation
• SMOP and kp6 fit together nicely

Jesse:

• I started a wiki page on the Perl 6 wiki at Perl_6_Implementations

Patrick:

• I don't know that it says how things fit together

Jesse:

• I tried to encourage other people to contribute stuff
• didn't get much uptake

Nicholas:

• should we suggest to Daniel that he should help explain things?

Jesse:

• that's more likely to get people contributing to it

Will:

• there's definitely some confusion about it within the Grants Committee

Jerry:

• SMOP has the highest documentation-to-line-of-code ratio of any implementation

Patrick:

• it needs a good overview though

Nicholas:

• I'll ask Daniel to explain more
• especially its relationship to Parrot and Rakudo

Jerry:

• it sounds like it could be an alternate runcore for Perl 5 as well

Jesse:

• tried a few different things
• decided to write a test for Rakudo
• tried a simple arithmetic test pulled from Pugs
• found that Rakudo didn't implement a function specified in the S29 draft
• Patrick helped me write a couple of lines of code to implement it
• then discovered that fudge didn't support try blocks in a specific way
• Larry patched that
• then found that incrementing an undefined value didn't work in Rakudo
• that was the end of my day
• I still need to write up my findings
• how easy is it for someone without experience in Rakudo and its internals to pick things up and contribute something?
• more difficult than I thought it might be, but it's getting more doable
• it's important to understand how it might fail before trying to get people to do it
• then I started trying to play with MAD on the weekend
• found and fixed a bug in its XML
• refactored it such that you can run MAD's tests in the core if you add a copy of XML::Parser to the core
• it's not far enough yet, but it's a start

Nicholas:

• is it going to be difficult to restructure the Parrot foundation from 501(c)(3) to 501(c)(6)?

Allison:

• you can do pretty much the same thing
• sponsors are on the board in a c6
• they're only advisory in a c3
• the sponsors we've talked to are mostly only interested in getting regular status reports and the like

Jesse:

• is there any jumping around to transfer copyright to the new foundation?

Allison:

• we'll do a copyright assignment from the Perl Foundation to the Parrot Foundation
• all of the CLAs that went into the code up to the point of signover will be fine
• but we'll essentially copy the Perl Foundation CLA to a Parrot Foundation CLA

Will:

• do we need to contact committers who haven't signed a CLA?

Patrick:

• where does Rakudo fall?

Allison:

• still under the Perl Foundation
• it doesn't move at all

Will:

• do we want to split up the repository at that point?

Allison:

• eventually, we'll want to do so anyway
• it's not an urgent thing

Jerry:

• what would it take to version a Perl6Regex frontend to PGE?
• let grammars specify a version of the grammar

Patrick:

• I did that before by having a separate compiler
• you're talking about something a bit finer grained
• I don't want to do that
• as we get closer to 1.0, that'd be fine
• I already have enough to do keeping up with the latest versions

Will:

• I don't think we want to keep up old versions

Patrick:

• I don't mind sticking to our deprecation cycle
• I hadn't put the change from today into the deprecation list yet
• we'll get to it in a couple of weeks

Jerry:

• just trying to figure out how to push forward with changes to PGE without having to update every language in the repository

Patrick:

• freeze S05?
• not a great solution

Larry:

• I heard that

Patrick:

• the last few changes have been great
• I'm not really serious about that

Larry:

• some of them you even asked for

c:

• it's an advantage to have these languages in the repository
• we can update them
• but only if we can run the tests before and after and know that they pass

Will:

• we might consider removing languages with failing tests and no recent updates
• there are 17 grant proposals, some of them Perl 6-related
• please comment on the TPF blog
• it'll help

Jesse:

• blog.perlfoundation.org

One of my favorite books is The Origin of Consciousness in the Breakdown of the Bicameral Mind, by Julian Jaynes, a psychologist at Princeton University. Nearly everyone seems to agree that this is either a work of profound genius, or of profound crackpottery, and also that they aren't sure which it is. Jaynes' theory, as nearly as I can summarize the book, is something like this:

Human consciousness (which Jaynes describes and defines in considerable detail) is a relatively recent development, dating back at most only about 3,000 years or so.

There is the shocking part of the theory. Most people probably imagine consciousness arising much, much earlier, perhaps before language. Jaynes disagrees. In his theory, language, and in particular its mediation of thought through the use of metaphors, is an essential prerequisite for consciousness. And his date for the development of consciousness means that human consciousness would postdate several other important developments, such as metalworking, large-scale agriculture, complex hierarchical social structures, and even writing. Jaynes thinks that the development of consciousness is a historical event and is attested to by written history. He tries to examine the historical record to find evidence not only of preconscious culture, but of the tremendous upheavals that both caused and were the result of the arrival of consciousness.

If preconscious humans farmed, built temples and granaries, and kept records, they must have had some sort of organizing behavior that sufficed in place of consciousness. Jaynes believes that prior to the development of consciousness, humans had a very different mentality. When you or I need to make a decision, we construct a mental narrative, in which we imagine ourselves trying several courses of action, and attempt to predict the possible consequences. Jaynes claims that Bronze Age humans did not do this. What then?

Instead, says Jaynes, the two halves of the brain were less well-integrated in preconscious humans than they are today. The preconscious mentality was "bicameral", with the two halves of the brain operating more independently, and sometimes at odds with each other. The left hemisphere, as today, was usually dominant. Faced with a difficult decision, preconscious human would wait, possibly undergoing (and perhaps even encouraging) an increasingly agitated physical state, until they heard the voice of a god directing them what to do. These hallucinated voices were generated by the right hemisphere of the brain, and projected internally into the left hemisphere.

For example, when the Iliad says that the goddess Athena spoke to Achilles, and commanded and physically restrained him from killing Agamemnon, it is not fabulating: Achilles' right brain hallucinated the voice of the Goddess and restrained him.

In Jaynes' view, there is a large amount of varied literary, anthropological, and neurological evidence supporting this admittedly bizarre hypothesis. For example, he compares the language used in the Biblical Book of Amos (bicameral) with that in Ecclesiastes (conscious). He finds many examples of records from the right period of history bewailing the loss of the guidance of the gods, the stilling of their voices, and the measures that people took, involving seers and prophets, to try to bring the guiding voices back.

Jaynes speculates that mental states such as schizophrenia, which are frequently accompanied by irresistible auditorily hallucinated commands, may be throwbacks to the older, "bicameral" mental state.

Whether you find the theory amazingly brilliant or amazingly stupid, I urge to to withhold judgment until you have read the book. It is a fat book, and there is a mass of fascinating detail. As I implied, it's either a work of profound genius or of profound crackpottery, and I'm not sure which. (Yaakov Sloman tells me that the response to Wittgenstein's Tractatus Logico-Philosophicus was similarly ambivalent when it was new. I think the consensus is now on the genius side.) Either way, it is quite fascinating. There needs to be some theory to account for the historical development of consciousness, and as far as I know, this is the only one on offer.

Anyway, I did not mean to get into this in so much detail. The reason I brought this up is that because of my continuing interest in Jaynes' theory, and how it is viewed by later scholars, I am reading Muses, Madmen, and Prophets: Rethinking the History, Science, and Meaning of Auditory Hallucination by Daniel B. Smith. I am not very far into it yet, but Smith has many interesting things to say about auditory hallucinations, their relationship to obsessive-compulsive disorder, and other matters.

On page 37 Smith mentions a paper, which as he says, has a wonderful title: "Involuntary Masturbation as a Manifestation of Stroke-Related Alien Hand Syndrome". Isn't that just awesome? It gets you coming and going, like a one-two punch. First there's the involuntary masturbation, and while you're still reeling from that it follows up with "alien hand syndrome".

To save you the trouble of reading the paper, I will summarize. The patient is a 72-year-old male. He has lesions in his right frontal lobe. He is experiencing "alien hand syndrome", where his hand seems to be under someone else's control, grabbing objects, like the TV remote control, or grabbing pieces of chicken off his plate and feeding them to him, when what he wanted to do was feed himself with the fork in his right hand. "During his hospital stay, the patient expressed frustration and dismay when he realized that he was masturbating publicly and with his inability to voluntarily release his grasp of objects in the left hand."

Reaction time tests of his hands revealed that when the left hand was under his conscious control, it suffered from a reaction time delay, but when it was under the alien's control, it didn't.

Whee, freaky.

The Portugeuse Perl Workshop is less than a month away, and I'm giving a keynote about "Why People Are Passionate About Perl" on June 4. So, why are you passionate about Perl?

I've been thinking about this quite a bit since we selected the topic. I can only really talk about why I am passionate about Perl. I want to talk more about that in the keynote. Post your story somewhere. If you don't think I'll find it, send me a link. I'll google the title "Why People Are Passionate About Perl" though :)

I've also been thinking that I want to do this keynote without slides, or at least the boring sort that are really just talking points. Chicago doesn't have 10 of the world most dangerous anythings, so Damian's approach is right out (and I don't have the Chicago accent anyway). I can't think fast enough or clever enough to use the Takahashi Method. I don't especially like orange, so copying Leon's slides don't quite fit either.

I figured I'd take a few good stories of people and their passion for Perl and show their picture on the screen while I told it. Then, in the audio that I'll record for Perlcast, I'll figure out how to make the album artwork be the slides. Or do something else like that.

Here's a set of questions, mini interview style, to get you started on your own post:

• The person who introduced me to Perl showed me that...
  
• I first starting using Perl to...
  
• I kept using Perl because...
  
• I can't stop thinking about Perl...
  
• I'm still using Perl because...
  
• I get other people to use Perl by...
  
• I also program in ... and ..., but I like Perl better since...
  

[Note: I've turned off comments. If you want to tell me why you are passionate about Perl, make a blog post somewhere. Explain to the world why you really, really like Perl (and I mean like like!). I think they call this thing a meme. ]

[Note note: Steal this meme for Python, or Ruby, or Java, or LISP, or whatever. I think the most enjoyable posts from my "Five Things I Hate About Perl" where actually the ones from other languages. I learned the most from those. ]

Take a look at this block of code and see if you can guess what it does.

END: {
    print "Exiting...\n";
}

print "s = $s\n";

BEGIN: {
    $s = 'Hello from the BEGIN phase';
}

You'd expect that the BEGIN block sets $s at compile time, so then that gets printed in the print statement, and then the END block executes at the end of the program. Yes, it would, except that those aren't BEGIN & END blocks. They're normal blocks that just happen to be named BEGIN and END.

Credit Randy Lauen for bringing this to my attention. Today at my day job he found a horrifying code construct, one that I'd been using (incorrectly) for quite some time now. You see, these two blocks are not the same:

BEGIN {
    # Here's some code that gets executed at compile time
}

BEGIN: {
    # This code gets executed at run time, because the colon
    # after BEGIN means it's just a plain old label
}

Randy had been tracking down a bug for over an hour and couldn't figure out why his variables weren't getting set at compile time. Turns out the BEGIN block he was modifying was actually a not-special-at-all BEGIN: block. And he found 40+ instances of the spurious colon in the codebase. Of course, INIT, CHECK, UNITCHECK and END are also rendered non-special by a colon as well.

Most of the time, this is invisible. People tend to put their BEGIN blocks at the top of the program, so the difference between a real BEGIN block and a block that happens to be named BEGIN are not noticed. Still, it's a a bug waiting to happen. Sounds like an ideal check for Perl::Critic policy, no? Mike O'Regan jumped to it, and submitted a patch to Perl::Critic, so those of you using Perl::Critic (and if you're not, you should be) can get warned of using colons on those specially-named blocks once a new version is released, shortly I hope.

I'm not the only one to have been doing this. Google Code Search turns up over 100 instances, including some in the core Perl tests (heck, maybe from me!), as well as mailman and POE.

Check your codebases. If you're using ack, it's just:

$ ack --perl '^(BEGIN|END|INIT|CHECK|UNITCHECK):\s*{'

Have you fallen prey to this? Let me know.

I'm spending May in Lille, France, so on May 13, I'm going to visit Bruxelles.pm just up the road. It's a quick train trip. I'll be at the Greenwich Café around 18:00 if anyone wants to meet earlier. I have to catch the last train back to Lille at 21:00, so don't come late!

I thought about visiting London.pm tonight, which is also not that far by train. Eurostar wants $800 per person for the trip, so I guess that is right out. Even booking two weeks in advance they want a lot more than I'm willing to pay to have a few drinks with one of the biggest Perl Mongers groups. So, sorry London :(.

Er, no it's not. This brief (and snarky) blog post pretty much sums up my feelings.

I've already been pretty critical of Arc, but pretty much ignored it after that. I've taken another peek and, once again, I'm still terribly impressed. Paul Graham has made arc code smaller for those projects he wants to focus on. He does this with decent libraries (well, decent if you don't mind 'ASCII-only' and layout controlled via HTML tables).

Welcome to the Intarweb, circa 1999.

I probably wouldn't be this harsh, but Graham has spent so long talking about "superior" programming languages that he pretty much had to knock this out of the park or tone his rhetoric down.

So, Irish voters will soon be voting in a state-wide referendum on the upcoming Treaty of Lisbon — the latest set of amendments to how the European Union is run.

Since ratification will require changes to the Irish constitution, we get to vote on these intricacies where most EU inhabitants do not. Unfortunately this means it’s not particularly “sexy” — it’s a pretty obtuse and boring set of issues, and deciding which way to vote is not easy, with such snore-worthy stuff at stake.

One of the organisations campaigning for a “no” vote in the referendum is called Libertas. Aileen forwarded on a very interesting article by Chekov Feeney on Indymedia Ireland about them, which is well worth a read if you’re interested in Irish politics and the international reach of US lobbying. Here’s some snippets:

Declan Ganley, president of Libertas, happens to be president of Rivada Networks, a US defence contractor (they supply emergency communications networks to the US intelligence community).

[...]

On Sunday April 20th, Libertas announced that Ulick McEvaddy was “joining the No To Lisbon Campaign” and publicised the event with a photo-opportunity of the two ‘entrepreneurs’ in front of the Libertas Campaign bus. McEvaddy is the first member of the Irish business and political elite to join the Libertas campaign since it emerged under the stewardship of Declan Ganley.

What’s particularly interesting about this is that McEvaddy is the CEO of Omega Air, a US defence contractor (they supply cargo planes and inflight refuelling services to the US military). [...] According to the [ US Air Force's Integrator Magazine ], “industry insiders say [McEvaddy's] company has even approached U.S. intelligence agencies about tanking services for detainee transfers, to reduce dependence on foreign air fields.” In other words, offering to provide inflight refuelling services to rendition flights so that they wouldn’t have to stop over at foreign airports such as Shannon on their way to “interrogate” suspects. A very accommodating offer indeed.

McEvaddy was also the figure who got himself appointed to the board of Knock airport with a view to opening it up to US military flights.

Nice guys, then.

The article goes on, and on, and on, detailing some shady transactions involving these guys and their US military/intelligence connections, the “astroturf” nature of the Libertas organisation, and the odd behaviour of the Libertas campaign in general.

It comes to this conclusion:

This article has examined the reality behing the Libertas campaign, the connections of its two high-profile backers, the implausibility of its message, the peculiar nature of its campaign and some of the underlying strategic differences at play. The conclusion is that the evidence suggests that Libertas is most likely to serve primarily as a vehicle for advancing US strategic interests.

Check it out — it’s a must-read.

As many of you know (all, like, 12 subscribers or whatever there are here), I took a job with Metaweb, who run Freebase, in January this year. So since then I’ve been in charge of the Freebase blog and haven’t been blogging here.

I’ve been wondering what to do with this domain, and I’ve decided to keep it running and to use it for personal/unofficial notes on Freebase-related topics. While the Freebase blog is a sort of official channel, I’ll use freebasing.org for less formal topics.

So, I suppose a disclaimer is in order here: Views and opinions posted on this blog do not necessarily represent those of my employer.

It seems tough to find a well-written SQL formatter out there which is both accurate and open-source. I looked at SQL::Tidy as an inspiration, but it's tokenizer is bad and there are plenty of issues with the code. A quick bit of hacking has gotten it to the point where it will take this:

SELECT me.import_id, me.timestamp, DATE_FORMAT ( timestamp, '%Y-%m-%d' ) as day
FROM import me LEFT JOIN import_error import_errors ON ( me.import_id =
import_errors.import_id AND import_errors.type IN (
'X::IMPORTER::VALIDATION::BRANDTITLEMISSING')) WHERE ( ( ( timestamp >=
'2008-05-01T00:00:00' ) AND ( timestamp <= '2008-05-06T13:12:30' ))
) GROUP BY me.import_id

And turn it into this:

SELECT me.import_id, me.timestamp, DATE_FORMAT ( timestamp, '%Y-%m-%d' )
           as day
      FROM import me LEFT
      JOIN import_error import_errors
      ON ( me.import_id = import_errors.import_id
               AND import_errors.type
               IN (
                        'X::IMPORTER::VALIDATION::BRANDTITLEMISSING'
               )
      )
WHERE ( ( ( timestamp >= '2008-05-01T00:00:00' )
               AND ( timestamp <= '2008-05-06T13:12:30' )
             )
           ) GROUP BY me.import_id

That's pretty ugly, but at least it makes it a lot easier to read auto-generated SQL that's all on one line. Still, it's only for quick cleanup. The tokenizer needs a lot of work before we can even think of a reasonable reformatter.

Amusingly, though, I accidentally ran Perl::Tidy over it.

SELECT me . import_id, me . timestamp,
    DATE_FORMAT(timestamp, '%Y-%m-%d')
    as day FROM import me LEFT JOIN import_error import_errors ON(me
        . import_id
        = import_errors
        . import_id AND import_errors
        . type IN('X::IMPORTER::VALIDATION::BRANDTITLEMISSING')) WHERE((
    (timestamp >= '2008-05-01T00:00:00')
    AND(timestamp <= '2008-05-06T13:12:30'))
        ) GROUP BY me
    . import_id

OK, not great, but not significantly worse than the SQL::Tidy.

Parrot r27355 was fun to write.

One of the persistent error messages Parrot emits for compiler writers is Null PMC access in invoke(). If you've had your hands deep in the guts of Parrot, you know what that means -- you tried to call a Sub PMC when you don't have a Sub PMC, you have no PMC. (If you don't know what that means, this entry is for you.)

Sometimes this means that there's a problem in Parrot. We've fixed almost all of those problems though, so the error usually comes from elsewhere. If you're writing a compiler, or running a compiler built on Parrot, the error usually means "You tried to call a function that doesn't exist."

Parrot's optimizer does something interesting at the end of compilation time. You've probably heard that Parrot's compiler, IMCC, translates PIR into PBC. That is, it turns source code into bytecode, which Parrot can either serialize t to disk or execute immediately. That bytecode is just a chunk of linear data in memory. It's not really a data structure. (Okay, it's a C array, but that doesn't make it a data structure.)

After IMCC has finished building a standalone chunk of bytecode, it performs a constant fixup phase. The notable part of this phase is that it edits the bytecode in place to replace all named invocations of functions known at fixup time with offset invocations.

The previous code looks something like:

invoke known_function
null    # padding
null    # padding

If IMCC has already seen known_function by this time, the direct invocation of known_function can continue. There's no runtime lookup necessary; all functions already compiled and ready are available in the bytecode.

If IMCC hasn't seen that function, runtime lookup is necessary, and so this function replaces the bytecode earlier with the equivalent of:

.local pmc func
func = find_name 'unknown_function'
invoke func

(I've simplified what actually happens slightly, because the concepts are more important than the details. Hopefully you see why the padding is necessary. If not, just imagine trying to splice additional opcodes into what may presumably be a lengthy C array -- like I said, barely a data structure.)

The problem with this second form occurs when find_name returns a NULL PMC, which it can legitimately do. In that case, the invoke opcode tries to invoke a NULL PMC and fails, and Parrot throws an exception saying "There's nothing here to invoke." There's the error message.

It's clear why that happens, but it's not useful. It would be more useful to see the name of the function you tried to invoke in the error message. Unfortunately, by the time Parrot calls the invoke op, that name is long gone.

My first idea was to rewrite the dynamic lookup form into something resembling:

.local pmc func
func = find_name 'unknown_function'
if defined func goto call_it
die "Can't invoke 'unknown_function'"

call_it:
invoke func

Unfortunately, I didn't have the space in the bytecode stream to insert that many ops, and I had no desire to move chunks of memory around in that C array. I could have added more padding after an invocation, but to be fair I'm only mostly sure that it exists there in the first place.

I had room for one op with a destination PMC and a string constant argument. I added an experimental op called find_sub_not_null which does the same thing as find_name but throws an exception which includes the requested name if Parrot can't find a PMC of that name.

This isn't entirely an ideal situation. It's a special case op, and I prefer to remove ops where possible. It's also nearly code duplication, though it's effectively three lines of code in an op, which isn't awful. I still want to be able to perform these kinds of transformations in PBC itself, but we need a different way to generate PBC and perform op-level transformations in PIR before we can do this effectively.

There are always tradeoffs, though. Doing this check in C is slightly faster than doing it in PIR. The standard Perl 5 rule of optimization applies even in Parrot -- the fewer ops, the mostly faster you can go. As well, I was able to improve the warning message today, rather than at some point in the future when we have better PBC optimization possibilities.

After all, I can always remove this op in the future.

I've just come back from the London Python Meetup. There was an informal survey, and I was the person who had the least Python experience (although I have written a little app). Around 50 people turned up and there was free pizza and beer.

There were some lightning talks, including one on Google App Engine. Another talk was a relevation someone has bought a bunch of servers for platform testing. The lightning talks were a bit hit-and-miss and would have been much better if they hadn't tried to have slides.

The main talk was Jacob Kaplan-Moss giving details about the history of Django, mostly little anecdotes. I was surprised to hear that PyCon had almost 1,100 people this year and that in the Django talk, half of the attendees (200 people) had learnt Python in order to use Django. That's some killer app.

Perl people as are pretty pragmatic, using the best tool for the job and learning about everything. However almost every talk tonight contained a dig about Perl, it was really dishearting. Python people: grow up and pay attention to what other people are doing - you might learn something.

Yesterday we had our monthly Vienna.pm TechSocial Meeting. I talked about 45 minutes about the Olso QA Hackathon (here are my slides). Then we watched the App::Asciio screencasts.

For the social part we went to Tin-Tan for cheap cocktails (happy hour all night on Monday, yay!) and good mexican food. And lots of discussions / chatting.

Oh, and I really do hate mobile devices with internet connection. Imagine you're sitting in a nice place drinking some cocktails, when suddenly somebody pulls out some gadget and shows you an RT ticket that has been open for 10 months :-). On the other hand, that's a good method to make me fix it. Which I just did...

YAPC::Asia has become huge. This year we've got 550 registrations and I think this is one of the biggest YAPCs ever.

However, our organization team has been getting smaller year by year, maybe because we knew we can do this. I live in San Francisco, USA and remotely organize the conference for this 2 years, just like any other project managers do for a project. That means, the real tough work has been done by our staff in Tokyo (I think it's too early to name and thank them since the conference has not even started)

That said, I'd like to hand off this conference to someone else, since I don't have a plan to get back to Japan anytime soon.

We're looking for the organizers for YAPC::Asia 2009. It could be any other individuals or preferrably Perl Mongers group in Asia. We've been writing things down what's needed to run this conference and already started writing Post-Mortem document so that this conference could be even better next year. I could still help configuring Act site, doing public relations to guests (if needed) etc. that I can still do remotely.

Drop me an email (miyagawa at gmail.com) if you're interested. Hopefully we can announce in the closing ceremony in this YAPC::Asia (May 15-16), and people will stop asking me about "When and where is YAPC::Asia 2009?" :)

You see, the great thing about posting vim stuff here is that people (Smylers, Aristotle, etc.) immediately post corrections. I appreciate that :)

Problem: when in the MySQL client, you can type 'edit' and edit your last SQL statement (can I go back further? I thought so, but I can't recall how). However, I want my syntax highlighting, but the filetype isn't set. After a bit of testing, I found out that the filename is always /var/tmp/sql* (on my system). So the following addition to my .vimrc fixes this:

" this is for MySQL's 'edit' command while in the client
au! BufRead,BufNewFile /var/tmp/sql* :call SetMySQL()
function! SetMySQL()
    set ft=sql
    SQLSetType mysql
endfunction

Perl's big problem is one of perception. The other day, a job candidate asked me "You're moving your web apps from PHP to Perl? Shouldn't it be the other way around?" Why did he think that? The candidate knew no Perl, and only a bit of PHP, so had no technical reason to believe that PHP was better than Perl. So why did he think Perl was subservient to PHP in the web arena?

What I suspect is that Perl is just less visible, and not just in the sense of crap like the TIOBE index where it equates hits for "Foo programming" popularity of the Foo language.  I'm talking about in the real way people see the world of programming.

I'm certain that PHP has become a de facto choice for basic web apps because it's just How You Do It these days. You see enough PHP in the context of the web, it starts to sink in.

Why is Ruby on Rails so popular? Is it better than Perl on Catalyst? Or is it just that people hear about Rails more? I suspect the latter, because perception is reality. When people perceive Perl as being dead, or not as powerful as other tools, it might as well be.

There are three goals I'd like to address:

• Perl needs diversity, needs new blood, both in users and community as well as tools.
• Perl needs more active mindshare in the programming world.
• Perl 5 must continue to be seen as a viable language.

The three actions to take: Decentralize, diversify and colonize.

Perl must decentralize

Perl has a great infrastructure. We have the CPAN. We have the various sites of perl.org. We have Perl Mongers and perlmonks, and so on. Unfortunately, we've grown complacent. It's time to start looking elsewhere for infrastructure and community.

Standardization on tools makes sense if the tools are the best possible tools available. In many cases, the infrastructure already in place in the Perl community may not be the best there is.

Is the Mailman installation running on lists.perl.org the best mailing list solution for you when you need to start a mailing list? If not, then consider other options. Starting a list at groups.google.com is trivial, for example.

Is rt.cpan.org the best bug tracking solution for your modules or projects? If not, then look at other options. Many of my modules, such as ack and WWW::Mechanize are hosted at Google Code, where I use the bug tracking system there.

Is use.perl.org the best place to host your blog? Maybe you should post somewhere else. Certainly that was the case with me. I found that use.perl.org wasn't the news site I wanted it to be, so I started Perlbuzz as an alternative, literally doubling the number of news sites devoted to Perl.

Note that "alternative" does not me "replacement". It's possible to have two similar but diferent projects, websites or whatever that both fill a need. If, say, Perlbuzz replaced use.perl.org, we'd still only have one news site, which is no better a situation than we started with.

Some of the alternatives might not be community-based. Google Code, is not run by altruistic volunteers. Commercial interests will usually have more time and money to spend on providing useful infrastructure. That's no knock on the volunteers, but merely taking advantage of what the commercial interest has to offer.

(I know at some point someone will say "At some point decentralization is harmful." I agree, but we're nowhere near that point. Such a problem to have!)

Perl must diversify

The mantra "There's more than one way to do it" is at the core of Perl the language, but this is not always the case in the Perl community. We must always remember the guiding principle of TMTOWTDI.

The first place to diversify is with people. I suspect, but cannot prove, that the Perl community's size is a net loss as time goes by. We must constantly be trying to bring new people into the fold, to take advantage of their innovations, and to make what we do more fun.

Second, we must be willing to launch new projects, even if they're similar to existing projects. It could be a module, or a website, or an entire application. The knee-jerk resistance to diversity often sounds like this:

• "Why do we need Getopt::Whatever, we already have Getopt::Long"
• "Why do we need another templating system?"
• "Why do we need Jifty/Catalyst/whatever? We already have Maypole."
• "Why do we need Perlbuzz, we already have use.perl.org?"

I've never understood this fear. It sounds like the resistance is based on the premise of "I wouldn't want to do that project, so you shouldn't either." It's the refrain of someone with a closed mind, unable or unwilling to imagine what could be. In the case of similar modules, the refrain usually goes "It's too hard to find the module you want anyway," but that's not a problem of the module, but of the way things are found in CPAN itself. (And if you'd like to address that little bit of decentralization, please take a look at my Rethinking CPAN mailing list.)

Ultimately, fear of the new is counter to the principle of TMTOWTDI. Indeed, plurality is at the very heart of open source, and it's additive, not subtractive. Template Toolkit need not take away from Mason. Perlbuzz need not take away from use.perl.org. Perl didn't take away from awk and shell. In any of these cases, if the new solution does take away from another, then the other solution was an inferior one to begin with.

Decentralizing and diversifying do two things. First, it opens our minds to alternatives that we may not have considered. It makes us more likely to find solutions that are better than what we started with. Second, it helps with colonization, the ultimate goal.

Perl must colonize

The other day, Tim O'Reilly posted on his Twitter feed a link to a cool article about life in the universe which mentioned Von Neumann probes, theoretical space craft that can replicate themselves using materials found in their travels. A single probe could travel some distance, replicate a thousand cloned probes, which would then launch in a thousand different directions, repeating the cycle.

That's what we need to do: Colonize the mindshare of the world to let them know that Perl is still viable, and a hell of a lot of fun. These new Perl lovers will spread the love as well, and the cycle will continue. In terms of action, it means simply "helping make more people aware of Perl as a cool & useful language." Fortunately, it's a case where we can think globally and act locally, even passively.

The biggest effect I see you, the reader, as being able to have is by spreading out Perl's web footprint. I want more places for people to find out about Perl, rather than a few big ones.

This colonization approach goes hand-in-hand with decentralization. Take my ack project, for example. ack has a number of footprints out there:

• A project home page, entirely separate from any perl.org or cpan.org infrastructure.
• A Google Code home page with source repository and bug tracker
• The ready-made search.cpan.org page for ack which points to both.

That's three times places for people to stumble across Perl, to see it mentioned. A separate home page also makes it more likely to get linked from somewhere else, as when Daring Fireball linked to ack and thousands of people came to see about ack. That's thousands of people who went and saw "Huh, here's a tool in Perl."

Of course, a few thousand visitors isn't going to change any mindshare. That's why it's not just me that needs to work on this. Fortunately, decentralizing and diversifying makes colonization easy.

Our Google footprint is pretty bad, anyway. When I search for "perl" in Google, the top five hits are:

• perl.org, which is a good base.
• perl.com, O'Reilly's Perl page, also a good base.
• The wikipedia entry for Perl, which certainly isn't much help for a beginner
• www.cpan.org, with its front page that expects that you know what you want.
• An old perl 4 man page from Carnegie-Mellon University

Is this the best impression that we have to give the world?

Adam Kennedy gets it

Adam Kennedy's Strawberry Perl is a marvel of the principles I've talked about. Strawberry decentralizes, as it uses none of the core perl.org infrastructure, and it diversifies and colonizes by giving Windows users an alternative to ActivePerl, which for many users is not robust enough. When people talk about Strawberry Perl, it helps with mindshare as well.

Adam is a great example of someone who has set out to make improvements to a part of Perl, and implemented it without worrying about permission or duplication, helping Perl as a whole along the way. I thank him for it. I suspect his Perl On a Stick project will have similar results.

How you can help?

The strategy of decentralize/diversify/colonize takes actions at all different levels. You don't have to go create your own Perl distribution, or even write any code. Here are some other ideas to get you started.

• Post a Perl-related web page to Digg, reddit or any of the other social bookmarking websites.
• Bring someone new to a Perl Mongers meeting, even if Perl isn't his/her primary language. Especially if it isn't his/her primary language.
• Go to a user group meeting for another language. See what they're talking about. Share the Perl perspective.
• Start a blog outside of the use.perl.org hierarchy. If you already have a u.p.o journal, keep it, and post to both places.
• Start a mailing list related to a project of yours. Host it somewhere like Google Groups. Don't worry that "people won't be able to find it," because the list of groups at lists.perl.org is no model of organization, either. Besides, the only people who know to look at lists.perl.org are the people that already know there are lists at lists.perl.org. They're not who we're after.
• Start an alternative to perlmonks.org. Perlmonks is a fine site, but it's long in the tooth, daunting to newbies, and frustrating to search. Surely there's a different way to have an online Perl forum that is better in many cases.
• Work on a project that helps cross-pollinate Perl to the rest of the world. The Test Anything Protocol is a good example of this.
• Review a book about Perl. Get it posted to Slashdot or another big tech site.
• Write about Perl in your blog. Even if you think it's not interesting to Perl people, that's fine. Someone will want to read it, and you'll spread that mindshare.
• When you talk about Perl, don't be afraid to use the F-word and the L-word: "fun" and "love".

What do you think?

I'd like to hear your ideas. How can we expand Perl's reach? What have you done to help decentralize, diversify and colonize? How can we keep Perl fun and exciting?

Two comments I don't need to hear: "This will never work" and "This is a waste of time." If that's what you have to contribute, trust that I've taken your insight into account already, and save yourself some typing. Alternative courses of action, however, are more than welcome. Also unnecessary: "None of this would be a problem if not for Perl 6."

Acknowledgements

Thanks to a number of people who have helped discuss these ideas with me as I put them together. These include Pete Krawczyk, Jeffrey Thalhammer, Ricard Signes, Liz Cortell, David Hand, Jesse Vincent and Elliot Shank. I'm sure I've forgotten some, so I apologize if I left you off the list.

The minicpan tool in CPAN::Mini lets you keep a copy of the most recent revisions of each module on the CPAN. Having a mini-CPAN is a great tool for anyone with a laptop, or who wants to look at the CPAN as a whole, or who wants to create a mini-mirror of CPAN to support a large installation without having to hit the net for each module install. An entire mini-CPAN only takes up about a gigabyte of drive space.

Ricardo Signes, CPAN::Mini's author, wrote to tell me:

CPAN::Mini 0.569 includes an obvious optimization: instead of reconnecting to your remote mirror for every file that might need updating, `minicpan` will now keep one HTTP connection open for the entire update. While I can't give numbers that reflect the most common cases of usage, a run that checks every file and finds no updates goes, on my laptop, from about two minutes to about twenty seconds -- about 1/6 the time! It also puts less load on the remote server, making it a friendlier way to keep a local mirror.

Also, Adam Kennedy has just posted about a major upgrade to CPAN::Mini::Extract, a tool to make it easy to get individual files from tarballs, that speeds up extraction:

By shifting expansion to a one-shot extraction to a temp file, and then opening tarballs once from the temp file, I managed to get a two to three times speed up for file extraction. Combined with CPAN::Mini pipelining, this makes CPAN::Mini::Extract massively faster (a 200%-300% overall speed up).

Selena Deckelmann has come back from BarCampPortland with copies of every Post-It on the topic selection board. The topic selection board at an unconference like a BarCamp is where people write on a Post-It a topic they'd like to see presented, and put it on a board for all to see. Whichever topics people vote for are the topics that are presented.

Scanning through the photoset on Flickr is fascinating, as these often are. Topics range from Pirates Paying Artists to WordPress as CMS to How to lie with statistics to Should we replace Congress with a wiki?

Also fascinating to see how widespread Twitter has become, with half the Post-Its leaving @usernames as contact information.

Makes me want to start up a Bar Camp Chicago. And move to Portland.

You know, I would really like to be able to hide outer scopes in Perl. There are plenty of times that I've worked with bad code and I would love to quickly and easily see what variables are embedded in a chunk of code which are defined in another scope. I figured out one way to do this with a primitive refactoring tool I'm working on. I shove the code into a different namespace, wrap it in a subroutine to prevent it from being executed (the code will croak() if there is a BEGIN block), and then eval it with strict and warnings. The error messages are trivial to parse and I can figure out which variables are not defined there.

This leads to my next problem: which variables are defined after that chunk of code which are dependent on that chunk of code. Trying to figure out the "extract method" refactoring means I have nastiness like this:

my ( $foo, $bar );
($foo, $bar, $baz ) = $self->_extracted_method($baz);

That raises another issue. The code must now always work in list context (let's hope you don't have wantarray embedded in the extracted code) and if there is an embedded return (some languages only allow return at the end of a method/subroutine), then this could also screw things up.

The more I play with this, the more I understand why Adam Kennedy struggled with a refactoring editor.

However, if I go for an 80% solution and I write tests (which, as you know, I do), then I could execute a refactoring and then automatically run all tests for the refactored code.

I really like my Test::WWW::Mechanize::Catalyst module. Well, I don't like how it's such a hack internally for something which should be so simple. However, I need some help: the recent releases of libwww-perl break the tests and I can't figure out how to fix it. Could you help?

According to statistics by LaPerla, the freshest 25% of CPAN is newer than how old?

1. 3.8 weeks
2. 38 days
3. 3.8 months
4. 38 months
5. 3.8 years

This Fortnight on perl5-porters - 13-27 April 2008 "Perl simply isn't broken enough. Most things work too well, hence no-one finds that they need to fix their itch, so in turn, they don't get sucked into core development generally. Maybe we need to start adding bugs, somewhat like a protection racket." "Your program works very nicely. It would be a shame if something went wrong with it, wouldn't it? ..." -- Nicholas Clark, on possible future revenue schemes.

Read more of this story at use Perl.

Lyle writes "New Bristol and Bath Perl Mongers launched. The old bristol.pm.org and bath.pm.org groups have been revived and joined together to form a new group. If you are from the UK and live near Bristol or Bath please subscribe to our group and see what's going on ;) http://perl.bristolbath.org"

Read more of this story at use Perl.

Someone asked on IRC today what's a nice simple module to use for config files. Someone suggest YAML. Jeez, talk about not simple. And YAML is actually hard to write by hand. Someone else suggested XML with XML::Simple. Same thing - a large complex module and the config format isn't particularly user editable. No disrespect meant to those who suggested those things - I just think differently...

All you usually need is INI files. Here's a simple parser I often use:

sub parse_config {
    my ($self, $file) = @_;
    open(my $fh, $file) || die "open($file): $!";
    my @lines = <$fh>;
    close $fh;
    chomp(@lines);
    my $config = $self->{config} = {};
    my $section;
    for (@lines) {
        s/^\s*//;
        s/\s*$//;
        next unless /\S/;
        next if /^#/;
        if (/^ \[ (.*) \] $/x) {
            $section = $config->{uc$1} = {};
        }
        elsif (/^ (\w+) \s* = \s* (.*) $/x) {
            die "key=value pair outside of a section" unless $section;
            $section->{lc$1} = $2;
        }
        else {
            die "invalid line in $file: $_";
        }
    }
}

It's nice and simple, requires no external modules or CPAN downloads, and works pretty well for most situations where simple config files are needed.

In an interesting interview, Donald Knuth disses unit tests.

[The] idea of immediate compilation and "unit tests" appeals to me only rarely, when I’m feeling my way in a totally unknown environment and need feedback about what works and what doesn’t. Otherwise, lots of time is wasted on activities that I simply never need to perform or even think about. Nothing needs to be "mocked up."

He's also not fond of XP:

Let me just say that almost everything I’ve ever heard associated with the term "extreme programming" sounds like exactly the wrong way to go...with one exception. The exception is the idea of working in teams and reading each other’s code. That idea is crucial, and it might even mask out all the terrible aspects of extreme programming that alarm me.

Oh, and reusable code is also something he takes a swipe at:

I also must confess to a strong bias against the fashion for reusable code. To me, "re-editable code" is much, much better than an untouchable black box or toolkit. I could go on and on about this. If you’re totally convinced that reusable code is wonderful, I probably won’t be able to sway you anyway, but you’ll never convince me that reusable code isn’t mostly a menace.

So, let's see. He doesn't like unit tests, XP, or reusable code. These are three things that I suspect many reading this might be in favor of. In fact, if it were just about anyone else saying this, the