Perls of Wisdom (not)

I use to program for fun, but I don’t have time (or enough concentration) to write anything substantial any more. It’s still fun to hack out one-liners from time to time though.
The other day I decided to keep track of my Amazon.com sales rank on the front page of Leoville. To do this I’d have to write a program in perl that the web server could call using CGI, the common gateway interface. The program would return the rank which the web server would embed into my page.

The first iteration of the program was pretty simple, thanks to a perl library called LWP. The library provides built-in routines to access web pages. Using the LWP routine “get” I can fetch the contents of the Amazon.com page, then use Perl’s built-in text search features to extract the ranking.

I wrote the program in a few minutes:


use LWP::Simple;
my $webpagetext;

# access Amazon web page
$webpagetext=get("http://www.amazon.com/exec/obidos/ASIN/0789726912/qid=1007181368/sr=1-6/ref=sr_1_74_6/104-8979567-7976756");

# find sales rank
$webpagetext =~ /(Sales Rank: )(d+)/;

# output sales rank
print "Content-type: text/htmlnn"; # this text is required for CGI output

print $2;

If you’re not familiar with perl a few things might need explanation. All the real work is done in the line…

$webpagetext =~ /(Sales Rank: )(d+)/;

In English this would read something like: search the contents of $webpagetext for the text “Sales Rank: ” followed by one or more digits.

The parentheses in the phrase (Sales Rank: ) (d+) tell Perl to group the results. Perl assigns the value in the first group to the variable $1, the second group to $2, etc. I’ll use $2 later to output the rank.

Finally I print the results to the console. CGI routes the output back to the web server which inserts it into the web page that called it.

I use Apache’s server-side includes (SSI) to call the perl program and embed the results of the program. On my system that means putting the line:

&lt!--#include virtual="/cgi-bin/ranking.pl"--&gt

into the web page. When the web server sees it, it calls ranking.pl and sticks the result into the page at that point.

So far, so good. I could run the program locally and it worked fine, but it wouldn’t work on my server. Turns out the LWP module was never installed. I wasn’t sure how to get around that until I installed Movable Type. This blog software uses several modules that aren’t part of my web host’s perl installation. But I learned I could put the needed modules in a directory on the web host and tell your program to look for them there. Thus adding the line:

use lib "/cgi-bin/mt/extlib/";

at the beginning of the program and storing the LWP::Simple module in the extlib directory, fixed the problem, and version 1.0 of my program was up and running. Worked great, too, until my book fell below 999 in ranking. Amazon displays larger ranks with commas, and my program didn’t consider that. I changed the search to include commas by substituting the regular expression [0-9,] for d:

$webpagetext =~ /(Sales Rank: )([0-9,]+)/;

and it was working again.

Incidentally, I work in perl on both Windows and Macintosh. On Windows, I use an excellent shareware editor from DZSoft. On Mac OS X I use BBEdit from Barebones Software. Both really speed up the development cycle by letting you run the program from within the editor, with built-in FTP uploading, and a perl reference.

No program is ever done, and neither was this one. Next time, how I extended it to keep track of the peak scores. (And maybe one of you perl experts can help me with a bug that’s really been buggin me.)

32 Replies to “Perls of Wisdom (not)”

  1. Leo
    Looks great! Your web cam is working well today too. Looks like you’re tied up on the phone again – Probably Patrick or Martin, I’m sure…..
    I should go back and learn the perl! Looks like fun !!
    Happy Weekend
    Chris

  2. Leo,
    You should learn/use PHP, the possibilities with it are endless and it’s soo much nicer than PERL. I stopped using frames on all the sites I run and do tables with PHP include()s instead, it is a much better way to do pages. I don’t think there’s anything you can do in Perl that you can’t do in PHP, it’s easy to learn as well, and is now fairly standard on webservers. See php.net (I think) for more information…

  3. Looks fun! LWP was always useful when I coded a commercial script for someone. I do some freelance work. Anyways, congrats on the book. Its being given away at tssaddicts!

  4. You got to put some pressure on your host to fix that. I couldn’t live without PHP, well I suppose I could live without it, but I wouldn’t like it. *grin* I had a terrible time finding a hosting partner for my personal site, but then again I’m both picky and cheap, which don’t go together well, but now I pay $20/yr for a private IP and that’s it, I have unlimited bandwidth and 160MB disk space, but if I need more space I can get it, unlimited email accounts, a distribution list, and I’m allowed to host my own DNS. Everyone on the planet wanted to do all of it for me but required me to use their DNS which I refuse to do. I have a nice one time fee setup with MyDynDNS you see so I can have subdomains point to my own personal servers/computers and have them dynamically and automatically updated, anyway I finally got what I wanted 🙂 from the good folks at http://www.elitehosts.com I must say their POP3 server is a little slow right now for my tastes but I have a date to have their head tech call me (yeah a real person on a real phone!) and I’m going to help them fix that next week. Regardless PHP hosts aren’t hard to find at all, and I think anyone whos not getting PHP support from their host should tell their host they want it, or go somewhere else. If you’re serving yourself download and install PHP, heck I even had it running under a couple of different Windoze web servers at one time. It makes a template based site with all sorts of fantastic server based, fast executing scripts a simple reality.

  5. Everything that can be done in PHP can be done in Perl. PHP is better for databases than perl and perl is better for simple jobs than PHP.
    Cross working. 🙂
    I altered the script to output the prices on tssaddicts shopping page. Works a treat.
    Leo, check ya mail. :p

  6. I wouldn’t dare chime in on which is better in this instance, PERL or PHP due to the fact that the only thing I ever learned was BASIC on an APPLEIIe ( How pathetic is that? doh!) But dangit all… I’m impressed. It works. Is there no other reason than to that to celebrate Leo as our official geek guru? I think not!

  7. Thanks for the ‘Adventures in Coding’ story, Leo The problem I have is that I never needed to use Perl enough to become proficient at it. I find it hard to ‘read’ after doing work in the C-type languages. I suppose if I get a decent website I’ll be delving into it again sometime. The nice thing is there are all kinds of libraries out there for it beyond the default install…provided your webhost supports installing them. I suppose I need some PHP books now too. ::sigh::

  8. PHP is much easier to learn than Perl, but just as powerful. Perl is much more flexible, but more more of a hodgepodge. It’s more of a scripting language like shell script than a systems language like c. It strikes me as a very dangerous language. It’s not orthogonal – to use a computer science term. Too many side effects. I’m always worried about unintended consequences in my programs.
    To an old C hacker Perl is also very ugly. Larry Wall’s philosophy is TIMTOWTDI – there’s more than one way to do it. In Perl there are several ways to do pretty much anything, and there are far to many “omnibus” instructions for my taste. That is, words that do a whole bunch of things at once. There are also too many instructions that have multiple meanings. All of this makes Perl a very easy language to hack in – one liners are a breeze – but means that every programmer does it differently. Which makes it hard to look at other people’s code.
    I’m just an amateur hacker, so I’m not really qualified to judge any language, but my personal preference is for Python and C. On the other hand, computer languages are like any tool. You need to use the one that’s best suited for the job at hand. And Perl was perfect for the job in this case.
    I’m a language collector more than a programmer. I love to learn new languages then use them for small programs. But I haven’t written anything substantial in 10 years. In those days I used C and asm, and was happy.

  9. If you don’t mind me asking Leo, what was the last substantial thing you coded. If you don’t wanna give details that’s okay. I was just wondering.

  10. And in case anyone cares, heres the changes I made for the price.
    $webpagetext =~ m!Our Price: .*$([d.]+)!;
    and print $1 instead of $2 and your golden.

  11. I was and still am a C/C++ coder and I personalloy found PHP to be a much nicer language than PERL, which is why I consider it the premiere language for web scipts/sites I still like C for system compiled programs, I despise JAVA (way too slow, unless you compile it, and the defeats the point of writing in JAVA) but I use PHP for quickie system scripts on my linux boxes (along with my C stuff). I’ve recently been hearing good things about Python, but since I’m really a networking/hardware guy myself I don’t think I’ll learn it until the need arises for something I’m writing that I can’t do in C or PHP…

  12. cool leo,
    you are alot better at programming than i am. i have been haivng a devil of a time trying to understand free store and pointers in c++. is the book going into a third printing and are there any free perl compilers out there that i could use to do the same thing. i would ike to have tickers for leoville, techtv, lockergnome and slashdot along with cnn to keep the news going on site.

  13. Phillip,
    PERL isn’t meant to be compiled. It’s really a scripting language like PHP, Python, etc. Compiled on run time. There may be PERL compilers, but I don’t think it would be a good idea to use them. When the script is called from a web page it will runtime compile it. You should be able to do tickers without a problem in PHP or PERL, shouldn’t have to compile them. I know I’ve seen a lot of scripts for news tickers that pull headlines from other sites. Check out hotscripts.com and tear aaprt some of their stuff to see how it’s done.

  14. Yes, shell scripting is handy as all heck in *nix environment. That’s how I got started into programming in the first place. From being a mere ‘user’, to writing a few scripts, to going to school and learning (mainly) C/C++. The last project I worked on, one of my co-workers used Perl to do some proof-of-concept modifications to an existing program without disturbing the working prototype we had. Then we did side-by-side comparison testing and tweaking. I guess my main complaint is that it’s so darn cryptic. Like you said, trying to read what someone else has coded can be frustrating because of that “omnibus” thing, unless you work with them day-in, day-out.

  15. I have no idea what you posted above. Is there a video out for that? LOL!
    I tried programming a few times but found that I couldn’t sit still long enough for it to keep my interest. I would much rather deal with the electronics and hardware and let others worry about the programming. Most programmers are underpaid and overworked.

  16. I would let leo shoot me with the taser….. Then i could brag to everyone that The famous leo laporte almost killed me with a taser……heh

  17. Perl is a very accessible language (among programming languages, that is) and because of that it has many new users who often don’t bother to learn best-practices or the right way to do things.
    Perl has also evolved over its 15 years, and has changed quite significantly causing many people’s knowledge to be outdated. Unfortunately, they often release software and write books.
    This has spread a lot of bad information about what Perl is and can be.
    Many of these myths and misconceptions are not investigated (as is very evident by most the previous posts) by people enough to realize that many of the things they hear and think are either untrue or inaccurate.
    If anyone here is interested in actually learning about Perl and not just dismissing it based on faulty information:
    There is a helpful and friendly community of Perl people at http://www.perlmonks.org and the http://www.perl.com site is quite comprehensive.
    Almost all of the misinformation and complaints that have been posted here are answered in a single article. Please, Please, Please read it.
    http://www.perl.com/pub/a/2000/01/10PerlMyths.html
    Regarding this little Perl vs PHP ‘debate’:
    Allow me to quote Perl guru and author of best selling Perl books, Randal Shwartz below.
    ** Note: Apache::EmbPerl and HTML::Mason are Perl modules that offer a platform of the same variety as PHP.
    —-BEGIN—-
    Comparing PHP to CGI/Perl is pointless. Compare PHP to either Apache::EmbPerl or HTML::Mason, and you are starting to get a fair comparison.
    Having watched PHP develop over the years, it started out as a very simple Perl replacement, but then has been slowly adding features of Perl one by one, but three to five years later. In another five years, it’ll probably be where Perl is now.
    So why wait? With Perl, you get a mature language, and a language that works as well off the web as on. And HTML::Mason and everything else in the CPAN make leveraging other people’s implementation a snap for nearly any common task.
    PHP – it’s “training wheels without the bike”.
    — Randal L. Schwartz, Perl hacker
    —-END—-

  18. As far as dynamic web scripting languages go, which is what we’re talking about here, PHP is a better choice than perl. Probably for the simple reason that PHP was meant to be a web scripting language that is growing to be even more useful, while dynamic web scripting came later to perl. If you look at PHP it is quite obvious it was meant to work in webpages, you can drop php code into any webpage by changing the extension to php and putting your code in (anglebracket)? *code here* ?(anglebracket) Right now perl does have some advantages as a full language, but the gap is narrowing with every release of PHP and hands down PHP is a better dynamic web language.

  19. I partly agree with you Ben. For a personal home page (what PHP orignally stood for) or for small sites I think PHP is a great choice. It’s easy to use and can do most everything a small site requires.
    But if you’re working with separate designers and programmers on a large site. There are many things that change.
    The need to separate HTML from your code with templating becomes not only convenient but almost necessary.
    Being able to easily change from something as simple as MySQL to Oracle while making only very minor changes because of the portable DBI module is important.
    Utilizing the quality work of people who have come before you by using CPAN modules, which cover almost anything even remotely common allows you to add new advanced interfaces and features very quickly.
    Using the ‘scrict’ pragma, mandatory taint checking, and web security related modules can help prevent some common problems of having untrusted user input invovled with the execution of code on your server.
    PHP is used as an Apache module. Perl can be used like this aswell, with the use of mod_perl, which almost all big sites use instead of vanilla CGI.
    mod_perl allows you to achieve performance at least equal to PHP. Using the HTML::Mason or Apache::EmbedPerl modules you can use Perl the same way you use PHP, embedding code between special delimiters.
    The gap is not narrowing, Perl is very much a moving target and as Randal Shwartz said: “why wait?”.
    In the end, like everything else, it really does come down to preference and what is the best tool for the job. Sometimes that means PHP, other times Perl, or maybe Java, Python, ASP, or Cold Fusion.

  20. If anyone wants to do live tracking with PHP this will do what Leo’s does. Just change the ASIN # to track any product.
    <?
    $filename = “http://www.amazon.com/exec/obidos/ASIN/0789726912“;
    if ($fd= fread(fopen(“$filename”, “r”), 100000)){
    $start= strpos($fd, “Amazon.com Sales Rank: </b> “);
    $rank = split(“<“, str_replace(“Amazon.com Sales Rank: </b> “, “”, substr($fd, $start, “50”)));
    echo $rank[0];
    }else{
    echo “Data Load Failed”;
    }
    exit;
    ?>

  21. Doh! I’m sorry! Movable Type sucks at special character parsing apparantly. I sent off a bug report to MT but this should be fixed by killing the comment above.

  22. Considering it isn’t a bug, I would think that the post on your site should probably be deleted or amended.

  23. Everyone:
    I modded the script so you could get a price from any page on amazon, using the QUERY_STRING var in the $ENV Array. The syntax is price.pl?http://www.amazon.com/whatever_the_url_is
    #!/usr/bin/perl -w
    $url=$ENV{‘QUERY_STRING’};
    use LWP::Simple;
    my $webpagetext;
    # access Amazon web page
    $webpagetext=get($url);
    # find sales rank
    $webpagetext =~ m!Our Price: .*$([d.]+)!;
    # my $rank = $1;
    # output sales rank
    print “Content-type: text/htmlnn”; # this text is required for CGI output
    print $1;

Comments are closed.