> Changing it to just warn (with carp) instead might be an acceptable compromise.
That would probably be best way to do it. It allows you to use future graph types while alerting folks to where the problem may lie instead of silently "breaking," especially for those folks who may not be coding perl but passing data through the module in another fashion.
Thanks for all your work with Perl. I can't calculate how much time you've saved me with projects.
> I would like to avoid this kind of test. First because that allow me > to use the native type names directly,
The change doesn't prevent this. You can specify either the native name or the alias name.
> the module to support types that the web-service introduce after the > module was released.
How can I be a BOFH if I let people do what they want, when and how they want. ;-)
But I do understand this point. Making it "mostly future-proof" means less work for users and you to maintain it. My change certainly prevents that.
Since it's a one line change, I can easily track the differences between my local copy and your official distribution.
> I don't belive passing the wrong type will be a common error. You > will need to test the generated URLs anyway and the error will be > discovered then.
That's how I came up with the suggestion - I used your example in the POD. You used "line" instead of "lines" and generated a broken api call.
On Thu, Jun 11, 2009 at 17:01, Jeff Stoner<leapfrog@freeshell.org> wrote: > On Thu, 11 Jun 2009, Gisle Aas wrote: > >> I would like to avoid this kind of test. First because that allow me >> to use the native type names directly, > > The change doesn't prevent this. You can specify either the native name or > the alias name.
I misread you line of code. I've been doing to much Python programing lately where iterating over a hash just gives you the keys :-)
>> the module to support types that the web-service introduce after the >> module was released. > > How can I be a BOFH if I let people do what they want, when and how they > want. ;-) > > But I do understand this point. Making it "mostly future-proof" means less > work for users and you to maintain it. My change certainly prevents that.
Changing it to just warn (with carp) instead might be an acceptable compromise.
> Since it's a one line change, I can easily track the differences between my > local copy and your official distribution. > >> I don't belive passing the wrong type will be a common error. You >> will need to test the generated URLs anyway and the error will be >> discovered then. > > That's how I came up with the suggestion - I used your example in the POD. > You used "line" instead of "lines" and generated a broken api call.
Oh, I'll fix that broken docs then. There was even another severe typo in that code.
I would like to avoid this kind of test. First because that allow me to use the native type names directly, and secondly because this allow the module to support types that the web-service introduce after the module was released.
I don't belive passing the wrong type will be a common error. You will need to test the generated URLs anyway and the error will be discovered then.
--Gisle
On Wed, Jun 10, 2009 at 20:51, Jeff Stoner<leapfrog@freeshell.org> wrote: > URI::GoogleChart will pass $type through to the final URL, invalid chart > types will cause Google to choke, eg. > > my $var = GoogleChart->new("fred", 300, 100, data => [45, 80, 99, 33]); > > produces > > http://chart.apis.google.com/chart?cht=fred&chs=300x100&chd=t:18.2,71.2,100,0 > > To prevent users from passing an invalid chart type, I added this to the > beginning of new() > > croak("Unsupported chart type") unless (grep /^$type$/, %TYPE_ALIAS); > > > --Jeff
On Thu, Jun 4, 2009 at 3:39 PM, Gisle Aas <gisle@aas.no> wrote:
> On Thu, Jun 4, 2009 at 11:46, Alex Kapranoff <ka@nadoby.ru> wrote: > > Oh, I see. I suppose this can break old scripts that expect 8 bit > characters > > without UTF-8 flag come through unaltered. > > They would then have to set $form->accept_charset("latin-1") as a > workaround. I think that's acceptable. I would hate to make this the > default for backwards compatiblity.
This will also affect people who use koi8-r, cp1251 оr any of hundreds of 8 bit encodings that were in wide use :) I think this point should at least be documented in POD and Changes, then.
Something like this: HTML::Form now always encodes data from its inputs into destination encoding when generating HTTP::Request objects. To specify the destination encoding you can use accept_charset() method or standard "accept-charset" attribute of the <form> tag if you create HTML::Form instance using parse() constructor. Destination encoding defaults to UTF-8 (imitating modern browsers). If you want your 8 bit data to come through unchanged you have two choices: 1) either decode it from $your_charset into internal Unicode representation using Encode::decode() before feeding into HTML::Form and then specify accept-charset($your_encoding) or 2) call accept-charset("latin1"). The latter method is not recommended unless you really use latin1.
We should also find a way to propagate the original charset of the > HTML document that's parsed. This should the be the default > accept_charset(), what you get when the attribute is still 'UNKNOWN'. > For the HTML::Form->parse($response) case this should happen > automatically.
Yes, totally! That was my original intention behind the patch :) We do exactly that in our WWW::Mechanize scripts now -- that is, manually set accept_charset() on forms from document charset, it would be awesome to have some automatic propagation.
Oh, I see. I suppose this can break old scripts that expect 8 bit characters without UTF-8 flag come through unaltered.
On Thu, Jun 4, 2009 at 9:23 AM, Gisle Aas <gisle@aas.no> wrote:
> On Wed, Jun 3, 2009 at 23:46, Alex Kapranoff <ka@nadoby.ru> wrote: > > It's an optimization to avoid unnecessary calls to Encode.pm. It should > not > > affect the result. > > It does affect the result for strings that only contain chars in the > latin1 range. > > See < > http://github.com/gisle/libwww-perl/commit/e4380a1385f36a139977e61b1a7e0e72f42746c8 > > > > --Gisle >
On Thu, Jun 4, 2009 at 11:46, Alex Kapranoff <ka@nadoby.ru> wrote: > Oh, I see. I suppose this can break old scripts that expect 8 bit characters > without UTF-8 flag come through unaltered.
They would then have to set $form->accept_charset("latin-1") as a workaround. I think that's acceptable. I would hate to make this the default for backwards compatiblity.
We should also find a way to propagate the original charset of the HTML document that's parsed. This should the be the default accept_charset(), what you get when the attribute is still 'UNKNOWN'. For the HTML::Form->parse($response) case this should happen automatically.
It's an optimization to avoid unnecessary calls to Encode.pm. It should not affect the result.
On Thu, Jun 4, 2009 at 1:38 AM, Gisle Aas <gisle@aas.no> wrote:
> On Mon, Jun 1, 2009 at 23:43, Alex Kapranoff <ka@nadoby.ru> wrote: > > The patch is attached. > > Thanks! I've now applied your patch. I do think I would like to get > rid of the 'if utf8::is_utf8($fi)' condition on encoding values. I > don't think the semantics should depend on this. Will that break > stuff for you? > > --Gisle > > > This attribute is rarely used in HTML but the interesting functionality > is > > encoding character bodies of generated requests into bytes according to > the > > specified encoding. > > > > If you try to click() a form specified as <form method="POST" > > enctype="multipart/form-data"> with some Unicode data in its fields you > will > > get this error: > > HTTP::Message content must be bytes at blib/lib/HTTP/Request/Common.pm > line > > 91 > > > > My patch fixes that by always encoding wide characters to the encoding > > specified in "accept-charset" attribute or set via method. Defaults to > > utf-8. Avoids unneeded encodings. > > > > Also, caution taken not to break compatibility with older perls. > > > > Docs and tests included. > > >
On Wed, Jun 3, 2009 at 23:46, Alex Kapranoff <ka@nadoby.ru> wrote: > It's an optimization to avoid unnecessary calls to Encode.pm. It should not > affect the result.
It does affect the result for strings that only contain chars in the latin1 range.
See <http://github.com/gisle/libwww-perl/commit/e4380a1385f36a139977e61b1a7e0e72f42746c8>
On Mon, Jun 1, 2009 at 23:43, Alex Kapranoff <ka@nadoby.ru> wrote: > The patch is attached.
Thanks! I've now applied your patch. I do think I would like to get rid of the 'if utf8::is_utf8($fi)' condition on encoding values. I don't think the semantics should depend on this. Will that break stuff for you?
--Gisle
> This attribute is rarely used in HTML but the interesting functionality is > encoding character bodies of generated requests into bytes according to the > specified encoding. > > If you try to click() a form specified as <form method="POST" > enctype="multipart/form-data"> with some Unicode data in its fields you will > get this error: > HTTP::Message content must be bytes at blib/lib/HTTP/Request/Common.pm line > 91 > > My patch fixes that by always encoding wide characters to the encoding > specified in "accept-charset" attribute or set via method. Defaults to > utf-8. Avoids unneeded encodings. > > Also, caution taken not to break compatibility with older perls. > > Docs and tests included. >
I am trying to install Crypt::SSLeay and am having problems getting it to work. I had this running with Perl 5.8 and then upgraded my cygwin to 5.10 a month or so ago. Reinstalled openssl and that seemed to go fine. I have a self-signed certificate, that was newly generated. Crypt:SSLeay won't pass its tests. I don't have any idea what it wrong.
Any help would be greatly appreciated.
Les
The output of the CPAN install attempt is shown below:
cpan[2]> clean Crypt::SSLeay CPAN: Storable loaded ok (v2.18) Going to read '/home/Les/.cpan/Metadata' Database was generated on Mon, 01 Jun 2009 04:27:10 GMT CPAN: LWP::UserAgent loaded ok (v5.813) CPAN: Time::HiRes loaded ok (v1.9715)
I would like to connect to one of the following sites to get 'authors/01mailrc.txt.gz':
Is it OK to try to connect to the Internet? [yes] Fetching with LWP: http://www.perl.org/CPAN/authors/01mailrc.txt.gz CPAN: YAML loaded ok (v0.66) Going to read '/home/Les/.cpan/sources/authors/01mailrc.txt.gz' Going to read 10 yaml files from /home/Les/.cpan/build/ ............................................................................ DONE Restored the state of 9 (in 0.8750 secs) ............................................................................ DONE Fetching with LWP: http://www.perl.org/CPAN/modules/02packages.details.txt.gz Going to read '/home/Les/.cpan/sources/modules/02packages.details.txt.gz' Database was generated on Wed, 03 Jun 2009 01:29:51 GMT ............................................................................ DONE Fetching with LWP: http://www.perl.org/CPAN/modules/03modlist.data.gz Going to read '/home/Les/.cpan/sources/modules/03modlist.data.gz' ............................................................................ DONE Going to write /home/Les/.cpan/Metadata Running clean for module 'Crypt::SSLeay' Running make clean rm -f \ *.a core \ SSLeay.c core.[0-9] \ blib/arch/auto/Crypt/SSLeay/extralibs.all core.[0-9][0-9] \ SSLeay.bso pm_to_blib.ts \ core.[0-9][0-9][0-9][0-9] SSLeay.x \ SSLeay.bs perl.exe \ tmon.out *.o \ pm_to_blib blib/arch/auto/Crypt/SSLeay/extralibs.ld \ blibdirs.ts core.[0-9][0-9][0-9][0-9][0-9] \ *perl.core core.*perl.*.? \ Makefile.aperl perl \ SSLeay.def core.[0-9][0-9][0-9] \ mon.out libSSLeay.def \ perlmain.c perl.exe \ so_locations SSLeay.exp rm -rf \ test.config crypt_ssleay_version.h \ blib mv Makefile Makefile.old > /dev/null 2>&1 DLAND/Crypt-SSLeay-0.57.tar.gz /usr/bin/make clean -- OK
cpan[3]> install Crypt::SSLeay Running install for module 'Crypt::SSLeay' Running make for D/DL/DLAND/Crypt-SSLeay-0.57.tar.gz
CPAN.pm: Going to build D/DL/DLAND/Crypt-SSLeay-0.57.tar.gz
Found multiple possibilities for OpenSSL /usr (OpenSSL 0.9.8) /usr/local/ssl (OpenSSL 0.9.8) Which SSL install path do you want to use? [/usr]
BUILD INFORMATION ================================================ ssl library: OpenSSL 0.9.8 in /usr ssl header: openssl/ssl.h libraries: -L/usr/lib -lssl -lcrypto -lgcc include dir: -I/usr/include/openssl ================================================ Checking if your kit is complete... Looks good Note (probably harmless): No library found for -lgcc Writing Makefile for Crypt::SSLeay The test suite can attempt to connect to public servers to ensure that the code is working properly. If you are behind a strict firewall or have no network connectivity, these tests may fail (through no fault of the code).
chmod 755 blib/arch/auto/Crypt/SSLeay/SSLeay.dll test -s SSLeay.bs && cp SSLeay.bs blib/arch/auto/Crypt/SSLeay/SSLeay.bs && \ chmod 644 blib/arch/auto/Crypt/SSLeay/SSLeay.bs make: [blib/arch/auto/Crypt/SSLeay/SSLeay.dll] Error 1 (ignored) DLAND/Crypt-SSLeay-0.57.tar.gz /usr/bin/make -- OK Running make test /usr/bin/perl5.10.0.exe "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t t/00-basic........ok t/01-connect......1/8 # Failed test 'Net::SSL->new' # at t/01-connect.t line 25. # SSL negotiation failed: at t/01-connect.t line 11 # at t/01-connect.t line 11 # ; at t/01-connect.t line 11 # ; at t/01-connect.t line 11 # ; at t/01-connect.t line 11 # Looks like you failed 1 test of 8. t/01-connect...... Dubious, test returned 1 (wstat 256, 0x100) Failed 1/8 subtests (less 7 skipped subtests: 0 okay) t/02-live.........1/4 # config on cygwin # ssl OpenSSL 0.9.8 in /usr # lib -L/usr/lib -lssl -lcrypto -lgcc # inc -I/usr/include/openssl # cc gcc t/02-live.........ok
Test Summary Report ------------------- t/01-connect.t (Wstat: 256 Tests: 8 Failed: 1) Failed test: 1 Non-zero exit status: 1 Files=3, Tests=24, 7 wallclock secs ( 0.06 usr 0.05 sys + 2.29 cusr 0.97 csys = 3.37 CPU) Result: FAIL Failed 1/3 test programs. 1/24 subtests failed. make: *** [test_dynamic] Error 255 DLAND/Crypt-SSLeay-0.57.tar.gz /usr/bin/make test -- NOT OK //hint// to see the cpan-testers results for installing this module, try: reports DLAND/Crypt-SSLeay-0.57.tar.gz Running make install make test had returned bad status, won't install without force Failed during this command: DLAND/Crypt-SSLeay-0.57.tar.gz : make_test NO
Is there a way to get the HTTP::message->decoded_content to work after using: $ua->request( $request, $content_file )
or will it only work when requesting to the request object ?
_________________________________________________________________ Invite your mail contacts to join your friends list with Windows Live Spaces. It's easy! http://spaces.live.com/spacesapi.aspx?wx_action=create&wx_url=/friends.aspx&mkt=en-us
On Wed, May 27, 2009 at 16:43, Terrence Brannon <metaperl@gmail.com> wrote: > I have modified the code I read in an earlier thread on writing data to a > request stream: > http://www.mail-archive.com/libwww@perl.org/msg06373.html > > my $ua = LWP::UserAgent->new(); > my $req = HTTP::Request->new(POST => $self->url); > my $xml = $self->xml_output; > my $length = length($xml); > > $req->content($xml); > $req->header('Content-length' => $length); > $req->content_type("application/octet-stream"); > > my $res = $ua->request($req); > my $response_data = $res->content; > > But the error I am getting from them is 'Root node of XML is missing' - > which basically means to me that they did not get anything written to the > request stream.
It could also mean that your XML is in error.
> They provide an example of how the POST request should be done in .NET. So > my goal is to use LWP to mimic the exact .NET commands below. The only > difference that I can see is that I am not creating a "requestStream" object > and then manually issuing requestStream.Write(postDataBytes, 0, > postDataBytes.Length); but I assume that LWP is doing something similar > under the hood.
AFAICT you should end up with the same kind of request on the wire. The setting of Content-Length should not be needed, but it should not be harmfull either.
On Wed, May 27, 2009 at 16:29, Terrence Brannon <metaperl@gmail.com> wrote: > Hi, > > I'm wondering what the difference is between providing content to the > HTTP::Request instance: > > my $req = HTTP::Request->new(POST => $self->url); > $req->content($c); > > and creating a LWP::UserAgent instance and using the Content pseudo header.
It's just totally different. In the first case you end up with a HTTP::Request object and in the second case an LWP::UserAgent object :-)
my $res = $ua->request($req); my $response_data = $res->content;
But the error I am getting from them is 'Root node of XML is missing' - which basically means to me that they did not get anything written to the request stream.
They provide an example of how the POST request should be done in .NET. So my goal is to use LWP to mimic the exact .NET commands below. The only difference that I can see is that I am not creating a "requestStream" object and then manually issuing requestStream.Write(postDataBytes, 0, postDataBytes.Length); but I assume that LWP is doing something similar under the hood.
> > Hi, > > I found 'get' function can't retrieve a nonstandard webpage (I > mean the retrieved webpage excludes some kinds of > tags,e.g.<HTML> ,<BODY> ). > > The malicious url,hxxp://ibalefo.net/?click=D1E863, only includes > a <iframe> tag,so I guess that is the reason why 'get' function > can't work . > > Does anyone know how to solve it ?
I am not able to access this URL using other tools, so the problem has to do with the URL.
Keary Suska
> Date: Tue, 19 May 2009 18:14:10 +0800 > Subject: Re: question about "get" function of LWP::Simple > From: msmouse@gmail.com > To: mathewzhao@hotmail.com > > Everybody: > > Be careful. It seems to be a harmful URL. > > ---------------------------------- > msmouse@ir.hit.edu.cn > msmouse@gmail.com > > > 2009/5/19 zmerry <mathewzhao@hotmail.com> > > hi, > > I want to analyze a malicious url,and the malicious url exists ,but > I always can't get it using LWP::Simple . > > ps. using http://www.google.com repalces the malicious url, it can > work. > > --------------------------------------------------- > use LWP::Simple; > $content = get("http://ibalefo.net/?click=D1E863"); > die "Couldn't get it!" unless defined $content; > --------------------------------------------------- > > Could someone give me a hand?
zmerry wrote: > Hi, > > I found 'get' function can't retrieve a nonstandard webpage (I mean the retrieved webpage excludes some kinds of tags,e.g.<HTML> ,<BODY> ). > > The malicious url,hxxp://ibalefo.net/?click=D1E863, only includes a <iframe> tag,so I guess that is the reason why 'get' function can't work . > > > > Does anyone know how to solve it ? > > > > best, > > matt > > > > > > > > > > > > Date: Tue, 19 May 2009 18:14:10 +0800 > Subject: Re: question about "get" function of LWP::Simple > From: msmouse@gmail.com > To: mathewzhao@hotmail.com > > Everybody: > > Be careful. It seems to be a harmful URL. > > ---------------------------------- > msmouse@ir.hit.edu.cn > msmouse@gmail.com > > > > > 2009/5/19 zmerry <mathewzhao@hotmail.com> > > > hi, > > I want to analyze a malicious url,and the malicious url exists ,but I always can't get it using LWP::Simple . > > > > ps. using http://www.google.com repalces the malicious url, it can work. > > > --------------------------------------------------- > use LWP::Simple; > $content = get("http://ibalefo.net/?click=D1E863"); > die "Couldn't get it!" unless defined $content; > --------------------------------------------------- > > > > > > Could someone give me a hand? > > > > > > Best, > > Matt > > > _________________________________________________________________ > 上MClub和Messenger好友一起都市寻宝！ > http://club.msn.cn/?from=1 > > _________________________________________________________________ > 上Windows Live 中国首页，下载最新版Messenger！ > http://www.windowslive.cn >
I want to analyze a malicious url,and the malicious url exists ,but I always can't get it using LWP::Simple .
ps. using http://www.google.com repalces the malicious url, it can work.
--------------------------------------------------- use LWP::Simple; $content = get("http://ibalefo.net/?click=D1E863"); die "Couldn't get it!" unless defined $content; ---------------------------------------------------
I want to analyze a malicious url,and the malicious url exists ,but I always can't get it using LWP::Simple .
ps. using http://www.google.com repalces the malicious url, it can work.
--------------------------------------------------- use LWP::Simple; $content = get("http://ibalefo.net/?click=D1E863"); die "Couldn't get it!" unless defined $content; ---------------------------------------------------
When I execute "make test", this is the error that I put on DOS-Prompt terminal: mkdir LWP::.: Invalid argument at C:/Perl/lib/ExtUtils/Install.pm line 455
Follow there is the complete shell output. Best Regards. ------------------------------------------------------------------------------------------------ D:\basi karaoke\LWP\libwww-perl-5.826>make test MAKE Version 5.2 Copyright (c) 1987, 1999 Inprise Corp. cp push(@s, " cp you can cp my $response cp robot, take cp ($robot_res->content_type =~ cp $wait = cp C<use_sleep> and cp $robot_res->content; if cp robots.txt expired; cp strict; # cp $robot_req = cp ($allowed) { cp Note that cp Retry-After header cp methods. In cp $old; } cp $self->{'rules'}->no_visits($netloc) || cp of the cp etc.), and cp will be cp They should cp $self->{'delay'} = cp when it cp pass the cp WWW::RobotRules->new($cnf{agent}); } cp *host_count = cp access to cp { if cp $ua->as_string Returns cp # Then cp every ten cp $delay; # cp LWP::UserAgent options cp ) Returns cp wait before cp $ua = cp required') unless cp m,^text/, && cp a string cp mandatory. The mkdir LWP::.: Invalid argument at C:/Perl/lib/ExtUtils/Install.pm line 455
On Wed, Apr 22, 2009 at 01:01, Danny Thomas <d.thomas@its.uq.edu.au> wrote: > the non-proxy path in LWP::Protocol::HTTP::request has > $host = $url->host; > so it looks like the destination is always taken from the uri > now maybe setting the proxy will do what I want, but does that > have downsides in that the destination won't be a proxy ?
When you set the proxy then the URL passed on the request line will differ (be a full absolute URL with scheme and everyting) so it would not really be the same and the server will probably reject it.
I think the easiest way to fake the host that LWP actually connects to is to override it by setting @LWP::Protocol::http::EXTRA_SOCK_OPTS before the request is sent. A runnable example:
use LWP::UserAgent; my $ua = LWP::UserAgent->new; local @LWP::Protocol::http::EXTRA_SOCK_OPTS = (PeerAddr => "www.microsoft.com"); my $res = $ua->get("http://localhost"); $res->dump;
Regards, Gisle
> > Here's the background. > We use a set of LWP-based scripts, each of which performs a login > to one of our sites. The login status and time is sent to our monitoring > system and an alert generated if the login did not behave as expected, > or the login time was too long. This merely identifies a problem > which could come from the web-server, application server, database, > network etc. > > In one case the uri takes you directly to the page with the login form, > but in other cases redirects, and/or cookies and/or javascript is involved. > So I want to replace these scripts with one based on WWW::Mechanize > which makes the code a lot simpler. Of course we still have to manually > code for the effects of any javascript. > > Some of the sites to be tested are behind a load-balancer which is normally > configured for sticky connections, i.e. connections from the testing host > are directed to the same cluster member. We still want to test the load- > balanced uri, in case it has problems, but it would be really good to > individually test each cluster member. > > We can do that with the Blackboard servers, as they each run just one site, > so a uri of "https://$HOSTNAME/webapps/login/" does work. > > However other web-servers are running multiple sites, so we want to be > able to explicitly specify the ip-address/hostname the connection is made > to. > > I believe WWW::Mechanize makes connections thru LWP::UserAgent. > We always use https for login, but it would be good if http also worked. > > Is it possible to currently do what I need or would it be hard to extend > the code to support this functionaltity ? > > cheers > Danny Thomas > > While /etc/host entries could be used, I think that's really gross, > particularly > on a monitoring server which is doing a mix of tests to many hosts. You > don't want the ping test of the load-balanced service directed to one of > the cluster members. >
i'm not sure of your favorite browser... but if it's firefox, i would suggest you load a plugin called livehttpheaders (i think). this allows you to see exactly what's happening between the browser/server so you can see what's needed from the client side when you're attempting to create a client to interface with the server....
this process will tell you whether you're using get/post etc, and what the server is expecting, sending back to the client...
you can also use a cmdline app like 'curl' to quickly see if what you think is required, works.. once you get to this point, coding in perl/php/python than becomes an easy process...
good luck!
-----Original Message----- From: Keary Suska [mailto:hierophant@pcisys.net] Sent: Wednesday, April 22, 2009 7:16 AM To: Aaron Naiman Cc: Libwww Perl Subject: Re: 'POST' method leads to "411 Length Required"
On Apr 22, 2009, at 3:11 AM, Aaron Naiman wrote:
> Thank you for getting back to me. > > Some parts of the HTTP spec are ambiguous and left to > implementation. In >> this specific case, error 411 means the server is saying that you >> are not >> sending a necessary content-length header, which is required for POSt >> requests whether there is any content or not. It may be that LWP >> doesn't >> provide the header if there is no content. Did you inspect the >> request to >> verify that the content-length header is indeed being sent? >> >> If the header is indeed being sent, the server might be choking on >> a 0 >> length. Send bogus data to see if that helps. > > > Initially, I received www.google.com with GET, and "411 Length > Required" > with POST. > > Subsequently, I tried providing header and content: > > $request = new HTTP::Request 'GET', 'http://www.google.com', > [asdf => > 'fdsa'], "dummy content"; > > and now with GET I see: "411 Length Required", and with POST: "405 > Method > Not Allowed". > > So, I am pretty baffled as to what the rules are, and how I can make > this > work.
I don't know about the first case, other than Google has been known to behave in nonstandard ways. a GET request shouldn't have an entity body, as I understand, and so shouldn't require a content length header.
The second case--405 method not allowed--just means the site doesn't permit a POST request. A server is not required to accept POSTs, so this response is possible.
It seems to me that the problems you are experiencing have more to do with your lack of understanding of the HTTP protocol, rather than the way LWP works. I suggest picking up a book on HTTP--I would recommend "HTTP Developers Handbook" by Chris Shiflett.
I would also underscore to not base your understanding on how google behaves. Historically Google has been unfriendly towards any interaction with their servers not performed by well known web browsing software.
> Thank you for getting back to me. > > Some parts of the HTTP spec are ambiguous and left to > implementation. In >> this specific case, error 411 means the server is saying that you >> are not >> sending a necessary content-length header, which is required for POSt >> requests whether there is any content or not. It may be that LWP >> doesn't >> provide the header if there is no content. Did you inspect the >> request to >> verify that the content-length header is indeed being sent? >> >> If the header is indeed being sent, the server might be choking on >> a 0 >> length. Send bogus data to see if that helps. > > > Initially, I received www.google.com with GET, and "411 Length > Required" > with POST. > > Subsequently, I tried providing header and content: > > $request = new HTTP::Request 'GET', 'http://www.google.com', > [asdf => > 'fdsa'], "dummy content"; > > and now with GET I see: "411 Length Required", and with POST: "405 > Method > Not Allowed". > > So, I am pretty baffled as to what the rules are, and how I can make > this > work.
I don't know about the first case, other than Google has been known to behave in nonstandard ways. a GET request shouldn't have an entity body, as I understand, and so shouldn't require a content length header.
The second case--405 method not allowed--just means the site doesn't permit a POST request. A server is not required to accept POSTs, so this response is possible.
It seems to me that the problems you are experiencing have more to do with your lack of understanding of the HTTP protocol, rather than the way LWP works. I suggest picking up a book on HTTP--I would recommend "HTTP Developers Handbook" by Chris Shiflett.
I would also underscore to not base your understanding on how google behaves. Historically Google has been unfriendly towards any interaction with their servers not performed by well known web browsing software.
Note that this 'expects' is fairly soft and does not go much further than "Applications SHOULD use this field to indicate the transfer-length of the message-body, unless this is prohibited by the rules... " with the transmission rules (keep alive, chunked, etc) beeing the only exceptions overruling this for POST (and related). This has always* been a messy problem.
But in general - this is a client bug (it should have been a bit more strict in sending) - but arguably the server should (and can) be accomodating in what it accepts. So I would try to fix both sides - starting with your own. E.g. a simple content-length = $ENV{CONTENT_LENGTH} || 0 sort of thing :).
> -----Original Message----- > From: Bjoern Hoehrmann [mailto:derhoermi@gmx.net] > Sent: 21 April 2009 23:42 > To: Aaron Naiman > Cc: libwww@perl.org > Subject: Re: 'POST' method leads to "411 Length Required" > > * Aaron Naiman wrote: > >$request = new HTTP::Request 'POST', 'http://www.google.com'; > > > >$response = $ua->request($request); > > You are using POST but are not posting any data, and > apparently the server ... > > >An Error Occurred 411 Length Required > > ... expects you to post some data when using POST. > -- > Björn Höhrmann · mailto:bjoern@hoehrmann.de · > http://bjoern.hoehrmann.de Am Badedeich 7 · Telefon: > +49(0)160/4415681 · http://www.bjoernsworld.de > 25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · > http://www.websitedev.de/ >
http://www.bbc.co.uk/ This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this.
Some parts of the HTTP spec are ambiguous and left to implementation. In > this specific case, error 411 means the server is saying that you are not > sending a necessary content-length header, which is required for POSt > requests whether there is any content or not. It may be that LWP doesn't > provide the header if there is no content. Did you inspect the request to > verify that the content-length header is indeed being sent? > > If the header is indeed being sent, the server might be choking on a 0 > length. Send bogus data to see if that helps.
Initially, I received www.google.com with GET, and "411 Length Required" with POST.
Subsequently, I tried providing header and content:
> -----Original Message----- > From: Bjoern Hoehrmann [mailto:derhoermi@gmx.net] > Sent: 21 April 2009 23:42 > To: Aaron Naiman > Cc: libwww@perl.org > Subject: Re: 'POST' method leads to "411 Length Required" > > * Aaron Naiman wrote: > >$request = new HTTP::Request 'POST', 'http://www.google.com'; > > > >$response = $ua->request($request); > > You are using POST but are not posting any data, and > apparently the server ... > > >An Error Occurred 411 Length Required > > ... expects you to post some data when using POST. > -- > Björn Höhrmann · mailto:bjoern@hoehrmann.de · > http://bjoern.hoehrmann.de Am Badedeich 7 · Telefon: > +49(0)160/4415681 · http://www.bjoernsworld.de > 25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · > http://www.websitedev.de/ >
http://www.bbc.co.uk/ This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this.
> Dear Björn, > > Thank you again for your response. > > * Aaron Naiman wrote: >>>> You are using POST but are not posting any data, and apparently the >>>> server ... >>>> >>>>> An Error Occurred 411 Length Required >>>> >>>> ... expects you to post some data when using POST. >>> >>> But as can be seen from .../HTTP/Status.pm, this is a *client* >>> error, and >>> not a server error. >> >> You're sending somebody who expects a letter from you an empty >> envelope. > > > However, from all of the examples I have seen, one need not send > anything in > a POST.
Some parts of the HTTP spec are ambiguous and left to implementation. In this specific case, error 411 means the server is saying that you are not sending a necessary content-length header, which is required for POSt requests whether there is any content or not. It may be that LWP doesn't provide the header if there is no content. Did you inspect the request to verify that the content-length header is indeed being sent?
If the header is indeed being sent, the server might be choking on a 0 length. Send bogus data to see if that helps.
* Aaron Naiman wrote: > >> You are using POST but are not posting any data, and apparently the > >> server ... > >> > >> >An Error Occurred 411 Length Required > >> > >> ... expects you to post some data when using POST. > > > >But as can be seen from .../HTTP/Status.pm, this is a *client* error, and > >not a server error. > > You're sending somebody who expects a letter from you an empty envelope.
However, from all of the examples I have seen, one need not send anything in a POST.
The only difference between GET and POST, is that the former sends the parameters in the URL, and the latter, as part of the header. N'est pas?
Anyhow, I have tried sending header and content, and still get the same error.
Were you able to succeed with sending content (filling in the envelope), and using the POST method?
Thanx,
Aaron
-- Aaron ("Aharon") Naiman Jerusalem College of Technology--Machon Lev naiman@math.jct.ac.il http://math.jct.ac.il/~naiman
the non-proxy path in LWP::Protocol::HTTP::request has $host = $url->host; so it looks like the destination is always taken from the uri now maybe setting the proxy will do what I want, but does that have downsides in that the destination won't be a proxy ?
Here's the background. We use a set of LWP-based scripts, each of which performs a login to one of our sites. The login status and time is sent to our monitoring system and an alert generated if the login did not behave as expected, or the login time was too long. This merely identifies a problem which could come from the web-server, application server, database, network etc.
In one case the uri takes you directly to the page with the login form, but in other cases redirects, and/or cookies and/or javascript is involved. So I want to replace these scripts with one based on WWW::Mechanize which makes the code a lot simpler. Of course we still have to manually code for the effects of any javascript.
Some of the sites to be tested are behind a load-balancer which is normally configured for sticky connections, i.e. connections from the testing host are directed to the same cluster member. We still want to test the load- balanced uri, in case it has problems, but it would be really good to individually test each cluster member.
We can do that with the Blackboard servers, as they each run just one site, so a uri of "https://$HOSTNAME/webapps/login/" does work.
However other web-servers are running multiple sites, so we want to be able to explicitly specify the ip-address/hostname the connection is made to.
I believe WWW::Mechanize makes connections thru LWP::UserAgent. We always use https for login, but it would be good if http also worked.
Is it possible to currently do what I need or would it be hard to extend the code to support this functionaltity ?
cheers Danny Thomas
While /etc/host entries could be used, I think that's really gross, particularly on a monitoring server which is doing a mix of tests to many hosts. You don't want the ping test of the load-balanced service directed to one of the cluster members.
* Aaron Naiman wrote: >> You are using POST but are not posting any data, and apparently the >> server ... >> >> >An Error Occurred 411 Length Required >> >> ... expects you to post some data when using POST. > >But as can be seen from .../HTTP/Status.pm, this is a *client* error, and >not a server error.
You're sending somebody who expects a letter from you an empty envelope. -- Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de 25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/
* Aaron Naiman wrote: > >$request = new HTTP::Request 'POST', 'http://www.google.com'; > > > >$response = $ua->request($request); > > You are using POST but are not posting any data, and apparently the > server ... > > >An Error Occurred 411 Length Required > > ... expects you to post some data when using POST.
But as can be seen from .../HTTP/Status.pm, this is a *client* error, and not a server error.
Still stuck, ....
Thanx,
Aaron
-- Aaron ("Aharon") Naiman Jerusalem College of Technology--Machon Lev naiman@math.jct.ac.il http://math.jct.ac.il/~naiman
You are using POST but are not posting any data, and apparently the server ...
>An Error Occurred 411 Length Required
... expects you to post some data when using POST. -- Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de 25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/
I have spent way too many hours scratching my head on this one (ouch!).
The following code fails.
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv use HTTP::Request; use LWP::UserAgent; my ($ua) = new LWP::UserAgent; $ua->proxy(["http"], "http://wwwproxy. ... :3128");
while true; ps aux | grep perl | grep below.pl; sleep 1; done
shows a very steady growth.
Simplest script I could think off is shown below. Any one any ideas - almost looks as if the .49 patch:
+ Added patch from Pavel Hlavnicka for freeing memory leaks from SSL_CTX_use_pkcs12_file() whose functionality is triggered by the $ENV{HTTPS_PKCS12_*} settings
was somehow reverted. Suggestions appreciated ! Let me know if you need a test server/client-cert to test against.
Thanks,
Dw
- sample script
use LWP::UserAgent; $|++; my $ua = LWP::UserAgent->new(keep_alive => 10); while(1) { $i++; my $req = HTTP::Request->new('GET', 'https://xxxx.com'); my $res = $ua->request($req); print "."; };
- command sequence to get yourself a server/client cert and config for apache:
http://www.bbc.co.uk/ This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this.
On Tue, Mar 17, 2009 at 08:47, Constantin Stefanov <cstef@parallel.ru> wrote: > So there are 2 flaws: > 1. No 'boundary=...' part in Content-Type header of multipert/mixed > message, which is created only after calling as_string() on HTTP::Request
Fixed in <http://gitorious.org/projects/libwww-perl/repos/mainline/commits/c791df9d0f97df1befd6ef28ed49e20818c1d17a>
> 2. HTTP::Daemon waits forever on receiving mutlipart/mixed message, > which may be fixed with patch from CPAN RT.
Applied in <http://gitorious.org/projects/libwww-perl/repos/mainline/commits/e374aa6ca550f10c34aa36623f04613464577543>
I have simple client POSTing multipart HTTP::Message to HTTP::Daemon-based server. Here is client code: ======================================================================= #!/usr/bin/perl -T use LWP::UserAgent; my $ua = LWP::UserAgent->new; my $req = HTTP::Request->new('POST', 'http://localhost:8089/');
$req->content_type('text/plain'); $req->content('First part'); my $second_part = HTTP::Message->new; $second_part->content_type('text/plain'); $second_part->content('Second part');
$req->add_part($second_part); #(1) my $resp = $ua->request($req); ======================================================================= This client sends incorrect message to server. Its Content-Type header is 'multipart/mixed', no 'boundary=...' part, so HTTP::Daemon does not recognize it as multipart. If I insert $req->as_string() in the string where #(1) is, client seems to work OK, but the server breaks.
Here is server code: ======================================================================= #!/usr/bin/perl -T use HTTP::Daemon; my $d = HTTP::Daemon->new(LocalPort => 8089, ReuseAddr => 1) || die;
my $c = $d->accept; my $req = $c->get_request(); $c->send_response(HTTP::Response->new(200)); $c->close;
$d->close(); ======================================================================= When client send correct multipart message (with boundary=... part in Content-Type), server waits forever. This issue was addressed in bug #28970 on CPAN RT (http://rt.cpan.org/Public/Bug/Display.html?id=28970). After applying patch mentioned there server seems to work as it should.
So there are 2 flaws: 1. No 'boundary=...' part in Content-Type header of multipert/mixed message, which is created only after calling as_string() on HTTP::Request 2. HTTP::Daemon waits forever on receiving mutlipart/mixed message, which may be fixed with patch from CPAN RT.
the content of the resulting post was ''xxxxxxx&searchContent=%C2%B9%C2%AB%C3%8B%C2%BE" which was not correct. should be "%B9%AB%CB%BE"
I've tried to use utf8; and change the value to encode('gbk', '公司'); (of course also changed the encoding of the source to utf8) and the problem still there.
I debuged and found the problem was around URI::_query::query:
sub query { my $self = shift; $$self =~ m,^([^?\#]*)(?:\?([^\#]*))?(.*)$,s or die;
if (@_) { my $q = shift; $$self = $1; if (defined $q) { $q =~ s/([^$URI::uric])/ URI::Escape::escape_char($1)/ego; $$self .= "?$q"; } $$self .= $3; } $2; }
before the subtitution, $q was correct and after that it turns out wrong. than i looked at URI::Escape::escape_char:
sub escape_char { return join '', @URI::Escape::escapes{$_[0] =~ /(\C)/g}; }
this method was called for four times, each time a byte (half a Chinese character) was passed in. on the entry point, length($_[0]) eq 1 but the /(\C)/g matching produces two bytes, which couses the problem.
I further discovered that the problem was caused by a join of utf8-flaged string and a non-utf8-flaged string -- perl automatically marks the resulting string utf8, which influnces the behavior of \C.
here's a test program:
1 use strict; 2 use warnings; 3 use Encode; 4 use utf8; 5 6 my $s1 = decode('utf8', 's1'); 7 my $s2 = encode('gbk','公司'); 8 my $s3 = "$s1+$s2"; 9 10 print_is_utf8($s1); 11 print_is_utf8($s2); 12 print_is_utf8($s3); 13 14 15 print_str($s1); 16 print_str($s2); 17 print_str($s3); 18 19 sub print_str { 20 my $str = shift; 21 print "$str: "; 22 print unpack('H*', $str) . '=' . join('+', map {unpack('H*', $_)} ($str=~/(\C)/g)) . "\n"; 23 } 24 25 sub print_is_utf8 { 26 my $str = shift; 27 print +(Encode::is_utf8($str)?"y":"n"), "\n"; 28 } ~ ~ the result: y n y s1: 7331=73+31 公司: b9abcbbe=b9+ab+cb+be s1+公司: 73312bb9abcbbe=73+31+2b+c2+b9+c2+ab+c3+8b+c2+be
So, now it's clear: Mechanize automatically decoded the page content, so the existing keys and values of the form have utf8 flags. in URI::_query this strings are joined with user provided keys and values, which in my case are not utf8-flagged. And then URI::Escape use \C to mach the byte and produces the wrong answer.
I think this can be fix by either Mechnize (don't auto-decode) or URI::_query (turn off utf8-flag before joining the key and value) .
Compiling My Own perl—brian d foy (first page)
FMTIEWTK About Closures—Johan Lodin (first page)
Expecting Perl—Mark Schoonover (first page)
Perl and Undecidability—Jeffrey Kegler (first page)
The Year in Perl, 2007—brian d foy (first page)
Templating My Output—Alberto Manuel Simões (first page)
Making My Own CPAN—brian d foy (first page)
Programming Parrot—Jonathan Scott Duff (first page)
Komodo Test Drive—Jim Brandt (first page)
Named Captures in Perl 5.9.5—brian d foy (first page)
Simple Web Access—Alberto Manuel Simões (first page)
Parrot Status Report—Jonathan Scott Duff (first page)
Mapping Op Codes—Eric Maki (first page)
CPANdeps—David Cantrell (first page)
HTML Slides—Grant McLean (first page)
Alter Egos—Anno Siegel (first page)
Found Perl is section of The Perl Review's website where we posted pictures of Perl paraphernalia or the word "Perl" in the wild. We started it a long time ago, and were very slow in updating it when people would send in photos.
Now "Found Perl" is a photo pool on Flickr. Instead of sending them to TPR, just add them to the pool.
The Perl Review's website has had a section on Schwern's Shirt, the orange monstrosity that brian d foy bought at the charity auction for The Perl Foundation at the 2004 Open Source Convention.
Now we've moved the section of the website to a Flickr group for Schwern's shirt. This way, anyone can add their photos of Schwern's shirt to the group. Instead of being infrequently updated on the website, people can add them as soon as they upload them to Flickr.
If you don't have a Flickr account and don't want to create one, you can still send them to The Perl Review by mailing them to editors@theperlreview.com.
David Pogue had a momentary lapse of judgement when he proclaims in his blog that the date sequence 01:02:03 04/05/06 will only happen once in all of human history.
Besides the obvious gaffes of date formatting (which one is the month and which one is the year?), the red herring of leading zeros (to make the minute and second stand out), and so on, no one who's seen this has made the comment that calendars say whatever we want them to say and the numbers are only special because we set the calendar up that way in this one case. What about the Chinese, Hebrew, and Muslim calendars?
So this seems like a good challenge to publish in The Perl Review: using the Perl Date modules (or not, I guess), in how many different calendars and formats can you make this sequence? What else is special about those days (are they a weekend, fall on a full moon, have a solar eclipse, etc.)?
Powell's Technical Books in Portland (that's the one on 33 NW Park Avenue) is going to carry The Perl Review. It's our first newsstand distribution.
I had to set a newstand price. The deal basically works like this: bookstores keep most of the profit. Magazines make money when the single-issue buyers turn into subscribers. After Powell's cut, which we set at 40%, and my costs, $2 an issue, I have to figure out a price that also motivates people to give the money to The Perl Review directly instead of the book store. That's why you see big discounts for magazines when you subscribe: that's the real price, and everything else is markup. The Powell's price ends up being $5, which is 50 cents more than the subscription price.
That's not to say that newsstands are bad. It's like better-than-free advertising since it sits on the shelf and I cover my costs plus a little more for every issue sold.
Forget about absolute numbers for a moment. At my price point, if they sell 75% of the copies, I break even. That would be fine with me because any copy sitting on the actual magazine display means people see that issue. Some might subscribe later even if they don't buy it. Now that I have a price point, I have to figure out the right number of issues. That's something I just have to guess.
I left 16 copies of the Spring 2005 issue, but I also have to consider that I sold about 10 at the Intermediate Perl book signing. We'll see how that goes.
Now, a good magazine accountant has to keep track of the actual number of newsstand sales too. As much as I'd like to pretend that we sell every single copy, the Post Office wants to know where all the issues went to verfiy that we abide by all the periodical rules. It's not enough for the newsstand to simply tell me what they sold. They certainly aren't going to tell me they sold everything when they didn't since that's money out of their pocket. They can't really tell me they sold nothing because that's money out of my pocket.
If you're a late night person living in a city, you might have seen a bunch of guys tearing off the front pages of newspapers and magazines. Instead of sending back the unused copies, they send back the cover (and they do that for books too). Every cover they don't send back is a sale that they owe me money for.
You might think those unsold issues represent lost money, but they really don't. They are a sunk cost, meaning that I would have spent that money regardless of the sales. That starts way back at the printers when I have to decide how many issues I want. That number includes all subscriptions, complimentary copies, samples to user groups, and all the issues I'll need to fulfill orders for back issues. Not only that, but the more copies I print, the lower the incremental cost (the cost per each copy). Each printing job has a fixed overhead for the job preparation, machine set-up, and so on. That's the make ready. I end up printing many more copies than I need, partly to amoritize the make ready. Not selling at the newsstand is slightly better than not selling while sitting in boxes in the office. At least people see them at the newsstand.
Remember that magazines make money on subscriptions, so that's the goal. I don't care about selling more at the newsstands. If someone subscribes because they see an issue on the newsstand, the profit from the subscription pays for about three unsold newsstand copies, so five subscriptions from people seeing the issue at Powell's would make up for no sells. That's just breaking even, and nobody makes any money. That also means I'm spending $6 to get a new subscriber.
If you're already despondent, you don't want to read about distributors. Most bookstores don't want to deal with every individual publisher. They'd have to keep track of a separate deal for every magazine. Instead, they want to deal with a single source in the same way they deal with books. I know my costs, and I know the newsstands cut, and I have a price point that I can't change to much because people won't buy it at too high a price. If I use a distributor, perhaps to get into the big chain book stores, they are going to want a big cut too. I'll end up either breaking even or losing money on every newsstand copy, and I'll want to convert that to a subscription as soon as possible. That's why you see so many wonderful subscription cards in the magazines.
So far I've just talked about money from sales. We can also sell advertising, which we do for the special friends of the Perl community. Since magazines know they are going to lose money at the newsstand, they make up the difference with paid advertising. Ever wonder why magazines such as Wired are mostly advertisements? That's making up for the money they'll lose on the newsstands. Remember when I talked about keeping track of the number of copies sold? Advertisers want to know those numbers. They don't care how many copies the newsstand bought. They care about the number of copies that shoppers bought. That sets the rate at which the magazines can sell ads. More eyeballs equal more dollars. There's a separate industry of companies that audit magazines to verify the numbers. That's even more money that gets sucked away.
The short story? Subscribe to the magazines you like. It's the only way they can survive.
Last year, I started including interviews with Perl people on The Perl Review website, but I didn't add a feed for that. Now I have and it's on RSS page.
The web site is actually a directory processed by Template Toolkit, so what I really need to do is add indexing support as ttree goes through the directories, then spit out pages as it does that. That sounds like a magazine article...
Every issue I get a couple of book reviews that don't quite cut it. Techies tend to present too many sides of the story: rather than express their opinion, they equivocate by pointing out all the holes in their own opinion. In short, they are entirely too nice.
Being nice isn't a bad thing, but what people really want out a book review is a recommendation. "Should I buy this book?" It doesn't matter if people agree with you as long as you are fair to the book. After that, people want to know the particulars: who, what, when, where, and how.
So far, The Perl Review has taken reviews from several different people, and very few people have provided multiple reviews. That worked when we were first getting started, but now I think we need something different. Since book reviews are about opinions, and the reader doesn't have to agree with the reviewer, I think readers need to know the reputation and leaning of the reviewer to make their own decision. For instance, my wife doesn't agree with movie-reviewer Roger Ebert, but based on his negative reviews she knows which movies she will like. At the same time, she knows the certain people on LiveJournal likes the same sorts of movies she does, so she can trust them.
In line with all that, I think I'm going to move towards recurring book reviewers, and do more to establish their reputation. That also means that I want to get a couple of reviewers who think about books differently so I can give more readers someone that thinks like them. I've approached a couple of people, and we'll see how it turns out.
I've added two guides I hadn't run across previously:
It seems that every time I put out a new issue I have to leave town for a couple weeks. I don't think the two are related. Travel is just too busy this year. Add to that finishing up the Alpaca book and I've had a pretty busy week. I'll be back in the middle of December.
I'm sending out the email announcements as I post this. Once you get the new password, you'll be able to download the latest issue of The Perl Review.
In this issue:
Seven Sins of Perl OO Programming -- chromatic
Hash Anti-Patterns -- Alberto Manuel Simões
Haskell for Perlers -- Frank Antonsen
PerlWar -- Yanick Champoux
books reviews, commentary, news and more...
Some of you neglected to renew, but I won't bug you in email anymore. To find out more about the sampler on the cover, you'll have to renew that subscription.
If you haven't subscribed yet, now's the time because I have to raise prices next year when the US postal rates go up.
The next issue of The Perl Review comes out in a couple of weeks, so it's time to renew if you've already finished your first year subscription!
This may seem odd for some who just recently subscribed, but with the magic of time travel we let you pretend that your subscription started much earlier by allowing you to choose which issue to start your subscription. You may have subscribed yesterday but selected to start with issue 1.0 (Winter 2004), meaning that your issue ran out with our last issue, 1.3 (Fall 2005).
I've found that most new subscribers like to get most of the back issues at first, and it's worked out rather well for us. Now that we're starting our second year, though, I have to figure out how to make that work so people don't have to renew right away. I'll have to add a two-year subscription, I think.
I've been reading lot about magazine renewal rates. A lot of publications seem to be happy to get 25% renewals, and they actually spend a fair bit of money to get that. Ever wonder why you get some many magazine renewal letters in your postal mail?
For TPR, since I don't have a lot of money, I simply used email. The target audience is certainly technology-adept, so that's not so bad. So far I've sent out three renewal emails. One was the week before OSCON, and I got about a 45% response for that. That's already good for magazines, but not quite the 98% renewal rate TPJ had in its first year. I sent out a second email two weeks after that, and another one over the weekend. I think I'm floating somewhere around 75% renewal right now.
I have a list of all of the people who haven't renewed (it's an SQL view ;), and a lot of them should have and I would be surprised if they don't. Although email provided me with a virtually free way to get that 75%, I also think email has a tendency to get lost. First, if the subscriber can't deal with it right awy, it joins the long queue of messages that get ignored forever. Second, it has a pretty good chance of being blocked by a spam filter. Third, some people might get so mucch mail that they just don't get to ever see it. I do send each mail individually since each contains a persoanl renewal code, so I'm not blasting out spam to any user at each domain.
I am considering sending out some postcards to the hold-outs. I figure that will cost me about $50, which means that I need about four renewals to make up for it. If I get four renewals, that's paid for. I'm also going to try emailing people directly. It certainly helps when other people talk about TPR. I notice a big spike in renewals when Ovid talked about his upcoming Logic Programming article.
There's another trick to getting renewals: special promotional subscriptions. Magazines give you a special rate for a limited time then hit you up for the full price as quickly as possible. It's all part of the ad-selling game. Advertisers want to know how many "qualified subscribers" you have, which means how many people actually want your magazine enough to pay for it. Getting people to be full subscribers raises those numbers.
TPR is a bit different because I don't aim to make it an audited magazine, meaning that no one is going to come in and put a stamp on our books saying they verified our subscriber base. I'm not in it to sell advertising (let's see if I'm saying the same thing in three years). So far I've only taken advertising from people the Perl community already loves and trusts. Most of that is just filling up the other full color pages that come along with the cover.
I now have a big stack of emails confirming renewal transactions, and I need to shove those transactions into my local database so people keep getting their magazines.
Most of them are pretty easy because they are keyed on their unique ID in the database. A lot of people skipped the link I sent them and went to the subscription page directly, so I have to match up those with what I see in te database.
So far it works like this and handles 95% of the records:
Get all the matches by name
With one match, we're probably done
With no match, try different parts of the name (last, first)
With multiple matches, we need to match something else too.
From the name matches, compare email addresses. If they match, we're done.
At this point we probably have several candidate matches.
Look at parts of the address (country, city, address). If the first two are matches, we're pretty sure we have a match.
Look at the email if we might have a match and compare the user portion to the new email and the stored email. Do the same for the host portion. This is just to raise the confidence level a bit.
This leaves about 5% that I need to check by hand. It's the first time I've had to go through with this so I'm being very cautious. Thank god for test suites.
Now that we've made it through the first year, it's time to get people to renew. I've sent out renewal emails, but I'm guessing 10% of them will never make it into an inbox. Spam has just about ruined email for anything.
If you've already received four issues, it's time for you to renew, and you can use subscription form and subscribe again.
Either way, I have an interesting task ahead. Since I don't store any personal information on the Pair.com servers that power our website, and we never store credit card information or do recurring billing, I get to match up the renewals with existing subscribers. It's easy to know which transactions are renewals, and each email I've sent has a link with a query string that I can link back to a subscriber. However, people might not follow the link but go directly to the webiste, or all sorts of other things that don't let me see that code. We'll see how that goes.
In related matters, I was rewriting the code bits that parse the email I get so I can shove all that stuff into the database. I was doing fancy things with ergexen, then Template::Extract, and some other things, and although it was a lot of fun, it was a big waste of time. Since I really just wanted to suck it into another program, why not send it as a ready-to-use data structure? It's easy enough to freeze things and unthaw them later. I still see my nicely formatted template, but at the end I include the Perl ready data. Besides the trivial parsing to isolate that, I'm ready to important things into the database. Things can be too simple to be seen sometimes.
I finally got to meet Eric Maki, TPRs designer, in person. We went through the current issue and looked at a lot of the design things we might want to change, and we also have an idea for an upcoming project I'll have details later.
If you haven't seen the latest cover (Summer 2005), get yourself an issue. It's so nice that we're going to make posters of it.
After a long day at YAPC where I taught a 4-day Learning Perl course in a single day, and then a 5 hour boat dinner cruise, Josh McAdams of Perlcast interviewed me about The Perl Review. I'm not sure when he'll publish it.
The cool thing is that Josh is moving to Chicago. I might be able to get on Perlcast a bit more often. :)
I got an advnace copy of The Best Software Writing from Apress. It's a collection of writings compiled by Joel Spolsky. Anyone want to take a whack at reviewing this for the next issue?
The Perl Review Summer 2005 issues should be in the mail today. The printer was screwing around again. They've been a disaster. You know the sort of people: they can't just do something without sending four emails back and forth, each with some new reason why things can't happen right away.
I'm thinking about making a special edition print issue with the best articles from the first two years. Good or bad idea? How much would you pay for that sort of thing? Which articles would you want?
The next issue is scheduled for December 1, and I'm looking for book reviews. Any recent book might work, but I'm specifically interested in reviews of:
Reviews should be 200 to 300 words and should reflect your opinion. Say explicitly who should buy this book and why they should, or that nobody should buy the book. No matter what you say, be nice.
sorting a hash by keys - the $a and $b need to stay there, don't replace them they are for the sorting function.
@sortedKeys = sort { $a <=> $b } keys %hashName # numerical sort
@sortedKeys = sort { $a cmp $b } keys %hashName # ASCII-betical sort
@sortedKeys = sort { lc($a) cmp lc($b) } keys %hashName # alphabetical sort
sorting a hash by values - the $a and $b need to stay there, don't replace them they are for the sorting function.
@sortedKeys = sort { $hashName{$a} <=> $hashName{$b} } keys %hashName # numerical sort
@sortedKeys = sort { $hashName{$a} cmp $hashName{$b} } keys %hashName # ASCII-betical sort
@sortedKeys = sort { lc($hashName{$a}) cmp lc($hashName{$b}) } keys %hashName # alphabetical sort
2008 scandalz.net
You can do this in a number of ways. IBM chose to do all of them.
Why do you find that funny?
-- D. Taylor, Computer Science 350, University of Washington
![[info]](http://l-stat.livejournal.com/img/userinfo.gif){kind=small}
