I'm trying to install Crypt-SSLeay on a debian system. If I try to install with the command
apt-get install libcrypt-ssleay-perl
This fails with a message as follows;
The following packages have unmet dependencies: libcrypt-ssleay-perl : Depends: perlapi-5.10.0
I'm running perl 5.14.2 and I can't revert to 5.10 easily.
If I use CPAN or install from source, then make test fails the output is below;
Any input greatly appreciated.
make test cp lib/Crypt/SSLeay/X509.pm blib/lib/Crypt/SSLeay/X509.pm cp lib/Net/SSL.pm blib/lib/Net/SSL.pm cp SSLeay.pm blib/lib/Crypt/SSLeay.pm cp lib/Crypt/SSLeay/CTX.pm blib/lib/Crypt/SSLeay/CTX.pm cp lib/Crypt/SSLeay/Conn.pm blib/lib/Crypt/SSLeay/Conn.pm cp lib/Crypt/SSLeay/MainContext.pm blib/lib/Crypt/SSLeay/MainContext.pm cp lib/Crypt/SSLeay/Err.pm blib/lib/Crypt/SSLeay/Err.pm /usr/bin/perl /usr/local/share/perl/5.14.2/ExtUtils/xsubpp -typemap /usr/share/perl/5.14/ExtUtils/typemap -typemap typemap SSLeay.xs > SSLeay.xsc && mv SSLeay.xsc SSLeay.c cc -c -I/usr/include -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DVERSION=\"0.58\" -DXS_VERSION=\"0.58\" -fPIC "-I/usr/lib/perl/5.14/CORE" SSLeay.c SSLeay.xs: In function ‘XS_Crypt__SSLeay__CTX_new’: SSLeay.xs:133:13: warning: passing argument 1 of ‘SSL_CTX_new’ makes pointer from integer without a cast [enabled by default] /usr/include/openssl/ssl.h:1469:10: note: expected ‘const struct SSL_METHOD *’ but argument is of type ‘int’ Running Mkbootstrap for Crypt::SSLeay () chmod 644 SSLeay.bs rm -f blib/arch/auto/Crypt/SSLeay/SSLeay.so LD_RUN_PATH="/usr/lib" cc -shared -O2 -g -L/usr/local/lib -fstack-protector SSLeay.o -o blib/arch/auto/Crypt/SSLeay/SSLeay.so \ -L/usr/lib -lssl -lcrypto \
chmod 755 blib/arch/auto/Crypt/SSLeay/SSLeay.so cp SSLeay.bs blib/arch/auto/Crypt/SSLeay/SSLeay.bs chmod 644 blib/arch/auto/Crypt/SSLeay/SSLeay.bs PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t t/00-basic.t .... 1/12 # Failed test 'use Crypt::SSLeay;' # at t/00-basic.t line 6. # Tried to use 'Crypt::SSLeay'. # Error: Can't load '/root/Crypt-SSLeay-0.58/blib/arch/auto/Crypt/SSLeay/SSLeay.so' for module Crypt::SSLeay: /root/Crypt-SSLeay-0.58/blib/arch/auto/Crypt/SSLeay/SSLeay.so: undefined symbol: SSLv2_client_method at /usr/local/lib/perl/5.14.2/DynaLoader.pm line 184. # at (eval 4) line 2. # Compilation failed in require at (eval 4) line 2. # BEGIN failed--compilation aborted at (eval 4) line 2.
# Failed test 'use Crypt::SSLeay::CTX;' # at t/00-basic.t line 7. # Tried to use 'Crypt::SSLeay::CTX'. # Error: Attempt to reload Crypt/SSLeay.pm aborted. # Compilation failed in require at /root/Crypt-SSLeay-0.58/blib/lib/Crypt/SSLeay/CTX.pm line 2. # Compilation failed in require at (eval 5) line 2. # BEGIN failed--compilation aborted at (eval 5) line 2.
# Failed test 'use Crypt::SSLeay::Conn;' # at t/00-basic.t line 8. # Tried to use 'Crypt::SSLeay::Conn'. # Error: Attempt to reload Crypt/SSLeay.pm aborted. # Compilation failed in require at /root/Crypt-SSLeay-0.58/blib/lib/Crypt/SSLeay/Conn.pm line 2. # Compilation failed in require at (eval 6) line 2. # BEGIN failed--compilation aborted at (eval 6) line 2.
# Failed test 'use Crypt::SSLeay::Err;' # at t/00-basic.t line 9. # Tried to use 'Crypt::SSLeay::Err'. # Error: Attempt to reload Crypt/SSLeay.pm aborted. # Compilation failed in require at /root/Crypt-SSLeay-0.58/blib/lib/Crypt/SSLeay/Err.pm line 2. # Compilation failed in require at (eval 7) line 2. # BEGIN failed--compilation aborted at (eval 7) line 2.
# Failed test 'use Crypt::SSLeay::MainContext;' # at t/00-basic.t line 10. # Tried to use 'Crypt::SSLeay::MainContext'. # Error: Attempt to reload Crypt/SSLeay/CTX.pm aborted. # Compilation failed in require at /root/Crypt-SSLeay-0.58/blib/lib/Crypt/SSLeay/MainContext.pm line 8. # Compilation failed in require at (eval 8) line 2. # BEGIN failed--compilation aborted at (eval 8) line 2.
# Failed test 'use Net::SSL;' # at t/00-basic.t line 12. # Tried to use 'Net::SSL'. # Error: Attempt to reload Crypt/SSLeay.pm aborted. # Compilation failed in require at /root/Crypt-SSLeay-0.58/blib/lib/Net/SSL.pm line 19. # Compilation failed in require at (eval 10) line 2. # BEGIN failed--compilation aborted at (eval 10) line 2. Undefined subroutine &main::main_ctx called at t/00-basic.t line 36. # Looks like you planned 12 tests but ran 11. # Looks like you failed 6 tests of 11 run. # Looks like your test exited with 255 just after 11. t/00-basic.t .... Dubious, test returned 255 (wstat 65280, 0xff00) Failed 7/12 subtests t/01-connect.t .. Can't load '/root/Crypt-SSLeay-0.58/blib/arch/auto/Crypt/SSLeay/SSLeay.so' for module Crypt::SSLeay: /root/Crypt-SSLeay-0.58/blib/arch/auto/Crypt/SSLeay/SSLeay.so: undefined symbol: SSLv2_client_method at /usr/local/lib/perl/5.14.2/DynaLoader.pm line 184. at /root/Crypt-SSLeay-0.58/blib/lib/Net/SSL.pm line 19. Compilation failed in require at /root/Crypt-SSLeay-0.58/blib/lib/Net/SSL.pm line 19. Compilation failed in require at t/01-connect.t line 4. BEGIN failed--compilation aborted at t/01-connect.t line 4. # Looks like your test exited with 2 before it could output anything. t/01-connect.t .. Dubious, test returned 2 (wstat 512, 0x200) Failed 8/8 subtests t/02-live.t ..... 1/4 # Failed test 'use Net::SSL;' # at t/02-live.t line 7. # Tried to use 'Net::SSL'. # Error: Can't load '/root/Crypt-SSLeay-0.58/blib/arch/auto/Crypt/SSLeay/SSLeay.so' for module Crypt::SSLeay: /root/Crypt-SSLeay-0.58/blib/arch/auto/Crypt/SSLeay/SSLeay.so: undefined symbol: SSLv2_client_method at /usr/local/lib/perl/5.14.2/DynaLoader.pm line 184. # at /root/Crypt-SSLeay-0.58/blib/lib/Net/SSL.pm line 19. # Compilation failed in require at /root/Crypt-SSLeay-0.58/blib/lib/Net/SSL.pm line 19. # Compilation failed in require at (eval 4) line 2. # BEGIN failed--compilation aborted at (eval 4) line 2. # config on linux # ssl OpenSSL 1.0.0e in /usr; # lib -L/usr/lib -lssl -lcrypto -lgcc # inc -I/usr/include # cc cc # Looks like you failed 1 test of 4. t/02-live.t ..... Dubious, test returned 1 (wstat 256, 0x100) Failed 1/4 subtests (less 1 skipped subtest: 2 okay)
Test Summary Report ------------------- t/00-basic.t (Wstat: 65280 Tests: 11 Failed: 6) Failed tests: 1-5, 7 Non-zero exit status: 255 Parse errors: Bad plan. You planned 12 tests but ran 11. t/01-connect.t (Wstat: 512 Tests: 0 Failed: 0) Non-zero exit status: 2 Parse errors: Bad plan. You planned 8 tests but ran 0. t/02-live.t (Wstat: 256 Tests: 4 Failed: 1) Failed test: 1 Non-zero exit status: 1 Files=3, Tests=15, 0 wallclock secs ( 0.03 usr 0.11 sys + 0.24 cusr 0.32 csys = 0.70 CPU) Result: FAIL Failed 3/3 test programs. 7/15 subtests failed. make: *** [test_dynamic] Error 1
>>>>> "Andre-John" == Andre-John Mas <ajmas@sympatico.ca> writes:
Andre-John> The 6 June 2012 is being treated as World IPv6 Launch Day ( http://www.worldipv6launch.org/ ) and I see no official support in this library. Currently I am having to use a patched version of libwww to get IPv6 support.
Furthermore, IETF just made IPv6 REQUIRED, not OPTIONAL.
http://tools.ietf.org/html/rfc6540
-- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 <merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/> Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc. See http://methodsandmessages.posterous.com/ for Smalltalk discussion
The 6 June 2012 is being treated as World IPv6 Launch Day ( http://www.worldipv6launch.org/ ) and I see no official support in this library. Currently I am having to use a patched version of libwww to get IPv6 support.
Well, the way suggested by you must do the job. But I was hoping that somebody was already implementing it in the 'get' method, side-by-side with the «':content_file' => $filename» and I somehow missed it. OK! I'll do it the hard way...
Thank you all Meir
> -----Original Message----- > From: Jesper Persson [mailto:jesper.j.persson@gmail.com] > Sent: Monday, April 16, 2012 7:36 AM > To: Meir Guttman > Cc: Libwww Perl > Subject: Re: How to deflate a ':content_file' response saved to a file > > Hi Meir, > > you can set the reponse content and then use decoded_content. Something > like this: > > $response = $ua->request($request, $filename); > > undef $/; > open(FH, "<$filename"); > binmode(FH); > $pageContent = <FH>; > close (FH); > $/ = "\n"; > > $response->content($pageContent); > my $decoded_pagecontent = $response->decoded_content( raise_error => > 1); > > Regards, > Jesper > > On 14 April 2012 22:25, Meir Guttman <meir@guttman.co.il> wrote: > > > I would like to download gzip (or similarly encoded) HTML content. If > > I keep the response in a variable, I can decode the contents using > the > > 'decoded_content' method, as follows (Just extracted from a working > > script, but not tested): > > > > #################################### > > use strict; > > use warnings; > > use base 'HTTP::Message'; > > use LWP::UserAgent > > > > my $can_accept = HTTP::Message::decodable; > > > > my $ua = LWP::UserAgent->new; > > my $response = $ua->get($uri, 'Accept-Encoding' => > $can_accept); > > print $response->decoded_content, "\n"; > > #################################### > > > > On the other hand, another variation of the 'get' method can save a > > response to a file, as in: > > > > $ua->get($url, ':content_file' => $filename); > > > > But I couldn't find a way to first deflate the contents that is saved > > to file. > > > > Can someone point me to the right part of the LWP package? > > > > Thanks, > > Meir > > > > > > No virus found in this incoming message. > Checked by AVG - www.avg.com > Version: 9.0.929 / Virus Database: 271.1.1/4939 - Release Date: > 04/15/12 21:34:00
$response->content($pageContent); my $decoded_pagecontent = $response->decoded_content( raise_error => 1);
Regards, Jesper
On 14 April 2012 22:25, Meir Guttman <meir@guttman.co.il> wrote:
> I would like to download gzip (or similarly encoded) HTML content. If I > keep the response in a variable, I can decode the contents using the > 'decoded_content' method, as follows (Just extracted from a working > script, but not tested): > > #################################### > use strict; > use warnings; > use base 'HTTP::Message'; > use LWP::UserAgent > > my $can_accept = HTTP::Message::decodable; > > my $ua = LWP::UserAgent->new; > my $response = $ua->get($uri, 'Accept-Encoding' => $can_accept); > print $response->decoded_content, "\n"; > #################################### > > On the other hand, another variation of the 'get' method can save a > response to a file, as in: > > $ua->get($url, ':content_file' => $filename); > > But I couldn't find a way to first deflate the contents that is > saved to file. > > Can someone point me to the right part of the LWP package? > > Thanks, > Meir > >
On Sun, Apr 15, 2012 at 4:25 AM, Meir Guttman <meir@guttman.co.il> wrote:
> print $response->decoded_content, "\n"; > > On the other hand, another variation of the 'get' method can save a > response to a file, as in: > > $ua->get($url, ':content_file' => $filename); > > But I couldn't find a way to first deflate the contents that is > saved to file. > > Can someone point me to the right part of the LWP package? >
Is there a reason you can't print the decoded content?
I don't know how get method uses uncompress internally either. Like you describe, I save the file to a directory and deflate it using unzip.
Thanks Uday
On Sun, Apr 15, 2012 at 3:03 PM, Meir Guttman <meir@guttman.co.il> wrote: > Dear Uday, > > I don't know. May be, but if so, I am not sure how. You see, I know > very little about the inner workings of all these complex packages. > (As an aside, this is a testimony to the value of OOP. Without any > knowledge of the implementation, I am able to use it.) > > So, the question is how do I invoke such a deflating package where > the ':content_file' => $filename hash short circuits and directs the > response directly to a file. I am sure that the deflating package > is invoked by the $response->decoded_content method. But I don’t > know about any hash that will tell the 'get' method to deflate the > response on the way to $filename. > > I can of course use the decoded_content method and save the result > to file. But I was hoping that there is a method that will do it > for me... > > Thanks! > Meir > >> -----Original Message----- >> From: Uday Bhaskar [mailto:uday.bhaskar@gmail.com] >> Sent: Sunday, April 15, 2012 6:13 AM >> To: Meir Guttman >> Cc: Libwww Perl >> Subject: Re: How to deflate a ':content_file' response saved to a file >> >> Are you thinking about use IO::Uncompress::Unzip ? >> >> Thanks >> Uday >> >> On Sun, Apr 15, 2012 at 1:55 AM, Meir Guttman <meir@guttman.co.il> >> wrote: >> > I would like to download gzip (or similarly encoded) HTML content. If >> > I keep the response in a variable, I can decode the contents using >> the >> > 'decoded_content' method, as follows (Just extracted from a working >> > script, but not tested): >> > >> > #################################### >> > use strict; >> > use warnings; >> > use base 'HTTP::Message'; >> > use LWP::UserAgent >> > >> > my $can_accept = HTTP::Message::decodable; >> > >> > my $ua = LWP::UserAgent->new; >> > my $response = $ua->get($uri, 'Accept-Encoding' => >> > $can_accept); >> > print $response->decoded_content, "\n"; >> > #################################### >> > >> > On the other hand, another variation of the 'get' method can save a >> > response to a file, as in: >> > >> > $ua->get($url, ':content_file' => $filename); >> > >> > But I couldn't find a way to first deflate the contents that is saved >> > to file. >> > >> > Can someone point me to the right part of the LWP package? >> > >> > Thanks, >> > Meir >> > > >
I don't know. May be, but if so, I am not sure how. You see, I know very little about the inner workings of all these complex packages. (As an aside, this is a testimony to the value of OOP. Without any knowledge of the implementation, I am able to use it.)
So, the question is how do I invoke such a deflating package where the ':content_file' => $filename hash short circuits and directs the response directly to a file. I am sure that the deflating package is invoked by the $response->decoded_content method. But I don’t know about any hash that will tell the 'get' method to deflate the response on the way to $filename.
I can of course use the decoded_content method and save the result to file. But I was hoping that there is a method that will do it for me...
Thanks! Meir
> -----Original Message----- > From: Uday Bhaskar [mailto:uday.bhaskar@gmail.com] > Sent: Sunday, April 15, 2012 6:13 AM > To: Meir Guttman > Cc: Libwww Perl > Subject: Re: How to deflate a ':content_file' response saved to a file > > Are you thinking about use IO::Uncompress::Unzip ? > > Thanks > Uday > > On Sun, Apr 15, 2012 at 1:55 AM, Meir Guttman <meir@guttman.co.il> > wrote: > > I would like to download gzip (or similarly encoded) HTML content. If > > I keep the response in a variable, I can decode the contents using > the > > 'decoded_content' method, as follows (Just extracted from a working > > script, but not tested): > > > > #################################### > > use strict; > > use warnings; > > use base 'HTTP::Message'; > > use LWP::UserAgent > > > > my $can_accept = HTTP::Message::decodable; > > > > my $ua = LWP::UserAgent->new; > > my $response = $ua->get($uri, 'Accept-Encoding' => > > $can_accept); > > print $response->decoded_content, "\n"; > > #################################### > > > > On the other hand, another variation of the 'get' method can save a > > response to a file, as in: > > > > $ua->get($url, ':content_file' => $filename); > > > > But I couldn't find a way to first deflate the contents that is saved > > to file. > > > > Can someone point me to the right part of the LWP package? > > > > Thanks, > > Meir > >
Are you thinking about use IO::Uncompress::Unzip ?
Thanks Uday
On Sun, Apr 15, 2012 at 1:55 AM, Meir Guttman <meir@guttman.co.il> wrote: > I would like to download gzip (or similarly encoded) HTML content. If I > keep the response in a variable, I can decode the contents using the > 'decoded_content' method, as follows (Just extracted from a working > script, but not tested): > > #################################### > use strict; > use warnings; > use base 'HTTP::Message'; > use LWP::UserAgent > > my $can_accept = HTTP::Message::decodable; > > my $ua = LWP::UserAgent->new; > my $response = $ua->get($uri, 'Accept-Encoding' => $can_accept); > print $response->decoded_content, "\n"; > #################################### > > On the other hand, another variation of the 'get' method can save a > response to a file, as in: > > $ua->get($url, ':content_file' => $filename); > > But I couldn't find a way to first deflate the contents that is > saved to file. > > Can someone point me to the right part of the LWP package? > > Thanks, > Meir >
I would like to download gzip (or similarly encoded) HTML content. If I keep the response in a variable, I can decode the contents using the 'decoded_content' method, as follows (Just extracted from a working script, but not tested):
#################################### use strict; use warnings; use base 'HTTP::Message'; use LWP::UserAgent
my $can_accept = HTTP::Message::decodable;
my $ua = LWP::UserAgent->new; my $response = $ua->get($uri, 'Accept-Encoding' => $can_accept); print $response->decoded_content, "\n"; ####################################
On the other hand, another variation of the 'get' method can save a response to a file, as in:
$ua->get($url, ':content_file' => $filename);
But I couldn't find a way to first deflate the contents that is saved to file.
Can someone point me to the right part of the LWP package?
(Forgive me if I'm doing this wrong, I haven't used this mailing list before)
Jesper; I ran into the same issue and worked around it for now by making a small modification to the referenced Methods.pm: At line 442, right after: elsif ($_ eq "identity") { $_ = sub { $_[0] }; } Add: elsif ($_ eq "chunked") { pop(@te); #Workaround for multiple chunked entries } Much better perl programmers will probably tell me there's a better/cleaner way to do this (my method may well break the encoding handler for complex scenarios with multiple copies of "chunked" mixed in with other types) but it works well enough for me to continue developing/testing my current app until the dev responsible for the bug in the web app I'm coding against fixes his side of things to prevent the doubled chunk encoding in the first place.
(Forgive me if I'm doing this wrong, I haven't used this mailing list before)
Jesper; I ran into the same issue and worked around it for now by making a small modification to the referenced Methods.pm: At line 442, right after: elsif ($_ eq "identity") { $_ = sub { $_[0] }; } Add: elsif ($_ eq "chunked") { pop(@te); #Workaround for multiple chunked entries }
Much better perl programmers will probably tell me there's a better/cleaner way to do this (my method may well break the encoding handler for complex scenarios with multiple copies of "chunked" mixed in with other types) but it works well enough for me to continue developing/testing my current app until the dev responsible for the bug in the web app I'm coding against fixes his side of things to prevent the doubled chunk encoding in the first place.
Is it considered acceptable for a subclass of LWP::UserAgent to invent a new phase name, and process handlers for that phase in new and/or overridden methods? If so, are there any naming recommendations that are likely to avoid conflicts with phases added in future versions of LWP::UA? And is the run_handlers method available to subclasses for these purposes (despite not being documented at the moment)?
I'd be happy to submit a documentation patch with the answers to these questions, once I know them. :-)
I am seeking help on an issue that I am facing while compiling Crypt::SSLeay module. I have successfully installed Openssl on the AIX 5.3 V12 successfully through SMITTY but when I try to compile Crypt::SSLeay module it throws following errors:
ld: 0711-768 WARNING: Object SSLeay.o, section 1, function .strcmp: The branch at address 0x13c is not followed by a recognized no-op
Could you please let me know is there anything wrong with SSL library or is it something else. Early response would be highly appreciable.
On Mon, Mar 5, 2012 at 3:54 PM, Tim Brody <tdb2@ecs.soton.ac.uk> wrote:
> On Sat, 2012-03-03 at 15:25 +0700, Bill Moseley wrote: > > I use HTTP::Request::Common to build an application/x-www-form-urlencoded > > POST from a passed-in hash. The hash contains strings as values. > > > > $req = POST /foo, \%parameters; > > > > > <snip> > > > > The thing to notice here is how the encoding for $latin1 changed just > > because of the addition into the hash of the $unicode string. Things > thus > > break when the server tries to decode the query parameters on the server > > side if it assumes either latin1 or utf8 encoding. >
I'm sorry, I made a mistake in my example. I meant:
$req = POST /foo, Content => \%parameters
It's the request *body* that I'm talking about, not the query parameters.
I understand this is a known issue in URI:
The escaping (percent encoding) of chars in the 128 .. 255 range passed to the URI constructor or when setting URI parts using the accessor methods depend on the state of the internal UTF8 flag (see utf8::is_utf8) of the string passed. *If the UTF8 flag is set the* * UTF-8 encoded version of the character is percent encoded. If the* * UTF8 flag isn't set the Latin-1 version (byte) of the character is* * percent encoded. This basically exposes the internal encoding of* * Perl strings. *
And because the same character string can be represented either without (latin1) or with (utf8) the UTF8 flag, and Perl can upgrade character strings from latin1 to utf8 w/o me knowing, I cannot be be sure exactly what percent encoding will be used. It's not really a bug, rather it's just not clear what percent encoding will be used.
That is, with two hashes passed to query_form, the second only with the addition of the "unicode" key with the value that has the utf8 flag true, changes the percent encoding of "latin1" value.
Gisle, If I were to override query_form do you see any problems with either of these approaches to make sure that the final percent encoding is always of utf8 encoded octets? One by explicitly encoding everything to utf8 first:
I use HTTP::Request::Common to build an application/x-www-form-urlencoded POST from a passed-in hash. The hash contains strings as values.
$req = POST /foo, \%parameters;
This uses URI->query_form to build the url-encoded body, and sets the Content-Type to application/x-www-form-urlencoded (without any charset).
The problem I have is that the %parameters hash contains valid Perl character strings, but the resulting url-encoded request differs depending on the mix of Perl stings -- by mix I mean strings with and without Perl's utf8 flag. The resulting url-encoded request ends up either latin1 or utf-8 url-encoded octets.
It's not easy to know what charset to add to the request, and likewise, things break if the server handling the request assumes it's a utf8 url-encoded request.
Perhaps some code might help:
Consider these three very normal, valid, *character* strings in Perl:
my $ascii = 'Hello'; my $latin1 = 'Ue: ' . chr(220); my $unicode = "Happy \x{263A}";
What you would expect is that only the $unicode would have Perl' utf8 flag set. And indeed that is true:
print_var($_) for $ascii, $latin1, $unicode;
str [Hello] with flag: NO str [Ue: Ü] with flag: NO str [Happy ☺] with flag: YES
And if the strings are concatenated Perl will utf8::upgrade $latin1, and you can see that is true because the umlaut survived the trip from latin1 to utf8.
URI query_form = str [ascii=Hello] with flag: NO URI query_form = str [ascii=Hello&*latin1=Ue%3A+%DC*] with flag: NO URI query_form = str [ascii=Hello&unicode=Happy+%E2%98%BA&* latin1=Ue%3A+%C3%9C*] with flag: NO
The thing to notice here is how the encoding for $latin1 changed just because of the addition into the hash of the $unicode string. Things thus break when the server tries to decode the query parameters on the server side if it assumes either latin1 or utf8 encoding.
The problem is I have code that accepts a hash and passes it directly to POST. But, if there happens to be a latin1 string in there then the request changes depending if there's also a string with the utf8 flag set or not.
Am I missing something here? Seems like if query_form is passed a hash then the resulting encoding should not change based on what else is in that hash.
I can think of two solutions. One would be to build the query string in a different way by explicitly encoding to utf8 first:
On Thu, Mar 1, 2012 at 6:00 AM, Gisle Aas <gisle@activestate.com> wrote:
> > HTTP::Headers doesn't do any encoding for you, which basically means that > you > need to pass encoded strings as $name and $value, and make sure the result > ends > up as valid HTTP. >
Ok. I typically like to decode and encode at the "edges" of my code, but it's less clear to me if setting a header here is at the edge. Clearly special handling is needed in this case.
> RFC 5987 seem to be relevant for HTTP::Headers to support in this regard. > > This discussion about Content-Disposition seemed relevant to your original > question: > > > http://stackoverflow.com/questions/1361604/how-to-encode-utf8-filename-for-http-headers-python-django
I had read that and was left somewhat confused. There's also a lot of examples of code that inspects the user agent and adjusts the filename based on the client. I don't like that approach at all. I'd rather send down what I have and let the client decide how to handle.
As a lazy programmer if I'm given a filename in an upload (or anyway by a client) and then later return it as an attachement I don't think it's any of my business to modify it other than encode it for transport.
So, in short, I'm simply doing this:
my $encoded_filename = uri_escape_utf8( $filename );
> Sorry, this got sent too quickly: > > On Wed, Feb 29, 2012 at 2:15 PM, Bill Moseley <moseley@hank.org> wrote: > >> I have a user-supplied filename that could be anything, including path >> segments and fullset of unicode characters (It's decoded as UTF-8 when >> supplied by the user. >> >> There seems to be quite a bit of debate about what modern browsers will >> except. >> >> my $filename = qq{/some/funny looking/"name that the \\'user' provided' >> with a \x{263A} (smile).doc}; >> >> How can I pass that back in a Content-Disposition "filename" header >> escaped correctly? >> > > > And does $request->header( $name => $value ) > > expect $name and $value to be Character data (decoded into Perl characters) > or does it always need to be encoded first? Seems like the encoding should > happen when the request gets "rendered" for output.
HTTP::Headers doesn't do any encoding for you, which basically means that you need to pass encoded strings as $name and $value, and make sure the result ends up as valid HTTP.
RFC 5987 seem to be relevant for HTTP::Headers to support in this regard.
This discussion about Content-Disposition seemed relevant to your original question:
On Wed, Feb 29, 2012 at 2:15 PM, Bill Moseley <moseley@hank.org> wrote:
> I have a user-supplied filename that could be anything, including path > segments and fullset of unicode characters (It's decoded as UTF-8 when > supplied by the user. > > There seems to be quite a bit of debate about what modern browsers will > except. > > my $filename = qq{/some/funny looking/"name that the \\'user' provided' > with a \x{263A} (smile).doc}; > > How can I pass that back in a Content-Disposition "filename" header > escaped correctly? >
And does $request->header( $name => $value )
expect $name and $value to be Character data (decoded into Perl characters) or does it always need to be encoded first? Seems like the encoding should happen when the request gets "rendered" for output.
I have a user-supplied filename that could be anything, including path segments and fullset of unicode characters (It's decoded as UTF-8 when supplied by the user.
There seems to be quite a bit of debate about what modern browsers will except.
my $filename = qq{/some/funny looking/"name that the \\'user' provided' with a \x{263A} (smile).doc};
How can I pass that back in a Content-Disposition "filename" header escaped correctly?
I can't check at this moment whether this is new in 5.15.7
make test for WWW-Mechanize-1.71 t/credentials-api.t ...... Use of uninitialized value in lc at /opt/perl/uq.is.perl.rhel6-5.15.7-20120121/lib/site_perl/5.15.7/LWP/UserAgent.pm line 580.
sub credentials LWP::UserAgent { my $self = shift; my $netloc = lc(shift); <== line 580 my $realm = shift || ""; my $old = $self->{basic_authentication}{$netloc}{$realm};
but t/credentials-api.t says my @ua = $ua->credentials; isnt( "@ua", "mech2 mech2", 'LWP::UserAgent instance retains its old credentials' );
looks like that test is wrong as LWP::UserAgent pod says you need to specify netloc & realm
A website is sending the same http header twice: Client-Transfer-Encoding: chunked Client-Transfer-Encoding: chunked
Which gives an error: Can't handle transfer encoding 'chunked,chunked' at C:/Perl64/site/lib/Net/HTTP/Methods.pm line 443.
perl code:
use LWP::UserAgent; my $ua = new LWP::UserAgent; my $request = HTTP::Request->new('GET'); $request->url('http://www.virk.dk'); my $response = $ua->request($request); my $x_died = $response->header("X-Died"); print "x_died: $x_died\n"; ^D x_died: Can't handle transfer encoding 'chunked,chunked' at C:/Perl64/site/lib/Net/HTTP/Methods.pm line 443.
On 1/16/2012 9:52 PM, Bjoern Hoehrmann wrote: > * Christopher J. Madsen wrote: >>> Your UTF-8 validation code seems wrong to me, you consider the sequence >>> F0 80 to be incomplete, but it's actually invalid, same for ED 80, see >>> the chart in <http://bjoern.hoehrmann.de/utf-8/decoder/dfa/#design>. >> >> I guess the RE could be improved, but I'm not sure it's worth the effort >> and added complication to catch a tiny fraction of false positives. > > Why make the check at all if you don't care if it's right?
I can't use a simple utf8::decode check, because I read a fixed number of bytes, and that might have cut a multi-byte character in half. So I use Encode::FB_QUIET, and then check the leftovers to make sure that it's a single, plausible, partial UTF-8 character. I have to check the leftovers, or the whole test would be meaningless. I just make sure it's a start byte followed by an appropriate number of continuation bytes.
As you say, certain start bytes can't validly be followed by certain continuation bytes, but writing an RE for those rules is more complexity than I think the problem warrants. What are the odds that I had 1021 bytes of valid UTF-8 (including at least 1 multi-byte character) followed by bytes that match my current RE but a strict test could have rejected? I'm already just assuming that the next bytes would be additional continuation bytes.
>>> Anyway, if people think this is the way to go, maybe HTTP::Message can >>> adopt the Content-Type header charset extraction tests in HTML::Encoding >>> so they don't get lost as my module becomes redundant? >> >> I thought it already did that? > > Not as far as I can tell; links welcome though.
At the beginning of content_charset, it calls content_type_charset (which is actually a HTTP::Headers method).
Or were you talking about t/01http.t and its associated input files?
-- Chris Madsen perl@cjmweb.net -------------------- http://www.cjmweb.net --------------------
* Christopher J. Madsen wrote: >Dropping support for UTF-32 from HTTP::Message is a separate issue from >removing HTML::Parser. I've got no comment on that.
(It's not quite as black and white as that, "HTML5" could be exempted in the algorithm, for instance.)
>> Your UTF-8 validation code seems wrong to me, you consider the sequence >> F0 80 to be incomplete, but it's actually invalid, same for ED 80, see >> the chart in <http://bjoern.hoehrmann.de/utf-8/decoder/dfa/#design>. > >I guess the RE could be improved, but I'm not sure it's worth the effort >and added complication to catch a tiny fraction of false positives.
Why make the check at all if you don't care if it's right?
>> Anyway, if people think this is the way to go, maybe HTTP::Message can >> adopt the Content-Type header charset extraction tests in HTML::Encoding >> so they don't get lost as my module becomes redundant? > >I thought it already did that?
Not as far as I can tell; links welcome though. -- Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de 25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/
On 1/16/2012 6:53 PM, Bjoern Hoehrmann wrote: > * Christopher J. Madsen wrote: >> My repo is https://github.com/madsen/io-html but since it's built with >> dzil, I also made a gist of the processed module to make it easier to >> read the docs: https://gist.github.com/1623654
> It is not clear to me that the combination would actually conform to the > "HTML5" proposal, for instance, HTTP::Message seems to recognize UTF-32 > BOMs, but as I recall the "HTML5" proposal does not allow that.
Dropping support for UTF-32 from HTTP::Message is a separate issue from removing HTML::Parser. I've got no comment on that.
> Your UTF-8 validation code seems wrong to me, you consider the sequence > F0 80 to be incomplete, but it's actually invalid, same for ED 80, see > the chart in <http://bjoern.hoehrmann.de/utf-8/decoder/dfa/#design>.
I guess the RE could be improved, but I'm not sure it's worth the effort and added complication to catch a tiny fraction of false positives.
> Anyway, if people think this is the way to go, maybe HTTP::Message can > adopt the Content-Type header charset extraction tests in HTML::Encoding > so they don't get lost as my module becomes redundant?
I thought it already did that?
-- Chris Madsen perl@cjmweb.net -------------------- http://www.cjmweb.net --------------------
* Christopher J. Madsen wrote: >My repo is https://github.com/madsen/io-html but since it's built with >dzil, I also made a gist of the processed module to make it easier to >read the docs: https://gist.github.com/1623654 > >I took a quick look at HTTP::Message, and I think you'd just need to do > > elsif ($self->content_is_html) { > require IO::HTML; > my $charset = IO::HTML::find_charset_in($$cref); > return $charset if $charset; > } > >You're already doing the BOM and valid-UTF8 checks; all you need is the ><meta> check, which is what find_charset_in does.
It is not clear to me that the combination would actually conform to the "HTML5" proposal, for instance, HTTP::Message seems to recognize UTF-32 BOMs, but as I recall the "HTML5" proposal does not allow that.
Your UTF-8 validation code seems wrong to me, you consider the sequence F0 80 to be incomplete, but it's actually invalid, same for ED 80, see the chart in <http://bjoern.hoehrmann.de/utf-8/decoder/dfa/#design>.
Anyway, if people think this is the way to go, maybe HTTP::Message can adopt the Content-Type header charset extraction tests in HTML::Encoding so they don't get lost as my module becomes redundant? -- Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de 25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/
and a discussion here about removing HTTP::Message's dependency on HTML::Parser (which needs a C compiler) for charset sniffing.
As it happens, I'm about to release a new dist that implements the HTML5 encoding sniffing algorithm in pure-Perl with no non-core dependencies for 5.8+. While its primary function is to make it dead simple to open a HTML file and get the right encoding layer applied automatically, it also exposes the underlying mechanism.
My repo is https://github.com/madsen/io-html but since it's built with dzil, I also made a gist of the processed module to make it easier to read the docs: https://gist.github.com/1623654
I took a quick look at HTTP::Message, and I think you'd just need to do
elsif ($self->content_is_html) { require IO::HTML; my $charset = IO::HTML::find_charset_in($$cref); return $charset if $charset; }
You're already doing the BOM and valid-UTF8 checks; all you need is the <meta> check, which is what find_charset_in does.
One possible issue is that find_charset_in returns Perl's canonical name for the encoding, which is not necessarily the same as the MIME or IANA charset name. You could do
return Encode::find_encoding($charset)->mime_name if $charset;
if you want.
I'm planning to release this in a week or so, after I see if any more bugs pop up or I think of any API changes I should make.
-- Chris Madsen perl@cjmweb.net -------------------- http://www.cjmweb.net --------------------
I think we need a better solution for HTML::Tagset than to Just Add Tags to what's there, as is suggested here https://rt.cpan.org/Ticket/Display.html?id=67299
To be clear: I'm NOT looking for a patch that adds elements. I'm looking for how to make the existing code work with both HTML4 and HTML5. If I add a SECTION tag to the module, then <SECTION> will no longer be flagged as invalid when I check an HTML 4 website with it.
So how can we keep things compatible and forward-looking?
Thanks, xoxo, Andy
-- Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance
On Nov 1, 2011, at 7:54 PM, Father Chrysostomos wrote:
> Begin forwarded message: > >> From: Father Chrysostomos <sprout cpan.org> >> Date: November 1, 2011 7:52:15 PM PDT >> To: libwww perl.org >> Subject: Support for HTTPS over proxies >> >> Currently LWP does not support HTTPS over proxies. I’m trying to write a patch for just that, but I’m not sure whether the special-casing code should go in LWP::UserAgent::send_request or in LWP::Protocol::http::request. >> >> If an HTTPS connection is made through an HTTP proxy, the request method sent to the proxy must be CONNECT, followed by optional headers (probably default_headers). Then the client must wait for an HTTP 200 response, after which it must initiate an HTTPS request using the same connection. >> >> Should this sort of switching be in LWP::UserAgent::send_method? > > I mean send_request.
I’ve just discovered that Net::SSL supports proxies. So I’m satisfied. I’m afraid a patch will not be forthcoming. However, I have to do this dance to make things work properly:
This is the relevant error code. My guess would be there is a problem with the certificates involved, but a quick search does not support that. You could try running `openssl s_client -connect example.com:443` (and possibly manually type some HTTP request after that) and see if that provides any clues. There are also environment variables you can set to control whether certificates are verified to rule that out as a problem. -- Björn Höhrmann · mailto:bjoern@hoehrmann.de · http://bjoern.hoehrmann.de Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de 25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/
> From: Father Chrysostomos <sprout cpan.org> > Date: November 1, 2011 7:52:15 PM PDT > To: libwww perl.org > Subject: Support for HTTPS over proxies > > Currently LWP does not support HTTPS over proxies. I’m trying to write a patch for just that, but I’m not sure whether the special-casing code should go in LWP::UserAgent::send_request or in LWP::Protocol::http::request. > > If an HTTPS connection is made through an HTTP proxy, the request method sent to the proxy must be CONNECT, followed by optional headers (probably default_headers). Then the client must wait for an HTTP 200 response, after which it must initiate an HTTPS request using the same connection. > > Should this sort of switching be in LWP::UserAgent::send_method?
Currently LWP does not support HTTPS over proxies. I’m trying to write a patch for just that, but I’m not sure whether the special-casing code should go in LWP::UserAgent::send_request or in LWP::Protocol::http::request.
If an HTTPS connection is made through an HTTP proxy, the request method sent to the proxy must be CONNECT, followed by optional headers (probably default_headers). Then the client must wait for an HTTP 200 response, after which it must initiate an HTTPS request using the same connection.
Should this sort of switching be in LWP::UserAgent::send_method?
On Thu, 27 Oct 2011 19:28:17 +0200, Gisle Aas <gisle@activestate.com> wrote:
> On Oct 26, 2011, at 17:33 , H.Merijn Brand wrote: > > > LWP::Protocol::https::Socket: SSL connect attempt failed because of handshake problemserror:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure at /pro/lib/perl5/site_perl/5.10.1/LWP/Protocol/http.pm line 51. > > I don't really know much about this error or how to resolve this. Could you perhaps try to install IO::Socket::SSL and see if that helps. It's currently the preferred interface to openssl.
> LWP::Protocol::https::Socket: SSL connect attempt failed because of handshake problemserror:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure at /pro/lib/perl5/site_perl/5.10.1/LWP/Protocol/http.pm line 51.
I don't really know much about this error or how to resolve this. Could you perhaps try to install IO::Socket::SSL and see if that helps. It's currently the preferred interface to openssl.
We have a perl-5.10.1 install tree that is quite up to date. All SSL related things have been compiled and linked against OpenSSL-0.9.8r
We have a perl-5.14.2 tree ready to take over, but that has been compiled against OpenSSL-1.0.0e and therefor is not compatible with the situation currently available at the customers site. The upgrade will happen as soon as all the 5.14.2 tests have been finished and no problems remain. As that upgrade will also have to synchronized with the move to OpenSSL-1.0.0, we are pretty much bound to 5.10.1 for now.
When we (try to) communicate using SOAP with libwww-6.03, we get this error: --8<--- SOAP::Transport::HTTP::Client::send_receive: HTTP::Response=HASH(0x6000000001d00150) SOAP::Transport::HTTP::Client::send_receive: 500 Can't connect to bcgba.burgerservicenummer.nl:443 Content-Type: text/plain Client-Date: Wed, 26 Oct 2011 15:04:10 GMT Client-Warning: Internal response
Can't connect to bcgba.burgerservicenummer.nl:443
LWP::Protocol::https::Socket: SSL connect attempt failed because of handshake problemserror:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure at /pro/lib/perl5/site_perl/5.10.1/LWP/Protocol/http.pm line 51. SOAP::Deserializer::deserialize: () SOAP::Parser::decode: () -->8---
When I then downgrade libwww to 5.837 all goes smooth.
That gives me no good feeling in my possible move to 5.14.2
A serious problem that I encounter in debugging that, is that this process cannot be simulated on sites other than on the customer machine, as the test-environment of this service that we as software developers can access is not identical to the site that the customers (try to) access, so I can only test on one single (test) box when the customer allows me to access it (over a dial-in ISDN line).
Is this a bug in libwww-6 or are there other paths to consider.
On 21/05/2011 8:12 AM, Andy Lester wrote: >> My reason for raising this now is that the HTML5 spec elements are starting to appear on documents on the internet, and with these tags missing from HTML::Tagset, it impacts the effectiveness of other libraries that depend upon it. > Haven't looked at the patch, but I think we need to talk about high-level thoughts about handling HTML5. > > * Should we have separate sets of tags for HTML 4 and 5? > * If so, should it be handled be one module, or should it get split into to? > * Should we have strict and loose definitions of tags, going forward? One of the things I've run into is tag attributes that are recognized by browsers, but not in the spec. > Thoughts?
I'm surprised that 5 months have slipped by and no one is interested in discussing adding HTML5 entities to HTML::TagSet library, and the large swathes of modules this will impact and/or (hopefully) improve.
Right now, trying to use HTML::TreeBuilder::XPath is futile when processing HTML 5 documents, and there are probably other very good libraries out there that are similarly incapacitated. I'm not saying using HTML 5 now as a 'draft' is good practice, but browsers are implementing it, and sites are using it. Last draft was published 25 May 2011, current editors version is dated 21 Oct 2011 (so only 3 days ago). There's a core set of new elements that look like they are set to stay (section, canvas, etc), so should HTML::TagSet include them now, and be updated again for other major changes that come along?
I suggest that the major elements of HTML 5 are added to HTML::Tagset as is. At least we get some immediate relief to a burning issue. HTML::TagSet can then be split at leisure into separate modules (HTML::TagSet::HTML5, HTML::TagSet::HTML4, HTML::TagSet::XHTML, etc) and a wrapper on top to combine them (and resolve conflicts between them) over the next few years if required.
James -- /Mobile:/ +61 422 166 708, /Email:/ james_AT_rcpt.to
On 15/10/2011 07:15, Gisle Aas wrote: > On Oct 15, 2011, at 0:03 , Rob Dixon wrote: > >> Method HTML::Form::FileInput::form_name_value has been changed so that >> there is now a separate 'filename' field in the object. There is also >> code, marked as legacy, handling the old way of doing things. >> Unfortunately is is broken, and the problem has arisen from a query >> regarding WWW::Mechanize which still uses this old method. > > Are you able to modifiy the test suite to demonstrate under what conditions > this bug occur?
> Method HTML::Form::FileInput::form_name_value has been changed so that > there is now a separate 'filename' field in the object. There is also > code, marked as legacy, handling the old way of doing things. > Unfortunately is is broken, and the problem has arisen from a query > regarding WWW::Mechanize which still uses this old method. > > At the start, $filename is copied from the {filename} field of the > object, or from {value} if that field is empty. Later on in the legacy > code, line 1509, $fn is copied to $filename only if it is currently > undefined. Because the 'filename' method has defaulted it to the same > value as $file it will never be undefined at this point, and can be an > array reference if the old call method is used, giving actual an actual > value in the HTTP::Request of something like "ARRAY(0x9567570)". > > All that seems to be necessary to fix the problem is to remove the line > > $old = $self->file unless defined $old; > > from the 'filename' method.
Hi Rob,
Are you able to modifiy the test suite to demonstrate under what conditions this bug occur?
I sent this to Gisle on 4 October but have had no reply. Thinking his account may be whitelisted, I am posting it again here.
Since there are no bugs - even cleared ones - on the CPAN reporting system for this module, I suspect it is best to mail you this bug report.
Method HTML::Form::FileInput::form_name_value has been changed so that there is now a separate 'filename' field in the object. There is also code, marked as legacy, handling the old way of doing things. Unfortunately is is broken, and the problem has arisen from a query regarding WWW::Mechanize which still uses this old method.
At the start, $filename is copied from the {filename} field of the object, or from {value} if that field is empty. Later on in the legacy code, line 1509, $fn is copied to $filename only if it is currently undefined. Because the 'filename' method has defaulted it to the same value as $file it will never be undefined at this point, and can be an array reference if the old call method is used, giving actual an actual value in the HTTP::Request of something like "ARRAY(0x9567570)".
All that seems to be necessary to fix the problem is to remove the line
Compiling My Own perl—brian d foy (first page)
FMTIEWTK About Closures—Johan Lodin (first page)
Expecting Perl—Mark Schoonover (first page)
Perl and Undecidability—Jeffrey Kegler (first page)
The Year in Perl, 2007—brian d foy (first page)
Templating My Output—Alberto Manuel Simões (first page)
Making My Own CPAN—brian d foy (first page)
Programming Parrot—Jonathan Scott Duff (first page)
Komodo Test Drive—Jim Brandt (first page)
Named Captures in Perl 5.9.5—brian d foy (first page)
Simple Web Access—Alberto Manuel Simões (first page)
Parrot Status Report—Jonathan Scott Duff (first page)
Mapping Op Codes—Eric Maki (first page)
CPANdeps—David Cantrell (first page)
HTML Slides—Grant McLean (first page)
Alter Egos—Anno Siegel (first page)
Found Perl is section of The Perl Review's website where we posted pictures of Perl paraphernalia or the word "Perl" in the wild. We started it a long time ago, and were very slow in updating it when people would send in photos.
Now "Found Perl" is a photo pool on Flickr. Instead of sending them to TPR, just add them to the pool.
The Perl Review's website has had a section on Schwern's Shirt, the orange monstrosity that brian d foy bought at the charity auction for The Perl Foundation at the 2004 Open Source Convention.
Now we've moved the section of the website to a Flickr group for Schwern's shirt. This way, anyone can add their photos of Schwern's shirt to the group. Instead of being infrequently updated on the website, people can add them as soon as they upload them to Flickr.
If you don't have a Flickr account and don't want to create one, you can still send them to The Perl Review by mailing them to editors@theperlreview.com.
David Pogue had a momentary lapse of judgement when he proclaims in his blog that the date sequence 01:02:03 04/05/06 will only happen once in all of human history.
Besides the obvious gaffes of date formatting (which one is the month and which one is the year?), the red herring of leading zeros (to make the minute and second stand out), and so on, no one who's seen this has made the comment that calendars say whatever we want them to say and the numbers are only special because we set the calendar up that way in this one case. What about the Chinese, Hebrew, and Muslim calendars?
So this seems like a good challenge to publish in The Perl Review: using the Perl Date modules (or not, I guess), in how many different calendars and formats can you make this sequence? What else is special about those days (are they a weekend, fall on a full moon, have a solar eclipse, etc.)?
Powell's Technical Books in Portland (that's the one on 33 NW Park Avenue) is going to carry The Perl Review. It's our first newsstand distribution.
I had to set a newstand price. The deal basically works like this: bookstores keep most of the profit. Magazines make money when the single-issue buyers turn into subscribers. After Powell's cut, which we set at 40%, and my costs, $2 an issue, I have to figure out a price that also motivates people to give the money to The Perl Review directly instead of the book store. That's why you see big discounts for magazines when you subscribe: that's the real price, and everything else is markup. The Powell's price ends up being $5, which is 50 cents more than the subscription price.
That's not to say that newsstands are bad. It's like better-than-free advertising since it sits on the shelf and I cover my costs plus a little more for every issue sold.
Forget about absolute numbers for a moment. At my price point, if they sell 75% of the copies, I break even. That would be fine with me because any copy sitting on the actual magazine display means people see that issue. Some might subscribe later even if they don't buy it. Now that I have a price point, I have to figure out the right number of issues. That's something I just have to guess.
I left 16 copies of the Spring 2005 issue, but I also have to consider that I sold about 10 at the Intermediate Perl book signing. We'll see how that goes.
Now, a good magazine accountant has to keep track of the actual number of newsstand sales too. As much as I'd like to pretend that we sell every single copy, the Post Office wants to know where all the issues went to verfiy that we abide by all the periodical rules. It's not enough for the newsstand to simply tell me what they sold. They certainly aren't going to tell me they sold everything when they didn't since that's money out of their pocket. They can't really tell me they sold nothing because that's money out of my pocket.
If you're a late night person living in a city, you might have seen a bunch of guys tearing off the front pages of newspapers and magazines. Instead of sending back the unused copies, they send back the cover (and they do that for books too). Every cover they don't send back is a sale that they owe me money for.
You might think those unsold issues represent lost money, but they really don't. They are a sunk cost, meaning that I would have spent that money regardless of the sales. That starts way back at the printers when I have to decide how many issues I want. That number includes all subscriptions, complimentary copies, samples to user groups, and all the issues I'll need to fulfill orders for back issues. Not only that, but the more copies I print, the lower the incremental cost (the cost per each copy). Each printing job has a fixed overhead for the job preparation, machine set-up, and so on. That's the make ready. I end up printing many more copies than I need, partly to amoritize the make ready. Not selling at the newsstand is slightly better than not selling while sitting in boxes in the office. At least people see them at the newsstand.
Remember that magazines make money on subscriptions, so that's the goal. I don't care about selling more at the newsstands. If someone subscribes because they see an issue on the newsstand, the profit from the subscription pays for about three unsold newsstand copies, so five subscriptions from people seeing the issue at Powell's would make up for no sells. That's just breaking even, and nobody makes any money. That also means I'm spending $6 to get a new subscriber.
If you're already despondent, you don't want to read about distributors. Most bookstores don't want to deal with every individual publisher. They'd have to keep track of a separate deal for every magazine. Instead, they want to deal with a single source in the same way they deal with books. I know my costs, and I know the newsstands cut, and I have a price point that I can't change to much because people won't buy it at too high a price. If I use a distributor, perhaps to get into the big chain book stores, they are going to want a big cut too. I'll end up either breaking even or losing money on every newsstand copy, and I'll want to convert that to a subscription as soon as possible. That's why you see so many wonderful subscription cards in the magazines.
So far I've just talked about money from sales. We can also sell advertising, which we do for the special friends of the Perl community. Since magazines know they are going to lose money at the newsstand, they make up the difference with paid advertising. Ever wonder why magazines such as Wired are mostly advertisements? That's making up for the money they'll lose on the newsstands. Remember when I talked about keeping track of the number of copies sold? Advertisers want to know those numbers. They don't care how many copies the newsstand bought. They care about the number of copies that shoppers bought. That sets the rate at which the magazines can sell ads. More eyeballs equal more dollars. There's a separate industry of companies that audit magazines to verify the numbers. That's even more money that gets sucked away.
The short story? Subscribe to the magazines you like. It's the only way they can survive.
Last year, I started including interviews with Perl people on The Perl Review website, but I didn't add a feed for that. Now I have and it's on RSS page.
The web site is actually a directory processed by Template Toolkit, so what I really need to do is add indexing support as ttree goes through the directories, then spit out pages as it does that. That sounds like a magazine article...
Every issue I get a couple of book reviews that don't quite cut it. Techies tend to present too many sides of the story: rather than express their opinion, they equivocate by pointing out all the holes in their own opinion. In short, they are entirely too nice.
Being nice isn't a bad thing, but what people really want out a book review is a recommendation. "Should I buy this book?" It doesn't matter if people agree with you as long as you are fair to the book. After that, people want to know the particulars: who, what, when, where, and how.
So far, The Perl Review has taken reviews from several different people, and very few people have provided multiple reviews. That worked when we were first getting started, but now I think we need something different. Since book reviews are about opinions, and the reader doesn't have to agree with the reviewer, I think readers need to know the reputation and leaning of the reviewer to make their own decision. For instance, my wife doesn't agree with movie-reviewer Roger Ebert, but based on his negative reviews she knows which movies she will like. At the same time, she knows the certain people on LiveJournal likes the same sorts of movies she does, so she can trust them.
In line with all that, I think I'm going to move towards recurring book reviewers, and do more to establish their reputation. That also means that I want to get a couple of reviewers who think about books differently so I can give more readers someone that thinks like them. I've approached a couple of people, and we'll see how it turns out.
I've added two guides I hadn't run across previously:
It seems that every time I put out a new issue I have to leave town for a couple weeks. I don't think the two are related. Travel is just too busy this year. Add to that finishing up the Alpaca book and I've had a pretty busy week. I'll be back in the middle of December.
I'm sending out the email announcements as I post this. Once you get the new password, you'll be able to download the latest issue of The Perl Review.
In this issue:
Seven Sins of Perl OO Programming -- chromatic
Hash Anti-Patterns -- Alberto Manuel Simões
Haskell for Perlers -- Frank Antonsen
PerlWar -- Yanick Champoux
books reviews, commentary, news and more...
Some of you neglected to renew, but I won't bug you in email anymore. To find out more about the sampler on the cover, you'll have to renew that subscription.
If you haven't subscribed yet, now's the time because I have to raise prices next year when the US postal rates go up.
The next issue of The Perl Review comes out in a couple of weeks, so it's time to renew if you've already finished your first year subscription!
This may seem odd for some who just recently subscribed, but with the magic of time travel we let you pretend that your subscription started much earlier by allowing you to choose which issue to start your subscription. You may have subscribed yesterday but selected to start with issue 1.0 (Winter 2004), meaning that your issue ran out with our last issue, 1.3 (Fall 2005).
I've found that most new subscribers like to get most of the back issues at first, and it's worked out rather well for us. Now that we're starting our second year, though, I have to figure out how to make that work so people don't have to renew right away. I'll have to add a two-year subscription, I think.
I've been reading lot about magazine renewal rates. A lot of publications seem to be happy to get 25% renewals, and they actually spend a fair bit of money to get that. Ever wonder why you get some many magazine renewal letters in your postal mail?
For TPR, since I don't have a lot of money, I simply used email. The target audience is certainly technology-adept, so that's not so bad. So far I've sent out three renewal emails. One was the week before OSCON, and I got about a 45% response for that. That's already good for magazines, but not quite the 98% renewal rate TPJ had in its first year. I sent out a second email two weeks after that, and another one over the weekend. I think I'm floating somewhere around 75% renewal right now.
I have a list of all of the people who haven't renewed (it's an SQL view ;), and a lot of them should have and I would be surprised if they don't. Although email provided me with a virtually free way to get that 75%, I also think email has a tendency to get lost. First, if the subscriber can't deal with it right awy, it joins the long queue of messages that get ignored forever. Second, it has a pretty good chance of being blocked by a spam filter. Third, some people might get so mucch mail that they just don't get to ever see it. I do send each mail individually since each contains a persoanl renewal code, so I'm not blasting out spam to any user at each domain.
I am considering sending out some postcards to the hold-outs. I figure that will cost me about $50, which means that I need about four renewals to make up for it. If I get four renewals, that's paid for. I'm also going to try emailing people directly. It certainly helps when other people talk about TPR. I notice a big spike in renewals when Ovid talked about his upcoming Logic Programming article.
There's another trick to getting renewals: special promotional subscriptions. Magazines give you a special rate for a limited time then hit you up for the full price as quickly as possible. It's all part of the ad-selling game. Advertisers want to know how many "qualified subscribers" you have, which means how many people actually want your magazine enough to pay for it. Getting people to be full subscribers raises those numbers.
TPR is a bit different because I don't aim to make it an audited magazine, meaning that no one is going to come in and put a stamp on our books saying they verified our subscriber base. I'm not in it to sell advertising (let's see if I'm saying the same thing in three years). So far I've only taken advertising from people the Perl community already loves and trusts. Most of that is just filling up the other full color pages that come along with the cover.
I now have a big stack of emails confirming renewal transactions, and I need to shove those transactions into my local database so people keep getting their magazines.
Most of them are pretty easy because they are keyed on their unique ID in the database. A lot of people skipped the link I sent them and went to the subscription page directly, so I have to match up those with what I see in te database.
So far it works like this and handles 95% of the records:
Get all the matches by name
With one match, we're probably done
With no match, try different parts of the name (last, first)
With multiple matches, we need to match something else too.
From the name matches, compare email addresses. If they match, we're done.
At this point we probably have several candidate matches.
Look at parts of the address (country, city, address). If the first two are matches, we're pretty sure we have a match.
Look at the email if we might have a match and compare the user portion to the new email and the stored email. Do the same for the host portion. This is just to raise the confidence level a bit.
This leaves about 5% that I need to check by hand. It's the first time I've had to go through with this so I'm being very cautious. Thank god for test suites.
Now that we've made it through the first year, it's time to get people to renew. I've sent out renewal emails, but I'm guessing 10% of them will never make it into an inbox. Spam has just about ruined email for anything.
If you've already received four issues, it's time for you to renew, and you can use subscription form and subscribe again.
Either way, I have an interesting task ahead. Since I don't store any personal information on the Pair.com servers that power our website, and we never store credit card information or do recurring billing, I get to match up the renewals with existing subscribers. It's easy to know which transactions are renewals, and each email I've sent has a link with a query string that I can link back to a subscriber. However, people might not follow the link but go directly to the webiste, or all sorts of other things that don't let me see that code. We'll see how that goes.
In related matters, I was rewriting the code bits that parse the email I get so I can shove all that stuff into the database. I was doing fancy things with ergexen, then Template::Extract, and some other things, and although it was a lot of fun, it was a big waste of time. Since I really just wanted to suck it into another program, why not send it as a ready-to-use data structure? It's easy enough to freeze things and unthaw them later. I still see my nicely formatted template, but at the end I include the Perl ready data. Besides the trivial parsing to isolate that, I'm ready to important things into the database. Things can be too simple to be seen sometimes.
I finally got to meet Eric Maki, TPRs designer, in person. We went through the current issue and looked at a lot of the design things we might want to change, and we also have an idea for an upcoming project I'll have details later.
If you haven't seen the latest cover (Summer 2005), get yourself an issue. It's so nice that we're going to make posters of it.
After a long day at YAPC where I taught a 4-day Learning Perl course in a single day, and then a 5 hour boat dinner cruise, Josh McAdams of Perlcast interviewed me about The Perl Review. I'm not sure when he'll publish it.
The cool thing is that Josh is moving to Chicago. I might be able to get on Perlcast a bit more often. :)
I got an advnace copy of The Best Software Writing from Apress. It's a collection of writings compiled by Joel Spolsky. Anyone want to take a whack at reviewing this for the next issue?
The Perl Review Summer 2005 issues should be in the mail today. The printer was screwing around again. They've been a disaster. You know the sort of people: they can't just do something without sending four emails back and forth, each with some new reason why things can't happen right away.
I'm thinking about making a special edition print issue with the best articles from the first two years. Good or bad idea? How much would you pay for that sort of thing? Which articles would you want?
The next issue is scheduled for December 1, and I'm looking for book reviews. Any recent book might work, but I'm specifically interested in reviews of:
Reviews should be 200 to 300 words and should reflect your opinion. Say explicitly who should buy this book and why they should, or that nobody should buy the book. No matter what you say, be nice.
sorting a hash by keys - the $a and $b need to stay there, don't replace them they are for the sorting function.
@sortedKeys = sort { $a <=> $b } keys %hashName # numerical sort
@sortedKeys = sort { $a cmp $b } keys %hashName # ASCII-betical sort
@sortedKeys = sort { lc($a) cmp lc($b) } keys %hashName # alphabetical sort
sorting a hash by values - the $a and $b need to stay there, don't replace them they are for the sorting function.
@sortedKeys = sort { $hashName{$a} <=> $hashName{$b} } keys %hashName # numerical sort
@sortedKeys = sort { $hashName{$a} cmp $hashName{$b} } keys %hashName # ASCII-betical sort
@sortedKeys = sort { lc($hashName{$a}) cmp lc($hashName{$b}) } keys %hashName # alphabetical sort
2008 scandalz.net
Please try to limit the amount of "this room doesn't have any bazingas"
until you are told that those rooms are "punched out". Once punched
out, we have a right to complain about atrocities, missing bazingas,
and such.
-- N. Meyrowitz