Debugging CPAN problem

The Raspberry Pi based Curtain Controller project for my blind uncle is complete and almost ready to go to its new home. I have two RPi's - the one that is boxed up with the relay controller and one that is setup as a test environment so I can deal with any future problems or extra features.

In setting up the controller unit I obviously made a mess somewhere as nothing would install from CPAN whereas this was not a problem on the development unit. The development unit also has an HTTP server which is accessible via an ngrok tunnel ready for the unit at be Alexa enabled once I get Device Discovery working. This server needs adding to the controler and requires HTTP::Server::Simple so CPAN is necessary.

The development unit used an OS image with the network credentials added. This same image has now been used for the controller unit. So everything should be identical except that one has header pins soldered in and the other one doesn't!
Once the OS had installed and an SSH connection made I have changed the password then updated with:

sudo apt-get update
sudo apt-get upgrade
cpan install CPAN

With the development unit, running CPAN and typing install HTTP::Server::Simple installed lots and lots of dependencies taking nearly 4 hours but eventually installed.

On the unit that is to be shipped imminently, CPAN gets 'Killed' - this is the last block of what CPAN prints out:

Configuring E/ET/ETHER/Try-Tiny-0.30.tar.gz with Makefile.PL
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Try::Tiny
Writing MYMETA.yml and MYMETA.json
  ETHER/Try-Tiny-0.30.tar.gz
  /usr/bin/perl Makefile.PL INSTALLDIRS=site -- OK
Running make for E/ET/ETHER/Try-Tiny-0.30.tar.gz
cp lib/Try/Tiny.pm blib/lib/Try/Tiny.pm
Manifying 1 pod document
  ETHER/Try-Tiny-0.30.tar.gz
  /usr/bin/make -- OK
The current configuration of allow_installing_outdated_dists is 'ask/y
+es', but for this option we would need 'CPAN::DistnameInfo' installed
+. Please install 'CPAN::DistnameInfo' as soon as possible. As long as
+ we are not equipped with 'CPAN::DistnameInfo' this option does not t
+ake effect
Running make test for ETHER/Try-Tiny-0.30.tar.gz
PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::H
+arness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/l
+ib', 'blib/arch')" t/*.t
t/00-report-prereqs.t .......... #
# Versions for all modules listed in MYMETA.json (including optional o
+nes):
#
# === Configure Requires ===
#
#     Module              Want Have
#     ------------------- ---- ----
#     ExtUtils::MakeMaker  any 7.34
#
# === Configure Suggests ===
#
#     Module      Want    Have
#     -------- ------- -------
#     JSON::PP 2.27300 2.97001
#
# === Build Requires ===
#
#     Module              Want Have
#     ------------------- ---- ----
#     ExtUtils::MakeMaker  any 7.34
#
# === Test Requires ===
#
#     Module              Want     Have
#     ------------------- ---- --------
#     ExtUtils::MakeMaker  any     7.34
#     File::Spec           any     3.74
#     Test::More           any 1.302133
#     if                   any   0.0608
#
# === Test Recommends ===
#
#     Module         Want     Have
#     ---------- -------- --------
#     CPAN::Meta 2.120900 2.150010
#
# === Test Suggests ===
#
#     Module                    Want    Have
#     ------------------------ ----- -------
#     CPAN::Meta::Check        0.011 missing
#     CPAN::Meta::Requirements   any   2.140
#     Capture::Tiny             0.12 missing
#
# === Runtime Requires ===
#
#     Module   Want Have
#     -------- ---- ----
#     Carp      any 1.50
#     Exporter 5.57 5.73
#     constant  any 1.33
#     strict    any 1.11
#     warnings  any 1.42
#
# === Runtime Suggests ===
#
#     Module    Want    Have
#     --------- ---- -------
#     Sub::Name 0.08 missing
#     Sub::Util  any    1.50
#
# === Other Modules ===
#
#     Module           Have
#     ------------- -------
#     JSON::PP      2.97001
#     Pod::Coverage missing
#     Sub::Name     missing
#     YAML          missing
#     autodie          2.29
#
t/00-report-prereqs.t .......... ok
t/basic.t ...................... ok
t/context.t .................... ok
t/erroneous_usage.t ............ ok
t/finally.t .................... ok
t/given_when.t ................. skipped: Tests skipped on perl 5.27.7
++, pending resolution of smartmatch changes
t/global_destruction_forked.t .. ok
t/global_destruction_load.t .... skipped: Capture::Tiny 0.12 required
t/named.t ...................... ok
t/when.t ....................... skipped: Tests skipped on perl 5.27.7
++, pending resolution of smartmatch changes
t/zzz-check-breaks.t ........... ok
All tests successful.
Files=11, Tests=97, 21 wallclock secs ( 0.78 usr  0.10 sys + 17.95 cus
+r  1.02 csys = 19.85 CPU)
Result: PASS
Killed
pi@eric:~ $

I've tried using CPAN to install just Try::Tiny as this seems to be where things are failing. When I do this I get almost the same output with different wallclock values reported and Lockfile removed. displayed on the line before Killed and several minutes between them.

Searching for an answer suggests that this is the OS killing CPAN and the most probable cause is lack of memory. The development unit has a 32Gb SD card whereas the controller has 16Gb. But, there is plenty of space on the card:

pi@eric:~ $ df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root       14989480 3552696  10774908  25% /
devtmpfs          188088       0    188088   0% /dev
tmpfs             221112       0    221112   0% /dev/shm
tmpfs             221112    3212    217900   2% /run
tmpfs               5120       0      5120   0% /run/lock
tmpfs             221112       0    221112   0% /sys/fs/cgroup
/dev/mmcblk0p1    258095   55052    203043  22% /boot
tmpfs              44220       4     44216   1% /run/user/1000

Two ostensibly identical units seem to be working differently.
Can you suggest anything I can try to debug this problem?

[davies]

++ to those who recommended cpanm - it's one of the first things I install on a Pi. SD cards have a limited life - about 1K writes before things start to fail (they can usually handle small failing areas). All my production Pis therefore have a second SD card on a USB adapter. The following code is on them all & run whenever I make a material change. This means that I can swap the card immediately whenever one dies.

#!/usr/bin/perl
use strict;
use warnings;
use feature 'say';
use constant DEBUG => 0;

my %tree;
my @tree = split(/\n/, `lsblk -bnlo NAME,SIZE,MOUNTPOINT`);
for (@tree) {
    my ($key, $size, $mount) = (split(/\s+/, $_));
    $key =~ s/\d.*//;
    if ($key =~ m/^sd*/ or $key eq 'mmcblk') {
        $tree{$key}{'mount'} = $mount;
        $tree{$key}{'size'}  = $size unless defined $tree{$key}{'size'
+};
    }
}
my $target;
use Data::Dumper; print Dumper %tree if DEBUG;
for (keys %tree) {
    if ('' eq $tree{$_}{'mount'}) {
        $target = $_;
        last;
    }
}
die 'No target found!' unless $target;
my $cmd = "sudo dd if=/dev/mmcblk0 of=/dev/$target bs=1M";
say $cmd if DEBUG;
if ($tree{$target}{'size'} >= $tree{'mmcblk'}{'size'}) {
    say 'Backing up - target large enough.';
} else {
    die "Can't back up with source larger than target.";
}
exec $cmd unless DEBUG;

=pod

=head1 NAME AND SYNOPSIS

    sdbackup.pl

=head1 DESCRIPTION

This is a standalone programme to be invoked manually. It is
written for a Raspberry Pi with a second SD card attached via
a USB adaptor. On the original target, dns, there is a thumb
drive mounted for logging. The code is therefore designed to
look for an unmounted device on one of the "sd" interfaces. If
such a device is found, the main SD card will be copied to it
unless the target is smaller than the source. The backup will
take roughly 1.2 minutes per meg. It will back up to the
first unmounted attached block device that appears in the
hash created from the 'lsblk' command. If there are two or
more unmounted devices, the target may not be predictable.

=head2 Constants

The only constant is DEBUG. If this is set to a value Perl
treats as true, no backup will be performed and a hash will
dumped before the command candidate is printed. Otherwise the
backup will be started unless the target is smaller than the
source.

=head2 Constraints

The code relies on the behaviour of 'lsblk' and in particular
its property of listing the block device before the partitions.
It also relies on the name of the mounted SD card starting
with 'mmcblk' followed by a digit. If there is more than one
such device, behaviour may be unpredictable.

=head2 lsblk parameters

=over

* b - Show device sizes in bytes.

* n - No headings

* l - Use list format

* o - Output the columns in the list that follows.

=back

=cut

Regards,

John Davies

[stevieb]

"SD cards have a limited life - about 1K writes before things start to fail"

Holy crap! 1k writes only? What kind of cards are you buying? I have several Pis (eight, 10 if I include my zeroWs), a couple of them that have been running for several years (literally) doing all manner of intense disk activity constantly, and I've only ever had two cards fail on me over that span across all of my Pis, and one was a direct result of accidentally applying 48v DC to the power pin while I had it connected to an SD reader while I was writing software for a micro controller.

I use mainly Kingston cards.

[davies]

My father always used to ask, "Is that something you know or something you've been told?". This is what I have been told. SD cards have logic that spreads the writes around so that no one point gets lots of hits. If you do a complete overwrite - which is what my code does - 1K times, you are reaching the end of the useful life of the device. When I put this to a salesman of SSDs, he did not deny the basic arguments but said that the power consumption more than made up for the shorter life (than spinning rust) and that in most cases, data were written far fewer times than read and that the read speed was a further advantage and and and. But he didn't deny the effective write limit, although I have no hard evidence for the 1K figure I had been told. I'll bet that, over time, the number of writes will increase. The cards I have that have failed most are Verbatim, but I have other reasons for avoiding anything of that brand in future.

Regards,

John Davies

[stevieb]

It may be much easier to avoid troubleshooting. A quick way to get it back up and running is create an image of the good SD card, and restore that image to the bad one:

Put SD card from working unit into computer, and:

dd if=/dev/sdX conv=sync bs=4M of=disk.img

When done, put the bad SD card into the computer, and:

dd if=disk.img conv=sync bs=4M of=/dev/sdX

Where X is the letter of the inserted SD card (check with dmesg or the like). Double, triple and quadruple check that you've got the right disk, or you could very easily and quickly hose your system.

If you've got the capability of connecting two SD cards at once:

dd if=/dev/sdGOOD conv=sync bs=4M of=/dev/sdBAD

[Bod]

It may be much easier to avoid troubleshooting. A quick way to get it back up and running is create an image of the good SD card, and restore that image to the bad one

That is probably the best way. I will need to sort out hostnames etc but that is pretty trivial.

Double, triple and quadruple check that you've got the right disk, or you could very easily and quickly hose your system

Everything I have written, including the crontab entries are backed up safely - but yes, I shall take great care with which disk is which :)

I don't have a Linux box to do this on but Win32 Disk Imager should do the trick just as well thanks.

[marto]

Are you confusing memory and storage space? You don't say which models pi are involved. Did you forget about this?

[Bod]

Are you confusing memory and storage space?

Yes - I mean disk storage space rather than RAM.

You don't say which models pi are involved

Raspberry Pi Zero with just 512Mb RAM - but both units have the same amount of RAM.

Did you forget about this?

I did look at cpanm but it isn't installed, so I have not been using it. I have today tried installing it with sudo apt-get install cpanm which didn't work and with cpan APP::cpanminus which was 'Killed'.

What needs doing to install and use cpanm?

[marto]

Ignoring the fact that the first thing you did after running cpan was to install cpan, go back and read what I wrote :) If you run out of RAM expect daft things to happen. Use the OS package management to install things, you don't need to everything yourself. sudo apt install libhttp-server-simple-perl You're wasting time compiling on a low end/embedded system. apt isn't cpan, so don't expect packages to be the same as they do on cpan, which is probably why your apt-get install cpanm "didn't work" (this isn't a report anyone can help you with constructively :), sudo apt install cpanminus or something similar. Also The documentation often has hints and tips for installation.

Update: fixed autocorrect based typo.

[ikegami]

If a program crashes, it's SIGSEGV that kills it. This results in a different message.

$ perl -e'kill SEGV => $$'
Segmentation fault

Killed is received from SIGKILL.

$ perl -e'kill KILL => $$'
Killed

I've gotten this when I used too many resources on a shared system (e.g. too much memory or CPU time). Check your ulimits and with your administrator. (I have limits even though ulimit doesn't list any.)

---

Update: oh, I see you did mention something along those lines. But...

Searching for an answer suggests that this is the OS killing CPAN and the most probable cause is lack of memory. The development unit has a 32Gb SD card

RAM, not disk space.

[ikegami]

If it is using too much CPU in a short time, you might have luck lowering the priority of the process. nice cpan ....

[Bod]

RAM, not disk space

Both the production and the development machines have the same amount of RAM - 512Mb
One installs fine from CPAN and the other one doesn't...

It will probably always remain a bit of a mystery as I installed HTTP::Server::Simple using the information marto helpfully shared. The production unit is all boxed up ready to be connected to the curtains it will control once the immediate festive activity has passed.

Regardless - knowing that Killed is received from SIGKILL was very helpful, thanks.

[ikegami]

How much RAM a machine has is not particularly relevant. Other factors can limit a process's memory. I specifically mentioned you should be looking into limits imposed by the system as these have been known to send SIGKILL to handle processes exceeding the limits. Re: Debugging CPAN problem provides a concrete example.

[stevieb]

You should have sent the unit *before* the festivities wrapped up... that way you could say that you were responsible for the automation of the "closing of the curtains" of this dumpster fire that has been the year 2020 :D

[Anonymous Monk]

On the unit that is to be shipped imminently, CPAN gets 'Killed' - this is the last block of what CPAN prints out:

Killed is printed by the shell after its child process receives a SIGKILL. This looks like it could be the oom-killer's doing. Take a look at dmesg output after the process is killed - you'll see if the crash is due to having ran out of RAM.

[perlfan]

>almost ready to go to its new home

very cool - I was going to say might be getting OOM'd, too. Or did say. xD

[Bod]

very cool

There will be a blog post with the details to help others following a similar path...