using perl to pull values and files out of virtual spaces
Hello Monks,
I've been using the testing harness of perl to run a download and watermark script, and I've had quite a bit of success with it in testing. As I enter the monastery, I'm getting some false results that I want to explore using perl.
I got back to the literature and used Module::Starter, and now have something that could count as a distro. It's passed 12 rounds of testing, but I've made a big mistake because I backed off of stat'ing the file as an ultimate test of a success. Let me just start with perl as I present Frobnitz.pm:
package Acme::Frobnitz;
use strict;
use warnings;
use IPC::System::Simple qw(capturex);
use Cwd qw(abs_path);
use File::Spec;
use File::stat;
use File::Basename;
use FindBin;
use POSIX qw(strftime);
our $VERSION = '0.03';
That stack of modules just makes me happy to look at, like an xmas tree. And already version 3! IPC::System::Simple is a good choice for a grown-up's version of system() according to haukex's canonical treatment.
The methods wrap calls to bash:
sub new {
my ($class) = @_;
return bless {}, $class;
}
sub _get_script_path {
my ($class, $script_name) = @_; # Resolve base directory dynamical
+ly
my $base_dir = abs_path("$FindBin::Bin/.."); # One level up from b
+in
my $script_path = File::Spec->catfile($base_dir, 'bin', $script_na
+me);
unless (-x $script_path) {
die "Script $script_path does not exist or is not executable.\
+n";
}
return $script_path;
}
sub download {
my ($class, $hyperlink) = @_;
die "No hyperlink provided.\n" unless $hyperlink;
my $script_path = $class->_get_script_path("call_download.sh");
my $output;
eval {
$output = capturex("bash", $script_path, $hyperlink);
};
if ($@) {
die "Error executing $script_path with hyperlink $hyperlink: $
+@\n";
}
return $output;
}
I've been jammed up with version control, because I've been unable to reconstitute my gitlab circumstances (totally my fault). I take snapshots with
tar -czvf Acme-Frobnitz-Snapshot.tar.gz Acme-Frobnitz-Snapshot
They mostly look the same, but I port them back and forth from my mini-mac to a linux machine, with an obligatory-named usb as download target. That happens in the bash part. There's also the docker part and a python part. Much of this source deals with calling one from the other and returning values as well as files. I've wanted perl to "pull the cart" for the whole process, so that I could subject it to testing. LanX rightly laments the dearth of this in contemporaneous culture.
I haven't revealed my screw-up yet...this isn't easy...
I thought I passed the 12th round of testing but later came to realize I was firing blanks as to whether files ended up on the usb drive, and it brings me to a weird moment in testing, where I can use perl to figure out why...
Here's the new perl caller...this is what is "pulling the cart," whatever that means...
use strict;
use warnings;
use Acme::Frobnitz;
my $frobnitz = Acme::Frobnitz->new();
# URL of the video to download
my $video_url = "https://www.instagram.com/p/DDa_FxsNtTe/";
# Step 1: Download the video
my $downloaded_file = $frobnitz->download($video_url);
print "Downloaded video to: $downloaded_file\n";
# Step 2: Verify the downloaded file
print "Verifying downloaded file...\n";
if ($frobnitz->verify_file($downloaded_file)) {
print "File verification successful.\n";
} else {
die "File verification failed. Aborting further processing.\n";
}
# Step 3: Add watermark to the downloaded video
my $final_file = $frobnitz->add_watermark($downloaded_file);
print "Final watermarked video is at: $final_file\n";
I want to make a pure perl test here and use perl to figure out my privileges, and I'm using this. You should note that this is untested software from chatgpt today:
sub verify_file {
my ($class, $file_path) = @_;
die "File path not provided.\n" unless $file_path;
my $abs_path = abs_path($file_path) // $file_path;
if (-e $abs_path) {
print "File exists: $abs_path\n";
# File size
my $size = -s $abs_path;
print "File size: $size bytes\n";
# File permissions
my $permissions = sprintf "%04o", (stat($abs_path)->mode & 077
+77);
print "File permissions: $permissions\n";
# Last modified time
my $mtime = stat($abs_path)->mtime;
print "Last modified: ", strftime("%Y-%m-%d %H:%M:%S", localti
+me($mtime)), "\n";
# Owner and group
my $uid = stat($abs_path)->uid;
my $gid = stat($abs_path)->gid;
print "Owner UID: $uid, Group GID: $gid\n";
return 1; # Verification success
} else {
print "File does not exist: $abs_path\n";
my $dir = dirname($abs_path);
# Report directory details
print "Inspecting directory: $dir\n";
opendir my $dh, $dir or die "Cannot open directory $dir: $!\n"
+;
my @files = readdir $dh;
closedir $dh;
print "Directory contents:\n";
foreach my $file (@files) {
next if $file =~ /^\.\.?\$/; # Skip . and ..
my $file_abs = File::Spec->catfile($dir, $file);
my $type = -d $file_abs ? 'DIR ' : 'FILE';
my $size = -s $file_abs // 'N/A';
print "$type - $file (Size: $size bytes)\n";
}
return 0; # Verification failed
}
}
Adding a new method:
Ok, let's use the toolchain.
(base) fritz@laptop:~/Desktop/Acme-Frobnitz-Snapshot$ ls
Acme-Frobnitz-Snapshot.tar.gz Dockerfile Makefile MYMETA.yml
+ xt
bin ignore.txt Makefile.PL README
Changes lib MANIFEST requirements.t
+xt
conf logs MYMETA.json t
(base) fritz@laptop:~/Desktop/Acme-Frobnitz-Snapshot$ perl Makefile.PL
WARNING: Setting ABSTRACT via file 'lib/Acme/Frobnitz.pm' failed
at /usr/share/perl/5.30/ExtUtils/MakeMaker.pm line 754.
...Q1) Why do I get this warning?
Generating a Unix-style Makefile
Writing Makefile for Acme::Frobnitz
Writing MYMETA.yml and MYMETA.json
(base) fritz@laptop:~/Desktop/Acme-Frobnitz-Snapshot$ make
cp lib/Acme/._Frobnitz.pm.bak blib/lib/Acme/._Frobnitz.pm.bak
cp lib/python_utils/watermarker2.py blib/lib/python_utils/watermarker2
+.py
cp lib/python_utils/__pycache__/watermarker2.cpython-39.pyc blib/lib/p
+ython_utils/__pycache__/watermarker2.cpython-39.pyc
cp lib/python_utils/__pycache__/watermarker2.cpython-313.pyc blib/lib/
+python_utils/__pycache__/watermarker2.cpython-313.pyc
...^^^I see all these little barnacle files now...
...in 3 lines we cross into python logging...
t/00-load.t ....... 1/? # Testing Acme::Frobnitz 0.03, Perl 5.030000,
+/usr/bin/perl
t/00-load.t ....... ok
t/13.download.t ... 1/? 2024-12-17 06:28:26,320 - __main__ - INFO - At
+tempting to create directory: E4B0-3FC22024-12-17/
2024-12-17 06:28:26,320 - __main__ - INFO - Directory created or alrea
+dy exists: E4B0-3FC22024-12-17/
2024-12-17 06:28:26,321 - __main__ - INFO - Entering function: mask_me
+tadata
2024-12-17 06:28:26,321 - downloader5 - INFO - Masking metadata
2024-12-17 06:28:26,321 - downloader5 - INFO - Received parameters for
+ metadata extraction:
2024-12-17 06:28:26,322 - downloader5 - INFO - download_path: E4B0-3FC
+22024-12-17/
...oh damn, that's the problem right there........................^^^^
+^^^^^^
The python part
Good heck. The progress I make with this is a function of effective logging. Continuing on STDOUT:
2024-12-17 06:28:36,493 - __main__ - INFO - Entering function: create_
+original_filename
2024-12-17 06:28:36,494 - downloader5 - INFO - Generated original file
+name: E4B0-3FC22024-12-17/Rick_Astley_20091025.webm
2024-12-17 06:28:36,494 - __main__ - INFO - Entering function: downloa
+d_video
2024-12-17 06:28:36,494 - downloader5 - INFO - Received parameters: do
+wnload_video:
2024-12-17 06:28:36,494 - downloader5 - INFO - download_path: E4B0-3FC
+22024-12-17/
... again oop
+s^^^ .....
2024-12-17 06:28:36,496 - downloader5 - INFO - video_title: Rick_Astle
+y_-_Never_Gonna_Give_You_Up_(Official_Music_Video)
2024-12-17 06:28:36,496 - downloader5 - INFO - video_date: 20091025
...but here's the false positive:
2024-12-17 06:29:22,739 - downloader5 - INFO - Video download complete
+d.
2024-12-17 06:29:22,742 - downloader5 - INFO - Download completed in 4
+6.24 seconds
...this part is one of my favorites:
2024-12-17 06:29:22,742 - __main__ - INFO - Entering function: store_p
+arams_as_json
2024-12-17 06:29:22,743 - utilities1 - INFO - Params saved to JSON fil
+e: E4B0-3FC22024-12-17/Rick_Astley_20091025.json
2024-12-17 06:29:22,743 - __main__ - INFO - Returning original filenam
+e: E4B0-3FC22024-12-17/Rick_Astley_20091025.webm
t/13.download.t ... ok
...should have failed...
Result: PASS
(base) fritz@laptop:~/Desktop/Acme-Frobnitz-Snapshot$ sudo make instal
+l
[sudo] password for fritz:
Installing /usr/local/share/perl/5.30.0/Acme/Frobnitz.pm
...
The Debugger
Let's fire up the debugger, woohoo! This is why I can never leave perl:
(base) fritz@laptop:~/Desktop/Acme-Frobnitz-Snapshot$ perl -d bin/3.d
+river.pl
Loading DB routines from perl5db.pl version 1.55
Editor support available.
Enter h or 'h h' for help, or 'man perldebug' for more help.
main::(bin/3.driver.pl:5): my $frobnitz = Acme::Frobnitz->new();
DB<1>
Scout it out once with c, then set a break:
DB<1> R
+
Warning: some settings and command-line options may be lost!
Loading DB routines from perl5db.pl version 1.55
Editor support available.
Enter h or 'h h' for help, or 'man perldebug' for more help.
main::(bin/3.driver.pl:5): my $frobnitz = Acme::Frobnitz->new();
DB<0> b 16
+
DB<1>
Get in there...
2024-12-17 07:10:09,829 - downloader5 - INFO - Video download complete
+d.
2024-12-17 07:10:09,832 - downloader5 - INFO - Download completed in 9
+.76 seconds
2024-12-17 07:10:09,832 - __main__ - INFO - Entering function: store_p
+arams_as_json
...
Verifying downloaded file...
main::(bin/3.driver.pl:16): if ($frobnitz->verify_file($downloaded_
+file)) {
DB<1> s
+
Acme::Frobnitz::verify_file(/usr/local/share/perl/5.30.0/Acme/Frobnitz
+.pm:65):
65: my ($class, $file_path) = @_;
DB<1> v
+
62 }
63
64 sub verify_file {
65==> my ($class, $file_path) = @_;
66: die "File path not provided.\n" unless $file_path;
67
68: my $abs_path = abs_path($file_path) // $file_path;
69
70: if (-e $abs_path) {
71: print "File exists: $abs_path\n";
DB<1> s
+
Acme::Frobnitz::verify_file(/usr/local/share/perl/5.30.0/Acme/Frobnitz
+.pm:66):
66: die "File path not provided.\n" unless $file_path;
DB<1> s
+
Acme::Frobnitz::verify_file(/usr/local/share/perl/5.30.0/Acme/Frobnitz
+.pm:68):
68: my $abs_path = abs_path($file_path) // $file_path;
DB<1> s
+
Acme::Frobnitz::verify_file(/usr/local/share/perl/5.30.0/Acme/Frobnitz
+.pm:70):
70: if (-e $abs_path) {
DB<1> s
+
Unsuccessful stat on filename containing newline at /usr/local/share/p
+erl/5.30.0/Acme/Frobnitz.pm line 70.
at /usr/local/share/perl/5.30.0/Acme/Frobnitz.pm line 70.
Acme::Frobnitz::verify_file(Acme::Frobnitz=HASH(0x5653a1900ad8), "
+2024-12-17 00:09:54 - Detected OS: Linux\x{a}2024-12-17 00:09:54 "...
+) called at bin/3.driver.pl line 16
Acme::Frobnitz::verify_file(/usr/local/share/perl/5.30.0/Acme/Frobnitz
+.pm:92):
92: print "File does not exist: $abs_path\n";
DB<1> p $abs_path
+
2024-12-17 00:09:54 - Detected OS: Linux
2024-12-17 00:09:54 - Ensuring Docker is running...
2024-12-17 00:09:54 - Docker is already running.
2024-12-17 00:09:54 - Loading configuration from /home/fritz/Desktop/A
+cme-Frobnitz-Snapshot/bin/../conf/app_config.json...
2024-12-17 00:09:54 - Target USB mount name from config: E4B0-3FC2
2024-12-17 00:09:54 - Detected USB mount point: /media/fritz/E4B0-3FC2
2024-12-17 00:09:54 - Attempting to create directory: /media/fritz/E4B
+0-3FC2/2024-12-17/
2024-12-17 00:09:54 - Directory created or already exists: /media/frit
+z/E4B0-3FC2/2024-12-17/
2024-12-17 00:09:54 - Docker image 'my_dl:latest' found.
2024-12-17 00:09:54 - Running Docker with the specified volume and con
+fig path...
2024-12-17 00:10:13 - File validation successful: [Instagram] Extracti
+ng URL: https://www.instagram.com/p/DDa_FxsNtTe/
Adding lib_path to sys.path: /app/bin/../lib/python_utils
[Instagram] DDa_FxsNtTe: Setting up session
[Instagram] DDa_FxsNtTe: Downloading JSON metadata
[Instagram] Extracting URL: https://www.instagram.com/p/DDa_FxsNtTe/
[Instagram] DDa_FxsNtTe: Setting up session
[Instagram] DDa_FxsNtTe: Downloading JSON metadata
[info] DDa_FxsNtTe: Downloading 1 format(s): dash-560108753314303v+das
+h-1118756702933936ad
[download] Destination: E4B0-3FC2/2024-12-17/Tim_Ballard_20241211.fdas
+h-560108753314303v.mp4
[download] 100% of 37.69MiB in 00:00:07 at 5.24MiB/s:00
[download] Destination: E4B0-3FC2/2024-12-17/Tim_Ballard_20241211.fdas
+h-1118756702933936ad.m4a
[download] 100% of 1.14MiB in 00:00:00 at 2.32MiB/s:00
[Merger] Merging formats into "E4B0-3FC2/2024-12-17/Tim_Ballard_202412
+11.mp4"
Deleting original file E4B0-3FC2/2024-12-17/Tim_Ballard_20241211.fdash
+-1118756702933936ad.m4a (pass -k to keep)
Deleting original file E4B0-3FC2/2024-12-17/Tim_Ballard_20241211.fdash
+-560108753314303v.mp4 (pass -k to keep)
E4B0-3FC2/2024-12-17/Tim_Ballard_20241211.mp4
DB<2>
Q3) What the damn heck is that?
Alright, well I found enough things wrong in there to need to go back and fix. It's funny how you find stuff during the writeup. Without a githost I don't have a means to provide an SSCCE without violating the first S.
For creating the virtual environment with some diagnostics, I've been using this Dockerfile, which has a nice amount of tread on it:
# Start with a base image
FROM python:3.10-slim
# Set PYTHONPATH to include the python_utils directory
ENV PYTHONPATH="/app/lib/python_utils:/app/lib"
# Update apt repository and install necessary dependencies, including
+ffmpeg
RUN apt-get update && \
apt-get install -y --no-install-recommends \
ffmpeg \
python3 \
python3-pip && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
# Set the working directory
WORKDIR /app
# Copy all files into the Docker image
COPY bin /app/bin
COPY lib /app/lib
COPY conf /app/conf
COPY requirements.txt /app/requirements.txt
COPY Dockerfile /app/Dockerfile
# Install Python dependencies
RUN pip3 install --upgrade pip && pip3 install -r requirements.txt --r
+oot-user-action=ignore
# Debug: List all files in /app and verify the content of lib/python_u
+tils
RUN ls -la /app && ls -la /app/lib && ls -la /app/lib/python_utils
# Debug: Verify requirements.txt is in the container
RUN cat /app/requirements.txt
# Debug: Verify Python environment and MoviePy installation
RUN python3 -c "import sys; print(sys.version)"
RUN python3 -c "import site; print(site.getsitepackages())"
RUN python3 -c "import moviepy; print('MoviePy is correctly installed'
+)"
The image is minimal. Execution is done from within bash:
original_filename=$(docker run --rm \
-e PYTHONPATH="/app/lib/python_utils:/app/lib" \
-v "$Config_Path":/app/conf/app_config.json \
-v "$usb_mount_point":"$usb_mount_point" \
my_dl python3 /app/bin/call_download.py "$1")
Docker isn't the easiest thing to get values out of, and this worked. Glad for any critical comments about the code. I gotta figure out how I threw a monkey wrench into what used to work, and I don't find that wrench without gdb. I'm not aware of a python equivalent.
May your solstice be filled with perl
