Organising the Archive

Lia — Mon, 15 Aug 2022 01:21:04 GMT

That works :D
Made a few tweaks that seemed to take fine.

Changed the button text from to else it generates green link text because of the css which could be hard to read on the blue button. Tried to override it with some style but gave up and made it span instead :) Added after the so it creates a little space between itself and the voting bit. Minor grammatical update to the missing post bit.

Full modified script below:

Spoiler

#!/usr/bin/perl =head1 NAME forum-archive - Put a google chached community.onewheel.com thread back together =cut use IO::Handle; use HTTP::Request; use LWP::UserAgent; use File::Copy; use File::Path; use File::Find; use Pod::Usage; use POSIX; my($HEADER, @POSTS, $FOOTER, $COUNT, $WEIRD); my(%META)=( 'base' => 'https://archive.owforum.co.uk/', 'logo' => 'http://archive.owforum.co.uk/Images/OWForumArchive.png', 'logo_ht' => '60', 'icon' => 'https://archive.owforum.co.uk/assets/resources/OWForumArchiveIcon.png', 'profiles' => '../../../assets/uploads/profile', 'resources' => '../../../assets/resources', 'system' => '../../../assets/uploads/system', ); my(%RESOURCES)=( 'fonts' => 'https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/all.min.css', 'style' => 'https://owforum.co.uk/assets/client-darkly.css', 'broken' => 'https://icon-icons.com/downloadimage.php?id=5390&root=39/PNG/128/&file=brokenfile_5952.png', ); my(%TOPICS)=(); my(%SEARCH_PATH)=(); =head1 SYNOPSIS forum_archive [directory2] [directory3] [...] =head1 DESCRIPTION This script takes a set of files downloaded from the google page cache for the community.onewheel.com NodeBB forum and tries to put it back together. =head2 Content Management A small number of resources are available from the internet. B will download these assets, if needed, for inclusion to the archive file structure. If assets have been downloaded recently (within the last day) then new downloads are not attempted. This should keep B from slamming remote resources during testing phases. =cut sub wget { my($src, $asset, $dst)=@_; my($file)=IO::Handle->new; my($req)=HTTP::Request->new( 'GET' => $src ); my($get)=LWP::UserAgent->new; my($response); $asset=~s|^[/.]+||; if(!-s "$asset/$dst" || -M "$asset/$dst" > 1 ) { $response=$get->request($req); if($response->is_success) { File::Path::make_path($asset, { 'chmod' => 0755 }); open($file, '>', "$asset/$dst"); print $file $response->decoded_content; close($file); } } } =pod B will dynamically create the C and C directory structures as needed to store content found within the post. In an effort to increase efficiency for commonly used content, such as avatars, actual copying of these files will not occur each time such content is seen, after its initial copy. =cut sub location { my($file)=@_; my($try); if(-r $file) { return($file); } else { foreach my $dir (keys(%SEARCH_PATH)) { $try=join('/', $dir, $file); return($try) if(-r $try); } } &wget($RESOURCES{'broken'}, $META{'resources'}, 'broken-file.png'); return(join('/', $META{'resources'}, 'broken-file.png')); } sub copy { my($src, $asset, $dst)=@_; $asset=~s|^[/.]+||; if(!-s "$asset/$dst" || -M "$asset/$dst" > 1 ) { File::Path::make_path($asset, { 'chmod' => 0755 }); File::Copy::copy(&location($src), "$asset/$dst"); } } =pod Avatar image are stored in a central location, shared by the entire archive. This lowers the space requirements of the archive and increases page load times and browser cache efficiency. =cut sub avatar { my($img)=@_; my($dst)=$img; $dst=~s|^.*/||; ©($img, $META{'profiles'}, $dst); return(join('/', $META{'profiles'}, $dst)); } =pod Uploaded images which are stored in the archive may be named slightly differently on the archive than on the original. NodeBB has gone through a couple iterations about how to handle this conflict, and B tries to handle this by using the more unique C tag element parameter name. When that doesn't work, the original name is kept. Images are also grouped by post, to avoid naming conflicts between different posts. Additionally, if an image is referenced in a post, but is not contained in the archive, a standard broken file image is substituded. =cut sub upload { my($src, $alt)=@_; my($new)=$src; $new=~s|^.*/||; if($alt=~m/\.\w+$/) { $new=join('_', $META{'postid'}, $alt); } else { $new=join('_', $META{'postid'}, $new); } ©($src, $META{'path'}, $new); return(sprintf(' }.$end); } =pod One of the differences which makes the archive different is, unfortunately, that some posts are missing. When this occurs, B inserts a break in the timeline with a note about the message IDs which are absent. =cut sub missing_post { my($id)=@_; return(qq{

Post(s) $id are missing from the archive :(
Know where these posts are? Visit the new forum for how to help get them added :)

}); } =pod Although most of this script works to remove unnecessary content from archive posts, one little thing is added. A button is added to help people copy a posts permalink to their clipboard to facilitiate sharing. =cut sub share_fn { return(qq{ }); } sub share_btn { return(qq{ }); } =pod An interactive, HTML5 based NodeBB forum requires a lot of javascript to work. Since the archive is a static copy of that data, all of the javascript is removed, and the archive works nearly identically on all platforms. =cut sub global { s|https?://community.onewheel.com/|$META{'base'}|sg; s|||sg; s|||sg; s|||sg; s|\s+

||s; s|\s+||m; s|class="posts"|class="posts timeline"|mg; s|\n\n

\n||sg; if(m|(\d+)|m) { $COUNT=$1; } } =pod B assumes that all the headers from all the source files are identical, and uses the first one it finds. With that content, the new banner is inserted, interactive metadata and buttons are removed, and the new style is setup. B also collects important information like the page path and total message count. =cut sub header { local($_)=@_; #Cleanup to a reasonable starting header only s/(

\s+).*$/$1/s; s/().*$/$1/m; #Grab some important info if(m||) { $META{'url'}=$1; $META{'path'}=$2; } #reset links #strip out unneeded content s|(|$1">|mg; s|\s+||sg; s|\s+||sg; s|\s+||sg; s|\s+||sg; s|\s+||sg; s|\s+||sg; s|\s+||sg; s|\s+||sg; s|\s+||sg; s|\s+||sg; s|(|$1$META{'icon'}">|mg; &wget($RESOURCES{'style'}, $META{'resources'}, 'client-darkly.css'); s||\n\t|s; if(m|forum-logo" src="(.*?/site-logo.png)"|m) { ©($1, $META{'system'}, 'site-logo.png'); s|forum-logo" src=".*?"|forum-logo" src="$META{'system'}/site-logo.png"|mg; } s|(

|$1 style="padding-top: 50px;">|m; s|\s+
||s; #Insert new banner s|(
).?.?
|&banner($1, $2, '')|se; #Remove unnecessary buttons s|\s+||mg; s|\s+
||s; s|
[^RL]+Register.?
||gs; s|
[^RL]+Login.?
||gs; s|\s||s; s|
.?
||s; s|Loading More Posts ||mg; s|class="slideout-panel" style=".?"|class="slideout-panel"|m; #s|*

The Archive

Organising the Archive