Site Map - skip to main content - dyslexic font - mobile - text - print

Hacker Public Radio

Your ideas, projects, opinions - podcasted.

New episodes Monday through Friday.

In-Depth Series

Bash Scripting

This is an open series in which Hacker Public Radio Listeners can share their Bash scripting knowledge and experience with the community. General programming topics and Bash commands are explored along with some tutorials for the complete novice.

Are bash local variables local? - clacke | 2019-05-07

https://en.wikipedia.org/wiki/Scope_%28computer_science%29

In hpr2739, Dave talked briefly about local variables. But what are they?

In most modern languages, especially in compiled languages, "local" means that the value of a variable cannot be directly known, by looking up the name, outside the bounds of that function, but that’s not how it works in bash.

Languages like C and Python have lexical scope. Lexical scope means local variables are local in the text. The names are local.

If I’m writing code that is textually located outside the function, I cannot even describe how to access the variables within the function, because myvariable in my function is not the same variable, not the same place, as myvariable in your function.

Languages like Bash and Elisp have dynamic scope. That means local variables are local in time. The names are global.

What happens when you declare a variable local in bash is that the existing value of that variable is stowed away, to be brought back when your function exits.

#!/usr/bin/env bash
function sayscope() {
    echo The scope is $whatsmyscope
}

function globalscope() {
    whatsmyscope=global
}

function dynamicscope() {
    whatsmyscope=dynamic
}

function localscope() {
    local whatsmyscope=local
    sayscope
    dynamicscope
    sayscope
}

globalscope
sayscope
localscope
sayscope
The scope is global
The scope is local
The scope is dynamic
The scope is global

Perl has both, and it calls them local (dynamic scope, like bash) and my (lexical scope):

#!/usr/bin/env perl
use v5.10;

sub sayscope {
    say "Dynamic scope is $whatsmyscope";
}

sub globalscope {
    $whatsmyscope="global";
}

sub dynamicscope {
    $whatsmyscope="dynamic";
}

sub lexicalscope {
    my $whatsmyscope="lexical";
    say "Lexical scope is $whatsmyscope";
    sayscope;
}

sub localscope {
    local $whatsmyscope="local";
    sayscope;
    dynamicscope;
    sayscope;
    lexicalscope;
}

globalscope;
sayscope;
localscope;
sayscope;
Dynamic scope is global
Dynamic scope is local
Dynamic scope is dynamic
Lexical scope is lexical
Dynamic scope is dynamic
Dynamic scope is global

You almost never want to use local in Perl, it’s mostly there for historical reasons — lexical scope is a Perl 5 feature. https://perl.plover.com/local.html covers well the remaining few and narrow exceptions where local might be useful.

As dynamic scope has some valid use, it’s available in some otherwise lexically scoped languages. For example, Common LISP has the special form, and several Schemes and Racket have parameter objects:

To dig fully into the history and flora of dynamic and lexical scope merits another episode.

bash coproc: the future (2009) is here - clacke | 2019-04-17

If you want the full manuscript, that’s at gitlab: hpr2793_bash_coproc_manuscript.adoc. It’s almost a transcript, but I added spontaneous commentary while reading the examples, so that’s not in the manuscript.

Episode errata:

  • Command substitution with $() is perfectly valid according to POSIX, and is accepted both by dash and by bash --posix. It’s not to be considered a bashism.

  • I fumbled the pronunciation of the printf format string in one place and said "parenthesis" instead of "percentage sign".

  • I tried to say "space" every time there’s a space, but I know I forgot it in a few places. But you probably need to look at the show notes to really make sense of the commands anyway.

Example #1:

More on command substitution in Dave’s hpr1903: Some further Bash tips.

Example #2:

You can also combine process substitution with redirection.

Example #3:

More on process substitution in Dave’s hpr2045: Some other Bash tips.

For a description of a hack for creating bidirectional anonymous pipes in bash, see my Fediverse post on this, and I owe you a show.

A coprocess in bash is a subshell to which you have access to two file descriptors: Its stdin and its stdout.

The two file descriptors will be put in a bash array. To learn more about arrays, check out Dave’s series within the bash series, a whopping five-part quadrology including hpr2709, hpr2719, hpr2729, hpr2739 and hpr2756.

You create a coprocess using the coproc keyword, brand spanking new since bash 4 from 2009. I am filing issues to pygments and GNU src-highlite to support it.

There are two ways to call coproc. The first way is to give coproc a simple command.

Example #4:

The other way is to give coproc an explicit name and a Command Grouping.

Example #5:

Slightly less contrived example #6:

$ coproc GREP (grep --line-buffered pub); printf '%s\n' hacker public radio >&${GREP[1]}; cat <&${GREP[0]}
[1] 25627
public
^C
$ kill %1
[1]+  Terminated              coproc GREP ( grep --color=auto --line-buffered pub )

Here grep and cat wait forever for more input, so we have to kill them to continue our lesson.

But we know that GREP will only return one line, so we can just read that one line. And when we are done feeding it lines, we can close our side of its stdin, and it will notice this and exit gracefully.

I’m glad I stumbled over that {YOURVARIABLE}>&- syntax for having a dereferenced variable as the left FD of a redirection. Originally I used an ugly eval.

Example #7:

$ coproc GREP (grep --line-buffered pub); printf '%s\n' hacker public radio >&${GREP[1]}; head -n1 <&${GREP[0]}; exec {GREP[1]}>&-
[1] 25706
public
[1]+  Done                    coproc GREP ( grep --color=auto --line-buffered pub )

There we go! Not the most brilliant example, but it shows all the relevant moving parts, and we covered a couple of caveats.

Now go out and play with this and come back with an example on how this is actually useful in the real world, and submit a show!

Bash Tips - 20 - Dave Morriss | 2019-02-25

Tidying loose ends (Some collateral Bash tips)

Deleting arrays

I forgot to cover one thing on my list when doing the last show: I forgot to explain how to delete arrays and array elements. I’ll cover that topic in this episode.

Positional and Special parameters

I have also avoided talking much about the positional and special parameters in Bash: '$1', '$2', '$#' and the rest. I will cover (some of) these in this episode.

Silly titles

I stopped doing the weird episode titles by episode 14 because I thought the joke was getting tired. However, I think a few people missed them (and a certain HPR colleague was found vandalising my new titles as they were being posted ;-), so I have added them inside the notes on the older shows and am adding one here – as a homage to silliness.

Long notes

I have provided detailed notes as usual for this episode, and these can be viewed here.

Bash Tips - 19 - Dave Morriss | 2019-01-31

Arrays in Bash (Supplemental auxiliary Bash tips)

This is the fourth and last of a small group of shows on the subject of arrays in Bash. It is also the nineteenth show in the Bash Tips sub-series.

In the last show we continued with the subject of parameter expansion in the context of arrays. There are other aspects of this that could be looked at, but we’ll leave it for the moment and may revisit it in the future.

In this episode we will look in more depth at the declare (typeset) built in command and at some commands that are related (readonly and local), We will also look at the commands that assist with loading data into arrays: mapfile (readarray) and read.

Long notes

As usual I have provided detailed notes and examples for this episode, and these can be viewed here.

Response to show 2720 - Dave Morriss | 2019-01-28

Introduction

On January 4th 2019 Ken Fallon did a great show entitled hpr2720 :: Download youtube channels using the rss feeds where he presented a Bash script called youtube-rss.bash for managing YouTube downloads through RSS feeds.

Ken said he welcomed constructive feedback ☺

When I see a Bash script these days I usually find myself looking for ways to rewrite it to make it fit in with what I have been learning while doing my Bash Tips sub-series. Either that or I find it’s got some better ideas than I’ve been using which I have to find out about.

I also spend time going over my own old scripts (I was writing them in the 1990’s in some cases) and trying to incorporate newer Bash features.

Suffice it to say that I spotted some areas for improvement in Ken’s script and thought this might be the way to share my thoughts about them. We’re low on shows as I write this, so that gave me more motivation to make a show rather than add a comment or send Ken an email.

Apology: I’m still suffering from the aftermath of some flu-like illness so have had to edit coughing fits out of the audio at various points. If you detect any remnants then I’m sorry!

Long notes

I have provided detailed notes as usual for this episode, and these can be viewed here.

Bash Tips - 18 - Dave Morriss | 2019-01-17

Bash Tips - 18 (Extra auxiliary Bash tips)

Arrays in Bash

This is the third of a small group of shows on the subject of arrays in Bash. It is also the eighteenth show in the Bash Tips sub-series.

In the last show we looked at ways of accessing elements with negative indices and how to concatenate arrays. We then launched into parameter expansion in the context of arrays.

There are a few more parameter expansion operations to look at in this episode, then in the next episode we will look in more depth at the declare built in command and at some of the commands that assist with loading data into arrays.

Long notes

I have provided detailed notes as usual for this episode, and these can be viewed here.

Bash Tips - 17 - Dave Morriss | 2019-01-03

Bash Tips - 17 (Additional auxiliary Bash tips)

Arrays in Bash

This is the second of a small group of shows on the subject of arrays in Bash. It is also the seventeenth show in the Bash Tips sub-series.

In the last show we saw the two types of arrays, and learned about the multiple ways of creating them and populating them. We also looked at how array elements and entire arrays are accessed.

Now we want to continue looking at array access and some of the various parameter expansion operations available.

Long notes

I have provided detailed notes as usual for this episode, and these can be viewed here.

Bash Tips - 16 - Dave Morriss | 2018-12-20

Bash Tips - 16 (Further auxiliary Bash tips)

Arrays in Bash

This is the first of a small group of shows on the subject of arrays in Bash. It is also the sixteenth show in the Bash Tips sub-series.

We have encountered Bash arrays at various points throughout this sub-series, and have even seen a number of examples, but the subject has never been examined in detail. This group of shows intends to make good this deficiency.

Long notes

I have provided detailed notes as usual for this episode, and these can be viewed here.

Links

Bash Tips - 15 - Dave Morriss | 2018-12-06

Bash Tips - 15 (More auxiliary Bash tips)

Pitfalls for the unwary Bash loop user

This is the fifteenth episode covering useful tips for Bash users. In the last episode we looked at the 'for' loop, and prior to that we looked at 'while' and 'until' loops. In this one I want to look at some of the loop-related issues that can trip up the unwary user.

Loops in Bash are extremely useful, and they are not at all difficult to use in their basic forms. However, there are some perhaps less than obvious issues that can result in unexpected behaviour.

Long notes

I have provided detailed notes as usual for this episode, and these can be viewed here.

Bash Tips - 14 - Dave Morriss | 2018-11-22

Bash Tips - 14 (Some auxiliary Bash tips)

More about loops

This is the fourteenth episode covering useful tips about using Bash. Episodes 9-13 covered Making Decisions in Bash and in these episodes we looked at while and until loops, but not for loops. This episode is making good this deficiency, and is also looking at break and continue which are very useful when using loops.

Long notes

I have provided detailed notes as usual for this episode, and these can be viewed here.

Extra ancillary Bash tips - 13 - Dave Morriss | 2018-11-08

Extra ancillary Bash tips - 13

Making decisions in Bash

This is the thirteenth episode in the Bash Tips sub-series. It is the fifth and final of a group of shows about making decisions in Bash.

In the last four episodes we saw the types of test Bash provides, and we looked briefly at some of the commands that use these tests. We looked at conditional expressions and all of the operators Bash provides to do this. We concentrated particularly on string comparisons which use glob and extended glob patterns then we devoted an episode to Bash regular expressions.

Now we want to look at the final topic within regular expressions, the use of capture groups.

Long notes

I have provided detailed notes as usual for this episode, and these can be viewed here.

Additional ancillary Bash tips - 12 - Dave Morriss | 2018-10-25

Additional ancillary Bash tips - 12

Making decisions in Bash

This is the twelfth episode in the Bash Tips sub-series. It is the fourth of a group of shows about making decisions in Bash.

In the last three episodes we saw the types of test Bash provides, and we looked briefly at some of the commands that use these tests. We looked at conditional expressions and all of the operators Bash provides to do this. We concentrated particularly on string comparisons which use glob and extended glob patterns.

Now we want to look at the other form of string comparison, using regular expressions.

Long notes

I have provided detailed notes as usual for this episode, and these can be viewed here.

Further ancillary Bash tips - 11 - Dave Morriss | 2018-10-11

Further ancillary Bash tips - 11

This is the eleventh episode in the Bash Tips sub-series. It is the third of a group of shows about making decisions in Bash.

In the last two episodes we saw the types of test Bash provides, and we looked briefly at some of the commands that use these tests. Now we want to start examining the expressions that can be used in these tests, and how to combine them. We will also start looking at string comparisons in extended tests.

Long notes

I have provided detailed notes as usual for this episode, and these can be viewed here.

More ancillary Bash tips - 10 - Dave Morriss | 2018-09-27

More ancillary Bash tips - 10

Making decisions in Bash

This is my tenth contribution to the Bash Scripting series under the heading of Bash Tips. The previous episodes are listed below in the Links section.

We are currently looking at decision making in Bash, and in the last episode we examined the tests themselves. In this episode we’ll look at the constructs that use these tests: looping constructs, conditional constructs and lists of commands.

Note: this episode and the preceding one were originally recorded as a single episode, but because it was so long it was split into two. As a consequence the audio contains references to examples such as bash9_ex2.sh where the true name is bash10_ex1.sh. The notes have been updated as necessary but not the audio.

Long notes

I have provided detailed notes as usual for this episode, and these can be viewed here.

Some ancillary Bash tips - 9 - Dave Morriss | 2018-09-13

Some ancillary Bash tips - 9

Making decisions in Bash

This is my ninth contribution to the Bash Scripting series under the heading of Bash Tips. The previous episodes are listed below in the Links section.

It seems to me that it would be worthwhile looking at how Bash can be used to make decisions, such as how many times a loop should cycle (looping constructs) or to choose between multiple choices (conditional constructs). Of course we need to look at some of the expressions used in conjunction with the commands that do these tasks – the tests themselves – and we’ll do this in this episode.

This is a complex area which I had some trouble with when I first started using Bash, and there is a lot to say about it all. I have prepared a group of HPR shows about this subject, in order to do it justice, and this is the first of the group.

Long notes

I have provided detailed notes as usual for this episode, and these can be viewed here.

Making a Raspberry Pi inventory - Dave Morriss | 2018-02-26

Making a Raspberry Pi inventory

Introduction

I have a number of Raspberry Pis -- possibly too many -- and I sometimes lose track of which is which, what model, size, name, address each one is. I wanted to be able to keep an inventory of them all, and to this end I wrote myself a little script that can be run on any Pi which will report useful information about it.

Every Pi has a unique serial number. Actually it's randomly generated so there may be a few collisions but it's close to unique! It also contains a revision number which encodes various items of information about it such as release date, model, PCB revision and memory. My script decodes this revision number for you based on a published table.

I run a Wikimedia instance on a Pi and have used this script to record details of my Pis there as well as what they are being used for and any planned projects. I now feel more organised!

Long notes

The full-length notes (available here) contain a listing of the script, a brief description of it, and some example output.

Useful Bash functions - part 4 - Dave Morriss | 2018-02-07

Useful Bash functions - part 4

Overview

This is the fourth show about the Bash functions I use, and it may be the last unless I come up with something else that I think might be of general interest.

There is only one function to look at this time, but it's fairly complex so needs an entire episode devoted to it.

As before it would be interesting to receive feedback on this function and would be great if other Bash users contributed ideas of their own.

Full Notes

Since the notes explaining this subject are long, they have been placed here.

A flight itinerary in Bash - Dave Morriss | 2018-01-18

A flight itinerary in Bash

My daughter flew out to New Zealand before Christmas 2017 to spend some time with her brother, who had been there with his girlfriend since November. I saw her flight itinerary from the airline, but had no idea of how the times related to time back home, so I wrote a little Bash script to calculate times in UTC (my local timezone).

Both of my children have travelled a fair bit in the past few years. I like to keep track of where they are and how they are progressing through their journeys because otherwise I tend to worry. This one was a reasonably simple journey, two flights via Doha in Qatar, with not too long a wait between them. The overall journey was long of course.

When my daughter flew out to Indonesia in 2015 (4 flights and a boat trip, over 38 hours travel time) I built a spreadsheet. Just whatever provides a good distraction!

The rest of the notes, including details of the date command and the script I wrote can be found here.

  • GNU documentation for date (You can also use man date or info date for the full details. I prefer the HTML version because I don't like the info tool very much).
  • The GNU Bash Reference Manual
  • Dann Wasko's "Linux in the Shell" episode hpr1182 :: LiTS 023: Date, which is full of useful information.
  • Resources:
    • The script I wrote, called edi_akl (named to denote the starting and ending airports).

Useful Bash functions - part 3 - Dave Morriss | 2017-12-20

Useful Bash functions - part 3

Overview

This is the third show about Bash functions. These are a little more advanced than in the earlier shows, and I thought I'd share them in case they are useful to anyone.

As before it would be interesting to receive feedback on these functions and would be great if other Bash users contributed ideas of their own.

Full Notes

Since the notes explaining this subject are long, they have been placed here.

Customizing my bash prompt - Windigo | 2017-08-03

Basic bash prompt information

  • Variables and files
    • .bashrc: the RC file where all of this stuff can be set
    • PS1: main prompt variable
    • PS2: continuation prompt
    • PROMPT_COMMAND: a bash function name, run every time prompt is displayed
  • Colors
    • Uses escape sequences
    • There are problems with prompts and escape sequences
      • Multiple escaping
      • String interpretation and variable expansion
    • tput to the rescue!
      • Takes away the need for complex escape codes
      • Must run tput init at the beginning of your .bashrc file
      • TL;DR: Use tput for color strings, add them at the last possible moment, with brackets and backslashes
  • Embedding bash scripts
    • Single quotes are king
    • Will be run every time PS1 is evaluated
    • Otherwise, only run at time of assignment

Layout of my prompt

  • Two lines
    • Information/status line
    • Prompt line
  • Status line
    • Starts with current username
      • Changes color when user has mail
    • Next is the hostname (truncated)
      • Separated by an @ symbol, like an email address
      • Changes color when the system is in need of a reboot
      • Checks for /run/reboot_required
    • Current directory
      • Separated from previous items by a pipe
      • Truncated with a tilde if user's home is in the path
      • Prepended with a number indicating the directory stack, if present
      • Appended with the git (±) symbol if we're in a git branch, followed by the name of the branch
  • Prompt line
    • A blue » character
    • Prepended with the number of background processes spawned from this terminal

Screenshot:

A screenshot of my bash prompt, both initially and with all of the bells and whistles showing

Link to Git Repository

Editor's Note: added 2017-08-05

Bash snippet - extglob and scp - Dave Morriss | 2017-06-20

Bash snippet - extglob and scp

The Problem

Following on from my last show on filename expansion, concentrating on extended patterns and the extglob option, I was asked a question by Jon Kulp in the comment section.

Jon was using ls *(*.mp3|*.ogg) to find all OGG and MP3 files in a directory which also held other files. However, when he wanted to copy this subset of files elsewhere he had problems using this expression in an scp command.

Having done some investigations to help solve this I thought I'd put what I found into an HPR episode and share it, and this is the show.

Along the way clacke commented too and this led me to more investigations!

Long notes

As often happens, my idea of a brief episode turned into something much longer, so I converted the notes into long notes which you can find here. In them I have marked some sections which you might want to skip over -- unless you are as much of a geek as I am! I have not covered these sections in detail in the audio.

Bash snippet - nullglob - Dave Morriss | 2017-05-30

Bash snippet - nullglob

I recently did an HPR show about Bash filename expansion and described the 'shopt' command and its options. One of the options I talked about was 'nullglob' which controls what is returned from an expansion when no files match.

When 'nullglob' is enabled, and a pattern does not match, nothing is returned. When it is disabled (the default) then the pattern itself is returned.

Although I didn't think I'd ever need to, I recently wrote a script where I used 'nullglob', and thought I would share a snippet of the code to demonstrate what I did.

The script is for managing mail messages containing tag and summary updates. I use Thunderbird for my mail and have configured it to drop these messages into a directory so I can process them. I use Thunderbird's message filters to do this. A certain amount of Spam is also received, and sometimes valid messages need a bit of work before they can be processed.

The directory where the messages are saved (the spool area) is stored in the variable 'MAILDROP' earlier in the script.

  1 #
  2 # Find the files and store their names in an array. Use 'nullglob' so we get
  3 # nothing when there is nothing, then revert to the original setting
  4 #
  5 NG="$(shopt -p nullglob)"
  6 shopt -s nullglob
  7 MESSAGES=( $MAILDROP/*.eml )
  8 eval "$NG"
  9 
 10 #
 11 # Exit if there's nothing to do or report what's there
 12 #
 13 if [[ ${#MESSAGES[@]} -gt 0 ]]; then
 14     echo "Files in the spool area:"
 15     printf "%s\n" "${MESSAGES[@]}"
 16 else
 17     echo "The spool area is empty"
 18     exit
 19 fi

The variable 'NG' holds the state of 'nullglob' before the script modifies it. Remember that 'shopt -p' returns a list of commands that will revert the named options to their current state.

Next (line 6) the 'nullglob' option is enabled.

The array 'MESSAGES' is created on line 7 to hold the list of mail files found in the spool area. This is done with a pattern which matches files that end with the string '.eml'. If we didn't have 'nullglob' enabled then when there were no files the array would contain the pattern - which would be misleading.

Having collected the file details 'nullglob' is turned off by executing the command in the variable 'NG' on line 8.

You might think that the script could just turn 'nullglob' on then turn it off again when it's no longer needed. However, I prefer to use the technique I have shown here because it needs to have no knowledge of the state of the option before it's set, and restores that state afterwards.

By line 13 the array 'MESSAGES' either contains a list of files or is empty. The script checks for these two cases by determining how many elements are in the array. Greater than zero means we have files to process and they are listed in lines 14 and 15. The script then goes on to do various things with the files.

If there were no files then the script reports this and exits.

That's it! This is not the only way to do this, but I like to write scripts that call as few sub-processes as I can, and this way appeals for that reason.

More supplementary Bash tips - Dave Morriss | 2017-05-17

More supplementary Bash tips

Pathname expansion; part 2 of 2

Expansion

As we saw in the last episode 2278 (and others in this sub-series) there are eight types of expansion applied to the command line in the following order:

  • Brace expansion (we looked at this subject in episode 1884)
  • Tilde expansion (seen in episode 1903)
  • Parameter and variable expansion (this was covered in episode 1648)
  • Command substitution (seen in episode 1903)
  • Arithmetic expansion (seen in episode 1951)
  • Process substitution (seen in episode 2045)
  • Word splitting (seen in episode 2045)
  • Pathname expansion (the previous episode 2278 and this one)

This is the last topic in the (sub-) series about expansion in Bash.

In this episode we will look at extended pattern matching as also defined in the “Manual Page Extracts” section at the end of the long notes.

Long Show Notes

I have written out a moderately long set of notes about this subject and these are available here.

Some supplementary Bash tips - Dave Morriss | 2017-04-26

Some supplementary Bash tips

Pathname expansion; part 1 of 2

Expansion

As we saw in the last episode 2045 (and others in this sub-series) there are eight types of expansion applied to the command line in the following order:

  • Brace expansion (we looked at this subject in episode 1884)
  • Tilde expansion (seen in episode 1903)
  • Parameter and variable expansion (this was covered in episode 1648)
  • Command substitution (seen in episode 1903)
  • Arithmetic expansion (seen in episode 1951)
  • Process substitution (seen in episode 2045)
  • Word splitting (seen in episode 2045)
  • Pathname expansion (this episode and the next)

This is the last topic in the (sub-) series about expansion in Bash. However, when writing the notes for this episode it became apparent that there was too much to fit into a single HPR episode. Consequently I have made it into two.

In this episode we will look at simple pathname expansion and some of the ways in which its behaviour can be controlled. In the next episode we’ll finish by looking at extended pattern matching. Both are included in the “Manual Page Extracts” section at the end of the long notes.

Long Show Notes

I have written out a moderately long set of notes about this subject and these are available here.

Audio speedup script - Dave Morriss | 2016-10-07

Audio speedup script

Back in 2015 Ken Fallon did a show (episode 1766) on how to use sox to truncate silence and speed up audio.

Inspired by this I wrote a Bash script to aid my use of the technique, which I thought I’d share with you.

I have written out detailed notes for this episode describing the script and examining how it works and these are available here http://hackerpublicradio.org/eps/hpr2135/full_shownotes.html.

Useful Bash functions - part 2 - Dave Morriss | 2016-08-15

Useful Bash functions - part 2

Overview

This is the second show about Bash functions. In this one I revisit the yes_no function from the last episode and deal with some of the deficiencies of that version.

As before it would be interesting to receive feedback on these versions of the function and would be great if other Bash users contributed ideas of their own.

Full Notes

Since the notes explaining this subject are long, they have been placed here.

Some other Bash tips - Dave Morriss | 2016-06-03

Some other Bash tips

Expansion

As we saw in the last episode 1951 (and others in this sub-series) there are eight types of expansion applied to the command line in the following order:

  • Brace expansion (we looked at this subject in episode 1884)
  • Tilde expansion (seen in episode 1903)
  • Parameter and variable expansion (this was covered in episode 1648)
  • Command substitution (seen in episode 1903)
  • Arithmetic expansion (seen in episode 1951)
  • Process substitution
  • Word splitting
  • Pathname expansion

We will look at process substitution and word splitting in this episode but since there is a lot to cover in these subjects, we'll save pathname expansion for the next episode.

I have written out a moderately long set of notes about this subject and these are available here http://hackerpublicradio.org/eps/hpr2045/full_shownotes.html.

Using the Incron file watching daemon - b-yeezi | 2016-04-06

Using the Incron file watching daemon

Check out the man page for incron and also this write-up by Nixcraft.

basic usage:

incrontab -e

In your editor of choice, follow this syntax:

<path-to-watch> <event mask> command

Links

Pomodoro Timer - The Evolution of a Script part deux - Nacho Jordi | 2016-03-18

The Script: http://hackerpublicradio.org/eps/hpr1990.bash

Pomodoro Timer - The Evolution of a Script (pt 1) - Nacho Jordi | 2016-03-15

Time now Ladies and Gents - Ken Fallon | 2016-01-26

In the show "hpr1943 :: HPR AudioBook Club 11.5 - Interview with David Collins-Rivera" pokey asked if there was a way to get the duration for media. The following three options springs to mind immediately.

The first option is fix_tags and was written by our own Dave Morriss.

$ date --utc --date="@$(echo $(fix_tags *mp3 *ogg 2>/dev/null | \
awk -F '\\(|\\)' '/length/ {print $2}' | \
sed 's/ sec//g' ) | \
sed 's/ /+/g' | bc )"  +"%T"
03:09:49

Next up is mediainfo which provides a lot of information on media files.

$ date -ud @$(echo $(mediainfo --full --Output=XML *mp3 *ogg | \
xmlstarlet sel -T -t -m "Mediainfo/File/track[@type='Audio']/Duration[1]" -v "." -n - | \
sed 's/.\{3\}$//') | \
sed 's/ /+/g' | bc)  +"%T"
03:09:49

The last option is to use ffprobe from the ffmpeg team.

$ date -ud @$(echo $(for i in *mp3 *ogg;\
do  \date -ud 1970-01-01T$(ffprobe -i $i 2>&1 | \
grep Duration | awk '{print $2}'| \
sed 's/,//g' ) +%s;done) | \
sed 's/ /+/g' | bc)  +"%T"
03:09:49

For complete shownote please visit http://hackerpublicradio.org/eps/hpr1952.html

Some additional Bash tips - Dave Morriss | 2016-01-25

Some additional Bash tips

Expansion

As we saw in the last episode 1903 there are seven types of expansion applied to the command line in the following order:

  • Brace expansion (we looked at this subject in episode 1884)
  • Tilde expansion (seen in episode 1903)
  • Parameter and variable expansion (this was covered in episode 1648)
  • Command substitution (seen in episode 1903)
  • Arithmetic expansion
  • Word splitting
  • Pathname expansion

There is also another, process substitution, which occurs after arithmetic expansion on systems that can implement it.

We will look at one more of these expansion types in this episode but since there is a lot to cover, we'll continue in a later episode.

I have written out a moderately long set of notes about this subject and these are available here http://hackerpublicradio.org/eps/hpr1951_full_shownotes.html.

Audio Note

This time, in the spirit of experimentation and as a way of learning Audacity I processed my audio thus:

  • Turned the stereo tracks to mono

  • Used a Noise Gate plug-in to reduce background noise (after "training" it on some silence)

  • Performed a Truncate Silence pass to reduce the length of pauses

  • Applied a small amount of amplification

Let me know if this had any positive or negative effects on the end product.

Some further Bash tips - Dave Morriss | 2015-11-18

Some further Bash tips

Expansion

There are seven types of expansion applied to the command line in the following order:

  • Brace expansion (we looked at this subject in the last episode 1884)
  • Tilde expansion
  • Parameter and variable expansion (this was covered in episode 1648)
  • Command substitution
  • Arithmetic expansion
  • Word splitting
  • Pathname expansion

We will look at some more of these in this episode but since there is a lot to cover, we'll continue in a later episode.

I have written out a moderately long set of notes about this subject and these are available here http://hackerpublicradio.org/eps/hpr1903_full_shownotes.html.

Some more Bash tips - Dave Morriss | 2015-10-22

Some more Bash tips

We looked at Parameter Expansion back in HPR episode 1648 where we saw how Bash variables could be used, checked and edited. There are other sorts of expansions within Bash, and we'll look at one called "Brace Expansion" in this episode, which follows on from episode 1843 "Some Bash tips".

I have written out a moderately long set of notes about this subject and these are available here http://hackerpublicradio.org/eps/hpr1884_full_shownotes.html.

Some Bash tips - Dave Morriss | 2015-08-26

Today I want to talk about three Bash commands:

  • pushd
  • popd
  • dirs

These let you change directory on a Linux system (and others which support Bash) but keep a record of where you have been in a stack structure. The stack can be viewed and manipulated with these commands as well.

I have written out a moderately long set of notes about these commands and these are available here http://hackerpublicradio.org/eps/hpr1843_full_shownotes.html.

Waking up with Windigo - Windigo | 2015-08-19

This is a quick summary of my alarm clock system, written in bash and highly unreliable.

Hardware

My preferred hardware platform is a Dell Mini 9.

Software

My alarm clock is an embarrassing combination of bash scripts and Audacious, my favorite media player. Any media player will do, as long as it's scriptable.

How It Works

There are currently two bash scripts in my crappy alarm setup. One script is called "wakeup" and the other is called "wakeup-at".

wakeup is simply a wrapper that adds some error handling around audacious. It launches audacious if it can't find an instance running already, waits five seconds for it to get itself together, and then causes it to play. It is also currently broken, so the 'launching audacious' part doesn't work. I have to manually start audacious myself. FAILURE.

wakeup script:

#!/bin/bash
audacious &

sleep 5s

audacious -p &

You've noticed that the "wakeup" script doesn't actually have any timing involved; If you want to use it as an alarm, you get to combine it with the bash "sleep" command. This is not a failure, this is by design! An example alarm: 

sleep 8h; wakeup

One problem with this methodology is that it requires math, and is prone to errors. If I'm going to sleep at 10:46:33 PM and need to wake up at 7:00 AM, I need to chain sleep commands together for each unit of time: 

sleep 7h; sleep 14m; sleep 27s; wakeup

Get some of that math wrong, and you wake up at the wrong time. FAILURE.

"wakeup-at" is a wrapper around "wakeup" that uses the "at" utility to schedule the wakeup script. So, instead of using multiple sleep commands, it accepts any of the time formats that at accepts: 

wakeup-at 7:00 AM
wakeup-at 6:00AM 2018-02-02
wakeup-at teatime

Here is the wakeup-at script: 

#!/bin/bash

## Make sure we have enough arguments
if [ $# -lt 1 ]
then
  echo "Usage: `basename $0` <time>"
  exit 1
fi

echo "$@"

## Add custom time keywords
case "$1" in
"eternaldarkness")
	echo wakeup | at 3:33 AM
	;;

## Catch-all; send all arguments to at
*)
	echo wakeup | at $@
	;;
esac

If you make a syntax error, "at" tells you about it immediately. Its only failings are what it inherits from the original "wakeup" script.

Organizing Photos with Bash - Tony Pelaez | 2015-06-15

Summary

In this episode I provide an overview of how I use bash to automate my process for orgainizing photographs on my computer.

There are two main objectives of this script:

  1. Organize photographs in a folder structure that makes sense to me, e.g. 2015/2015-05-22
  2. Allow me to back up my photographs using a variety of methods.

Download the Script

This script is hosted on Github and you can download the latest version using following command:

git clone https://gist.github.com/81e489b2a7397bb17305.git

Script


#!/bin/bash

shopt -s -o nounset

# Create variables and configure script.
declare -rx SCRIPT=${0##*/}
declare TMPDIR=/tmp/photos
declare -r CURRENTDIR=`pwd`
declare FILES=$TMPDIR/*
declare DESTINATION=/media/Tyr/Pictures/Photos
declare -r GOOGLEUSER="tnyplz@gmail.com"
declare -r  OPTSTRING="-h, -d:"
declare -r  LONGOPTSTRING="help, destination-directory, no-google-backup, sd-card, tmp-dir, no-delete, backup"
declare RESULT
declare GOOGLE_BACKUP=true
declare SD=false
declare SDDIR
declare NODELETE=false
declare S3=false

# Executable dependencies
declare -rx find="/usr/bin/find"
declare -rx gphoto2="/usr/bin/gphoto2"
declare -rx google="/usr/bin/google"
declare -rx dcraw="/usr/bin/dcraw"
declare -rx rsync="/usr/bin/rsync"
declare -rx rename="/usr/bin/rename"
declare -rx tar="/usr/bin/tar"
declare -rx s3cmd="/usr/bin/s3cmd"

# Sanity Checks
if test -z $BASH; then
    printf "$SCRIPT:$LINENO: please run this script with the BASH shell\n" >&2
    exit 192
fi
# check for find
if test ! -x $find; then
    printf "$SCRIPT:$LINENO: the $find command is not available -- \
aborting\n" >&2
    exit 192
fi
# check for gphoto2
if test ! -x $gphoto2; then
    printf "$SCRIPT:$LINENO: the $gphoto2 command is not available -- \
aborting\n" >&2
    exit 192
fi
# check for google
if test ! -x $google; then
    printf "$SCRIPT:$LINENO: the $google command is not available -- \
aborting\n" >&2
fi
# check for dcraw
if test ! -x $dcraw; then
    printf "$SCRIPT:$LINENO: the $dcraw command is not available -- \
aborting\n" >&2
fi
# check for rename
if test ! -x $rename; then
    printf "$SCRIPT:$LINENO: the $rename command is not available -- \
aborting\n" >&2
    exit 192
fi
# check for rsync
if test ! -x $rsync; then
    printf "$SCRIPT:$LINENO: the $rsync command is not available -- \
aborting\n" >&2
fi
# check for tar
if test ! -x $tar; then
    printf "$SCRIPT:$LINENO: the $tar command is not available -- \
aborting\n" >&2
fi
# check for glacier-cmd
if test ! -x $s3cmd; then
    printf "$SCRIPT:$LINENO: the $s3cmd command is not available -- \
aborting\n" >&2
fi


# Check for Options
# =================

getopt -T
if [ $? -ne 4 ]; then
    printf "$SCRIPT:$LINENO: %s\n" "getopt is in compatibility mode" >&2
    exit 192
fi

RESULT=$(getopt --name "$SCRIPT" --options "$OPTSTRING" --longoptions "$LONGOPTSTRING" -- "$@")
if [ $? -gt 0 ]; then
    exit 192
fi

eval set -- "$RESULT"

while [ $# -gt 0 ]; do
    case "$1" in
    -h | --help) # show help
        printf "%s\n" "
This script helps you automate the process of downloading photos from
your camera, uploading backups to Google Picasa, and syncing the files
with a specified directory.

Dependendies:
  gphoto2
  dcraw
  googlecl
  rsync
  s3cmd

usage: $SCRIPT [options]

Options:
  -h | --help                        Show help for $SCRIPT
  --destination-directory {LOCATION} Set the location where the photos will be
                                     copied to.
  --tmp-dir {LOCATION}               Set the temporary directory where images
                                     will be downloaded to initially. The
                                     default is /tmp/photos.
  --no-google-backup                 Disable uploading low rez copies to Google
                                     Plus.
  --sd-card {LOCATION}               Set the location of the sd card.
  --no-delete                        Do not delete from temp file.
  --backup {FOLDER} {S3 BUCKET}      Create archive from folder and upload to S3.
  "
        exit 0
        ;;
    --destination-directory ) shift
        if [ $# -eq 0 ]; then
            printf "$SCRIPT:$LINENO: %s\n" "Invalid argument for destination. No destination given." >&2
            exit 192
        fi
        DESTINATION="$1"
        ;;
    --tmp-dir ) shift
        if [ $# -eq 0 ]; then
            printf "$SCRIPT:$LINENO: %s\n" "Invalid argument for tmp-dir.  No temporary directory given." >&2
            exit 192
        fi
        TMPDIR="$1"
        FILES=$TMPDIR/*
        ;;
    --no-google-backup ) shift
        GOOGLE_BACKUP=false
        ;;
    --sd-card ) shift
        SD=true
        if [ $# -eq 0 ]; then
            printf "$SCRIPT:$LINENO: %s\n" "Invalid argument for sd directory. No sd card directory given." >&2
            exit 192
        fi
        SDDIR="$1"
        ;;
    --no-delete ) shift
	NODELETE=true
	;;
    --backup ) shift
        if [ $# -eq 0 ]; then
            printf "$SCRIPT:$LINENO: %s\n" "Invalid argument for AWS Glacier Backup. Backup folder and vault must be specified."
        fi
        S3=true
        BACKUP_FOLDER="$1"
        BUCKET="$2"
        ;;
    esac
    shift
done

# Functions
# =========

# function to convert a raw image to jpg.
# input: requires the user to specify the file extention ($1).
function convert_to_jpg () {
  FILES2CONVERT=$TMPDIR/*"$1"
  for FILE in $FILES2CONVERT
  do
    FILE2BACKUP=$TMPDIR/Backup/`basename "$FILE" "$1"`'.jpg'
    if [ -e $FILE2BACKUP ]; then
        printf "$SCRIPT:$LINENO: Skipping $FILE, jpg file already exists\n"
    elif [ -e $FILE ]; then
        printf "$SCRIPT:$LINENO: Converting $FILE to $FILE2BACKUP\n"
        $dcraw -cvz -w -o 1 -q 3 "$FILE" | cjpeg -quality 80 -optimize > "$FILE2BACKUP"
    else
        printf "Did not convert $FILE\n"
    fi
  done
}

# function to resize jpeg to upload to picasa
function resize_to_thumb () {
    FILES2RESIZE=$TMPDIR/Backup/* # TODO pass this in as argument along with destination directory
    for FILE in $FILES2RESIZE
    do
        printf "$SCRIPT:$LINENO: Creating thumbnail for $FILE..."
        convert $FILE -resize 2048x2048 $TMPDIR/Backup/Upload/`basename "$FILE" ".jpg"`'_thumb.jpg'
        printf "done\n"
    done
}

# function to import photos
function import_photos () {
    printf "$SCRIPT:$LINENO: Importing Photos\n"
    if $SD; then
        cp -p "$SDDIR"/* .
    else
        $gphoto2 --quiet --get-all-files
    fi
}

# function to remove spaces in file names
function remove_spaces () {
    $find $1 -depth -name "* *" -execdir $rename 's/ /_/g' "{}" \;
}

# function to sort images into direcotries based on date.
# input: directory to sort ($1)
#        directory to sort into ($2)
function sort_images () {
    SORTDIR=$2'/Sorted/'
    for FILE in $1
    do
        printf "$SCRIPT:$LINENO: Sorting $FILE\n"
        DATEDIR=$SORTDIR`date -r "$FILE" +%Y`'/'`date -r "$FILE" +%Y-%m-%d`
        mkdir -p $DATEDIR
        cp "$FILE" $DATEDIR/
    done
}

# function create archive and upload to AWS S3
# input: directory to create an archive for ($1)
#        s3 bucket name ($2)
function archive_folder () {
    ARCHIVE=$TMPDIR/$(basename $1).tar.bz2
    printf "$SCRIPT:$LINENO: archiving $ARCHIVE\n"
    $tar -cvjf $ARCHIVE $1
    $s3cmd put $ARCHIVE $2
}

# Create temporary directory
mkdir -p $TMPDIR
cd $TMPDIR

# Create AWS Glacier archive
if $S3; then
    archive_folder $BACKUP_FOLDER $BUCKET
    cd $CURRENTDIR
    if [ $NODELETE = false ]; then
        rm -rf $TMPDIR
    fi
    exit 0
fi

# Import files from camera
import_photos
printf "$SCRIPT:$LINENO: Importing Photos Done!\n"

# Remove Spaces in Filenames
remove_spaces $TMPDIR

#Convert all files to lower case
printf "$SCRIPT:$LINENO: Converting Photos to Lower Case.\n"
for FILE in *
do
    f=`echo $FILE | tr '[:upper:]' '[:lower:]'`
    mv "$FILE" "$f"
done
printf "$SCRIPT:$LINENO: Converting Photos to Lower Case Done!\n"

# Sort files
sort_images "$FILES" "$TMPDIR"
printf "$SCRIPT:$LINENO: Sorting Images Done!\n"

# Create backup jpgs and upload them to Picassa
mkdir -p $TMPDIR/Backup
cp $TMPDIR/*.jpg $TMPDIR/Backup/

convert_to_jpg ".nef"
convert_to_jpg ".nrw"
mkdir -p $TMPDIR/Backup/Upload
resize_to_thumb

if $GOOGLE_BACKUP; then
    # Upload jpgs to Picassa
    # Requires that you authorize googlecl through the web browser.
    $google picasa create --user $GOOGLEUSER --title "Backup "`date +%Y-%m` $TMPDIR/Backup/Upload/*
fi

# Copy files to final locations
$rsync -ravv $TMPDIR/Sorted/ $DESTINATION # TODO test to make sure destination works

cd $CURRENTDIR

# Remove temp folder
if [ $NODELETE = false ]; then
    rm -rf $TMPDIR
fi

printf "$SCRIPT:$LINENO: Processing Complete!\n"
exit 0

Useful Bash functions - Dave Morriss | 2015-04-28

Overview

I enjoy writing Bash scripts to solve various problems. In particular I have a number of scripts I use to manage the process of preparing a show for HPR, which I am developing at the moment.

My more complex Bash scripts use a lot of functions to perform the various tasks, and, in the nature of things, some of these functions can be of use in other scripts and are shared between them.

I thought I would share some of these functions with HPR listeners in the hopes that they might be useful. It would also be interesting to receive feedback on these functions and would be great if other Bash users contributed ideas of their own.

Full Notes

Since the notes explaining this subject are long, they have been placed here: http://hackerpublicradio.org/eps/hpr1757_full_shownotes.html, and an experimental ePub version is available here: http://hackerpublicradio.org/eps/hpr1757_full_shownotes.epub.

  1. Bash Support Vim plugin: http://www.vim.org/scripts/script.php?script_id=365
  2. HPR episode Bash parameter manipulation: http://hackerpublicradio.org/eps.php?id=1648
  3. How to write functions (from The Linux Documentation Project):
  4. Download the pad and yes_no functions: http://hackerpublicradio.org/eps/hpr1757_functions.sh

Windows Remote Desktop on GNU/Linux - Ken Fallon | 2015-02-20

Recorded using Easy Voice Recorder Pro

I wrote a bash script to connect to various different windows servers from my GNU/Linux desktops. I had a few different requirements:

  • I should be able to call it based on hostname.
  • All windows should be 90% smaller than my screen.
  • It should map my keyboard.
  • It should map my local disk.
  • It should quickly timeout if the port is not available.

You can get the full script here, but let’s walk through it:

The first line calls bash and then gets the server name from the symlink that is calling the script. The port is set as “3389”, but you can change that if you like. 

#!/bin/bash
SERVER=`basename $0`
PORT="3389"

The next few lines finds the smallest vertical and horizontal sizes, even if you are running multiple screens. Then it calculates 90% of that to use as the size.

h=$(echo "scale=0;(($(xrandr | grep '*+' | sed 's/x/ /g' | awk '{print $1}' | sort -n | head -1 )/100)*90)" | bc)
v=$(echo "scale=0;(($(xrandr | grep '*+' | sed 's/x/ /g' | awk '{print $2}' | sort -n | head -1 )/100)*90)" | bc)
SIZE=${h}x${v}

Next we set the default username and password. I have it ask me for my password but I put it in here as an example.

PASSWORD='defaultpassword'
USERNAME='administrator'
WORKGROUP='workgroup'

In some cases the credentials may be different, so I have a case statement that will cycle through the servers and apply the differences. Depending on your naming schemes you may be able to use regular expressions here to filter out groups of servers. 

case "${SERVER}" in
  *server*) echo "Server ${SERVER}"
    PASSWORD='work_password'
    USERNAME='administrator'
    WORKGROUP='WORKGROUP'
    ;;
 
  *colo*) echo "Server ${SERVER}"
    PASSWORD='colo_server_password'
    USERNAME='administrator'
    WORKGROUP='COLODOMAIN'
    ;;
     
  some_server ) echo "Server ${SERVER}"
    PASSWORD='some_server_password'
    USERNAME='some_server_password'
    ;;
  *) echo "No match for ${SERVER}, using defaults"
    ;;
esac

Next we use an inbuilt bash command to see if a remote port is open and timeout after one second. 

timeout 1 bash -c "echo >/dev/tcp/${SERVER}/${PORT}"

I used to connect to rdp using the program rdesktop, but it is now of limited value due to the fact that there are many open bugs that are not getting fixed. Bugs such as Bug 1075697 - rdesktop cannot connect to systems using RDP version 6 or newer and Bug 1002978 - Failed to negotiate protocol, retrying with plain RDP . I then switch to using xfreerdp. This is the client that is behind remmina.

You can use xfreerdp /kbd-list to get a list of the available keyboard layouts.

if [ $? -eq 0 ]; then
  echo "${SERVER}:${PORT} is open"
  xfreerdp /v:${SERVER} /size:${SIZE} /kbd-type:0x00000409 /t:${SERVER} /d:${WORKGROUP} /u:${USERNAME} /p:${PASSWORD} /a:drive,pc,/ /cert-ignore &
else
  echo "${SERVER}:${PORT} is closed"
fi

Next you will need to be sure that your host names are available, either in dns or in your /etc/hosts/ file. For example:

10.1.0.1 server1
10.1.0.2 server2
10.1.0.3 server3
10.2.0.1 coloserver1
10.2.0.2 coloserver2
10.2.0.3 coloserver3
192.168.1.1 some_server

Edit the script to your liking and then put it into your a directory in your path, possibly /usr/local/bash or ~/bin/. You can then make symbolic links to the servers to the bash script, also in a directory in your path, using the command:

ln -s /usr/local/bash/rdp.bash ~/bin/some_server
chmod +x ~/bin/some_server

Which links the global rdp.bash script to your personal symlink, and makes it executable.

All that you need to do then is type the name of the server and a rdp screen should pop up.

In our example:

$ some_server

From there your Windows Server session should pop up.

Bash parameter manipulation - Dave Morriss | 2014-11-26

Bash parameter manipulation

I'm a great fan of using the Linux command line and enjoy writing shell scripts using the Bash shell.

  • BASH (or more usually Bash or bash) is the name of a Unix shell. The name stands for Bourne Again SHell, which is a play on words. Bash is an extension of the shell originally written by Stephen Bourne in 1978, usually known as SH.

  • Bash was written as part of the GNU Project which forms part of the Linux Operating System.

  • A shell is the part of the operating system that interprets commands, more commonly known as the command line.

  • A knowledge of Bash is very helpful if you would like to be able to use the power of the command line. It is also the way to learn how to build Bash scripts for automating the tasks you need to perform.

In this episode we look at what parameters are in Bash, and how they can be created and manipulated. There are many features in Bash that you can use to do this, but they are not easy to find.

As I was learning my way around Bash it took me a while to find these. Once I had found them I wanted to make a "cheat sheet" I could stick on the wall to remind me how to do things. I am sharing the result of this process with you.

The version of Bash which I used for this episode is 4.3.30(1)-release

The full notes for this episode are to be found here: http://hackerpublicradio.org/eps/hpr1648_full_notes.html

thebestofyoutube.com download script - Ken Fallon | 2014-01-24

In episode "Thu 2013-12-19: hpr1404 Editing pre-recorded audio in Audacity" I walked you through editing a podcast, by the magic of editing this is been posted after the other show has aired. The plan here is to get people to share their useful hacks to show how elegant, or in my case ugly, code can be. As Knightwise says "Getting technology to work for you."™
Feel free to share your own hacks with us.

http://hackerpublicradio.org/eps.php?id=1404
http://hackerpublicradio.org/eps/hpr1430-downloader.bash.txt


#!/bin/bash
# Downloads videos from youtube based on selection from http://thebestofyoutube.com
# (c) Ken Fallon http://kenfallon.com
# Released under the CC-0

maxtodownload=10
savepath="/mnt/media/Videos/tv/youtube/bestofyoutube"
savedir="${savepath}/$(\date -u +%Y-%m-%d_%H-%M-%SZ_%A)"
mkdir -p ${savedir}
logfile="${savepath}/downloaded.log"

# Gather the list
seq 1 ${maxtodownload} | while read videopage;
do 
  thisvideolist=$(wget --quiet "http://bestofyoutube.com/index.php?page=${videopage}" -O - | 
  grep 'www.youtube.com/embed/' | 
  sed 's#^.*www.youtube.com/embed/##' | 
  awk -F '"|?' '{print "http://www.youtube.com/watch?v="$1}')
  for thisvideo in $(echo $thisvideolist);
  do 
    if [ "$( grep "${thisvideo}" "${logfile}" | wc -l )" -eq 0 ];
    then
      echo "Found the new video ${thisvideo}"
      echo ${thisvideo} >> ${logfile}_todo
    else
      echo "Already downloaded ${thisvideo}"
    fi
  done
done

# Download the list
if [ -e ${logfile}_todo ];
then
  tac ${logfile}_todo | youtube-dl --batch-file - --ignore-errors --no-mtime --restrict-filenames \
    --max-quality --format mp4 --write-auto-sub -o ${savedir}'/%(autonumber)s-%(title)s-%(id)s.%(ext)s'
  cat ${logfile}_todo >> ${logfile}
  rm ${logfile}_todo
fi

Recording Terrestrial Radio with bash scripts and cron jobs - Jon Kulp | 2013-05-14

In this episode I talk about my solution for capturing terrestrial radio so that I can listen to it at my own convenience. I use a bash script, cron jobs, and the streamripper package. here are some links to things I mentioned in the podcast.

Jezra's command-line audio player sap (simple audio player): http://www.jezra.net/projects/sap

Streamripper: http://streamripper.sourceforge.net/

Radio station KRVS 88.7 FM, Lafayette, Louisiana, USA http://krvs.org/

And you can see the whole radio-recording script here: http://jonathankulp.org/archives/647

My Magnatune Downloader - Dave Morriss | 2013-03-14

The Problem

I'm a fan of Magnatune (http://magnatune.com/) and have been buying music from them for 7 or 8 years. The Magnatune website itself is good for exploring and downloading, and interfaces for browsing and purchasing are available in a number of players on Linux. I have direct experience of:

  • Amarok: allows you to browse, purchase, examine artist information and album details.
  • Rhythmbox: the plugin, which used to allow browsing and purchasing, is currently unavailable, but is apparently due to return soon.
  • Gnome Music Player Client: (a front-end to the Music Player Daemon, mpd) offers a Magnatune browser plugin
  • Magnatune Web 2.0 player: a web-based tool which will browse, play and download Magnatune music.
  • Magnatune Android player: a fairly basic browser and player for Android 2.0 and up.

The Magnatune Web 2.0 player is the best of the bunch as far as I am concerned, particularly since it allows me to explore the music collection whilst listening to streamed music at the same time. However, none of these interfaces provide me with exactly what I want in terms of the download process, so I decided to write my own.

The Plan

I currently host my music on my HP Proliant microserver, share it across the home network, and play it with the Music Player Daemon (http://sourceforge.net/projects/musicpd/) on my desktop system. I normally keep the album cover image, artwork and related material in the same directory as the album itself, and I want to be able to save all files in their appropriate places automatically.

Magnatune provides an API which is documented at http://download.magnatune.com/info/api, though this information is only available to members. Data is available in several formats: XML, SQlite and MySQL.

Design

I didn't want to launch into building a full-blown application, especially since I only needed a downloader, so I decided to create a collection of Bash and Perl scripts.

I decided to use the XML data organised by album. This is updated on about a weekly or two weekly basis, and there is a signalling mechanism through a downloadable file containing a checksum. When this changes the large data file has changed and can be downloaded. At the time of writing I simply run this by hand when I receive an email alert from Magnatune.

Magnatune uses an unique key made from the artist and album names which it refers to as the SKU (Stock Keeping Unit) or albumsku. They use this as an URL component and in XML tags. I use it to identify the stuff I download and to keep a simple inventory.

I decided to write some basic scripts:

  • To download the catalogue
  • To extract information from the catalogue
  • To download an album
  • To unpack the downloaded items into the target directory

I wanted to learn more about manipulating XML data, so I decided to use XSL, the Extensible Stylesheet Language. This lets you define stylsheets for XML data, including ways of identifying XML components with XPath and of transforming XML with XSLT.

I have included a number of links to the resources I used in the shownotes.

Repository

I have placed all of the scripts, their associated files, and HTML and PDF README files (extended shownotes) in a GitLab repository. This can be browsed at https://gitlab.com/davmo/magnatune-downloader or, if a copy is required it can be obtained with the command: 

  git clone https://gitlab.com/davmo/magnatune-downloader.git

This makes a local git repository containing a copy of all of the files in the current directory.

Note: The code was originally hosted on Gitorious (https://gitorious.org/magnatune-downloader), but with the demise of this service it was moved to GitHub and the details above updated. Then since the Microsoft takeover of GitHub, it has been moved to GitLab and the details updated as needed.

Scripts

  • update_albums: a Bash script to download a new version of the album catalogue, as a bzipped XML file, if it is different from the current version. It generates a summary of the catalogue for simple searching using XSLT.
  • report_albumsku: a Bash script to take a SKU code and look up the album details in the XML file.
  • get_album: a Bash script to download an album, cover images and artwork. It takes the SKU as an argument and uses it to make an URL for an XML file which points at all of the components, and this is downloaded (with authentication). The script then parses this file to get the necessary URLs for downloading. I only use the OGG format but it could easily collect any or all formats available from Magnatune. The script records the fact that this particular SKU code has been downloaded so that it isn't collected again in error. All downloaded files are given names beginning with the SKU code and are stored for the installation phase.
  • install_download: a Perl script which unpacks the downloaded zip file to its final destination then adds the cover images and artwork to the same place. I used Perl because it allowed me to query the zip file to determine the name of the directory that was going to be created.

Further Developments

I have added further scripts to this system since I created it. I have one that synchronises the music files from my workstation to the server, and two that give me a simple wish-list or queue functionality.

Since I have a 200GB download limit per month on my broadband contract I try not to download music too often and avoid contention with the rest of the family. My queueing system is used to keep a list of stuff I'd like to buy from Magnatune, and I simply feed the top element from the queue into my download script every week or so.

In the future I expect to be refining all of these scripts and making them less vulnerable to errors. For example, I have found a few cases where Magnatune's XML is not valid and this causes the xsltproc tool to fail. I'd like to be able to recover from such errors more elegantly than I'm doing now.

At some point I may well be tempted to consolidate all of the current functions into a single Perl script.

Disclaimer

I have no connections to Magnatune other than being a contented customer.

Links

What I do with bash scripts - Jon Kulp | 2013-03-05

In this episode I talk about the way I use shell scripting on a day-to-day basis. I am not employed in a technical field, so the fact that I use shell scripts at all surprises most people. I am just a music history professor with an enthusiasm for Linux and free software. Although I have dabbled a bit with Python, I don't feel nearly as comfortable with Python as I do with bash, so all of the scripts I mention in this episode are written for bash.

Here are links to blog posts about some of the scripts mentioned in the show.

markdown2latex: http://jonathankulp.org/archives/570

Cowsay stuff: http://jonathankulp.org/archives/346

"stick" scp script: http://jonathankulp.org/archives/441

MyIP: http://jonathankulp.org/archives/620

Bash Scripting: Episode 2 Command Line Basics - Ken Fallon | 2010-11-17

In the second installment Ken resolves to not do any work and so get's permission from Chess Griffin to reuse extracts from Linux Reality Episode 14 - Command Line Basics May 17, 2006 http://www.linuxreality.com/archives.php#14 Shownotes can be found at http://hackerpublicradio.org/shownotes/hpr0598.html

Introduction to bash scripting - Ken Fallon | 2010-08-11

http://en.wikipedia.org/wiki/Bourne_shell
http://en.wikipedia.org/wiki/Command-line_interface

A list of "Hello World" programs in many different computer languages: 
http://www.roesler-ac.de/wolfram/hello.htm 

For Windows:
Editor: http://notepad-plus-plus.org/
Bash (and more): http://x.cygwin.com/ 
(run setup, and selecting the 'xinit' package from the 'X11' category.)

$ echo '#!/bin/bash' > hello.bash
$ echo "echo hello world" >> hello.bash

$ cat hello.bash 
#!/bin/bash
echo hello world

$ chmod +x hello.bash

$ ./hello.bash
hello world

feedback-(a)-kenfallon.com
More information http://www.kenfallon.com

Xoke's Podcasting Script - Xoke | 2010-05-24

Xoke talks about his podcasting script that is available on xoke.org

bash loops - Ken Fallon | 2010-03-24

user@pc:~$ for number in 1 2 3
> do
> echo my number is $number
> done
my number is 1
my number is 2
my number is 3

user@pc:~$ for number in 1 2 3 ; do echo my number is $number; done
my number is 1
my number is 2
my number is 3

user@pc:~$ cat x.txt|while read line;do echo $line;done
one-long-line-with-no-spaces
one ling line with spaces

user@pc:~$ for line in `cat x.txt`;do echo $line;done
one<-long-line-with-no-spaces
one
ling
line
with
spaces