Finishing off the subject of expansion in Bash (part 2)
<< First, < Previous, Next >, Latest >>

Hosted by Dave Morriss on Wednesday 2017-05-17 is flagged as Explicit and is released under a CC-BY-SA license.
Tags: Bash, expansion, filename expansion, extglob, extended pattern matching.
Listen in ogg,
spx, or
mp3 format. | Comments (8)
This is an open series in which Hacker Public Radio Listeners can share their Bash scripting knowledge and experience with the community. General programming topics and Bash commands are explored along with some tutorials for the complete novice.
More supplementary Bash tips
Pathname expansion; part 2 of 2
Expansion
As we saw in the last episode 2278 (and others in this sub-series) there are eight types of expansion applied to the command line in the following order:
- Brace expansion (we looked at this subject in episode 1884)
- Tilde expansion (seen in episode 1903)
- Parameter and variable expansion (this was covered in episode 1648)
- Command substitution (seen in episode 1903)
- Arithmetic expansion (seen in episode 1951)
- Process substitution (seen in episode 2045)
- Word splitting (seen in episode 2045)
- Pathname expansion (the previous episode 2278 and this one)
This is the last topic in the (sub-) series about expansion in Bash.
In this episode we will look at extended pattern matching as also defined in the “Manual Page Extracts” section at the end of the long notes.
Long Show Notes
I have written out a moderately long set of notes about this subject and these are available here.
Links
- Other HPR series referenced:
- Wikipedia article on glob patterns
- Advanced Bash Scripting Guide: “Globbing”
- Article on Greg’s Wiki entitled “Globs”
- Questions about Bash extended globbing on Stack Exchange:
Show Transcript
Automatically generated using whisper
whisper --model tiny --language en hpr2293.wav
Comments
Comment #1 posted on 2017-05-21T16:42:38Z by Jonathan Kulp
What about with SCP?
Hi Dave,
This was a really excellent show! Just the kind of stuff that I wish I had known for about the last 10 years. I tried something after listening to this that worked wonderfully as long I was long as I was just doing a list command, but when I tried the same thing using secure copy to get the same list of arguments, it didn't work.
What I wanted to do was to push all MP3 and OGG files in a given directory over to my server in a single command, excluding the HTML and markdown files in the same directory.
The following command worked perfectly to **list** all of the MP3s and OGGs: "ls *(*.mp3|*.ogg)", but when I tried the same arguments with SCP it failed. Have you tried doing these kinds of expansions with secure copy?
Comment #2 posted on 2017-05-21T18:08:45Z by Dave Morriss
SCP is a bit weird
Hi Jon,
Glad you enjoyed the show.
To look into this I did the usual thing of creating a directory 'scptest' and making files in it:
$ touch scptest/{a,b}{00..10}{a..z}.{dat,txt}
I could copy selected files TO a remote directory:
$ scp scptest/*(???y.txt|???y.dat) dave@rpi5:test/
However, on rpi5 I couldn't copy FROM the other machine.
I did find a solution, but it's quite long for a comment, and I'm not 100% sure I understand it. I tried it and it did copy the files I specified.
Ken would suggest a show on the subject, but perhaps if I pointed you to the link I found it might do the job :-)
https://unix.stackexchange.com/questions/103058/exclude-characters-for-scp-filepattern
See what you think.
Actually, I shunt files around a lot between systems, but I often tend to use 'rsync'. However, that's a whole other subject.
Comment #3 posted on 2017-05-22T08:44:32Z by Dave Morriss
SCP without extended globs
It didn't occur to me at the time to try this (on rpi5, pulling files off a remote machine):
$ scp dave@desktop:'scptest/a??w.{txt,dat}' .
dave@desktop's password: ...
This works. The quotes prevent there being two scp invocations with associated password prompts.
Comment #4 posted on 2017-05-22T11:12:48Z by Jonathan Kulp
Details, details...
Aha! As always the devil is in the details. This last comment you left gave me the hint I needed. I was putting the asterisk in the wrong place and also using parentheses instead of curly braces. The following command works just like I want:
scp *.{mp3,ogg}
jonserver:~/destination/dir/
Thanks, Dave!
Comment #5 posted on 2017-05-22T11:29:36Z by Dave Morriss
TMTOWTDI
"There's more than one way to do it" - Larry Wall (actually he was talking about Perl, but it works here)
I'm glad I helped you to get where you wanted to be (even though I realise I wasn't quite answering your original question).
Having researched this and thought about it a bit I started putting together a brief(ish) HPR show about it.
Comment #6 posted on 2017-05-22T12:26:21Z by Jonathan Kulp
Ken is smiling
Dave, somehow I suspected when I asked whether this worked with secure copy that it would end up becoming another show from you. You're the best!
Comment #7 posted on 2017-05-24T04:52:46Z by clacke
scp brace expansion??!
Dave, I had no idea scp would do brace expansion on the server side. That's quite unexpected, and quite the discovery!
I tried it with rsync, and rsync *also* supports it. That made me suspicious.
As I feared, it seems to mean it runs the server side of rsync (and of scp) through the shell. Testing confirms it. If I want to (explicitly) copy files with spaces in the names, the quoting nightmare starts. :-(
So what started out as a happy discovery just turned into another disappointment in how broken our software is.
All this time, I had assumed that rsync started a server with no specific arguments, and then communicated over the rsync protocol which files to get. I guess I've been lucky all these years and avoided explicitly naming weird file names.
Comment #8 posted on 2017-05-24T10:09:52Z by Dave Morriss
scp is a bit of a hack!!
Hi clacke.
I had been messing with scp using the -v option which generates a lot of information. You can see it connecting via ssh then, if the original command was:
scp -v dave@rpi4:'scptest/*.{mp3,ogg}' .
it sends across:
scp -v -f scptest/*.{mp3,ogg}
where the quotes prevent local expansion and the (undocumented) -f option in the command marks it as running on the remote end.
I don't know how this mechanism deals with names with spaces and so forth, but I imagine it's nasty. It needs some experiments. My ideas for a "brief" show about this subject look doomed to be "l o n g" :-)
I think rsync will talk to a remote server (never used it) but I'd expect it would need to exist before the transfer.
<< First, < Previous, Next >, Latest >>
Leave Comment
Note to Verbose Commenters
If you can't fit everything you want to say in the comment below then you really should record a response show instead.
Note to Spammers
All comments are moderated. All links are checked by humans. We strip out all html. Feel free to record a show about yourself, or your industry, or any other topic we may find interesting. We also check shows for spam :).