Site Map - skip to main content

Hacker Public Radio

Your ideas, projects, opinions - podcasted.

New episodes Monday through Friday.


hpr3423 :: "upg.sh" my "dump.txt" to "note.md"

I upgraded my scripts.

<< First, < Previous, Latest >>

Hosted by Some Guy On The Internet on 2021-09-15 is flagged as Clean and is released under a CC-BY-SA license.
Tags: Bash Scripting, sed, awk, xargs, markdown, notes.
Listen in ogg, spx, or mp3 format. | Comments (1)

upg.sh my dump.txt to note.md

SYNOPSIS: upg.sh

  • Upgrade your system and store stdout into a markdown file.
#!/bin/bash
# upg.sh

FILENAME=sys-upgrade$(date +%m-%d-%Y).md
DIRECTORY="${HOME}/Documents/"

# step 1: formatting.
echo -e "# **System Upgrade:** $(date)\n" \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "**Command:** \`sudo apt-get update; sudo apt-get upgrade --yes\`\n" \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "**Command Breakdown:**" \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "- \`sudo\`, Admin Privilages." \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "- \`apt-get\`, Package Manager." \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "- \`update;\`, Package Manager's task; update the system software repositories." \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "- \`sudo apt-get upgrade\`, Perform system upgrade with updated repositories." \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "- \`--yes\`, Answers yes to the prompt." \
    | tee -a ${DIRECTORY}${FILENAME}

# step 2: run commands with formatting.
echo -e "\n**Command std-output:**\n" \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "\`\`\`" \
    | tee -a ${DIRECTORY}${FILENAME}
    echo $(date) \
    | tee -a ${DIRECTORY}${FILENAME}

sudo apt-get update \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "\n# System update completed.\n" \
    | tee -a ${DIRECTORY}${FILENAME}

sudo apt-get upgrade --yes \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "\n# System upgrade completed.\n" \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "\`\`\`\n" \
    | tee -a ${DIRECTORY}${FILENAME}

# step 3: additional details with more formatting.
echo -e "**Upgraded Package Details:**\n" \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "\`\`\`" \
    | tee -a ${DIRECTORY}${FILENAME}

PKGLIST=$(sed -n "/The following packages will be upgraded:/,/^.. upgraded/p" ${FILENAME} \
    | sed '1d;$d' | xargs -n 1 | sed '/:i386$/d') \

PKGCACHE=$(echo -e "${PKGLIST}\n" \
    | xargs -n1 -I _ apt-cache search _)
echo "${PKGCACHE}" > ${DIRECTORY}delete.txt

echo "${PKGLIST}" \
    | xargs -n 1 -I _ echo "sed -n '/^_ /p'" "${DIRECTORY}delete.txt" \
    | bash | tee -a ${DIRECTORY}${FILENAME};

echo -e "\`\`\`" \
    | tee -a ${DIRECTORY}${FILENAME}

rm -v ${DIRECTORY}delete.txt;
PKGLIST=
PKGCACHE=

# step 4: place EOF (end of file).
    sed -i '/EOF/d' ${DIRECTORY}${FILENAME}
echo "EOF" >> ${DIRECTORY}${FILENAME}
#EOF

Script breakdown: upg.sh

  • First, we declare bash as our shell with #!/bin/bash. We could also use #!/bin/sh for a more portable script.

  • I like to paste the name of the script we're working on into the script itself # upg.sh.

  • Setup a couple of variables to shorten the syntax.

FILENAME=sys-upgrade$(date +%m-%d-%Y).md
DIRECTORY="${HOME}/Documents/"
  • # step 1: formatting.
    • Build labels and a short breakdown of the update/upgrade commands used.
echo -e "# **System Upgrade:** $(date)\n" \                                                    <-- formatting: label with date.
    | tee -a ${DIRECTORY}${FILENAME}                                                           <-- path/to/file
echo -e "**Command:** \`sudo apt-get update; sudo apt-get upgrade --yes\`\n" \                 <-- formatting: command label.
    | tee -a ${DIRECTORY}${FILENAME}                                                           <-- path/to/file
echo -e "**Command Breakdown:**" \                                                             <-- formatting: label.
    | tee -a ${DIRECTORY}${FILENAME}                                                           <-- path/to/file
echo -e "- \`sudo\`, Admin Privilages." \                                                      <-- formatting: label.
    | tee -a ${DIRECTORY}${FILENAME}                                                           <-- path/to/file
echo -e "- \`apt-get\`, Package Manager." \                                                    <-- formatting: label.
    | tee -a ${DIRECTORY}${FILENAME}                                                           <-- path/to/file
echo -e "- \`update;\`, Package Manager's task; update the system software repositories." \    <-- formatting: label.
    | tee -a ${DIRECTORY}${FILENAME}                                                           <-- path/to/file
echo -e "- \`sudo apt-get upgrade\`, Perform system upgrade with updated repositories." \      <-- formatting: label.
    | tee -a ${DIRECTORY}${FILENAME}                                                           <-- path/to/file
echo -e "- \`--yes\`, Answers yes to the prompt." \                                            <-- formatting: label.
    | tee -a ${DIRECTORY}${FILENAME}                                                           <-- path/to/file
  • # step 2: run commands with formatting.,
    • Setup labels and an area for the stdout to be store with markdown formatting.
    • We place the time and date into the stdout area then run the commands.
echo -e "\n**Command std-output:**\n" \                                                        <-- formatting: label.
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "\`\`\`" \                                                                             <-- formatting: markdown.
    | tee -a ${DIRECTORY}${FILENAME}
    echo $(date) \                                                                             <-- command: date.
    | tee -a ${DIRECTORY}${FILENAME}

sudo apt-get update \                                                                          <-- command: update.
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "\n# System update completed.\n" \                                                     <-- formatting: label.
    | tee -a ${DIRECTORY}${FILENAME}

sudo apt-get upgrade --yes \                                                                   <-- command: upgrade with "--yes" option.
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "\n# System upgrade completed.\n" \                                                    <-- formatting: label.
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "\`\`\`\n" \                                                                           <-- formatting: markdown.
    | tee -a ${DIRECTORY}${FILENAME}
  • # step 3: additional details with more formatting.,
    • List the packages that were upgraded with details from system cache.
echo -e "**Upgraded Package Details:**\n" \                                                    <-- formatting: label.
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "\`\`\`" \                                                                             <-- formatting: markdown.
    | tee -a ${DIRECTORY}${FILENAME}

PKGLIST=$(sed -n "/The following packages will be upgraded:/,/^.. upgraded/p" ${DIRECTORY}${FILENAME} \  <--| variable with list of packages within it.
    | sed '1d;$d' | xargs -n 1 | sed '/:i386$/d') \                                            <--| sed: filter the first and last lines then remove the :i386 duplicate packages.

PKGCACHE=$(echo -e "${PKGLIST}\n" \                                                            <--| variable with massive apt-cache search results.
    | xargs -n1 -I _ apt-cache search _)                                                       <--| xargs runs the PKGLIST (the _ is the value of PKGLIST) into the apt-cache search.
echo "${PKGCACHE}" > ${DIRECTORY}delete.txt                                                    <--| I had to put the PKGCACHE in a file. I couldn't get sed to filter a variable (yet).

echo "${PKGLIST}" \                                                                            <--| use that PKGLIST to create a few sed commands to filter the file called "delete.txt".
    | xargs -n 1 -I _ echo "sed -n '/^_ /p'" "${DIRECTORY}delete.txt" \                        ^--| xargs is used to create the sed commands.
    | bash | tee -a ${DIRECTORY}${FILENAME};                                                   <--| run the sed commands through bash then store them.

echo -e "\`\`\`" \
    | tee -a ${DIRECTORY}${FILENAME}

rm -v ${DIRECTORY}delete.txt;                                                                  <--| use rm to delete the file called "delete.txt" it has the apt-cache search results in it.
PKGLIST=                                                                                       <--| empty the variable. why? why not!
PKGCACHE=                                                                                      <--| empty the variable. why? why not!
  • # step 4: place EOF (end of file).,
    • Add EOF (END OF FILE) to the end of the file. If one is already there, -
    • it's removed then replaced in the correct position.
    sed -i '/EOF/d' ${DIRECTORY}${FILENAME}                                                    <--| search for EOF then remove it. we don't want multiple EOF if we run the script multiple times in the same day.
echo "EOF" >> ${DIRECTORY}${FILENAME}                                                          ^--| adds the EOF (End Of File) at the end of the file.  I read it was a nice thing to do.
#EOF                                                                                           <--| Yep. it's there.


SYNOPSIS: note.sh "command" "filename"

  • example: note.sh "ls -lhA" "basic-list"
  • make markdown notes of your commands.
#!/bin/bash
# note.sh "command" "filename" no extentions.

# variables
FILENAME=$2$(date +%m-%d-%Y).md
DIRECTORY="${HOME}/Documents/"

# step 1: create file with formatting.
echo -e "# **Command:** \` $1 \`\n" \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "**Command Breakdown:**" \
    | tee -a ${DIRECTORY}${FILENAME}
echo "$1" | tr " " '\n' \
    | awk '{ print "- `" $0 "`, info." }' \
    | tee -a ${DIRECTORY}${FILENAME}

# step 2: run command with more formatting.
echo -e "\n**Command std-output:**" \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "\`\`\`\n$(date)" \
    | tee -a ${DIRECTORY}${FILENAME}

echo $1 | bash \
    | tee -a ${DIRECTORY}${FILENAME}

echo -e "\`\`\`" \
    | tee -a ${DIRECTORY}${FILENAME}

echo -ne "\n${FILENAME} has been updated $(date)."

# step 3: insert EOF (End Of File).
sed -i '/EOF/d' ${DIRECTORY}${FILENAME}
echo EOF >> ${DIRECTORY}${FILENAME}

Script breakdown: upg.sh

  • First, we declare bash as our shell with #!/bin/bash. We could also use #!/bin/sh for a more portable script.

  • I like to paste the name of the script we're working on into the script itself # upg.sh.

  • Setup a couple of variables to shorten the syntax.

FILENAME=$2$(date +%m-%d-%Y).md                                                                <--| the "$2" is the second user input (file name) from the commandline.
DIRECTORY="${HOME}/Documents/"
  • # step 1: create file with formatting.
    • Build labels for Command Name with a short breakdown of the command(s) used.
    • Note: the breakdown must be entered manually.
echo -e "# **Command:** \` $1 \`\n" \                                                          <--| the "$1" is the first user input (the command) from the commandline.
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "**Command Breakdown:**" \
    | tee -a ${DIRECTORY}${FILENAME}
echo "$1" | tr " " '\n' \                                                                      <--| This just breaks the command into parts then adds some markdown formatting for use to add -
    | awk '{ print "- `" $0 "`, info." }' \                                                    ^--| details to later. I just added the word info so you know to provide info about the command. -
    | tee -a ${DIRECTORY}${FILENAME}                                                           ^--| the formatting gets a bit crazy if you use something like: awk {' print $1 $2 $3 '} path/to/file;  each space becomes a newline with the markdown formatting.
  • # step 2: run command with more formatting.
    • Echo the Command into bash with markdown formatting for stdout.
echo -e "\n**Command std-output:**" \
    | tee -a ${DIRECTORY}${FILENAME}
echo -e "\`\`\`\n$(date)" \
    | tee -a ${DIRECTORY}${FILENAME}

echo $1 | bash \
    | tee -a ${DIRECTORY}${FILENAME}

echo -e "\`\`\`" \
    | tee -a ${DIRECTORY}${FILENAME}

echo -ne "\n${FILENAME} has been updated $(date)."
  • # step 3: insert EOF (End Of File).
    • Add EOF (END OF FILE) to the end of the file. If one is already there, -
    • it's removed then replaced in the correct position.
sed -i '/EOF/d' ${DIRECTORY}${FILENAME}
echo EOF >> ${DIRECTORY}${FILENAME}

Correspondent: Some Guy On The Internet.
Host ID: 391
E-mail: Lyunpaw.nospam@nospam.gmail.com


Comments

Subscribe to the comments RSS feed.

Comment #1 posted on 2021-09-15T21:23:22Z by Trey

Great work

Thank you for sharing episodes like these. Not only is the information you present valuable, but sharing your thought process helps provide context, as well as a launch point for others to build on.

I encourage you to keep it up, and start using git to manage and share your code and comments.

PS. I still dispise markdown. :)

<< First, < Previous, Latest >>

Leave Comment

Note to Verbose Commenters
If you can't fit everything you want to say in the comment below then you really should record a response show instead.

Note to Spammers
All comments are moderated. All links are checked by humans. We strip out all html. Feel free to record a show about yourself, or your industry, or any other topic we may find interesting. We also check shows for spam :).

Provide feedback
Your Name/Handle:
Title:
Comment:
Anti Spam Question: What does the P in HPR stand for ?
Are you a spammer →
Who hosted this show →
What does HPR mean to you ?