OPS145 Lab 6: Difference between revisions

From Littlesvr Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:
=TRANSFERRING FILES BETWEEN COMPUTERS=
= Special characters in the shell =
<br>
The terminal you've been using this whole time in the course is running an application called the shell. Specifically the bash shell.
===Main Objectives of this Practice Tutorial===


:* List '''common utilities''' contained in the '''ssh''' application framework
This application (bash) has been written to interpret input from the user as some sort of a command; or as an argument; or as data. It's a very powerful application with many abilities, but almost all the user input comes from the keyboard.


:* Securely '''copy''' files between Unix/Linux servers using the '''scp''' command
There are only so many keys of the keyboard, and most of them are used to input human-readable characters. That means some of those characters will necessarily have multiple meanings. We will look at several such characters in this lab.


:* Securely '''transfer''' copies of files between Unix/Linux servers using the '''sftp''' command
Here's most of the ASCII table (as much as I could fit on my screen). Note that most of the characters in this table are familiar to you, but some (especially the first 32) are probably brand new to you. You can see this table easily by running '''man ascii''':
[[File:LongerManAscii.png|center]]


:* Use the '''ssh''' command to run  and view commands on a '''remote computer''' from a '''local computer'''.
== . and .. ==
Coming from the Windows world you might be used to the concepts of "filename" and "extension". If you're not too familiar with that: you should configure your windows to always show file extensions (it doesn't by default).


:* Use the '''mail''' command to send email with '''file attachments''' to your Seneca email account
For example if you have a picture file called '''smily.jpg''': You might call '''smily''' the filename, and '''jpg''' the extension. But then what is the dot? You could call it a separator, but In fact the dot is just as much part of the filename as smily and jpg. Looking at the ASCII table: '''s''' and '''m''' are just as different from each other as '''s''' and '''.'''  
<br>


===Tutorial Reference Material===
And what's the extension in a file called '''lab1.tar.xz'''? Is it '''xz'''? Or '''tar.xz'''? What about in a file called '''Dr. Evil plan.txt'''?


{|width="100%" cellspacing="0" cellpadding="10"
When it comes to filenames and extensions: the use of a dot as a separator is just a convention to help people organize their files, and is barely useful from a technological point of view.


|- valign="top"
But there's another use of dots in filesystems, which is much more fundamental. Every directory contains at least these two records: a '''link to itself (.)''', and a '''link to its parent (..)'''


|colspan="2" style="font-size:16px;font-weight:bold;border-bottom: thin solid black;border-spacing:0px;"|Course Notes<br>
You can see these records when you run '''ls''' with the '''-a''' argument. Let's look at the SampleFiles directory you downloaded back in [[OPS145 Lab 2#ls -l -h|Lab 2]]:  


|colspan="2" style="font-size:16px;font-weight:bold;border-bottom: thin solid black;border-spacing:0px;padding-left:15px;"|Definitions / Commands<br>
* Open a terminal, and change the PWD to ~/Downloads/SampleFiles
* Run '''ls -a''' Note that in the output there is a '''.''' and a '''..'''
* Run '''ls -a -l''' Note that the '''.''' and '''..''' are directories.


|colspan="1" style="font-size:16px;font-weight:bold;border-bottom: thin solid black;border-spacing:0px;padding-left:15px;"|YouTube Videos<br>
[[File:Ls-al.png|center]]


|- valign="top" style="padding-left:15px;"
* Remember that you can give an argument to '''ls''' telling it to show the details of a specific file or directory. For example '''ls -l 1984.txt''' will show details about that specific file, and '''ls -l /''' will show details for the immediate contents of the root directory.
* Run '''ls -l -a .''' (including the dot in the end) and note that it shows the same output as '''ls -l -a''' That's because:
** '''ls''' will show the contents of PWD if you don't give it a specific path.
** Your PWD is ~/Downloads/SampleFiles/
** The '''.''' in SampleFiles points to itself (SampleFiles)
* Run '''ls -l -a ..''' and '''ls -l -a ~/Downloads''' Note that the output is the same. That's because:
** Your PWD is ~/Downloads/SampleFiles/
** The parent directory of ~/Downloads/SampleFiles/ is ~/Downloads/
** Inside ~/Downloads/SampleFiles/ '''..''' and '''~/Downloads''' are the same thing
* The same for every other directory. The only sort-of exception is the root directory. Because it's the root and it doesn't have a parent: its parent is itself:
[[File:FilesystemIntroWithDots.png|center]]


|colspan="2" |'''Slides:'''<ul><li>Week 6 Lecture 1 Notes:<br> [https://wiki.cdot.senecacollege.ca/uli101/slides/ULI101-6.1.pdf PDF] | [https://wiki.cdot.senecacollege.ca/uli101/slides/ULI101-6.1.pptx PPTX]</li></ul>
* Take some time to look at these many lines and understand that actually it's not that complicated. You just have to remember: dot is for itself, dot-dot is for its parent.


|  style="padding-left:15px;" |'''Definitions:'''
== Hidden files ==
* [https://en.wikipedia.org/wiki/Secure_copy Secure Copy]
Dots have another (mostly unrelated) special use when it comes to filesystems in the shell. Any file/directory name which begins with a dot is considered '''hidden'''.
* [https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol Secure File Transfer Protocol]
* [https://en.wikipedia.org/wiki/Mail_(Unix) Email]<br><br>


|  style="padding-left:15px;"|'''File Transfer Commands:'''
This is '''definitely not a security feature!''' It is purely a convenience.
* [http://man7.org/linux/man-pages/man1/scp.1.html scp]
* [http://man7.org/linux/man-pages/man1/sftp.1.html sftp]
* [https://man.openbsd.org/mail mail]


|colspan="1" style="padding-left:15px;" width="30%"|'''Instructional Videos:'''<ul><li>[https://www.youtube.com/watch?v=EBvxbuhDKBw Using scp Command]</li><li>[https://www.youtube.com/watch?v=QcKYFEgfV-I Using sftp Command]</li><li>[https://www.youtube.com/watch?v=Iuf_mLrvh2o Using the mail Command]</li></ul>
* Change your PWD to your home directory.
|}
* Run '''ls'''
* Run '''ls -a'''


= KEY CONCEPTS =
Notice there are many more files and directories in your home than you saw before. These were always there, the '''ls''' command doesn't show them by default. The '''-a''' argument tells '''ls''' to also show hidden files.


The '''ssh''' Linux command is a '''suite of tools''' to allow the user to issue Linux commands securely between <br>Unix / Linux servers, as well as securely '''copy''' and '''transfer''' files among Unix/Linux servers.
These are hidden to avoid clutter, to make it easier for you to find things you want. There's nothing special about these files and directories. They're just rarely manipulated by users.


In this tutorial, you will learn several different methods to securely transfer files from your Matrix Linux account<br>to other computers using Linux commands including '''scp''', '''sftp''' and '''mail'''.
This is also where the "extension" idea breaks down completely. If you insist that a file's extension is everything after the dot and the file's name is everything before the dot: all these hidden files have no names, and only extensions. That's clearly not the case. Almost all these files are plain text files, without a .txt extension.


* Use the cat command to look at the contents of .bash_history. This file is where the commands you ran previously are stored, so you can retrieve them using the arrow keys.
* Since you're at it: here's a new command. Run '''history''' on the terminal. It will show you all the previous commands you've run.


===Issuing Commands on Remote Unix/Linux Servers===
== Comments ==
[[Image:ssh-remote.png|thumb|right|350px|The '''ssh''' command can be used to <u>run</u> and <u>view</u> commands on '''remote computer''' from a '''local computer'''.
*Now that you're more comfortable with the command-line: download and extract the next tarball using commands instead of Firefox and a graphical archive manager:<syntaxhighlight lang="bash">
]]
cd ~/Downloads # To make sure it's downloaded here
You can use the '''ssh''' command to issue Unix/Linux commands on a '''remote''' server<br>from your <u>local</u> computer without logging into a remote server (such as Matrix).
ls -l -h # To confirm you didn't already download SpecialChars.tar.xz
wget http://ops345.ca/ops145/SpecialChars.tar.xz # This will download SpecialChars.tar.xz into the PWD
tar xvf SpecialChars.tar.xz # This will extract the contents of SpecialChars.tar.xz into the PWD
</syntaxhighlight>
In the block of commands above there are words which are clearly not intended to be executed by the shell (it's only so smart). They are for human beings to read. But you can run all the above without any difficulty because anything following the special character '''#''' is treated as a comment, meaning the shell ignores it no matter what it is.


The '''#''' is usually called a pound sign, or a number sign, or a hash.


''Command Usage:''
== The * wildcard ==


<span style="font-family:courier">'''ssh username@matrix.senecacollege.ca ls -l'''</span>
You've used the '''*''' wildcard (called a star, it's the multiplication sign on your keyboard) in [[OPS145 Lab 3#Deleting files|lab 3]] to delete several files at the same time. Now we'll look at it in more detail.


You will be prompted for your Matrix account password, then the contents of your home directory in your remote Matrix account will be displayed on your <u>local</u> computer's terminal.
Sometimes you want to do something will all the contents of a directory. At other times you might want to do something with all your .jpg files, or all the .txt files, or all the files which have backup in their name. Those are examples of when you use a '''*'''
<br><br>


===Secure Copy (scp)===
Inside the newly created ~/Downloads/SpecialChars/ directory is a '''SortMe''' subdirectory, which is a disorganized mess. We'll organize it in this lab. Remember that the whole point is to learn wildcards, so use commands in a terminal to move files around instead of dragging them in the graphical interface.


[[Image:scp-diagram.png|thumb|right|350px|The '''scp''' Unix/Linux command is used to securely copy files between Unix/Linux servers.]]
* Use the graphical file manager to look at the SortMe directory to get an idea of what's in there. Note that:
The '''scp''' command is used to <u>securely</u> copy files between your '''local''' computer and '''remote''' Unix/Linux server. The usage for the ''scp'' command is similar to the '''cp''' command with the addition of '''user name''' and '''host name'''.
** There are pictures and sounds
** Some of the pictures are small squares, and others are large photos
** Some of the sounds are .wav files and others are .oga files
** Some of the .wav files sound like speaker test sounds
* We're going to organize these into the following directory tree inside '''~/lab6''':


[[File:SortMeTree.png|center]]


''Command Usage:''
* Remember that filenames are case-sensitive. Open a terminal and create the directory tree above using as many commands as you like. But here's a handy argument you can give to the mkdir command: '''-p'''<syntaxhighlight lang="bash">
mkdir -p ~/lab6/sounds/WAV/SpeakerTest
</syntaxhighlight>When run with -p, the mkdir command will create not only the last directory on the path you give it, but all the other missing directories in that path as well. You might want to avoid using this if you're not very comfortable with the command-line yet, since it's very easy to create a hundred directories in the wrong place by giving '''mkdir -p''' the wrong path several times.
* Change your PWD to '''~/Downloads/SpecialChars/SortMe'''


<span style="font-family:courier">'''scp local.file username@host:destination-pathname'''<br>'''scp local.file username@host:'''<br>'''scp user@host:file-pathname local-pathname'''<br><br></span>
When the bash shell encounters a * character on the command-line: it assumes you're trying to use it as a wildcard for filenames, and does its best to replace the * with a list of all filenames which match the pattern.


The most common '''mistake''' that students make is forgetting to add<br>the '''colon''' character ":" <u>after</u> the remote hostname.
On its own: a * matches any number of any characters. We can test this easily using the ls command:<syntaxhighlight lang="bash">
ls *
</syntaxhighlight>This appears to print the same output as ls on its own, but in fact the ls command received from the shell a list of all the files in ~/Downloads/SpecialChars/SortMe as arguments.


The user name in the command can be '''omitted''' if it's the  same as on the <u>local</u> host.<br>Multiple file and recursive directory copy (i.e. option '''-r''') is supported.
You can combine the * wildcard with other characters, restricting what it will be expanded to. For example:<syntaxhighlight lang="bash">
<br><br>
ls *jpg
</syntaxhighlight>This time * still expands to anu number of any characters, but because it's combined with an explicit '''jpg''' in the end: the result will be a list of all the files which have jpg at the end of their name.


===Secure File Transmission Control Protocol (sftp)===
* Try running commands similar to the above with different extensions.


[[Image:sftp-diagram.png|thumb|right|350px|The '''sftp''' Unix/Linux command is used to securely transfer (copy) files between Unix/Linux servers.]]
The wildcard doesn't have to be at the beginning. It can be in the end. For example this is how you would get a list of all the files which have names starting with amanea:<syntaxhighlight lang="bash">
ls amanea*
</syntaxhighlight>The wildcard can be in the middle too. For example this is how you would get a list of all the files with _ somewhere in the name:<syntaxhighlight lang="bash">
ls *_*
</syntaxhighlight>Or a list of filenames which contain an _ and end with wav:<syntaxhighlight lang="bash">
ls *_*wav
</syntaxhighlight>


'''FTP''' stands for '''File Transfer Protocol''' which provides a set of '''rules''' on how to convert data that is transferred between computers (both identical and different operating systems). The '''sftp''' command performs file transfers <u>securely</u> using encryption.
The use of wildcards is completely unrelated to the ls command; ls just happens to be an easy program to show what the wildcard expands to.You can use it with equal effectiveness with other commands, like cp.If we did't have a graphical interface available and we wanted to copy all the jpg files to the ~/lab6/pictures directory you created earlier: that would involve a lot of typing, even with tab completion. Instead we can copy all of them quickly.


* First run '''ls''' and '''ls *jpg''' to confirm that *jpg expands to what you want (in this case all the jpg files). You should notice that it doesn't. There are two jpg files which have their '''JPG''' in uppercase.
* If you want to include those as well: use two arguments (bash will actually convert your two arguments to 26 arguments - all the jpg and JPG filenames)<syntaxhighlight lang="bash">ls *jpg *JPG</syntaxhighlight>


''Command Usage:''
* If you're satisfied with the list of filenames you're getting: you can use the same list with the cp command. The cp command is potentially distructive, since it will overwrite files by default. So think twice before you run it with unknown arguments.
* Also give cp the extra '''-v''' (verbose) argument so that it will print details about what it's doing:<syntaxhighlight lang="bash">
cp -v *jpg *JPG ~/lab6/pictures/
</syntaxhighlight>
* Run a command to copy all the '''wav files''' to '''~/lab6/sounds/WAV/'''
* Run a command to copy all the '''oga files''' to '''~/lab6/sounds/OGA'''
* Change your working directory to '''~/lab6/sounds/WAV'''


'''<span style="font-family:courier">sftp username@hostname</span>'''
We want to move the files Front_Right.wav, Rear_Center.wav, Side_Left.wav, Front_Center.wav, Rear_Left.wav, Side_Right.wav, Front_Left.wav, and Rear_Right.wav to the SpeakerTest subdirectory without typing in all those names. Luckily (would you believe it) these specific files have a specific filename pattern which makes it different from all the other files in sounds/: they all contain an underscore.


* Use ls to check that the *_* pattern will include all the speaker test files, and only the speaker test files:<syntaxhighlight lang="bash">
ls *_*
</syntaxhighlight>
* If it looks good: use that argument with the mv command. Note that mv also accepts a -v (verbose) argument:<syntaxhighlight lang="bash">
mv -v *_* SpeakerTest
</syntaxhighlight>
You should end up with the following arrangement in your sounds directory:
[[File:TreeOfSounds.png|center]]


When you login via the ''sftp'' command, the '''sftp prompt''' appears. The sftp prompt is like a Bash shell prompt, but with a limited number of commands. When issuing sftp commands, the local server relates to the server where you first issued the sftp command. Refer to the diagram on the right for <u>local</u> and <u>remote</u> ''sftp'' commands.
== The ? wildcard ==
[[Image:graphical-sftp-application.png|thumb|right|250px|'''Graphical''' SFTP application.]]
Like the '''*''' wildcard: the '''?''' wildcard stands for any character. But unlike the *: the ? stands for exactly one character.


To illustrate we're going to move all the small square shaped images into a new subdirectory.


'''<big>Graphical SFTP Applications</big>'''
* Create the '''~/lab6/pictures/square''' directory
* In ~/lab6/pictures: try '''ls *_*'''


Although it is important to know how to use the '''sftp''' command for ''quizzes'', ''midterm'' and ''final exam'',<br>there are '''graphical sftp applications''' that provide an alternative to issuing commands.
* Note that it shows all the files. That's because all of them contain an underscore. But we want only the filenames which begin with a single digit, then have an underscore, and then anything else. Use a ? to get that list:<syntaxhighlight lang="bash">ls ?_*</syntaxhighlight>You could make it more specific by adding jpg in the end, but in this case it's not necessary.


If you installed the graphical '''Secure Shell Client''' application in your Windows computer from performing [https://wiki.cdot.senecacollege.ca/wiki/Tutorial_1:_Using_Your_Matrix_Server_Account#CONNECTING_TO_YOUR_MATRIX_ACCOUNT_FROM_A_GRAPHICAL_WINDOWS_APPLICATION TUTORIAL 1 INVESTIGATION 1], you can use this application to transfer files between your computer and your Matrix account by graphically '''navigating''', '''selecting''' and '''dragging''' files between computers.
* If the output looks right: move all the square images into the square directory:<syntaxhighlight lang="bash">mv ?_* square</syntaxhighlight>
<br>


===Sending Emails with File Attachment (mail)===
You should end up with this arrangement:
[[File:TreeOfImages.png|center]]


You can use the '''mail''' command in Matrix to send email messages<br>to other email accounts via the Internet.
== Spaces in filenames ==
At this point in computer history people are used to the idea that a space is nothing special in a filename. That's because most people use GUIs to manipulate files.


On the command-line a space is a very special character which is used all the time: it's the separator between the command and the arguments, and the separator between multiple arguments.


[[Image:mail-diagram.png|thumb|right|400px|Using the '''mail''' command with '''redirection''' to send email<br>with file attachment.]]
From the filesystem's point of view: there is nothing special about a space, it's just another valid character from the ASCII table. The problem is the interface between the user and the filesystem, in our case that's the shell.
'''Sending a Simple Email Message:'''


:#Type: '''<span style="font-family:courier;">mail username@hostname''' and press '''ENTER''' </span><br>
* Inside your downloaded and extracted SpecialChars directory is a '''Spaces''' directory. Copy that to '''~/lab6/'''
:#Enter '''subject line''' and press '''ENTER'''<br>
* If you look at the contents in a graphical interface: you'll see there really isn't anything special about those files and directories.
:#Type the '''body of the message''' and then when finished,<br>press <span style="font-family:courier">'''ctrl+d'''</span> to send message<br><br>
* In your terminal: change your PWD to '''~/lab6/Spaces/'''
* Run '''ls'''. Note that the names have single quotes around them. This is relatively new behaviour for bash, many systems won't add the quotes. Note also that the spaces between the filenames in the output are no different than the spaces inside the filenames.
* Try to see the contents of '''Important Dir''' using '''ls'''<syntaxhighlight lang="bash">ls Important Dir</syntaxhighlight>
It doesn't work because the shell thinks you mean that '''Important''' and '''Dir''' are separate filenames. To tell the shell you mean the space is part a part of Important Dir: you need to escape the space. There are three ways to do this:
*Put a '''\''' (backslash) before the special character
*Enclose the entire argument in double quotes
*Enclose the entire argument in single quotes


'''Sending an Email Message with a File Attachment:'''
* Try again to see the contents of Important Dir using ls, but this time escape the space:<syntaxhighlight lang="bash">ls Important\ Dir</syntaxhighlight>You may have already found that tab completion does this for you automatically.
[[Image:mail-diagram-2.png|thumb|right|275px|Viewing email with file attachment in<br>'''Seneca email account'''.]]
*If the shell encounters a quote (either double quote or single quote): it will assume everything from that quote until the next quote is one argument. For example, you can run:<syntaxhighlight lang="bash">ls "Important Dir"
:#Type: '''<span style="font-family:courier;">mail username@hostname -a filepathname</span>''' and press '''ENTER''' <br>
ls 'Important Dir'</syntaxhighlight>
:#Enter '''subject line''' and press '''ENTER'''<br>
*Both of those will work.
:#Type the '''body of the message''' and then when finished, press <span style="font-family:courier">'''ctrl+d'''</span> to send message<br><br>
Spaces in filenames are not impossible to deal with on the Linux command line, but they make simple things difficult, and complicated things really difficult.


'''Alternative Method of Sending an Email Message with a File Attachment:'''
== Escaping other special characters ==
Like spaces: other special characters can have their special status stripped by escaping them.


:#Type: '''<span style="font-family:courier;">mail -s "subject line" username@hostname < filepathname</span>'''
* Inside your downloaded and extracted SpecialChars directory is a '''WeirdNames''' directory. Copy that to '''~/lab6/'''
:#Press '''ENTER''' to send<br><br>'''NOTE:''' You would have to use this method since you have used '''stdin''' redirection<br>to attach the file’s so you can’t input the subject line from the terminal!<br><br>


=INVESTIGATION 1: FILE TRANSFER (SECURE COPY)=
Normally you would avoid creating such filenames if you're expecting to work with them in the command line, but this is a learning exercise.


<span style="color:red;">'''ATTENTION''': This online tutorial will be required to be completed by '''Friday in week 8 by midnight''' to obtain a grade of '''2%''' towards this course</span><br><br>
* Change your PWD to ~/lab6/WeirdNames
* Try to run an ls command escaping only the space in '''Andrew's stuff.txt''':


[[Image:cmd.png|thumb|right|350px|Make certain to '''open a command-line terminal''' and<br><u>'''NOT'''</u> a graphic SSH application for this tutorial!.]]
[[File:EscapeQuote.png|center]]
The '''SSH''' package on your ''home computer'' and on the ''Matrix Linux server''<br>contain a '''suite''' (i.e. collection) of secure utilities including '''ssh''' and '''scp'''.
The shell will give you this strange prompt-like behaviour, as if it's waiting for you to type something in. What it's waiting for is the closing single quote for what looked like an opening single quote.


* Press '''Ctrl+c''' to abort the command you were trying to run.
* Re-run the ls command, this time escaping not only the space but the single quote as well.
* Similar problems will the * special character. A * is just a character like any other in the ASCII table. It has special meaning in the shell but not on the filesystem. Try it:


In this investigation, you will learn how to use the '''scp''' command to '''securely copy files''' between your computer and your Matrix Linux server. This methods is useful because it can be performed in the ''MS-Windows'', ''MacOSx'', and ''Unix/Linux'' operating systems.<br><br>You will also learn how to issue the '''ssh''' command to run commands on your<br><u>'''remote'''</u> Matrix server while remaining on your <u>'''local'''</u> computer.
[[File:EscapeStar.png|center]]


= Some more work for you =
By this point in the lab you should have:


'''Perform the Following Steps:'''
* Downloaded and extracted SpecialChars.tar.xz
* Created the ~/lab6 directory and the subdirectories for sorting pictures and sounds
* Copied/moved specific pictures and sounds into appropriate directories
* Copied Spaces and WeirdNames to ~/lab6


# Determine which operating system that your computer is using.<br><br>
Do a few more things for practice with special characters on the command-line:
# Connect to your Matrix account using the instructions in the table below based on <u>your</u> '''current operating system'''.<br><br>
<table style="margin-left:50px;">
  <tr valign="top" style="text-align:center;">
    <th>'''Newer Version of Windows 10:'''</th>
    <th>'''MacOSX:''' </th>
    <th>'''Linux:''' </th>
  </tr>
  <tr>
<td>
*From the start menu, type '''cmd''' and launch program
*In the command terminal, enter the following command:<br><span style="color:blue;font-weight:bold;font-size:.75em;">ssh senecausername@matrix.senecacollege.ca</span>
    </td>
    <td>
*Click ''Launchpad'' icon, type '''terminal'''<br>and press '''ENTER'''
*In the terminal, enter the following command:<br><span style="color:blue;font-weight:bold;font-size:.75em;">ssh senecausername@matrix.senecacollege.ca</span>
    </td>
    <td>
*From the menu, choose:<br>'''Applications''' > '''System Tools''' > '''Terminal'''
*In the terminal, enter the following command:<br><span style="color:blue;font-weight:bold;font-size:.75em;">ssh senecausername@matrix.senecacollege.ca</span>
    </td>
  </tr>
</table><br>
<ol><li value="3">'''NOTE:''' Make certain to open a '''command-line terminal''' and <u>NOT</u> a <u>graphical</u> SSH application for this tutorial.<br><br></li><li>After logging into your Matrix account, issue to the '''pwd''' command to confirm you are in your home directory.<br><br></li><li>Issue the following Linux command to create the following directory:<br><span style="color:blue;font-weight:bold;font-family:courier;">mkdir ~/remote</span><br><br></li><li>Change to the <b>~/remote</b> directory and confirm that you have changed to that directory.<br><br></li><li>Use a text editor to create a text file called <b>myfile.txt</b><br><br></li><li>Enter the following two lines displayed below in your editing session:<br><span style="font-family:courier;font-weight:bold;">This is my file<br>It is a small file</span><br><br></li><li><b>Save</b> editing changes to your ''myfile.txt'' file and exit your text editor.<br><br></li><li>Let's run a <b>shell script</b> to check that you created the <b>remote</b> directory<br>and that you created the <b>myfile.txt</b> file (with correct file contents) in that directory.<br><br></li></li>Enter the following command: <span style="color:blue;font-weight:bold;font-family:courier;">~uli101/week6-check-1</span><br><br></li><li>If you encounter errors, make corrections and then re-run the checking script until you receive<br>a congratulations message, and proceed to the next step.<br><br><b>NOTE:</b> We will now learn to transfer files between your <b><u>local</u> home computer</b> and your <b><u>remote</u> Matrix Linux server</b>.<br><br></li></ol>[[Image:cmd.png|thumb|right|350px|You are required to <u>'''remain'''</u> in your '''local''' computer's command terminal for the remainder of this ''INVESTIGATION'' and ''INVESTIGATION 2'']]
<ol><li value="12">'''Exit your Matrix ssh session''' but '''<u>remain</u>''' in the ''command terminal''<br>on your '''local''' computer..<br><br><span style="color:blue;">'''ATTENTION:''' You are required to <u>'''remain'''</u> in your '''local''' computer's command terminal for the remainder of this ''INVESTIGATION'' and ''INVESTIGATION 2''.</span><br><br></li><li>The <b>mkdir</b> command works with <i>MS Windows/UNIX/Linux/MacOSx computers</i>.<br>Issue the following command on your <u>local</u> computer to<br>create a directory called <b>local</b>: <span style="color:blue;font-family:courier;font-weight:bold;">mkdir local</span><br><br></li><li>The <b>cd</b> command works with <i>MS Windows/UNIX/Linux/MacOSx computers</i>.<br>Issue the following command on your <u>local</u> computer to<br>change to the <b>local</b> directory: <span style="color:blue;font-family:courier;font-weight:bold;">cd local</span><br><br></li><li>If you are using MS Windows on your local computer, issue the <span style="color:blue;font-weight:bold;font-family:courier;">dir</span> command to confirm you are<br>in the ''local'' directory; otherwise, use the <span style="color:blue;font-weight:bold;font-family:courier;">pwd</span> command.<br><br></li> <li>If you are in MS Windows, open the '''GRAPHICAL''' <span style="color:blue;font-family:courier;font-weight:bold;">NotePad</span> application to create a text file<br>(Otherwise, use the <b>nano</b> or <b>vi</b> text editor).<br><br></li><li>Enter a few lines of text, and if using '''Notepad''', then click on the <b>File</b> menu and select <b>save as</b><br>(save as the filename <span style="color:blue;font-weight:bold;font-family:courier;">other.txt</span> in your <span style="color:blue;font-weight:bold;font-family:courier;">local</span> directory) and then <b>exit</b> the <i>Notepad</i> text editor.<br><br>'''NOTE:''' if using another text editor, save your editing session and exit the text editor.<br><br></li><li>If your OS is MS Windows issue the <span style="color:blue;font-family:courier;font-weight:bold;">dir</span> Windows command to view the contents of your current directory<br>(otherwise, issue the <span style="color:blue;font-family:courier;font-weight:bold;">ls</span> command for other operating systems).<br><br>We will use the '''scp''' command to copy the <u>local</u> file called '''other.txt''' to your home directory on your <u>remote</u> Matrix Linux server.<br><br></li><li>Issue the following Linux command to copy the <b>other.txt</b> file from your local machine to your remote Matrix server<br>(replace yoursenecaid is <u>YOUR</u> Seneca ID and '''ADD A COLON : TO THE END OF THE COMMAND'''):<br><span style="color:blue;font-weight:bold;font-family:courier;">scp other.txt yoursenecaid@matrix.senecacollege.ca:</span><br><br></li><li>When prompted, enter your Matrix '''password'''.<br>[[Image:scp-diagram-2.png|thumb|right|350px|You can issue the ssh command, followed by a '''command''' that will be run on your '''remote''' computer,<br>but <u>display</u> command output on your '''local''' computer.]]<br><b>TIP:</b> You can issue the ssh command, followed by a command that will be run on your remote computer, but display on your local computer <b><u>without</u></b> having to establish a continuous connection to your remote Matrix server.<br><br></li><li>Issue the following command (using your matrix username):<br><span style="color:blue;font-weight:bold;font-family:courier;">ssh yoursenecaid@matrix.senecacollege.ca ls -l other.txt</span><br><br></li><li>When prompted, '''enter your password''' and press '''ENTER'''.<br><br>Do you see detailed information <b>other.txt</b> file? (look at bottom)<br>That command was run remotely on your Matrix server as confirmation that you securely copied that file to the home directory of the Matrix server.<br><br>Let's copy the file called '''myfile.txt''' in the '''~/remote''' directory that you created earlier in your Matrix account to your <b>local</b> directory on your home computer.<br><br></li><li>Issue the following Linux command (replace yoursenecaid is <u>YOUR</u> Seneca ID).<br> The period "." as '''second argument''' represents your <u>current</u> directory<br>on your <u>local</u> computer):<br><span style="color:blue;font-weight:bold;font-family:courier;">scp  yoursenecaid@matrix.senecacollege.ca:remote/myfile.txt .</span><br><br></li><li>Issue the <span style="color:blue;font-weight:bold;font-family:courier;">dir</span> or <span style="color:blue;font-weight:bold;font-family:courier;">ls</span> command (depending on the OS of your <u>local</u> computer) to confirmed your properly copied that file from Matrix.<br><br></li><li>Use the '''Notepad''' application (or vi for other OS types) to create a text file called <span style="color:blue;font-weight:bold;font-family:courier;">mytextfile.txt</span>,<br>type some text and then save in the <span style="color:blue;font-weight:bold;font-family:courier;">local</span> directory of your computer.<br><br><li>Issue the <b>dir</b> or <b>ls</b> command (depending on your OS) to confirm that your newly-created file exists in your <b>local</b> directory.<br><br></li><li>We are going to intentionally make a '''<u>mistake</u>''' with the '''scp''' command.<br>Issue the following Linux command to copy the <b>mytextfile.txt</b> file from your local machine to your remote Matrix server<br>(replace yoursenecaid is <u>YOUR</u> Seneca ID and DO <u>NOT</u> INCLUDE THE : at the end of the command so see what happens):<br><span style="color:blue;font-weight:bold;font-family:courier;">scp mytextfile.txt yoursenecaid@matrix.senecacollege.ca</span><br><br>Did you notice anything different (i.e. no password)?<br><br></li><li>Issue the following command (using your matrix username):<br><span style="color:blue;font-weight:bold;font-family:courier;">ssh yoursenecaid@matrix.senecacollege.ca ls -l mytextfile.txt</span><br><br></li><li>When prompted, enter your password and press ENTER.<br><br><b>The file mytextfile.txt does NOT appear in your home directory on your Matrix server!<br>Note that the COLON was NOT added to the end of the command! Therefore, you MUST<br>remember to include the COLON : at the end of the hostname, or it will NOT remotely copy the file!</b><br><br></li><li>Issue the following command to properly copy that same file to your Matrix server:<br><span style="color:blue;font-weight:bold;font-family:courier;">scp mytextfile.txt yoursenecaid@matrix.senecacollege.ca:</span><br><br></li><li>Issue the following command to confirm that it was remotely copied to your <b>home</b> directory in Matrix:<br><span style="color:blue;font-weight:bold;font-family:courier;">ssh yoursenecaid@matrix.senecacollege.ca ls -l /home/yoursenecaid/mytextfile.txt</span><br><br>Do you see the output for the detailed file listing of <b>mytextfile.txt</b>?<br>What does  this indicate?<br><br><li>Issue the following command to copy the '''other.txt''' file on your local computer to the '''~/remote''' directory in Matrix renaming it as '''different.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">scp other.txt yoursenecaid@matrix.senecacollege.ca:remote/different.txt</span><br><br></li><li>Issue the following command to confirm that the file was remotely copied to your '''~/remote''' directory in Matrix with a different filename:<br><span style="color:blue;font-weight:bold;font-family:courier;">ssh yoursenecaid@matrix.senecacollege.ca ls -l /home/yoursenecaid/remote/different.txt</span><br><br>Were you able to properly copy this file?<br><br>Let's issue a checking script remotely to see that you properly copied that file from your<br>local computer to your remote Linux server to both your <b>home</b> directory and <b>~/remote</b> directory.<br><br></li><li>Issue the following:<br><span style="color:blue;font-weight:bold;font-family:courier;">ssh yoursenecaid@matrix.senecacollege.ca ~uli101/week6-check-2</span><br><br>If you encounter errors, re-run the scp commands to correct and re-run the above command until you receive a congratulations message.<br><br></li><li>Remain in the terminal on your <u>local</u> computer and proceed to ''INVESTIGATION 2''.<br><br></ol>
 
:In the next investigation, you will use the '''sftp''' Linux command to transfer (i.e. copy) files between your local computer and the Matrix server.<br><br>
 
<!-- RESIDUAL FROM LESS EFFECTIVE METHOD OF COPYING FILES BETWEEN SERVERS
 
<span style="color:blue;">'''ATTENTION:''' Since the only server account you have is your '''Matrix''' server,<br>we will simulate using the scp command to copy from your '''Matrix''' server to<br>'''<u>another instance</u> of your Matrix server'''.<br><br>To make this work, we will need to edit a "start-up" file to place a special series of commands.<br>You will learn more about start-up files later in this course.</span><br><br>
# Use a text editor to edit the following start-up file: '''<span style="font-family:courier">~/.bashrc</span>'''<br><br>
# '''Copy''' and '''paste''' the following text (shown below) at the '''TOP''' of your startup file:<br><span style="font-family:courier;font-weight:bold;">&#91;&#91; $- == *i* &#93;&#93; || return  </span><br><br>
# '''Save''' editing changes to this startup file and exit your text editor.<br><br> -->
 
=INVESTIGATION 2: FILE TRANSFER (SECURE FTP) =
 
The '''SSH''' package on your ''home computer'' and on the ''Matrix Linux server''<br>contain a '''suite''' (i.e. collection)of secure utilities including '''ssh''' and '''sftp'''.
 
 
In this investigation, you will learn how to use the '''sftp''' command to '''transfer''' files between<br>Unix/Linux servers. This methods is useful because it can be performed in the<br>''MS-Windows'', ''MacOSx'', and ''Unix/Linux'' operating systems.<br><br>You will also learn how to issue the '''ssh''' command to run commands on your<br><u>'''remote'''</u> Matrix server while remaining on your <u>'''local'''</u> computer.
 
 
=== Command Line Terminal (CLI) ===
 
Let's look at using the '''sftp''' command on your '''local''' machine.
 
 
'''Perform the Following Steps:'''
[[Image:cmd.png|thumb|right|275px|Make certain to <u>remain</u> in the<br>command-line terminal in your '''local''' computer.]]
# Make certain that you are in a command terminal on your <u>local</u> computer<br>(i.e. do '''NOT''' log into your Matrix account).<br><br>
# Issue a command (depending on your OS) to confirm that you are located in the '''local''' directory in your home computer.<br><br>
# If you are in MS Windows, open the <span style="color:blue;font-family:courier;font-weight:bold;">NotePad</span> application to create a text file<br>(otherwise: use another text editor like <b>vi</b> or <b>nano)</b><br>.
# Enter a few lines of text, and then click on the <b>File</b> menu and select <b>save as</b><br>(save as the filename <span style="color:blue;font-weight:bold;font-family:courier;">thefile.txt</span> in your <span style="color:blue;font-weight:bold;font-family:courier;">local</span> directory) and then <b>exit</b> the <i>Notepad</i> text editor.<br><br>If you using another OS, then save-as using the same filename and directory location for the text editor you are using.<br><br>
# If your OS is MS Windows issue the <span style="color:blue;font-family:courier;font-weight:bold;">dir</span> Windows command to view the contents of your current directory<br>(otherwise, issue the <b>ls</b> command for other operating systems).<br><br>'''Note:''' the <u>relative</u> pathname symbols "'''.'''" and "'''..'''" work for the ''Windows''/''MacOSx''/''Unix''/''Linux'' operating systems.<br><br>
# Issue the following command to move to the '''parent''' directory: <span style="color:blue;font-family:courier;font-weight:bold;">cd ..</span><br><br>
# If your OS is MS Windows issue the <span style="color:blue;font-family:courier;font-weight:bold;">dir</span> Windows command to view the contents of that parent directory that you changed to<br>(otherwise, issue the <b>ls</b> command for other operating systems).<br><br>
# Issue the following command to start an '''sftp''' session (note: yoursenecaid is YOUR Seneca ID):<br><span style="color:blue;font-weight:bold;font-family:courier;">sftp yoursenecaid@matrix.senecacollege.ca</span><br><br>'''NOTE:''' You may be required to enter '''yes''' to have the public key shared.<br><br>[[Image:sftp-commands.png|thumb|right|350px|Common '''sftp''' commands to  manage the transfer<br>of files between computers.]]
# You should be in the '''sftp command prompt''' where you are expected to issue '''sftp commands'''. Please take a moment to view common<br><u>local</u> and <u>remote</u> ''sftp commands'' on the right-side table.<br><br>
# Issue the following ''sftp command'': <span style="color:blue;font-weight:bold;font-family:courier;">pwd</span><br><br>What is the pathname? Which server does this represent: local or remote?<br><br>
# Issue the following ''sftp command'': <span style="color:blue;font-weight:bold;font-family:courier;">lpwd</span><br><br>What is the pathname? Which server does this represent: local or remote?<br><br>
# Issue the following ''sftp command'' to create a directory on your remote server: <span style="color:blue;font-weight:bold;font-family:courier;">mkdir remote2</span><br><br>
# Issue the following ''sftp command'' to confirm that the '''remote2''' directory has been created<br>in your remote server's home directory:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls</span><br><br>
# Issue the following ''sftp command'' to change to the '''remote2''' directory on your ''remote'' server:<br><span style="color:blue;font-weight:bold;font-family:courier;">cd remote2</span><br><br>
# Issue the '''pwd''' ''sftp command'' to confirm that you have changed to the ''remote2'' directory on your remote server.<br><br>
# Issue the following sftp command to change to the '''local''' directory on your local computer:<br><span style="color:blue;font-weight:bold;font-family:courier;">lcd local</span><br><br>
# Issue the '''lpwd''' ''sftp command'' to confirm that you have changed to the ''local'' directory on your local computer.<br><br>
# Issue the following ''sftp command'' to transfer the file called '''thefile.txt''' to the '''~/remote2''' directory on your remote server:<br><span style="color:blue;font-weight:bold;font-family:courier;">put thefile.txt</span><br><br>
# Issue the '''ls''' sftp command to confirmed that you transferred the file called: '''thefile.txt'''<br><br>Let's create another directory on your local computer called '''local2''' so we can learn to <u>download</u> a file from your remote directory.<br><br>
# Issue the following ''sftp command'' to change to the '''parent''' directory on your local computer:<br><span style="color:blue;font-weight:bold;font-family:courier;">lcd ..</span><br><br>
# Issue the '''lpwd''' ''sftp command'' to confirm that your current working directory on your local computer is your home directory.<br><Br>
# Issue the following ''sftp command'' to create the following directory on your local computer:<br><span style="color:blue;font-weight:bold;font-family:courier;">lmkdir local2</span><br><br>
# Issue the following ''sftp command'' to change to the '''local2''' directory on your local computer:<br><span style="color:blue;font-weight:bold;font-family:courier;">lcd local2</span><br><br>
# Issue the '''lpwd''' ''sftp command'' to confirm you have changed to the '''local2''' directory on your local computer.<br><br>Let's learn to <u>download</u> a file from your remote server to your local computer.<br><br>
# Issue the following '''sftp command''' to transfer your '''thefile.txt''' file from the '''remote2''' directory<br>on your remote server to your local computer:<br><span style="color:blue;font-weight:bold;font-family:courier;">get thefile.txt</span><br><br>
# Issue the <span style="font-weight:bold;font-family:courier;">lls</span> ''sftp command'' to confirm that you transferred the file '''thefile.txt''' to your local computer.<br><br>
# Issue the following ''sftp command'' to exit the sftp utlilty: <span style="color:blue;font-weight:bold;font-family:courier;">exit</span><br><br>
# Issue the following Linux command to remotely run a checking script to ensure you created the correct directories<br>and properly transferred those created files: <br><span style="color:blue;font-weight:bold;font-family:courier;">ssh yoursenecaid@matrix.senecacollege.ca ~uli101/week6-check-3</span><br><br>
# If you encounter errors, make corrections and then re-run the checking script until you receive a congratulations message.<br><br>'''FYI:''' To run a checking program to check if you created the '''local''' and '''local2''' directories in MS Windows would require<br>running a local-based script (like '''PowerShell'''). Since this is a Unix/Linux based course, we don't have a PowerShell script,<br>so we will ignore checking for files transferred to your local computer.
 
<!-- OLDER CONTENT
 
# Make certain that you are logged into your Matrix account and are located in your '''home''' directory.<br><br>
# Issue the following Linux commands to create the following directories:<br><span style="color:blue;font-weight:bold;font-family:courier;">mkdir ~/local</span><br><br>
# Change to the '''~/local''' directory.<br><br>
# Use a text editor to create a text file called '''thefile.txt'''<br><br>
# Enter the following two lines displayed below for this text file:<br><span style="font-family:courier;font-weight:bold;">This is my file called thefile.txt<br>It is also a small file</span><br><br>
# '''Save''' editing changes to the text file and exit your text editor.<br><br>
# Change back to your '''home''' directory and confirm that you are located in your ''home'' directory.<br><br>Let's run a '''shell script''' to check to make certain that you created the correct directories<br>and that you created the '''myfile.txt''' file (with correct file contents) in the '''~/local''' directory<br>before learning to use the ''sftp'' Linux command.<br><br>
# Issue the following Linux command to run a checking script:<br><span style="color:blue;font-weight:bold;font-family:courier;">~uli101/week6-check-3</span><br><br>
# Issue the following Linux command to start an sftp session (note: yoursenecaid is YOUR Seneca ID):<br><span style="color:blue;font-weight:bold;font-family:courier;">sftp yoursenecaid@matrix.senecacollege.ca</span><br><br>'''NOTE:''' You may be required to enter '''yes''' to have the public key shared.<br><br>You are now in the '''sftp command prompt''' where you are expected to issue '''sftp commands'''.<br>Please refer to the table of ''sftp commands'' in the '''KEY CONCEPTS''' section for reference.<br><br>
# Issue the following ''sftp command'': <span style="color:blue;font-weight:bold;font-family:courier;">pwd</span><br><br>What is the pathname? Which server does this represent: local or remote?<br><br>
# Issue the following ''sftp command'': <span style="color:blue;font-weight:bold;font-family:courier;">lpwd</span><br><br>What is the pathname? Which server does this represent: local or remote?<br><br>
# Issue the following ''sftp command'' to create a directory on your remote server: <span style="color:blue;font-weight:bold;font-family:courier;">mkdir remote</span><br><br>
# Issue the following ''sftp command'' to confirm that the '''remote''' directory has been created in your remote server's home directory:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls</span><br><br>
# Issue the following ''sftp command'' to change to the '''remote''' directory on your ''remote'' server:<br><span style="color:blue;font-weight:bold;font-family:courier;">cd remote</span><br><br>
# Issue the '''pwd''' ''sftp command'' to confirm that you have changed to the ''remote2'' directory on your remote server.<br><br>
# Issue the following sftp command to change to the '''local''' directory on your local computer:<br><span style="color:blue;font-weight:bold;font-family:courier;">lcd local</span><br><br>
# Issue the '''pwd''' ''sftp command'' to confirm that you have changed to the ''local'' directory on your local computer.<br><br>
# Issue the following ''sftp command'' to transfer your '''~/local/thefile.txt''' file to the '''~/remote''' directory on your remote server:<br><span style="color:blue;font-weight:bold;font-family:courier;">put thefile.txt</span><br><br>
# Issue the '''ls''' sftp command to confirmed that you transferred/copied the file: '''thefile.txt'''<br><br>Let's create another directory on your local computer called '''local2''' so we can learn to <u>download</u> a file from your remote directory.<br><br>
# Issue the following ''sftp command'' (without an argument) to change to your '''home''' directory:<br><span style="color:blue;font-weight:bold;font-family:courier;">lcd ~</span><br><br>
# Issue the '''lpwd''' ''sftp command'' to confirm that your current working directory on your local computer is your home directory.<br><Br>
# Issue the following ''sftp command'' to create the following directory on your local computer:<br><span style="color:blue;font-weight:bold;font-family:courier;">lmkdir local2</span><br><br>
# Issue the following ''sftp command'' to change to the '''local2''' directory on your local computer:<br><span style="color:blue;font-weight:bold;font-family:courier;">lcd local2</span><br><br>
# Issue the '''lpwd''' ''sftp command'' to confirm you have changed to the ''local2'' directory on your local computer.<br><br>Let's learn to <u>download</u> a file from your remote server to your local computer.<br><br>
# Issue the following '''sftp command''' to transfer your '''thefile.txt''' file from your remote directory on your local computer:<br><span style="color:blue;font-weight:bold;font-family:courier;">get thefile.txt</span><br><br>
# Issue the '''lls''' '''sftp command''' to confirm that you transferred/copied the file: thefile.txt on your local computer.<br><br>
# Issue the following ''sftp command'' to exit the sftp utlilty: <span style="color:blue;font-weight:bold;font-family:courier;">exit</span><br><br>
# Issue the '''ssh''' command to login to your Matrix server account.<br><br>
# Issue the following Linux command to run a checking script to ensure you created the CORRECT directories, created and copied the correct files: <br><span style="color:blue;font-weight:bold;font-family:courier;">ssh yoursenecaid@matrix.senecacollege.ca ~uli101/week6-check-3</span><br><br>
# If you encounter errors, make corrections and then re-run the checking script until you receive a congratulations message.<br><br>
 
 
 
:Although it is important to learn how to use '''command-line sftp''' which is considered coverage for quizzes, midterm and final exam, there are '''graphical sftp applications''' available that are considered "user-friendly". The next investigation will show you how to transfer files between your local computer and your Matrix server graphically (assuming your local computer is running MS Windows).
<br>
 
=== Graphical Application (Secure Shell) ===
[[Image:graphical-sftp-application.png|thumb|right|200px|'''SSH Secure Shell Client''' has a graphical application to transfer files between computers.]]
If you installed the graphical Secure Shell application in [https://wiki.cdot.senecacollege.ca/wiki/Tutorial_1:_Using_Your_Matrix_Server_Account#CONNECTING_TO_YOUR_MATRIX_ACCOUNT_FROM_A_GRAPHICAL_WINDOWS_APPLICATION TUTORIAL 1 INVESTIGATION 1], <br>you can run a '''graphical''' application from your computer desktop to transfer files between<br>your computer and your Matrix account.
 
Let's run the '''Secure Shell SFTP application''' to transfer files between computers.<br>This investigation assumes that your computer is a '''Windows''' machine. If you local computer<br>is '''NOT''' a ''Windows'' machine, you can <u>skip</u> this investigation.
 
 
'''Perform the Following Steps:'''
 
# On your local computer, open the <span style="color:blue;font-family:courier;font-weight:bold;">NotePad</span> application to create a text file.<br><br>
# Enter a few lines of text, and then click on the <b>File</b> menu and select <b>save as</b><br>(save as the filename <span style="color:blue;font-weight:bold;font-family:courier;">yourfile.txt</span> in your <span style="color:blue;font-weight:bold;font-family:courier;">local</span> directory)<br>and then <b>exit</b> the <i>Notepad</i> text editor.<br><br><table align="right"><tr valign="top"><td>[[Image:graphical-sftp-icon.png|thumb|right|110px|Double Click on SFTP to launch graphical application.]]</td><td>[[Image:sftp-connect.png|thumb|right|200px|Using Quick Connect to login to your Matrix Linux server.]]</td></table>
# In your local computer's command terminal, make certain that you are in the '''local''' directory, and if not, change to that directory.<br><br>
# Issue the <span style="color:blue;font-weight:bold;font-family:courier;">dir</span> command for the '''local''' directory to confirm that you created the file called '''yourfile.txt'''<br><br>
# On your Windows desktop, point and '''double-click''' on the<br>'''SSH Secure file Transfer''' application icon.<br><br>
# The main Secure Shell SFTP application window should appear.<br><br>'''FYI:''' This application shows files on your home computer (on the left-side) and files on your remote Matrix computer on the right-side. You will NOT see files<br>for your Matrix server since you have NOT logged into the Matrix server.<br><br>
# Click the '''Quick Connect''' button located in the Secure SFTP application window.<br><br>'''NOTE:''' The Connect dialog box allows the user to specify the '''server name'''<br>and your '''account name''' to allow you to connect to the server.<br><br>
# Click on the textbox labelled '''Hostname''' and type the text: <span style="color:blue;font-weight:bold">matrix.senecacollege.ca</span><br><br>
# Click on the textbox labelled '''User Name''' and type your Seneca username<br>(i.e. same as your Seneca userid).<br><br>
# After entering the ''hostname'' and ''username'', click the '''Connect''' button.<br><br>
# Enter your Seneca password when prompted.<br><br>
# When you correctly connect to your Matrix account, you should see '''folders'''<br>in your remote server (i.e. Matrix server).<br><br>[[Image:graphical-sftp-application.png|thumb|right|250px|'''Click''' and '''drag''' file(s) to transfer between computers.]]
# There are two sub-windows to represent your local computer on the <u>left</u><br>and the remote (Matrix) server on the <u>right</u>.<br><br>
# In the <u>left</u> (local computer) window, point and '''double-click''' on your Windows<br>profile name (i.e. ''username'') and then locate and '''double-click''' on the '''local''' directory.<br><br>
# In the <u>right</u> (remote Matrix server) window, navigate to the '''remote2''' directory<br>and '''double-click''' to move to that directory.<br><br>
# In the left window, click and drag the '''yourfile.txt''' file to the <u>right</u> window.<br><br>
# Check to see that the '''yourfile.txt''' file appears in the <u>right</u> window to confirm<br>that the file has been transferred to your Matrix server.<br><br>'''NOTE:''' You can select multiple files by using ''SHIFT-CLICK'' or ''CTRL-CLICK'' methods<br>and then click and drag file selections between computers.<br><br>
# Take a few moments to noter other buttons for both local and remote windows to<br>'''refresh the view''', '''delete file(s)''', '''navigate up to parent directory''', etc.<br><br>
# Click the '''File''' menu, and select '''Disconnect''' and then click '''OK''' to disconnect your sftp connection.<br><br>
# Now that you have disconnected, '''close''' the sftp application window.<br><br>
# Return to your command terminal on your <u>local</u> computer (do '''NOT''' connect to Matrix!).<br><br>
# Issue the following Linux command to run a checking script to confirm that you transferred<br>the most recently-created textfile to the '''remote2''' directory in Matrix:<br><span style="color:blue;font-weight:bold;font-family:courier;">ssh yoursenecaid@matrix.senecacollege.ca ~uli101/week6-check-4</span><br><br>
# When prompted, enter your password.<br><br>
# If you encounter errors, make corrections and then re-run the checking script until you receive a congratulations message.<br><br>
 
-->
 
:In the next investigation, you will learn an alternative way to transfer a file to another computer server<br>by sending an '''e-mail message with an attached file'''.
 
=INVESTIGATION 3: FILE TRANSFER (EMAIL) =
 
 
The '''Matrix''' server is also an '''email server''' that can allow you to '''send''' emails messages to other email accounts.
 
In this investigation, you will learn how to '''transfer''' a file from your Matrix server to another computer by sending<br>an '''email message''' with a '''file attachment'''.
 
 
'''Perform the Following Steps:'''
 
# Make certain that you connect and login to your '''Matrix''' server and confirm that you are located in your '''home''' directory.<br><br>
# Issue the following Linux command (using your Seneca-ID):<br><span style="color:blue;font-weight:bold;font-family:courier;">mail yoursenecaid@myseneca.ca</span><br><br>
# When prompted, enter the '''subject line''':  <span style="color:blue;font-weight:bold;font-family:courier;">Test Message</span><br>and press <span style="color:blue;font-weight:bold;font-family:courier;">ENTER</SPAN><br><br>
# In the email message '''BODY''' section, type the following text displayed below (and press '''ENTER'''):<br><span style="font-weight:bold;font-family:courier;">This is a test email message</span><br><br>
# Press <span style="color:blue;font-weight:bold;font-family:courier;">ctrl-d</span> to send your email message.<br><br>Did any output display? What you do think '''EOT''' stands for?<br><br>
# Launch a '''web-browser''', login into your '''Seneca email''' account and check for new email messages.<br>Did you receive the email message that you sent from your Matrix server?<br><br>If you did NOT receive an e-mail message, check the '''JUNK''' or '''CLUTTER''' folders.<br>If you still did not receive an email message, return to your terminal and re-issue the '''mail''' command<br>making certain that you pressed <span style="color:blue;font-weight:bold;font-family:courier;">ctrl-d</span> instead of pressing '''ctrl-c'''<br><br>
# Return to your terminal (i.e. Linux Bash shell) and issue the following Linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;">mail -a ~/remote/myfile.txt yoursenecaid@myseneca.ca</span><br><br>
# When prompted, enter the subject line: <span style="color:blue;font-weight:bold;font-family:courier;">Test Message with Attachment</span><br> and press <span style="color:blue;font-weight:bold;font-family:courier;">ENTER</span><br><br>
# In the email message '''BODY''' section, type the following text displayed below (and press '''ENTER'''):<br><span style="font-weight:bold;font-family:courier;">This is a test email message with a file attachment</span><br><br>
# Press <span style="color:blue;font-weight:bold;font-family:courier;">ctrl-d</span> to send your message.<br><br>
# Switch to your Seneca email and check for new email messages.<br><br>Did you receive that email message? Does the email contain a file attachment?<br><br>
# Return to your Linux Bash shell and issue the following Linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;">mail  yoursenecaid@myseneca.ca &lt; ~/remote/myfile.txt</span><br><br>What happened? Were you prompted for subject and could you enter text in email body?<br>Did you see a file attachment as a separate file, or just text?<br><br>
# Check your email to see if you received your email message. If you did, what do you notice regarding the subject line?<br><br>You should have noticed that there was '''NO''' customized '''subject line''',<br>since you redirected '''standard input''' (''stdin'') from the file, so there was no way<br>for the user to send a subject line.<br><br>You can use the '''-s''' option, followed by text (in quotes) to specify a '''subject line'''.<br><br>
# Return to your Linux Bash shell and issue the following Linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;">mail -s "email with attachment" yoursenecaid@myseneca.ca &lt; ~/remote/myfile.txt</span><br><br>
# Check your email to see if you received your email message. If you did, what do you notice this time?<br><br>
# After completing this INVESTIGATION, perform the LINUX PRACTICE QUESTIONS at the end of the tutorial.<br><br>
 
= LINUX PRACTICE QUESTIONS =
 
The purpose of this section is to obtain '''extra practice''' to help with '''quizzes''', your '''midterm''', and your '''final exam'''.
 
Here is a link to the MS Word Document of ALL of the questions displayed below but with extra room to answer on the document to
simulate a quiz:
 
https://wiki.cdot.senecacollege.ca/uli101/files/uli101_week6_practice.docx
 
Your instructor may take-up these questions during class. It is up to the student to attend classes in order to obtain the answers to the following questions. Your instructor will NOT provide these answers in any other form (eg. e-mail, etc).
 
 
'''Review Questions:'''
 
# Write a Linux command to copy a file in the current directory called '''mytext.txt''' from your Matrix account to your account called '''user1'''<br>on the Linux server domain name called '''tech.myserver.com''' to that user’s home directory.
# Write a Linux command similar to the previous question, but rename the file on the remote Linux server to '''yourtext.txt'''
# Write a Linux command to copy a file called '''~/project/linux.txt''' to the remote server called '''linux.techie.org'''<br>(your username for this remote server is the same username for your local server).<br><br>
# Write a Linux command to connect to the username '''saulm''' for the server domain name '''tux.senecac.on.ca''' to transfer files between Linux servers.
# Assuming that you are connected to that server in ''question #4''. What is the sftp command to display your current working directory on your local server?
# Assuming that you are connected to that server in ''question #4''. What is the sftp command to view files in your local server?<br>What is the sftp command to view files in your remote server?
# Assuming that you are connected to that server in ''question #4''. What is the sftp command to <u>download</u> the file answers.txt from the current directory of your remote server?
# Assuming that you are connected to that server in ''question #4''. What is the sftp command to <u>upload</u> the file questions.txt from your local server to the '''~/documents/tests''' directory on your remote server?
# Assuming that you are connected to that server in ''question #4''. What is the sftp command to quit your current session?<br><br>
# Write a Linux command to send the attached file '''message.txt''' to the email address '''murray.saul@senecacollege.ca''' with the subject line: '''Important Message'''
# Create a '''table''' listing each Linux command, useful options  and command purpose for the following Linux commands: '''scp''' , '''sftp''' , '''mail'''.
# Create a '''table''' listing each '''sftp command''' and it's purpose.
 
 
_________________________________________________________________________________
 
Author:  Murray Saul
 
License: LGPL version 3
Link:    https://www.gnu.org/licenses/lgpl.html
 
_________________________________________________________________________________


* Move the three files '''ThisHas OneSpace.txt''',  '''This Has Three Spaces.txt''', and '''WhereIsTheSpace.txt''' from '''~/lab6/Spaces''' to '''~/lab6/Important Dir''' using either one or multiple commands.
* Rename '''NotA*Wildcard''' to '''Not a * wildcard''' in '''~/lab6/WeirdNames'''


= Submit evidence of your work =
After you finish the lab: run the following commands to submit your work:<syntaxhighlight lang="bash">
cd ~
wget http://ops345.ca/check/ops145-lab6-check.sh # Download the check script
chmod 700 ops145-lab6-check.sh # Make the downloaded file executable
./ops145-lab6-check.sh # Run the check script
</syntaxhighlight>If it says "Your lab6 has been submitted": make a screenshot, and you're done. If it gives you any warnings or errors: you have to fix them and try the ./ops145-lab6-check.sh command again.


[[Category:OPS145]]
[[Category:OPS145]]

Latest revision as of 21:55, 29 February 2024

Special characters in the shell

The terminal you've been using this whole time in the course is running an application called the shell. Specifically the bash shell.

This application (bash) has been written to interpret input from the user as some sort of a command; or as an argument; or as data. It's a very powerful application with many abilities, but almost all the user input comes from the keyboard.

There are only so many keys of the keyboard, and most of them are used to input human-readable characters. That means some of those characters will necessarily have multiple meanings. We will look at several such characters in this lab.

Here's most of the ASCII table (as much as I could fit on my screen). Note that most of the characters in this table are familiar to you, but some (especially the first 32) are probably brand new to you. You can see this table easily by running man ascii:

LongerManAscii.png

. and ..

Coming from the Windows world you might be used to the concepts of "filename" and "extension". If you're not too familiar with that: you should configure your windows to always show file extensions (it doesn't by default).

For example if you have a picture file called smily.jpg: You might call smily the filename, and jpg the extension. But then what is the dot? You could call it a separator, but In fact the dot is just as much part of the filename as smily and jpg. Looking at the ASCII table: s and m are just as different from each other as s and .

And what's the extension in a file called lab1.tar.xz? Is it xz? Or tar.xz? What about in a file called Dr. Evil plan.txt?

When it comes to filenames and extensions: the use of a dot as a separator is just a convention to help people organize their files, and is barely useful from a technological point of view.

But there's another use of dots in filesystems, which is much more fundamental. Every directory contains at least these two records: a link to itself (.), and a link to its parent (..)

You can see these records when you run ls with the -a argument. Let's look at the SampleFiles directory you downloaded back in Lab 2:

  • Open a terminal, and change the PWD to ~/Downloads/SampleFiles
  • Run ls -a Note that in the output there is a . and a ..
  • Run ls -a -l Note that the . and .. are directories.
Ls-al.png
  • Remember that you can give an argument to ls telling it to show the details of a specific file or directory. For example ls -l 1984.txt will show details about that specific file, and ls -l / will show details for the immediate contents of the root directory.
  • Run ls -l -a . (including the dot in the end) and note that it shows the same output as ls -l -a That's because:
    • ls will show the contents of PWD if you don't give it a specific path.
    • Your PWD is ~/Downloads/SampleFiles/
    • The . in SampleFiles points to itself (SampleFiles)
  • Run ls -l -a .. and ls -l -a ~/Downloads Note that the output is the same. That's because:
    • Your PWD is ~/Downloads/SampleFiles/
    • The parent directory of ~/Downloads/SampleFiles/ is ~/Downloads/
    • Inside ~/Downloads/SampleFiles/ .. and ~/Downloads are the same thing
  • The same for every other directory. The only sort-of exception is the root directory. Because it's the root and it doesn't have a parent: its parent is itself:
FilesystemIntroWithDots.png
  • Take some time to look at these many lines and understand that actually it's not that complicated. You just have to remember: dot is for itself, dot-dot is for its parent.

Hidden files

Dots have another (mostly unrelated) special use when it comes to filesystems in the shell. Any file/directory name which begins with a dot is considered hidden.

This is definitely not a security feature! It is purely a convenience.

  • Change your PWD to your home directory.
  • Run ls
  • Run ls -a

Notice there are many more files and directories in your home than you saw before. These were always there, the ls command doesn't show them by default. The -a argument tells ls to also show hidden files.

These are hidden to avoid clutter, to make it easier for you to find things you want. There's nothing special about these files and directories. They're just rarely manipulated by users.

This is also where the "extension" idea breaks down completely. If you insist that a file's extension is everything after the dot and the file's name is everything before the dot: all these hidden files have no names, and only extensions. That's clearly not the case. Almost all these files are plain text files, without a .txt extension.

  • Use the cat command to look at the contents of .bash_history. This file is where the commands you ran previously are stored, so you can retrieve them using the arrow keys.
  • Since you're at it: here's a new command. Run history on the terminal. It will show you all the previous commands you've run.

Comments

  • Now that you're more comfortable with the command-line: download and extract the next tarball using commands instead of Firefox and a graphical archive manager:
    cd ~/Downloads # To make sure it's downloaded here
    ls -l -h # To confirm you didn't already download SpecialChars.tar.xz
    wget http://ops345.ca/ops145/SpecialChars.tar.xz # This will download SpecialChars.tar.xz into the PWD
    tar xvf SpecialChars.tar.xz # This will extract the contents of SpecialChars.tar.xz into the PWD
    

In the block of commands above there are words which are clearly not intended to be executed by the shell (it's only so smart). They are for human beings to read. But you can run all the above without any difficulty because anything following the special character # is treated as a comment, meaning the shell ignores it no matter what it is.

The # is usually called a pound sign, or a number sign, or a hash.

The * wildcard

You've used the * wildcard (called a star, it's the multiplication sign on your keyboard) in lab 3 to delete several files at the same time. Now we'll look at it in more detail.

Sometimes you want to do something will all the contents of a directory. At other times you might want to do something with all your .jpg files, or all the .txt files, or all the files which have backup in their name. Those are examples of when you use a *

Inside the newly created ~/Downloads/SpecialChars/ directory is a SortMe subdirectory, which is a disorganized mess. We'll organize it in this lab. Remember that the whole point is to learn wildcards, so use commands in a terminal to move files around instead of dragging them in the graphical interface.

  • Use the graphical file manager to look at the SortMe directory to get an idea of what's in there. Note that:
    • There are pictures and sounds
    • Some of the pictures are small squares, and others are large photos
    • Some of the sounds are .wav files and others are .oga files
    • Some of the .wav files sound like speaker test sounds
  • We're going to organize these into the following directory tree inside ~/lab6:
SortMeTree.png
  • Remember that filenames are case-sensitive. Open a terminal and create the directory tree above using as many commands as you like. But here's a handy argument you can give to the mkdir command: -p
    mkdir -p ~/lab6/sounds/WAV/SpeakerTest
    
    When run with -p, the mkdir command will create not only the last directory on the path you give it, but all the other missing directories in that path as well. You might want to avoid using this if you're not very comfortable with the command-line yet, since it's very easy to create a hundred directories in the wrong place by giving mkdir -p the wrong path several times.
  • Change your PWD to ~/Downloads/SpecialChars/SortMe

When the bash shell encounters a * character on the command-line: it assumes you're trying to use it as a wildcard for filenames, and does its best to replace the * with a list of all filenames which match the pattern.

On its own: a * matches any number of any characters. We can test this easily using the ls command:

ls *

This appears to print the same output as ls on its own, but in fact the ls command received from the shell a list of all the files in ~/Downloads/SpecialChars/SortMe as arguments. You can combine the * wildcard with other characters, restricting what it will be expanded to. For example:

ls *jpg

This time * still expands to anu number of any characters, but because it's combined with an explicit jpg in the end: the result will be a list of all the files which have jpg at the end of their name.

  • Try running commands similar to the above with different extensions.

The wildcard doesn't have to be at the beginning. It can be in the end. For example this is how you would get a list of all the files which have names starting with amanea:

ls amanea*

The wildcard can be in the middle too. For example this is how you would get a list of all the files with _ somewhere in the name:

ls *_*

Or a list of filenames which contain an _ and end with wav:

ls *_*wav

The use of wildcards is completely unrelated to the ls command; ls just happens to be an easy program to show what the wildcard expands to.You can use it with equal effectiveness with other commands, like cp.If we did't have a graphical interface available and we wanted to copy all the jpg files to the ~/lab6/pictures directory you created earlier: that would involve a lot of typing, even with tab completion. Instead we can copy all of them quickly.

  • First run ls and ls *jpg to confirm that *jpg expands to what you want (in this case all the jpg files). You should notice that it doesn't. There are two jpg files which have their JPG in uppercase.
  • If you want to include those as well: use two arguments (bash will actually convert your two arguments to 26 arguments - all the jpg and JPG filenames)
    ls *jpg *JPG
    
  • If you're satisfied with the list of filenames you're getting: you can use the same list with the cp command. The cp command is potentially distructive, since it will overwrite files by default. So think twice before you run it with unknown arguments.
  • Also give cp the extra -v (verbose) argument so that it will print details about what it's doing:
    cp -v *jpg *JPG ~/lab6/pictures/
    
  • Run a command to copy all the wav files to ~/lab6/sounds/WAV/
  • Run a command to copy all the oga files to ~/lab6/sounds/OGA
  • Change your working directory to ~/lab6/sounds/WAV

We want to move the files Front_Right.wav, Rear_Center.wav, Side_Left.wav, Front_Center.wav, Rear_Left.wav, Side_Right.wav, Front_Left.wav, and Rear_Right.wav to the SpeakerTest subdirectory without typing in all those names. Luckily (would you believe it) these specific files have a specific filename pattern which makes it different from all the other files in sounds/: they all contain an underscore.

  • Use ls to check that the *_* pattern will include all the speaker test files, and only the speaker test files:
    ls *_*
    
  • If it looks good: use that argument with the mv command. Note that mv also accepts a -v (verbose) argument:
    mv -v *_* SpeakerTest
    

You should end up with the following arrangement in your sounds directory:

TreeOfSounds.png

The ? wildcard

Like the * wildcard: the ? wildcard stands for any character. But unlike the *: the ? stands for exactly one character.

To illustrate we're going to move all the small square shaped images into a new subdirectory.

  • Create the ~/lab6/pictures/square directory
  • In ~/lab6/pictures: try ls *_*
  • Note that it shows all the files. That's because all of them contain an underscore. But we want only the filenames which begin with a single digit, then have an underscore, and then anything else. Use a ? to get that list:
    ls ?_*
    
    You could make it more specific by adding jpg in the end, but in this case it's not necessary.
  • If the output looks right: move all the square images into the square directory:
    mv ?_* square
    

You should end up with this arrangement:

TreeOfImages.png

Spaces in filenames

At this point in computer history people are used to the idea that a space is nothing special in a filename. That's because most people use GUIs to manipulate files.

On the command-line a space is a very special character which is used all the time: it's the separator between the command and the arguments, and the separator between multiple arguments.

From the filesystem's point of view: there is nothing special about a space, it's just another valid character from the ASCII table. The problem is the interface between the user and the filesystem, in our case that's the shell.

  • Inside your downloaded and extracted SpecialChars directory is a Spaces directory. Copy that to ~/lab6/
  • If you look at the contents in a graphical interface: you'll see there really isn't anything special about those files and directories.
  • In your terminal: change your PWD to ~/lab6/Spaces/
  • Run ls. Note that the names have single quotes around them. This is relatively new behaviour for bash, many systems won't add the quotes. Note also that the spaces between the filenames in the output are no different than the spaces inside the filenames.
  • Try to see the contents of Important Dir using ls
    ls Important Dir
    

It doesn't work because the shell thinks you mean that Important and Dir are separate filenames. To tell the shell you mean the space is part a part of Important Dir: you need to escape the space. There are three ways to do this:

  • Put a \ (backslash) before the special character
  • Enclose the entire argument in double quotes
  • Enclose the entire argument in single quotes
  • Try again to see the contents of Important Dir using ls, but this time escape the space:
    ls Important\ Dir
    
    You may have already found that tab completion does this for you automatically.
  • If the shell encounters a quote (either double quote or single quote): it will assume everything from that quote until the next quote is one argument. For example, you can run:
    ls "Important Dir"
    ls 'Important Dir'
    
  • Both of those will work.

Spaces in filenames are not impossible to deal with on the Linux command line, but they make simple things difficult, and complicated things really difficult.

Escaping other special characters

Like spaces: other special characters can have their special status stripped by escaping them.

  • Inside your downloaded and extracted SpecialChars directory is a WeirdNames directory. Copy that to ~/lab6/

Normally you would avoid creating such filenames if you're expecting to work with them in the command line, but this is a learning exercise.

  • Change your PWD to ~/lab6/WeirdNames
  • Try to run an ls command escaping only the space in Andrew's stuff.txt:
EscapeQuote.png

The shell will give you this strange prompt-like behaviour, as if it's waiting for you to type something in. What it's waiting for is the closing single quote for what looked like an opening single quote.

  • Press Ctrl+c to abort the command you were trying to run.
  • Re-run the ls command, this time escaping not only the space but the single quote as well.
  • Similar problems will the * special character. A * is just a character like any other in the ASCII table. It has special meaning in the shell but not on the filesystem. Try it:
EscapeStar.png

Some more work for you

By this point in the lab you should have:

  • Downloaded and extracted SpecialChars.tar.xz
  • Created the ~/lab6 directory and the subdirectories for sorting pictures and sounds
  • Copied/moved specific pictures and sounds into appropriate directories
  • Copied Spaces and WeirdNames to ~/lab6

Do a few more things for practice with special characters on the command-line:

  • Move the three files ThisHas OneSpace.txtThis Has Three Spaces.txt, and WhereIsTheSpace.txt from ~/lab6/Spaces to ~/lab6/Important Dir using either one or multiple commands.
  • Rename NotA*Wildcard to Not a * wildcard in ~/lab6/WeirdNames

Submit evidence of your work

After you finish the lab: run the following commands to submit your work:

cd ~
wget http://ops345.ca/check/ops145-lab6-check.sh # Download the check script
chmod 700 ops145-lab6-check.sh # Make the downloaded file executable
./ops145-lab6-check.sh # Run the check script

If it says "Your lab6 has been submitted": make a screenshot, and you're done. If it gives you any warnings or errors: you have to fix them and try the ./ops145-lab6-check.sh command again.