|
|
| Line 1: |
Line 1: |
| =USING REGULAR EXPRESSIONS= | | = The ''server'' concept = |
| <br>
| | For most people a '''server''' is something on the internet you '''connect to''', and a client is something you connect from. That sort of view is mostly valid, but it breaks down a little when the server and the workstation are the same machine. |
| ===Main Objectives of this Practice Tutorial===
| |
|
| |
|
| :* Define the term '''Regular Expressions''' | | A couple of examples: |
|
| |
|
| :* Explain the difference between '''Regular Expressions''' and '''Filename Expansion''' | | * When you connect to wiki.littlesvr.ca in Firefox: |
| | ** Firefox is the client, it's making the request for a web page |
| | ** The web server (Apache) on wiki.littlesvr.ca is the server, responding the the request with the contents of the web page |
| | * When you use your phone to check your email: |
| | ** The email application on your phone is the client, making the request to get new email |
| | ** The email server (e.g. Postfix, Gmail, Office365) is the server, responding to the request with a list of new emails |
|
| |
|
| :* Explain the purpose of '''Literal (Simple)''' Regular Expressions
| | Most of a Linux system administrator's work is done in a terminal, and most of that is done on remote machines. Few companies can afford to hire in-house administrators, and those that can afford it have too many machines to connect keyboards and monitors to. But everything is connected to a network. |
|
| |
|
| :* Understand and use common symbols for '''Complex''' Regular Expressions and their purpose | | The same client-server model applies here. You connect from a terminal on your workstation to the machine you want to work with. In this case: |
|
| |
|
| :* Understand and use command symbols for '''Extended''' Regular Expressions and their purpose
| | * The '''ssh''' program is the client, connecting to the ssh server, sends what you type to the server and prints the output which those commands print on the server. |
| | * The ssh server (sshd) is the software on the destination machine which receives commands from the client, executes them, and sends the output of those commands back to the client. |
|
| |
|
| :* List several Linux commands that can use regular expressions
| | = Temporary account on ops345.ca = |
| <br>
| | I've created an account for you on ops345.ca which you can use for SSH practice. |
|
| |
|
| ===Tutorial Reference Material===
| | You should have received an email like this: |
| | [[File:OPS345NewAccountEmail.png|center|border]] |
| | Note that your username has an ops145_ prefix. Your password is randomly generated. |
|
| |
|
| {|width="100%" cellspacing="0" cellpadding="10" | | {{Admon/important|Please don't abuse this account|For example: don't upload 1GB videos to it. There are many students sharing this server, and I'm not an international corporation with money to burn. It's a single server on a relatively slow internet connection.}} |
|
| |
|
| |- valign="top"
| | = ssh to ops345.ca = |
| | You will be connecting to the ops345.ca server from your workstation using the '''ssh''' client. |
|
| |
|
| |colspan="2" style="font-size:16px;font-weight:bold;border-bottom: thin solid black;border-spacing:0px;padding-left:15px;"|Linux Command/Shortcut Reference<br>
| | The syntax for a basic ssh command is:<syntaxhighlight lang="bash"> |
| | ssh usernameOnServer@serverAddress |
| | </syntaxhighlight>The username is in your email. The server address is ops345.ca. After you run the ssh command: it will ask you for a password, which is the password in your email. |
|
| |
|
| |- valign="top" style="padding-left:15px;"
| | * Go ahead and use ssh to connect to '''ops345.ca''' as '''ops145_yourusername''' |
| | style="padding-left:15px;" |'''Regular Expressions:'''
| |
| * [https://techterms.com/definition/regular_expression#:~:text=A%20regular%20expression%20(or%20%22regex,wildcards%2C%20and%20ranges%20of%20characters.&text=A%20regular%20expression%20can%20be,%2C%20such%20as%20%22app%22. Definition]
| |
| * [https://en.wikipedia.org/wiki/Regular_expression#:~:text=Regular%20expressions%20are%20used%20in,built%2Din%20or%20via%20libraries. Purpose (WIKI)]<br><br>
| |
|
| |
|
| | The first time you ssh to a server: the ssh client will give you a scary warning message about authenticity. It's a very interesting message which is worth understanding, but we don't have time to talk about it in this course. Just enter "yes" in the prompt. |
| | [[File:FirstSSHtoops345.ca.png|center]]From this point on anything you type at the terminal will be sent to the server you're SSHed to, the commands you run will be executed on the server, and the standard output/error they produce will be sent back to your terminal. |
|
| |
|
| | style="padding-left:15px;"|'''Linux Commands:'''
| | You can notice that: |
| * [https://ss64.com/bash/egrep.html egrep]
| |
| * [https://www.man7.org/linux/man-pages/man1/man.1.html man]
| |
| * [https://man7.org/linux/man-pages/man1/more.1.html more] / [https://www.man7.org/linux/man-pages/man1/less.1.html less]
| |
| * [https://man7.org/linux/man-pages/man1/vi.1p.html vi] / [http://linuxcommand.org/lc3_man_pages/vim1.html vim]
| |
| * [https://man7.org/linux/man-pages/man1/sed.1p.html sed]
| |
| * [https://man7.org/linux/man-pages/man1/awk.1p.html awk]
| |
| * [https://linux.die.net/man/1/wget wget]
| |
| |}
| |
|
| |
|
| = KEY CONCEPTS =
| | # Your prompt is different. This is usually the first hint that you are SSHed to another machine. |
| | # Your username is different. |
| | # Your password is different. |
| | # None of your files are in your home directory. |
| | # There appear to be almost no files on this system at all (look in '''/''' and '''/bin''') |
|
| |
|
| ===Regular Expressions===
| | Your account allows you to SSH into a chroot, which is another thing we don't have time to talk about. |
|
| |
|
| <i>A '''regular expression''' is a combination of two types of characters: '''literals''' and '''special characters'''.<br>Strings of text can be compared to this pattern to see if there is a match.</i>
| | * You can end your session by running "exit" |
| | * Reconnect to ops345.ca and create a directory named '''lab8''' inside your home directory. |
| | * On the server there is an /srv/art/ directory, with some ASCII art in text files. Copy that '''art''' directory and its contents into the lab8 directory you just created. |
| | * Use the '''cat''' command to look at the contents of the text files. |
| | * Change the '''permissions''' on the '''art''' directory to '''700''' so that noone else can read its contents. |
| | * You may look in other students' home directories under /home - the default permissions will allow you to do this. That's somewhat of a tradition from the old days. |
| | * If you like: change the permissions on your home directory so that other users cannot see what's in it. |
|
| |
|
| This usually refers to text that is <u>contained</u> inside a '''file''' or text as a result<br>of issuing Linux commands using a '''Linux pipeline command'''.
| | {{Admon/tip|You're working on a remote machine|It's important to understand that everything you've done in the ssh session is done on the remote machine. Open another terminal in your workstation and look for /home/ops145_yourusername, the lab8 directory you just created, or the ASCII art files. None of that stuff should be on your workstation.}} |
| <br><br>
| |
|
| |
|
| ===Literal (Simple) Regular Expressions===
| | * Back in your terminal on ops345: create another directory named '''SampleFilesBackup''' under lab8. Leave that new directory empty for now. |
| | * Leave your ssh session running, but set this terminal window aside for a bit. |
|
| |
|
| [[Image:re-3.png|thumb|right|200px|A '''simple''' ('''literal''') regular expression is a series of letters and numbers (tabs or spaces).]]
| | = Copy files using scp = |
| The simplest regular expression is a series of letters and numbers, (tabs or spaces).<br>A '''simple''' ('''literal''') regular expression consists of normal characters, which used to match patterns.<br><br>
| | You can copy files from one Linux machine to another relatively easily. |
| Although there are many Linux commands that use regular expressions, the '''grep''' command is a useful command to learn how to display matches of patterns of strings within text files.<br><br>
| |
| For example:
| |
| <span style="color:blue;font-weight:bold;font-family:courier;">grep Linux document.txt</span><br><br>
| |
|
| |
|
| === Complex / Extended Regular Expressions === | | The syntax for a basic scp command is a more complicated than the basic ssh command, but it's similar because scp uses an SSH tunnel to do its work:<syntaxhighlight lang="bash"> |
| | scp fileOnLocalMachine usernameOnServer@serverAddress:destinationDirectory |
| | </syntaxhighlight>The username, server, and password are the same as the ones you used with ssh. The fileOnLocalMachine is the name of the file you want to copy to the server. The destinationDirectory specifies which directory on the server you want to put the file into. |
|
| |
|
| '''Complex Regular Expressions'''
| | You can also copy files from the remote server to your local machine: just reverse the arguments. |
| <br><br>
| |
| The problem with just using '''simple''' ('''literal''') regular expressions is that only <u>simple</u> or <u>general</u> patterns are matched.
| |
|
| |
|
| ''Complex Regular Expressions'' use symbols to help match text for more <u>precise</u> (complex) patterns.<br>The most common complex regular expression symbols are displayed below: | | * In a terminal on your workstation: copy the file '''Snow.jpg''' from SampleFiles into your ~/lab8/SampleFilesBackup/ directory on the ops345 server:<syntaxhighlight lang="bash"> |
| <br><br> | | scp Snow.jpg ops145_yourusername@ops345.ca:~/lab8/SampleFilesBackup/ |
| :'''Anchors: ''' <span style="color:blue;font-family:courier;font-weight:bold;">^</span> , <span style="color:blue;font-family:courier;font-weight:bold;">$</span><br>Match lines the begin (^) or end ($) with a pattern.<br> | | </syntaxhighlight> |
| :'''Single Character:''' <span style="color:blue;font-family:courier;font-weight:bold;">.</span><br>Represents a single character that can be any type of character.<br> | | * In your ssh session: confirm that the file has been copied. You can run '''ls -l''' to check that it has the same number of bytes on your workstation and on the server. |
| :'''Character Class:''' <span style="color:blue;font-family:courier;font-weight:bold;">[ ]</span> , <span style="color:blue;font-family:courier;font-weight:bold;">[^ ]</span><br>Represents a single character but with restrictions.<br>
| | * You can use copy more than one file at a time, and if you want: you can use wildcards. Copy both '''NotPlainText.odt''' and '''NotPlainText2.pdf''' from ~/Downloads/SampleFiles/ on your workstation to ~/lab8/SampleFilesBackup/ on ops345.ca:<syntaxhighlight lang="bash"> |
| :'''Zero or More Occurrence:''' <span style="color:blue;font-family:courier;font-weight:bold;">*</span><br>Zero or more occurrences of previous character.<br><br>
| | scp NotPlainText* ops145_yourusername@ops345.ca:~/lab8/SampleFilesBackup/ |
| | </syntaxhighlight> |
| | *Copy both '''1984.txt''' and '''AnimalFarm.txt''' from ~/Downloads/SampleFiles/ on your workstation to ~/lab8/SampleFilesBackup/ on ops345.ca. Note that both these files (and only these files in SampleFiles) end with .txt. So you can use the '''*''' wildcard to copy them both at the same time. |
| | After you're done you should have at least these contents on the server:[[File:ScpSomeFilesDone.png|center]] |
|
| |
|
| :Examples of '''complex regular expressions''' are displayed below:
| | == Ownership == |
| | |
| <table align="left"><tr valign="top"><td>[[Image:re-4.png|thumb|right|200px|Example of using '''anchors'''.]]</td><td>[[Image:re-5.png|thumb|right|175px|Example of matching by '''character(s)'''.]]</td><td>[[Image:re-6.png|thumb|right|220px|Example of using '''character class'''.]]</td><td>[[Image:re-7.png|thumb|right|200px|Example of matching '''zero or more occurrence of preceding character'''.]]</td></tr></table>
| |
| <br><br><br><br><br><br><br><br><br><br>
| |
| | |
|
| |
| '''Extended Regular Expressions'''
| |
| | |
| ''Extended Regular Expressions'' consist of additional special characters to “extend”<br>the capability of regular expressions. You must use the '''egrep''' or '''grep -E''' commands<br>in order to properly use extended regular expressions.
| |
| | |
| | |
| :'''Repetition:''' <span style="color:blue;font-family:courier;font-weight:bold;">{min,max}</span><br>Allows for more precise repetitions. Using braces, you can specify<br>the '''minimum''' and/or '''maximum''' number of repetitions.
| |
| | |
| :'''Groups:''' <span style="color:blue;font-family:courier;font-weight:bold;">( )</span><br>Allows you to search for repetition for a '''group of characters''', a '''word''', or a '''phase'''.<br>You enclose them within brackets <span style="font-family:courier;font-weight:bold;">( )</span> to specify a '''group'''.
| |
| | |
| :'''or Condition:''' <span style="color:blue;font-family:courier;font-weight:bold;">|</span><br>Can be used with '''groups''' to match a variety of character(s), words or phases.<br>The | symbol is used to separate the variety of character(s) within a ''group''.<br><br>
| |
| | |
| :Examples of how to use '''extended regular expressions''' with the '''egrep''' command are displayed below:<br><br>
| |
| | |
| <table align="left"><tr valign="top"><td>[[Image:re-8.png|thumb|right|280px|Example of using '''repetition'''.]]</td><td>[[Image:re-9.png|thumb|right|250px|Example of using '''groups'''.]]</td><td>[[Image:re-10.png|thumb|right|250px|Example of using '''or''' condition with '''groups'''.]]</td></tr></table>
| |
| <br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>
| |
| | |
| =INVESTIGATION 1: SIMPLE & COMPLEX REGULAR EXPRESSIONS=
| |
| | |
| <span style="color:red;">'''ATTENTION''': This online tutorial will be required to be completed by '''Friday in week 10 by midnight''' to obtain a grade of '''2%''' towards this course</span><br><br>
| |
| | |
| In this investigation, you will learn how to use the '''grep''' command with '''simple and complex regular expressions'''<br>to help search for ''patterns'' contained in text files.
| |
| | |
| | |
| '''Perform the Following Steps:'''
| |
| | |
| # '''Login''' to your matrix account.<br><br>
| |
| # Issue a Linux command to '''confirm''' you are located in your '''home''' directory.<br><br>
| |
| # Issue the following linux Linux command to '''copy''' a text file to ''your'' '''home''' directory from the ULI101 home directory:<br><span style="color:blue;font-weight:bold;font-family:courier;">cp ~uli101/tutorialfiles/textfile1.txt ~/</span><br><br>
| |
| # View the contents of the '''textfile1.txt''' file using the '''more''' command see what data is contained in this file.<br><br>Although there are several Linux commands that use regular expressions,<br>we will be using the '''grep''' command for this investigation.<br><br>[[Image:regexps-1.png|thumb|right|250px|Output of '''grep''' command matching simple regular expression "'''the'''" (only lowercase). Notice the pattern matches larger words like "'''their'''" or "'''them'''".]]
| |
| #Issue the following Linux command to match the pattern '''the''' within '''textfile1.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "the" textfile1.txt</span><br><br>Take a few moments to view the output and observe the matched patterns.<br><br>
| |
| # Issue the grep Linux command with the <span style="font-weight:bold;font-family:courier;">-i</span> option to ignore case sensitively:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -i "the" textfile1.txt</span><br><br>What do you notice is different when issuing this command?<br><br>You will notice that the pattern "'''the'''" is matched including larger words like "'''them'''" and "'''their'''".<br>You can issue the '''grep''' command with the <span style="font-weight:bold;font-family:courier;">-w</span> option to only match the pattern as a '''word'''.<br><br>
| |
| # Issue the following Linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "the" textfile1.txt</span><br><br>You should now see only strings of text that match the word '''the''' (upper or lower case).<br><br>Matching literal or simple regular expressions can be useful, but are '''limited'''<br>in what pattens they can match. For example, you may want to<br>search for a pattern located at the '''beginning''' or '''end''' of the string.<br><br>There are other regular expression symbols that provide more '''precise''' search pattern matching.<br>These special characters are known as '''complex''' and '''extended''' regular expressions symbols.<br><br>For the remainder of this investigation, we will focus on '''complex regular expressions''' and then<br>focus on ''extended regular expressions'' in INVESTIGATION 2.<br><br><table align="right"><tr valign="top"><td>[[Image:regexps-2.png|thumb|right|280px|Anchoring regular expressions at the '''beginning''' of text.]]</td><td>[[Image:regexps-3.png|thumb|right|250px|Anchoring regular expressions at the '''ending''' of text.]]</td></tr></table>
| |
| # Issue the following Linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "^the" textfile1.txt</span><br><br>The '''^''' symbol is referred to as an '''anchor'''.<br>In this case, it only matches<br>the word "'''the'''" (both upper or lowercase) at the <u>beginning</u> of the string.<br><br>
| |
| # Issue the following Linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "the$" textfile1.txt</span><br><br>The '''$''' symbol is used to anchor patterns at the <u>end</u> of the string.<br><br>
| |
| # Issue the following Linux command to anchor the <u>word</u> "'''the'''"<br>'''simultaneously''' at the <u>beginning</u> and <u>end</u> of the string:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "^the$" textfile1.txt </span><br><br>What do you notice?<br><br>Anchoring patterns at both the <u>beginning</u> and <u>ending</u> of strings can greatly assist<br>for more '''precise''' search pattern matching.<br><br>We will now be demonstrate the '''effectiveness''' of <u>combining</u><br> '''anchors''' with <u>other</u> complex regular expressions symbols.<br><br><table align="right"><tr valign="top"><td>[[Image:regexps-4.png|thumb|right|280px|Anchoring regular expressions using '''period''' symbols at the '''beginning''' of text.]]</td><td>[[Image:regexps-5.png|thumb|right|250px|Anchoring regular expressions using '''period''' symbols simultaneously at the '''beginning''' and '''ending''' of text.]]</td></tr></table>
| |
| # Issue the following Linux command to match strings that '''begin with 3 characters''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^..." textfile1.txt</span><br><br>What do you notice? Can lines that contain '''less than 3 characters''' be displayed?<br><br>
| |
| # Issue the following Linux command to match strings that '''begin <u>and</u> end with 3 characters''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^...$" textfile1.txt</span><br><br>What do you notice compared to the previous command?<br><br>
| |
| # Issue the following Linux command to match strings that '''begin with 3 digits''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9][0-9]" textfile1.txt</span><br><br>What did you notice?<br><br>
| |
| # Issue the following Linux command to match strings that '''end with 3 uppercase letters''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "[A-Z][A-Z][A-Z]$" textfile1.txt</span><br><br><table align="right"><tr valign="top"><td>[[Image:regexps-6.png|thumb|right|220px|Anchoring '''3 digits''' at the '''beginning''' and '''ending''' of text.]]</td><td>[[Image:regexps-7.png|thumb|right|250px|Anchoring '''3 alpha-numeric characters''' at the '''beginning''' and '''ending''' of text.]]</td></tr></table>What type of strings match this pattern?<br><br>
| |
| # Issue the following Linux command to match strings that '''consist of only 3 digits''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9][0-9]$" textfile1.txt</span><br><br>What did you notice?<br><br>
| |
| # Issue the following Linux command to match strings that '''consist of only 3 alphanumeric digits''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[a-zA-Z0-9][a-zA-Z0-9][a-zA-Z0-9]$" textfile1.txt</span><br><br>What did you notice?<br><br>The <span style="font-weight:bold;font-family:courier;">"*"</span> complex regular expression symbol is often confused with the "*" '''filename expansion''' symbol.<br>In other words, it does NOT represent zero or more of '''any character''', but zero or more '''occurrences'''<br>of the character that comes '''before''' the <span style="font-weight:bold;font-family:courier;">"*"</span> symbol.<br><br>
| |
| # To demonstrate, issue the following Linux command to display '''zero or more occurrences''' of the letter "'''x'''":<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "x*" textfile1.txt</span><br><br>You will most likely notice most lines of the file is displayed.<br><br>
| |
| # Let's issue a Linux command to display strings that contain '''more than one occurrence''' of the letter "'''x'''":<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "xx*" textfile1.txt</span><br><br>Why did this work? because the pattern indicates one occurrence of the letter "x",<br>followed by '''zero or MORE occurrences''' of the <u>next</u> letter "x".<br><br>If you combine the complex regular expression symbols <span style="font-weight:bold;font-family:courier;">".*"</span> it will act like<br>zero or more occurrences of <u>any</u> character (i.e. like <span style="font-weight:bold;font-family:courier;">"*"</span> did in filename expansion).<br><br>
| |
| # Issue the following Linux command to match strings begin and end with a number with nothing or anything inbetween:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9].*[0-9]$" textfile1.txt</span><br><br>Using '''simultaneous anchors''' combined with the <span style="font-weight:bold;font-family:courier;">".*"</span> symbol(s) can help you to refine your search patterns of strings.<br><br>
| |
| # Issue the following Linux command to display strings that begin with a capital letter,<br>end with a number, and contains a capital X somewhere inbetween:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[A-Z].*X.*[0-9]$" textfile1.txt</span><br><br>Let's look at another series of examples involving searching for strings that only contain '''valid numbers'''.<br>We will use '''pipeline commands''' to both display stdout to the screen and save to files<br>for confirmation of running these pipeline commands when run a '''checking-script''' later in this investigation.<br><br>
| |
| # Issue the following Linux command to create the '''regexps''' directory: <span style="color:blue;font-weight:bold;font-family:courier;">mkdir ~/regexps</span><br><br>
| |
| # Change to the '''regexps''' directory and confirm that you have moved to this directory.<br><br>
| |
| # First, issue the following Linux command to copy another data file called '''numbers1.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">cp ~uli101/tutorialfiles/numbers1.dat ~/regexps</span><br><br>
| |
| # View the contents of the '''numbers.dat''' file using the '''more''' command and quickly view the contents of this file.<br>You should notice '''valid''' and '''invalid''' numbers contained in this file. When finished, exit the more command.<br><br>
| |
| # Issue the following linux pipeline command to display only '''whole''' numbers (i.e. no '''+''' or '''-''' sign):<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9]*$" numbers1.dat | tee faulty.txt</span><br><br>You may have noticed that the command '''does not entirely work'''. You may notice an '''empty line'''<br>(which is NOT a whole number). This occurs since the * regular expression symbol represents<br>ZERO or MORE occurrences of a number. You can use an additional numeric character class<br>with the * regular expression symbol to search for one or more occurrences of a number.<br><br>
| |
| # Issue the following Linux pipeline command to display only whole numbers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9]*$" numbers1.dat | tee whole.txt</span><br><br>You should see that this now works.<br><br>
| |
| # Issue the following Linux pipeline command to display <u>only</u> '''signed''' integers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-][0-9][0-9]*$" numbers1.dat | tee signed.txt</span><br><br>What did you notice? Positive and negative numbers display, not '''unsigned''' numbers.<br><br>[[Image:regexps-8.png|thumb|right|300px|Simultaneous '''anchoring''' of regular expressions using '''character class''' and '''zero or more occurrences''' to display '''signed''' and '''unsigned''' integers.]]
| |
| # Issue the following Linux pipeline command to display '''signed''' or '''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]*[0-9][0-9]*$" numbers1.dat | tee all.txt</span><br><br>Did this command work?<br><br>
| |
| # Issue the following command to check that you created those hard links: <br><span style="color:blue;font-weight:bold;font-family:courier;">~uli101/week9-check-1</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script. If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>You can also use the '''grep''' command using ''regular expression'' as a '''filter''' in pipeline commands.<br><br>
| |
| # Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | grep "[0-9].*dat$"</span><br><br>What did this pipeline display?<br><br>
| |
| # Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | grep "[a-z].*txt$"</span><br><br>What did this pipeline display?<br><br>
| |
| | |
| : Although very useful, '''complex''' regular expressions do NOT <u>entirely</u> solve our problem of displaying<br> '''valid''' unsigned and signed numbers (not to mention displaying decimal numbers).<br><br>In the next investigation, you will learn how to use '''extended''' regular expressions that will completely solve this issue.<br>
| |
| | |
| : You can proceed to INVESTIGATION 2.
| |
| <br>
| |
| | |
| =INVESTIGATION 2: EXTENDED REGULAR EXPRESSIONS =
| |
| | |
| <br>
| |
| In this investigation, you will learn how to use '''extended regular expressions''' with the '''egrep''' command<br>to further refine your search patterns.
| |
| | |
| | |
| '''Perform the Following Steps:'''
| |
| | |
| # Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account.<br><br>
| |
| # Issue the following Linux command to copy another data file called '''numbers2.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">cp ~uli101/tutorialfiles/numbers2.dat ~/</span><br><br>
| |
| # View the contents of the '''numbers2.dat''' file using the '''more''' command and quickly view the contents of this file.<br>You should notice ''valid'' and ''invalid'' numbers contained in this file. When finished, exit the more command.<br><br>[[Image:eregexps-1.png|thumb|right|300px|'''Weakness''' of '''complex''' regular expressions that do not '''limit''' the number of '''positive''' or '''negative''' signs.]]
| |
| # Issue the following Linux command to display '''signed''' or '''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]*[0-9][0-9]*$" numbers2.dat</span><br><br>You should notice '''multiple''' '''+''' or '''-''' '''signs''' appear <u>prior</u> to some numbers.<br>This occurs since you are searching or one or MORE occurrences of a + or - sign.<br><br>Using '''extended regular expression''' symbols to specify '''minimum''' and '''maximum''' repetitions: '''{min,max}''' can solve that problem.<br><br>
| |
| # Issue the following Linux command (using extended regular expression symbols)<br>to display '''signed''' or '''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]{0,1}[0-9]{1,}$" numbers2.dat</span><br><br>'''NOTE: No output will be displayed! Why?'''<br><br>This is due to the fact that the '''grep command was NOT issued correctly to use extended regular expression symbols'''.<br>You would need to issue either '''grep -E''', or just issue the '''egrep''' command. The egrep command works with<br>'''all''' regular expression symbols, and should be used in the future <u>instead</u> of the older grep command.<br><br>We will use '''pipeline commands''' to both display stdout to the screen and save to files<br>for confirmation of running these pipeline commands when run a '''checking-script''' later in this investigation.<br><br>
| |
| # Issue the following Linux pipeline command using '''egrep''' instead of ''grep'':<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]{0,1}[0-9]{1,}$" numbers2.dat | tee better-number1.txt</span><br><br>[[Image:eregexps-2.png|thumb|right|300px|Using '''extended''' regular expression symbols (such as '''repetition''') to refine matches of ''signed'' and ''unsigned'' integers.]]You should have noticed that the command worked correctly this time because you used the '''egrep''' command.<br><br>'''NOTE:''' With extended regular expressions, the '''?''' symbol can be used to represent the '''{0,1}''' repetition symbols and the '''+''' symbol can be used to represent the '''{1,}''' repetition symbols<br><br>
| |
| # Issue the following Linux pipeline command using the repetition shortcuts <span style="font-weight:bold;font-family:courier;">"+"</span> and <span style="font-weight:bold;font-family:courier;">"?"</span>:<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]?[0-9]+$" numbers2.dat | tee better-number2.txt</span><br><br>You should have seen the '''same results''', but less typing was required.<br><br>
| |
| # Issue the following Linux pipeline command to display '''signed''', '''unsigned''', '''whole''', and '''decimal''' numbers:<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]{0,1}[0-9]{1,}[.]{0,1}[0-9]*$" numbers2.dat | tee better-number3.txt</span><br><br>Were all signed and unsigned intergers and decimal numbers displayed?<br><br>
| |
| # Issue the follwoing command to check that you correctly issued<br>those ''Linux pipeline commands'': <br><span style="color:blue;font-weight:bold;font-family:courier;">~uli101/week9-check-2</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.<br>If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>You can also use extended regular expression symbols for '''grouping'''.<br>For example, you can search for repetitions of GROUPS of characters (like a word)<br>as opposed to just a single character or a GROUP of numbers as opposed to a single digit.<br><br>
| |
| # Issue the following linux pipeline command to copy another data file called '''words.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">cp ~uli101/tutorialfiles/words.dat ~/</span><br><br>
| |
| # View the contents of the '''words.dat''' file using the '''more''' command and quickly view the contents of this file.<br>Within this file, you should notice some lines that contain repetitions of words. When finished, exit the more command.<br><br>
| |
| # Issue the following linux pipeline command to display '''two or more occurrences''' of the word "the":<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the){2,}" words.dat | tee word-search1.txt more</span><br><br>'''NOTE: No output is displayed! Why?'''<br><br>This is due to the fact that a <u>space</u> should be included at the end of the word "'''the'''".<br>Usually words are separated by spaces; therefore, there were no matches since there were not occurrences<br>of "thethe" as opposed to "'''the the'''" (i.e. no space after repetition of the pattern).<br><br>
| |
| # Reissue the previous pipeline command with the word the followed by a '''space''' within the brackets:<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the ){2,}" words.dat | tee word-search2.txt</span><br><br>[[Image:eregexps-3.png|thumb|right|330px|Using '''extended''' regular expression symbols (such as '''grouping''') to refine matches of repetition of '''words''' (as opposed to ''characters'').]]The <span style="font-weight:bold;font-family:courier;">"|"</span> (or) symbol (same symbol as "pipe") can be used within the grouping symbols to allow matching of additional groups of characters.<br>Again, it is important to follow the character groupings with the space character<br><br>
| |
| # Issue the following linux pipeline command to search for '''two or more occurrences''' of the word "'''the '''" <u>or</u> '''two or more occurrences''' of the word "'''and '''":<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the |and ){2,}" words.dat | tee word-search3.txt</span><br><br>
| |
| # Issue the following Linux command to check that you correctly issued<br>those ''Linux pipeline commands'' using the '''tee''' command to create those text files:<br><span style="color:blue;font-weight:bold;font-family:courier;">~uli101/week9-check-3</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.<br>If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>Let's issue a Linux '''pipeline''' command using the '''egrep''' command as a '''filter'''<br>using <u>both</u> '''complex''' and '''extended''' regular expressions.<br><br>
| |
| # Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | egrep "[a-z]{1,}.*[0-9]"</span><br><br>What did this Linux pipeline command display?<br><br>
| |
| : The '''grep''' and '''egrep''' Linux commands are NOT the only Linux commands that use regular expressions.<br>In the next investigation, you will apply regular expressions to a number of Linux commands<br>that you already learned in this course.
| |
| | |
| : You can proceed to INVESTIGATION 3<br>
| |
| <br>
| |
| | |
| =INVESTIGATION 3: OTHER COMMANDS USING REGULAR EXPRESSIONS =
| |
| <br>
| |
| In this investigation, you will see commands other than '''grep''' or '''egrep''' that can use regular expressions.
| |
| | |
| | |
| '''Perform the Following Steps:'''
| |
| | |
| # Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account.<br><br>
| |
| # Let's look at using regular expressions with the '''man''' command.<br>Issue the following linux command :<br><span style="color:blue;font-weight:bold;font-family:courier;">man ls</span><br><br>[[Image:other-re-1.png|thumb|right|300px|Entering '''/sort''' in the '''man''' command can search for the string "'''sort'''".]]
| |
| # We want to search for an option that can sort the file listing.<br>Type the following regular expression below and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/sort</span><br><br>'''FYI:''' The '''grep''' and '''egrep''' Linux commands contain the regular expressions within quotes, but '''most''' other Linux commands specify regular expressions using<br>'''forward slashes''' (e.g. <span style="font-weight:bold;font-family:courier;">/regular expression</span> or <span style="font-weight:bold;font-family:courier;">/regular expression/</span>).<br><br>
| |
| # Scroll throughout the man pages for the ls command to view matches for the pattern "'''sort'''"<br>(You can press '''SPACE''' or key combination '''alt-b''' to move forward and backwards one screen respectively).<br><br>
| |
| # Press the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span> to '''exit''' the ''man'' pages for '''ls'''.<br><br>Let's use regular expressions with the '''less''' command.<br><br>
| |
| # Issue the following Linux command to copy another data file called '''large-file.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">cp ~uli101/tutorialfiles/large-file.txt ~/</span><br><br>[[Image:other-re-2.png|thumb|right|300px|Entering '''/uli101''' in the '''less''' command can display all matches of "'''uli101'''" throughout the text file.]]
| |
| # Issue the following Linux command to view the contents of the '''large-file.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">less large-file.txt</span><br><br>
| |
| #We want to search for a pattern '''uli101''' within this text file.<br>Type the following regular expression and press ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">/uli101</span><br><br>You should see the pattern "uli101" throughout the text file.<br><br>
| |
| # Press the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span> to exit the '''less''' command.<br><br>
| |
| # Try the same search techniques with the '''more''' command.<br><br>Does it work the same for the ''less'' command?<br><br>[[Image:other-re-3.png|thumb|right|300px|Entering '''/uli101''' in the '''vi''' command can search for the string "'''uli101'''".]]Let's learn how to perform a simple '''search and replace''' within the '''vi''' utility<br>by using regular expressions.<br><br>
| |
| # Issue the following Linux command to edit the '''large-file.txt''' file:<br><span style="color:blue;font-weight:bold;font-family:courier;">vi large-file.txt</span><br><br>Let's first perform a simple search within this text file.<br><br>
| |
| # Press the '''ESC''' key to make certain you are in '''COMMAND''' mode.<br><br>
| |
| # Type the following and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/uli101</span><br><br>You should notice the pattern "'''uli101'''" highlighted for ALL occurrences in this text file.<br><br>Let's '''search''' for the '''uli101''' pattern, and '''replace''' it in capitals (i.e '''ULI101''').<br><br>In vi, to issue a command, you need to enter '''LAST LINE''' MODE then issue a command.<br>Let's issue a command from '''LAST LINE''' MODE to search and replace '''uli101''' to '''ULI101'''.<br><br>[[Image:other-re-4.png|thumb|right|500px|In l'''ast line''' MODE in the '''vi''' text editor, issuing a command using regular expressions to convert '''uli101''' to '''ULI101'''.]]
| |
| # Making certain that you are '''COMMAND''' MODE in vi,<br>type the following and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">:%s/uli101/ULI101/g</span><br><br>'''NOTE:''' The letter '''g''' after the replace regular expression represents "'''global'''" and will replace ALL occurrences of uli101 in the text document (as opposed to replacing the first occurrence for every line).<br><br>
| |
| # Type the following (in uppercase letters) and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/ULI101</span><br><br>You should notice the pattern "'''ULI101'''" highlighted for ALL occurrences in this text file.<br><br>
| |
| # Navigate throughout the text file to confirm that ALL occurrences of '''uli101''' have been <u>replaced</u> with '''ULI101'''.<br><br>
| |
| # Save changes to your vi editing session and exit by typing the following and pressing ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">:x</span><br><br>
| |
| | |
| = LINUX PRACTICE QUESTIONS =
| |
| | |
| The purpose of this section is to obtain '''extra practice''' to help with '''quizzes''', your '''midterm''', and your '''final exam'''.
| |
| <br><br>
| |
| ==REVIEW QUESTIONS: SIMPLE & COMPLEX REGULAR EXPRESSIONS==
| |
| | |
| Here is a link to the MS Word Document of ALL of the questions displayed below but with extra room to answer on the document to
| |
| simulate a quiz:
| |
| | |
| https://wiki.cdot.senecacollege.ca/uli101/files/uli101_command_practice_9a.docx
| |
| | |
| Your instructor may take-up these questions during class. It is up to the student to attend classes in order to obtain the answers to the following questions. Your instructor will NOT provide these answers in any other form (eg. e-mail, etc).
| |
| | |
| | |
| '''Part A: Display Results from Linux Commands using Simple & Complex Regular Expressions'''
| |
| | |
| Note the contents from the following tab-delimited file called '''~uli101/cars''':
| |
| | |
| <pre>
| |
| Plym fury 77 73 2500
| |
| chevy nova 79 60 3000
| |
| ford mustang 65 45 10003
| |
| volvo gl 78 102 9850
| |
| ford ltd 83 15 10507
| |
| chevy nova 80 50 3503
| |
| fiat 600 65 115 450
| |
| honda accord 81 30 6000
| |
| ford thundbd 84 10 17000
| |
| toyota tercel 82 180 755
| |
| chevy impala 65 85 1553
| |
| ford bronco 83 25 9505
| |
| </pre>
| |
| | |
| Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.
| |
| | |
| 1. <span style="font-family:courier;font-weight:bold">grep plym ~uli101/cars</span><br>
| |
| 2. <span style="font-family:courier;font-weight:bold">grep -i fury ~uli101/cars</span><br>
| |
| 3. <span style="font-family:courier;font-weight:bold">grep “^[m-z]” ~uli101/cars</span><br>
| |
| 4. <span style="font-family:courier;font-weight:bold">grep -i “^[m-z]” ~uli101/cars</span><br>
| |
| 5. <span style="font-family:courier;font-weight:bold">grep “3$” ~uli101/cars</span><br>
| |
| 6. <span style="font-family:courier;font-weight:bold">grep -i “c.*5$” ~uli101/cars</span><br>
| |
| | |
| | |
| | |
| '''Part B: Writing Linux Commands Using Regular Expressions'''
| |
| | |
| Write a single Linux command to perform the specified tasks for each of the following questions.
| |
| | |
| 7. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains the pattern:
| |
| the<br>
| |
| 8. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains the word:
| |
| the<br>
| |
| 9. Write a Linux command to display all lines in the file called '''~/text.txt''' that begin with a number.<br>
| |
| 10. Write a Linux command to display all lines in the file called '''~/text.txt''' that end with a letter
| |
| (either upper or lowercase).<br>
| |
| 11. Write a Linux command to display all lines in the file called '''~/text.txt''' that begin and end with a number.<br>
| |
| 12. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains exactly 3 characters that can be anything.<br>
| |
| 13. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains exactly 3 numbers.<br>
| |
| 14. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains 1 or more “C” characters.<br>
| |
| | |
| <br><br>
| |
| | |
| ==REVIEW QUESTIONS: REGULAR EXPRESSIONS (INCLUDING EXTENDED REGULAR EXPRESSIONS)==
| |
| | |
| Here is a link to the MS Word Document of ALL of the questions displayed below but with extra room to answer on the document to
| |
| simulate a quiz:
| |
| | |
| https://wiki.cdot.senecacollege.ca/uli101/files/uli101_command_practice_9b.docx
| |
| | |
| Your instructor may take-up these questions during class. It is up to the student to attend classes in order to obtain the answers to the following questions. Your instructor will NOT provide these answers in any other form (eg. e-mail, etc).
| |
| <br><br>
| |
| '''Part A: Display Results from Linux Commands using Regular Expressions'''
| |
| | |
| Note the contents from the following tab-delimited file called '''~uli101/numbers.txt''':
| |
| | |
| <pre>
| |
| +123
| |
| ---34
| |
| +++++++++++17
| |
| -45
| |
| 45p8
| |
| 25.6
| |
| 11
| |
| </pre>
| |
| | |
| Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.
| |
| | |
| 1. <span style="font-family:courier;font-weight:bold">grep "^[-+]" ~uli101/numbers.txt</span><br>
| |
| 2. <span style="font-family:courier;font-weight:bold">grep "^[-+]*.[0-9]" ~uli101/numbers.txt</span><br>
| |
| 3. <span style="font-family:courier;font-weight:bold">grep "^[+-]?[0-9]" ~uli101/numbers.txt</span><br> (Why?)<br>
| |
| 4. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]" ~uli101/numbers.txt</span><br>
| |
| 5. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]+$" ~uli101/numbers.txt</span><br>
| |
| 6. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]+[.]?[0-9]+$" ~uli101/numbers.txt</span><br>
| |
| | |
| | |
| '''Part B: Writing Linux Commands Using Regular Expressions'''
| |
| | |
| Write a single Linux command to perform the specified tasks for each of the following questions.
| |
| | |
| | |
| 7. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with 1 or more occurrences of an UPPERCASE letter.
| |
| | |
| 8. Write a Linux command to display all lines in the file called '''~/data.txt''' that ends with 3 or more occurrences of the number 6
| |
| | |
| 9. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with 2 or more occurrences of the word “the” (upper or lower case).
| |
| | |
| 10. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with 2 or more occurrences<br> of the word “the” <u>or</u> the word “but” (upper or lower case).
| |
| | |
| 11. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with a minimum of 2 occurrences<br> and a maximum of 4 occurrences of the word “the” or the word “but” (upper or lower case).
| |
| | |
| | |
| _________________________________________________________________________________
| |
| | |
| Author: Murray Saul
| |
| | |
| License: LGPL version 3
| |
| Link: https://www.gnu.org/licenses/lgpl.html
| |
| | |
| _________________________________________________________________________________
| |
|
| |
|
| | * Run ls -l on the files you uploaded to the server. |
| | * Run ls -l on the same files on your workstation. |
|
| |
|
| | Note that everything you uploaded the server is owned by the username you logged in as after running the scp command. Your regular user from your workstation doesn't exist on the server. |
|
| |
|
| | =Submit evidence of your work= |
| | After you finish the lab: run the following command to submit your work on the ops345.ca server:<syntaxhighlight lang="bash"> |
| | ops145-lab8-check.sh |
| | </syntaxhighlight>If it says "Your lab5 has been submitted": make a screenshot, and you're done. If it gives you any warnings or errors: you have to fix them and try the ./ops145-lab5-check.sh command again. |
| [[Category:OPS145]] | | [[Category:OPS145]] |
