Revision as of 09:30, 21 February 2025

!!!THIS LAB IS NOT READY YET!!!

So far, you have created Python programs to prompt a user to input data from the keyboard. When creating Python programs, you may also need to process large volumes of information, or store processed data. Since many tasks that system administrators perform deal with files: reading and writing files is a crucial skill.

It is very important to provide logic in your Python program in case it encounters an error. An example would be an invalid path or trying to close a file that is already closed. The second part in this lab will look into how Python interpreters handle errors (commonly referred to as exception handling) at run time, and learn how to write Python code that will run gracefully even when problems occur during program execution.

Working with Files

You will now learn how to open text files, read the contents within a text file, process the contents, and finally write the processed contents back into a file. These operations are very common, and are used extensively in programming. Examples of file operations would include situations such as logging output, logging errors, reading and creating configuration/temporary files, etc.

Files are accessed through the use of file objects. An object is a variable which stores data in the form of attributes (data contents) and methods (functions).

Reading Data From Files

Create a new python file for testing.

Create a new text file in the lab5 directory:

cd ~/ops435/lab5
vi ~/ops435/lab5/data.txt

Place the following content inside the new text file and save it:

This is the second line
Third line
Last line

In order to read data from a text file, we need to create an object that will be used to access the data in a file. In some programming languages (like C) this is called a file descriptor, or a file pointer. In Python, it's an object.

Define a variable called f in order to help retrieve content from our text file.:
```
f = open("data.txt", "r")
```

The open() function takes two string arguments: a path to a file, and a mode option (to ask for reading, writing, appending, etc). The open() function will return a file object to us, this file object will allow us to read the lines inside the file. Here are the most useful functions for text file manipulation:

f.read()       # read all lines and stores in a string
f.readlines()  # read all lines and stores in a list
f.readline()   # read first line, if run a second time it will read the second line, then third
f.close()      # close the opened file

Next, read data from the opened file and store the contents in a variable called read_data, and then confirm the contents of the variable read_data:
```
data_from_file = f.read()
print(data_from_file)
```

After you're done working with file, you should close the file. There are good reasons not to forget this. It is sometimes useful to first confirm that the file is still open prior to closing it. But really you should know - it's your code that would have opened it:

f.close()                               # This method will close the file

Let's take some time to revisit the file read sequence. The following code sequence will open a file, store the contents of that file into a variable, and close the file:

f = open('data.txt', 'r')   # Open file
data_from_file = f.read()   # Read from file
f.close()                   # Close file

data_from_file in this case contains the data from the file in a single long string. The end of each line in the file is part of that string, as the special character '\n' which represents the newline character.

Often it is convenient to split the string on newline characters, so each line is stored as an item in a list:

data_from_file.split('\n')                   # Returns a list
list_of_lines = data_from_file.split('\n')   # Saves returned list in variable
print(list_of_lines)

There are other methods the we can use with our object f to place lines from our file into a list. Try these two:

# METHOD 1:

f = open('data.txt', 'r')
# The method1 list is constructed by reading lines from the file descriptor:
method1 = list(f)
f.close()
print(method1)

# METHOD 2:

f = open('data.txt', 'r')
# The readlines() method returns a list with each element being one line from the file:
method2 = f.readlines()
f.close()
print(method2)

Practice Reading Files

Create the ~/ops435/lab5/lab5a.py program. Use the following as a template:

#!/usr/bin/env python3

def read_file_string(file_name):
    # Takes a filename string, returns a string of all lines in the file

def read_file_list(file_name):
    # Takes a filename string, returns a list of lines without new-line characters

if __name__ == '__main__':
    file_name = 'data.txt'
    print(read_file_string(file_name))
    print(read_file_list(file_name))

This program will read the same file (data.txt) that you previously created
The read_file_string() function should return a string
The read_file_list() function should return a list
The read_file_list() function must remove the new-line characters from each line in the list
Both functions must accept one argument which is a string

Sample Run 1:

python3 lab5a.py 
Hello World
This is the second line
Third line
Last line
['Hello World', 'This is the second line', 'Third line', 'Last line']

Sample Run 2 (with import):

import lab5a        
file_name = 'data.txt'                             
print(lab5a.read_file_string(file_name))
# Will print 'Hello World\nThis is the second line\nThird line\nLast line\n'
print(lab5a.read_file_list(file_name))
# Will print ['Hello World', 'This is the second line', 'Third line', 'Last line']

Download the checking script and check your work:

cd ~/ops435/lab5/
pwd #confirm that you are in the right directory
ls CheckLab5.py || wget http://ops345.ca/ops445/CheckLab5.py
python3 ./CheckLab5.py -f -v lab5a

Before proceeding, make certain that you identify all errors in lab5a.py. When the checking script tells you everything is OK - proceed to the next step.

Writing To Files

Up to this point, you have learned how to access text from a file. In this section, you will learn how to write text to a file. Writing data to a file is useful for creating new content in a file or modifying existing data contained within a file. If you don't have programming experience: the modifying part is much more complicated than you might imagine.

Risk of Losing File Contents
A common problem that new programmers may encounter is to accidentally erase existing contents of a file when writing new data to a file. When opening files for writing (using the 'w' open function option), Python assumes existing content in the file is no longer wanted and it's immediately deleted; therefore, if you wish to write data to a file but keep existing content, you need to use the open file option 'a' (append new data to a file).

When opening a file for writing, the 'w' parameter is passed to the open() function. When the 'w' option is chosen: previous (existing) content inside the file is deleted. This deletion takes place the moment the open() function is executed, not when writing to the file. If the file that is being written to doesn't exist, the file will be created when open() is called.

Create a temporary Python file and open a non-existent data file (called file1.txt) for writing:
```
f = open('file1.txt', 'w')
```
To confirm that the new file now exists and is empty, run this in the shell:
```
ls -l file1.txt
```

To add text to the file, you can use the write() method for the file object. Usually you end every line in a text file with the special character '\n' to represent a "new line". Multiple lines may be written using a single write operation: simply put the special character '\n' wherever each line should end.

Add three line to your data file:

f.write('Line 1\nLine 2 is a little longer\nLine 3 is too\n')

Once the write() method has completed, the final step is to close() the file. The file MUST be closed properly or else data will not consistently be written to the file (the same way you have to eject a USB storage device in your operating system before you yank it out):
```
f.close()
```
View the contents of the file in the shell to make sure the data was written successfully:
```
cat file1.txt
```

Now create a new file called file2.txt, but this time call multiple write() methods in sequence. You will often write to a file multiple times inside a loop:

f = open('file2.txt', 'w')
f.write('Line 1\nLine 2 is a little longer\nLine 3 is as well\n')
f.write('This is the 4th line\n')
f.write('Last line in file\n')
f.close()

Run the following shell command to confirm that the contents were written to file2.txt:
```
cat file2.txt
```

Make Backup Copies of Your Data Files
Since you might make a mistake, and accidentally destroy file contents when writing to your file: it is highly recommended to make backup copies of your files prior to running your Python programs in this lab. This can be particularly useful when performing any future assignment which involves writing to files.

Make Backup Copies of ALL your files
Since you are now writing code that opens files for writing you may accidentally truncate the wrong file (like your assignment file, for example). Make regular backups of all your work. Just copy it to a USB stick, or if you're bored: learn version control.

Backup both of your newly-created files and confirm that they were backuped up:
```
cp file1.txt file1.txt.bk
cp file2.txt file2.txt.bk
ls -l file*
```
Let's see what can happen if you perform an incorrect write() operation. Add this to the end of your temporary python program:
```
f = open('file2.txt', 'w')
```
And then check file2.txt:
```
cat file2.txt
```

You should notice that the previous content in your file2.txt file was lost. Why do you you think the previous data is no longer there?

Restore your file from the backup and verify the backup restoration:
```
cp file2.txt.bk file2.txt
cat file2.txt
```

To avoid overwriting the contents of a file, we can open the file for appending instead of writing (the difference in those two words is technical language, not human english).

Use the option a instead of w to to append to a file:

f = open('file1.txt', 'a')
f.write('This is the 4th line\n')
f.write('Last line in file\n')
f.close()

Check your file1.txt contents, you'll see that the two lines written above are appended to the previous content in the file.

The final thing to consider when writing to files is that write() will only accept a string as an argument. This means that before trying to write integers, floats, lists, or dictionaries to a file, first either convert the value using str() function or extract the specific strings from items in the list.

In this example we convert a single number and all the numbers in a list to strings before writing them to a file:

my_number = 1000
my_list = [1,2,3,4,5]

f = open('file3.txt', 'w')
f.write(str(my_number) + '\n')

for num in my_list:
    f.write(str(num) + '\n')

f.close()

Check your file3.txt contents to confirm that all the numbers have been written to it.

Practice Writing to Files

Copy ~/ops435/lab5/lab5a.py to ~/ops435/lab5/lab5b.py

Add the following functions below the two functions that you have already defined:

def append_file_string(file_name, string_of_lines):
    # Appends the string_of_lines to the end of the file named file_name.

def write_file_list(file_name, list_of_lines):
    # Writes all items from list_of_lines to the file named file_name 
    # where each item is one line.

def copy_file_add_line_numbers(file_name_read, file_name_write):
    # Reads data from the file named file_name_read, writes that data 
    # to the file named file_name_write.
    # Adds a line number to each line in the new file.

Replace the main section of your program with the following:

if __name__ == '__main__':
    file1 = 'seneca1.txt'
    file2 = 'seneca2.txt'
    file3 = 'seneca3.txt'
    
    string1 = 'First Line\nSecond Line\nThird Line\n'
    list1 = ['Line 1', 'Line 2', 'Line 3']
    
    append_file_string(file1, string1)
    print(read_file_string(file1))
    
    write_file_list(file2, list1)
    print(read_file_string(file2))
    
    copy_file_add_line_numbers(file2, file3)
    print(read_file_string(file3))

Implement the functions following these specifications:
- append_file_string():
  - Takes two string arguments.
  - Appends to the file (Argument 1) all data from the string (Argument 2).
- write_file_list():
  - Takes two arguments: a string and a list.
  - Writes to the file (Argument 1) all the lines from the list (Argument 2).
- copy_file_add_line_numbers():
  - Takes two arguments: Both are filenames (which are strings).
  - Reads all the lines from the first file (Argument 1), and writes all those lines into second file (Argument 2), while adding line numbers.
  - Line numbers should be added to the beginning of each line with a colon next to them (see sample output below for reference).
  - Hint: Use an extra variable for the line number.

Sample Run 1:

rm seneca1.txt seneca2.txt seneca3.txt
./lab5b.py 
First Line
Second Line
Third Line

Line 1
Line 2
Line 3

1:Line 1
2:Line 2
3:Line 3

Sample Run 2 (run second time):

python3 lab5b.py 
First Line
Second Line
Third Line
First Line
Second Line
Third Line

Line 1
Line 2
Line 3

1:Line 1
2:Line 2
3:Line 3

Sample Run 3 (with import):

import lab5b
file1 = 'seneca1.txt'
file2 = 'seneca2.txt'
file3 = 'seneca3.txt'
string1 = 'First Line\nSecond Line\nThird Line\n'
list1 = ['Line 1', 'Line 2', 'Line 3']

lab5b.append_file_string(file1, string1)

lab5b.read_file_string(file1)
# Will return 'First Line\nSecond Line\nThird Line\nFirst Line\nSecond Line\nThird Line\n'

lab5b.write_file_list(file2, list1)

lab5b.read_file_string(file2)
# Will return 'Line 1\nLine 2\nLine 3\n'

lab5b.copy_file_add_line_numbers(file2, file3)

lab5b.read_file_string(file3)
# Will return '1:Line 1\n2:Line 2\n3:Line 3\n'

Download the checking script and check your work:

cd ~/ops435/lab5/
pwd #confirm that you are in the right directory
ls CheckLab5.py || wget wget http://ops345.ca/ops445/CheckLab5.py
python3 ./CheckLab5.py -f -v lab5b

Before proceeding, make certain that you identify all errors in lab5b.py. When the checking script tells you everything is OK - proceed to the next step.

Exceptions and Error Handling

Every programmer and every program runs into errors all the time. You should expect that it will happen for any code that you write. In Python, when an error occurs, the python runtime raises an exception. This section will teach you to catch these exceptions when they happen and to allow the program to continue running, or to stop program execution with a readable error message.

There exists a massive number of exceptions. If you write a lot of a certain type of code: you'll become familiar with the exceptions the methods you typically use throw. Online references can be useful for new-to-you methods. You also have an option to not handle exceptions at all (as you did so far in this course), but your resulting program may not be acceptable, depending on where and how it's used.

In this section, we will provide examples of how to handle a few exceptions in Python.

To start, create a new temporary python program. Before attempting to handle exception errors, let's create an error, and then see how to can "handle" it:

print('5' + 10)

You should get an exception error similar to the following:

---------------------------------------------------------------------------
Traceback (most recent call last)
 Fiel "<stdin>", line 1, in <module>
TypeError: Can't convert 'int' object to str implicitly

Click on the link [1] and scroll down to TypeError. You've seen this error before so you already know what the problem is, but take a couple of minutes to read the documentation. It will be hard to understand what they're talking about, but this sort of exercise is worth the time and effort.

If we don't want the user of our program to have to learn how to read Python exceptions (which is a very good idea), we can catch/trap/handle this exception when it happens. This is done with a specific block of code called a try clause where you place code in-between the try: and the except: coding blocks. In a general sense, it works like a modified if-else statement, where the try statement acts as a test, and the except statement will or will not handle the exception depending if it occurs or does NOT occur. That is to say, If no error occurs in the code contained in the try section, the script will continue as usual but if an error occurs in the "try" section, then it can be handled in whatever way you feel is appropriate (like an user-friendly error message).

Let's demonstrate handling our TypeError error with code that first does not contain an error and then similar code that DOES generate an error.

The following code does NOT generate an error:

try:
    print(5 + 10)
    print("Still inside try block")
except TypeError:
    print('At least one of the values is NOT an integer')

print("End of program")

The following code handles an exception error to provide user-friendly feedback that at least one of the values is not an integer:

try:
    print(5 + 'ten')
    print("Still inside try block")
except TypeError:
    print('At least one of the values is NOT an integer')

print("End of program")

Walk through what happens in each of the two examples below. Notice that once an exception is thrown inside a try block: execution of the code stops, and jumps to the except block.

If an exception happens and it's not handled with try/except: your entire program will crash. That is by design, and is a good thing. You need to handle exceptional circumstances in your program, the computer can't do that for you.

Exceptions work the same way in all the programming languages that I've seen that have them, so the mechanics of handling exceptions is worth learning.

Let's generate another type of error where we try to open a file that doesn't exist:
```
f = open('filethatdoesnotexist', 'r')
```

Now, catch and handle this exception error:

try:
    f = open('filethatdoesnotexist', 'r')
    f.write('hello world\n')
    f.close()
except FileNotFoundError:
    print('no file found')

Multiple exceptions can also be caught at the same time, such as does not exist, is a directory, or we don't have permission.

To test out the error handling code (previously issued), try removing permissions from the file, or specify a directory instead of a regular file, and then try to open it:

try:
    f = open('filethatdoesnotexist', 'r')
    f.write('hello world\n')
    f.close()
except (FileNotFoundError, PermissionError, IsADirectoryError): 
    print('failed to open file')

By taking the time to view the Python Exception Hierarchy, you can see how errors get caught in python. The options FileNotFoundError, PermissionError, and IsADirectory are all inherited from OSError. This means that while using more specific errors might be useful for better error messages and handling, it's not always possible to catch every error all the time.
Another way to catch multiple exceptions is with separate except bloks:
```
try:
    f = open(abc, 'r')
    f.write('hello world\n')
    f.close()
except (FileNotFoundError, PermissionError): 
    print('file does not exist or wrong permissions')
except IsADirectoryError:
    print('file is a directory')
except OSError:
    print('unable to open file')
except:
    print('unknown error occured')
    raise
```
When catching multiple exceptions, make certain to catch the lowest ones contained in the exception-hierarchy first. For example, if you put 'Exception' first, both 'OSError' and 'FileNotFoundError', would never get caught.

TIP: In python it's usually best to 'try:' and 'except:' code rather than to attempt to anticipate everything that could go wrong with if statements. For example, instead of checking to see if a file exists and we have read permissions, it can be better to just try and read the file and fail and catch any errors with 'OSError'.

Create a Python Script Which Handles Errors

Create the ~/ops435/lab5/lab5c.py script.

Use the following as a template:

#!/usr/bin/env python3

def add(number1, number2):
    # Add two numbers together, return the result, if error return string 'error: could not add numbers'

def read_file(filename):
    # Read a file, return a list of all lines, if error return string 'error: could not read file'

if __name__ == '__main__':
    print(add(10,5))                        # works
    print(add('10',5))                      # works
    print(add('abc',5))                     # exception
    print(read_file('seneca2.txt'))         # works
    print(read_file('file10000.txt'))       # exception

Sample Run 1:

python3 lab5c.py 
15
15
error: could not add numbers
['Line 1\n', 'Line 2\n', 'Line 3\n']
error: could not read file

Sample Run 2 (with import):

import lab5c                

lab5c.add(10,5)             
15                          

lab5c.add('10',5)           
15                          

lab5c.add('10','5')         
15                          

lab5c.add('abc','5')                                                                                                                     
'error: could not add numbers'                                  

lab5c.add('hello','world')                                                                                                               
'error: could not add numbers'                                  

lab5c.read_file('seneca2.txt')
['Line 1\n', 'Line 2\n', 'Line 3\n']           

lab5c.read_file('file10000.txt')   
error: could not read file'

3. Exit the ipython3 shell, download the checking script and check your work. Enter the following commands from the bash shell.

cd ~/ops435/lab5/
pwd #confirm that you are in the right directory
ls CheckLab5.py || wget https://raw.githubusercontent.com/Seneca-CDOT/ops435/master/LabCheckScripts/CheckLab5.py
python3 ./CheckLab5.py -f -v lab5c

4. Before proceeding, make certain that you identify any and all errors in lab5c.py. When the checking script tells you everything is OK before proceeding to the next step.

LAB 5 SIGN-OFF (SHOW INSTRUCTOR)

Have Ready to Show Your Instructor:

✓ Output of: ./CheckLab5.py -f -v

✓ Output of: cat lab5a.py lab5b.py lab5c.py

LAB REVIEW

What is the purpose of a file object?
Write a Python command to open the text file called customers.txt for read-only operations.
Write Python code to efficiently store the contents of the file in question #2 as a large string (including new-line characters) called customer-data.
Write Python code to store the contents of the file in question #2 as a list, removing the new-line characters.
What is the purpose of closing an open file? Write a Python command to close the file opened in question #2.
Write the Python command to confirm you successfully closed the customers.txt file in question #5. What is the returned status from that command to indicate that the file has been closed?
What is the difference between opening a file for writing data as opposed to opening a file for appending data? What can be the consequence if you don't understand the difference between writing and appending data?
Write a Python command to open the file customer-data.txt for writing data.
Write a Python command to save the text: customer 1: Guido van Rossum (including a new-line character) to the opened file called customer-data.txt
Briefly explain the process writing a list as separate lines to an open file.
What is the purpose of handling exception errors?
Write a Python script to prompt a user for the name of the file to open. Use exception error handling to provide an error message that the specific file name (display that exact name) does not exist; otherwise, open the file for reading and display the entire contents of the file (line-by-line).

OPS445 Lab 5: Difference between revisions

Revision as of 09:30, 21 February 2025

Contents

!!!THIS LAB IS NOT READY YET!!!

Working with Files

Reading Data From Files

Practice Reading Files

Writing To Files

Practice Writing to Files

Exceptions and Error Handling

Create a Python Script Which Handles Errors

LAB 5 SIGN-OFF (SHOW INSTRUCTOR)

LAB REVIEW

Navigation menu

OPS445 Lab 5: Difference between revisions

Revision as of 09:30, 21 February 2025

!!!THIS LAB IS NOT READY YET!!!

Working with Files

Reading Data From Files

Practice Reading Files

Writing To Files

Practice Writing to Files

Exceptions and Error Handling

Create a Python Script Which Handles Errors

LAB 5 SIGN-OFF (SHOW INSTRUCTOR)

LAB REVIEW

Navigation menu

Search