Vasudev Ram: A basic file compare utility in Python

By Vasudev Ram


Weighing dishes from the island of Thera, Minoan civilization, 2000–1500 BC

Image attribution: Norbert Nagel

Recently, I had written a simple version of the Unix split command in Python. In order to check whether a split command works correctly, you have to join the files it creates - the split files - back into a single file, and then compare that new file to the original input file. So you need a file comparison utility. While both Unix and Windows have such utilities (cmp and fc.exe respectively), I thought of writing a simple one in Python. I did that, and then tested it with a few pairs of input files.

Here is the code, in file_compare.py:

# file_compare.py
# A simple file comparison utility.
# Author: Vasudev Ram
# Copyright 2016 Vasudev Ram

import sys
import os
from os.path import exists, getsize

def out_write(msg):
    sys.stdout.write(msg)

def err_write(msg):
    sys.stderr.write(msg)

def usage():
    err_write("Usage: {} file_a file_b\n".format(sys.argv[0]))

def file_object_compare(in_fil_a, in_fil_b):
    '''Logic: Assume files are equal to start with.
    Read both files, character by character.
    Compare characters at corresponding byte offsets. 
    If any pair at the same offset don't match, the files 
    are unequal. If we reach the end of the files, and 
    there was no mismatch, the files are equal.  We do not
    check for one file being a strict subset of the other, 
    because we only enter this function if the files are 
    of the same size.'''

    files_are_equal = True
    pos = 0
    while True:
        ca = in_fil_a.read(1)
        if ca == '':
            break
        cb = in_fil_b.read(1)
        if cb == '':
            break
        if ca != cb:
            files_are_equal = False
            break
        pos += 1
        if pos % 10000 == 0:
            print pos, 

    if files_are_equal:
        return (True, None)
    else:
        return (False, "files differ at byte offset {}".format(pos))

def file_compare(in_filename_a, in_filename_b):
    '''Compare the files in_filename_a and in_filename_b.
    If their contents are the same, return (True, None).
    else return (False, "[reason]"), where [reason] 
    is the reason why they are different, as a string.
    Reasons could be: file sizes differ or file contents differ.'''

    if getsize(in_filename_a) != getsize(in_filename_b):
        return (False, "file sizes differ")
    else:
        in_fil_a = open(in_filename_a, "rb")
        in_fil_b = open(in_filename_b, "rb")
        result = file_object_compare(in_fil_a, in_fil_b)
        in_fil_a.close()
        in_fil_b.close()
        return result

def main():
    if len(sys.argv) != 3:
        usage()
        sys.exit(1)

    try:
        # Get the input filenames.
        in_filename_a, in_filename_b = sys.argv[1:3]
        # Check they exist.
        for in_filename in (in_filename_a, in_filename_b):
            if not exists(in_filename):
                err_write(
"Error: Input file '{}' not found.\n".format(in_filename))
                sys.exit(1)
        # Don't allow comparing a file with itself.
        if in_filename_a == in_filename_b:
            out_write("No sense comparing {} against itself.".format(in_filename_a))
            sys.exit(0)
        # Compare the files.
        result = file_compare(in_filename_a, in_filename_b)
        if result[0]:
            out_write("Files compare equal.")
        else:
            out_write("Files compare unequal: {}".format(result[1]))
        sys.exit(0)
    except IOError as ioe:
        sys.stderr.write("Caught IOError: {}\n".format(str(ioe)))
    except Exception as e:
        sys.stderr.write("Caught Exception: {}\n".format(str(e)))

if __name__ == '__main__':
    main()

And here are a few input files I ran it with (containing differences at progressive character positions), a few runs of the program, and the output of those runs:

$ type f0.txt
file 1

$ type f1.txt
file 1

$ type f2.txt
file 2

$ type f3.txt
file 3

$ type f4.txt
mile 1

$ type f5.txt
fale 1

$ type f6.txt
fire 1

$ python file_compare.py
Usage: file_compare.py file_a file_b

$ python file_compare.py a b
Error: Input file 'a' not found.

$ python file_compare.py f0.txt f1.txt
Files compare equal.

$ python file_compare.py f0.txt f2.txt
Files compare unequal: files differ at byte offset 5

$ python file_compare.py f1.txt f2.txt
Files compare unequal: files differ at byte offset 5

$ python file_compare.py f2.txt f2.txt
No sense comparing f2.txt against itself.

$ python file_compare.py f1.txt f3.txt
Files compare unequal: files differ at byte offset 5

$ python file_compare.py f1.txt f4.txt
Files compare unequal: files differ at byte offset 0

$ python file_compare.py f1.txt f5.txt
Files compare unequal: files differ at byte offset 1

$ python file_compare.py f1.txt f6.txt
Files compare unequal: files differ at byte offset 2

$ python file_compare.py f1.txt f7.txt
Error: Input file 'f7.txt' not found.

$ python file_compare.py f64MB f64MB2
Files compare equal.

Most of the files tested were small, but I also tested with some files of tens of lines, and the last pair of files tested was 64 MB each.

Note:

These two lines:

if pos % 10000 == 0:
            print pos, 

are present in order to display a progress counter, for comparisons on large files. You can delete them if you don't want to monitor the progress of the comparison.

Currently I am using read(1), which means Python is reading the file character by character. There are potentially many levels of buffering that happen anyway, such as at the level of the C stdio library that underlies CPython's I/O, the OS, the hard disk controller, and even the CPU. But it may be possible to improve the performance of this program, by specifying some buffer when opening the files.

See:

Python open function
and
Default buffer size for a file

It's possible to write a much shorter version of this program, subject to certain limitations :-) Can you guess how? If you have an idea, mention it in the comments.

The image at the top is of weighing dishes from the island of Thera, Minoan civilization, 2000–1500 BC.

- Enjoy.

- Vasudev Ram - Online Python training and programming

Signup to hear about new products and services I create.

Posts about Python  Posts about xtopdf

My ActiveState recipes