John Hobbs

There is a relatively prevalent quote in the programming world, bandied about by programmers of all creeds.

“Premature optimization is the root of all evil.”

- Donald Knuth

I agree. What we intuit about optimization is usually wrong. And what’s more, until you hit a bottleneck, it’s often a waste of time. Moore has been good to us, and CPU bound problems aren’t as common as they once were.

That said, I think people should not cling to this. It’s dumb.

Today I was looking for a better way to compute sha1 file hashes in PHP. I know about sha1() obviously, but that’s not how it should be done for files.

Python provides the excellent hashlib that lets you update a hash with blocks of data. It’s excellent for file hashing, because you can read in data in chunks and update the hash as you go, thus avoiding reading the whole file into memory at once. Here, have a sample program that reads in 512 byte chunks.

1
2
3
4
5
6
7
8
9
10
11

>>> import hashlib
>>> hash = hashlib.sha1()
>>> with open( 'test.jpg', 'r' ) as handle:
...     while True:
...             data = handle.read(512)
...             if not data:
...                     break
...             hash.update( data )
... 
>>> hash.hexdigest()
'9e8d5ee361c6988baf7f75999f2c854a765f3eca'

So I quickly found the sha1_file() function. Perfect, I’m sure this reads in chunks, otherwise they would not have bothered to make the function.

Then I scroll down to check the user contributed notes for anything interesting. The top two notes are examples of using this snippet to get a sha1 hash of a file:

$hash = sha1( file_get_contents( $file ) );

I was stunned. The definition of file_get_contents() is as follows:

“Reads entire file into a string.”

Surely they are not suggesting that I read the entire file into memory, as a PHP variable, and then pass it on to the hashing function? Who would think that is a good solution, when there is a perfectly good built in function to do this for you, in chunks, in a neater fashion?

My only explanation is ignorance, or utter lack of consideration. I gave it a test with these two PHP scripts.

1
2
3

<?php
        echo sha1( file_get_contents( 'test.jpg' ) ) . "\n";
        echo memory_get_peak_usage() . "\n";

1
2
3

<?php
        echo sha1_file( 'test.jpg' ) . "\n";
        echo memory_get_peak_usage() . "\n";

The first version weighed in as using slightly more memory than second one. About 89k actually. And test.jpg is 88k.

Imagine if that picture was a few megs bigger. Imagine if I could somehow trigger this process on your website, over and over again with ab or something. It’s a DOS in the making.

Develop however you want to, just please, please don’t be stupid.

Related posts:

1. PHP Virtual Machine, Version 0.01
2. Thursday Quote: Rob Pike
3. Charting Weight Change With Google Visualizations

“It is not about solving puzzles and being the brightest kid in the class. It is about realizing that the complexity of software dwarfs even the most brilliant human; that cleverness cannot win. The only weapons we have are simplicity and convention.”

- Jonathan Edwards
Mea Culpa

Related posts:

1. Thursday Quote: David Heinemeier Hansson
2. Thursday Quote: Ed Nather
3. Thursday Quote: Harold Ableson

Yesterday I was stuck at the office while Darcy got a flat replaced. I decided to make a Markdown editor real quick, since I’m always using the web dingus to preview it, and that gets old.

I found the python-markdown2 library, and then decided to try out the WebKit integration in Qt4.

The best word for it is painless. Seriously, I got the first version glued together in about ten minutes, including finding the markdown library and reading all the docs for QWebView. It’s a brutal hack, but it worked just fine, and only took up 45 lines of code.

I’ve since dropped in three more markup languages, as well as cleaned it up to something readable. But I doubt I have more than a couple hours invested thus far.

I think this is a quintessential demonstration of the power of Python, which breaks down to three things.

1. Fast development
   No compile cycle, it reduces the barrier to creating a prototype.
2. Huge Ecosystem
   There is a library for everything and it’s all easy to get to and use.
3. Good Documentation & Intuitive Behavior
   Most packages have good documentation, and those that don’t still behave intuitively. Just pydoc around until you find what you need.

MarkedUp is now feature complete, stable, and available at your local github.

Related posts:

1. Looking up words in a Dictionary using Python
2. I am a fool….
3. Charting Weight Change With Google Visualizations