John Hobbs

After six years of sameness, I finally redesigned the blog.

While I was at it, I made it responsive, so it should look good on mobile too.

Some content is bound to still look funny as I haven’t reviewed all of it, but for the most part I think it looks good!

Special thanks to @johnhenrymuller for design help and my cool new logo.

Related posts:

1. BlowPass V0.60 – A Major Revision
2. In-Omaha.com

“Just edit the makefile directly. autoconf and friends are the devil’s tools.”

- Linus Torvalds
subsurface README

Related Posts

1. Thursday Quote: Linus Torvalds
2. Thursday Quote: Danny O’Brien
3. Thursday Quote: Reg Braithwaite

On a recent project I had the need to implement a basic search functionality into a web interface. This isn’t unusual, but what was different is that the data in this project was fairly static, and was kept in JSON on disk (and cached into memory). Additionally, there would be a point where it was moved off of our servers, onto an entirely different stack (hence the JSON).

There was minimal server side processing so far, and I did not want to add more overhead that would need to be ported. So, I decided to implement my search in JavaScript (with some help from Python). My idea was to do very basic string matching on a pre-built index. In this article I am going to lay out my implementation, but on a dummy data set.

The Process

This search breaks down into three steps:

1. Build Search Index
2. Perform Search
3. Connect To Interface

Before we start writing that, let’s cover the data.

The Data

Here is the data I will be sifting through. To keep it simple, I’m just searching through some sentences with associated ID’s. It’s short and simple, but with some tweaks you can apply this to bigger data sets, as I did.

My data set will be represented in JSON for portability and easy interpretation from JavaScript.

{
  "entries": [
    {
      "id": 1,
      "title": "The lazy white cat slept.",
      "body": "What a lazy cat."
    },
    {
      "id": 2,
      "title": "George, though angry, didn't make a sound.",
      "body": "George is a quiet man."
    },
    {
      "id": 3,
      "title": "Anyone could see that white didn't suit her.",
      "body": "Plus, it's after Labor Day."
    },
    {
      "id": 4,
      "title": "By Thor's Hammer, I will have my revenge.",
      "body": "Also, by Odin's Eye"
    },
    {
      "id": 5,
      "title": "Get off the couch you lazy bum.",
      "body": "Yeah, it's way better to sit at a computer desk."
    }
  ]
}

The Index

To facilitate easy matching, I’m going to build a search index that will be a dictionary, with individual words (tokens) as the keys, and arrays of ID’s as the values.

I’ve decided on Python for my index builder. Handling JSON from Python is easy, though if you have a version older than 2.6 you will need to use a different import, such as simplejson.

The core of this functionality is the tokenizer. To build this, we need to determine our rules. Since this is a simple search, I’m going to tokenize on word boundaries, it will be case insensitive, and I will only accept the characters A-Z, single quote and dash inside of a word.

Here is my implementation of the tokenizer, that follows these rules:

6
7
8
9

def tokenize ( string ):
  # Strip extra punctuation
  string = re.sub( r'[^a-z0-9A-Z \'\-]', '', string.lower() )
  return string.split( ' ' )

The remainder of the Python is simply opening and parsing the JSON.

13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

  index = {}
 
  with open( 'data.json', 'r' ) as handle:
    obj = json.loads( handle.read() )
 
    for entry in obj['entries']:
      # Break up both title and body
      tokens = tokenize( entry['title'] )
      tokens.extend( tokenize( entry['body'] ) )
 
      # Make them unique by casting to set
      tokens = set( tokens )
 
      # Now add them to the index
      for token in tokens:
        # Make a new entry for the token if it doesn't exist
        if token not in index.keys():
          index[token] = []
        # Add this id to the list of matches for this token
        index[token].append( entry['id'] )

Run that, and we get the dictionary we wanted:

$ python build_index.py
{u'a': [1, 2, 5],
 u'after': [3],
 u'also': [4],
 u'angry': [2],
 u'anyone': [3],
 ...

JavaScript

Now that we have an index to work with, let’s convert it to JSON. This just takes a little tweak to the Python.

33
34

  with open( 'index.json', 'w' ) as handle:
    handle.write( json.dumps( index ) )

That gives us a JSON file to work with.

{"thor's": [4], "is": [2], "bum": [5], "didn't": [2, 3], "yeah": [5], ...

Now we’re going to load that JSON with an AJAX call, so we have the index to work with.

8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

  init: function () {
    Search.$results = $( '#search-results' );
 
    $.getJSON( 'index.json' )
      .error( Search.index_load_error )
      .success( Search.index_load_success );
  },
 
  index_load_error: function () {
    Search.$results.append( $('<li/>').text( 'Error Loading Index' ) );
  },
 
  index_load_success: function ( data ) {
    Search.index = data;
  },

Once we have that data, finding matches is just a matter of looking them up in Search.index. Below I’ve used a loop that will search through an array of terms, and then accumulate the matches.

25
26
27
28
29
30
31
32

    var matches = [];
 
    // For each term passed in, check it in the index
    $.each( terms, function ( i, term ) {
      if( Search.index[term] ) {
        matches = matches.concat( Search.index[term] );
      }
    } );

It works!

Hook It Up

Just a little bit remains to connect this thing. First we need to port over the tokenizer from Python to get a consistent result. Pretty easy port.

47
48
49
50
51

  tokenize: function ( string ) {
    string = string.toLowerCase();
    string = string.replace( /[^a-z0-9A-Z \'\-]/, '' );
    return string.split( ' ' );
  }

JavaScript tokenize in action.

Last we just need to apply our functions to some inputs, which we hook up with some jQuery.

17
18
19
20
21
22
23
24
25
26
27
28
29
30

      $( function () {
        $( '#search-terms' ).keyup( function () {
          var results = Search.search( Search.tokenize( $(this).val() ) ),
              $results = $( '#search-results' );
 
          // Clear current results
          $results.children().remove();
 
          // Add new ones
          $.each( results, function ( index, element ) {
            $results.append( $( '<li/>' ).text( element ) );
          } );
        } );
      } );

Make It Better

This is not, by any means, a complete system. You would want to grab the results and match up the ID’s to a more useful output. Additionally, you could drop the AJAX call and use a script tag to bring in the index.

There are plenty of improvements you can make, and I’d love to know if you make them!

For reference, here is the complete source: https://gist.github.com/1673557.

Related Posts

1. Replacing MySQL FULLTEXT With JavaScript
2. Linux Multiple File Search And Replace
3. Javascript Colorshifter

“It takes balls to abandon features — grow some.”

- Kyle Neath

Product Design at GitHub

Related Posts

1. Thursday Quote: Andy Rutledge
2. Thursday Quote: Antoine de Saint-Exupéry
3. Thursday Quote: Scott Belsky

Related Posts

No related posts.

I recently did some work with cache control in Kohana and found the documentation a little thin out there, so I thought I would share what I learned.

Kohana has nice built in functionality for ETag validations, so you don’t really need to roll your own cache headers.

If you need to brush up on web caching in general, I would recommend a quick read of Caching Tutorial for Web Authors and Webmasters, an excellent and concise reference.

Setup

For the purposes of this example, I’m creating a small controller which will use a short string as it’s response body. The implementation here is trivial.

1
2
3
4
5
6
7
8
9

<?php defined('SYSPATH') or die('No direct script access.');
 
  class Controller_Example extends Controller {
 
    public function action_index () {
      $this->response->body( 'Hello, world!' );
    }
 
  } // Controller_Example

Let’s look at the headers that are returned by default. Your headers may vary, so adjust accordingly.

If you look at the response headers from this request, you will note that there no Cache-Control, Last-Modified or ETag headers are returned. That gives us a blank slate to work with.

ETag

An ETag is a unique identifier string describing your content for cache validation. ETags have an advantage over Last-Modified headers in that there is no need to worry about clock synchronization. The server can determine how to generate ETags in any manner it desires.

The Response object in Kohana provides two methods useful for ETag based caching. The first is Response::generate_etag(), the second is Response::check_cache().

Response::generate_etag()

This method uses the sha1 hash to create a unique ETag based on the content of the rendered response. Because it renders and hashes the response before returning a result, there is a memory and CPU time hit, which increases with the size of your response.

Response::check_cache()

This method is the one we will use directly, as it compares the ETag of the response to request headers and takes the appropriate action.

It’s signature is ($etag = NULL, Request $request = NULL). This is a bit odd, because although both are NULL by default, and the Request parameter is second, it is not optional while $etag is.

If you provide NULL for $etag the method will use Response::generate_etag() to get a valid ETag. As mentioned above, this is not always the optimal choice, so if you have a unique identifier that you can provide, you should.

Since this is a simplistic example, I will let Response::generate_etag() create my ETag value.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<?php defined('SYSPATH') or die('No direct script access.');
 
  class Controller_Example extends Controller {
 
    public function after () {
      parent::after();
      $this->response->check_cache( null, $this->request );
    }
 
    public function action_index () {
      $this->response->body( 'Hello, world!' );
    }
 
  } // Controller_Example

Let’s see the response headers for this version. We now have an ETag header, at the very bottom.

If we refresh the page again, we see that the browser sends an “If-None-Match” request header, which Response::check_cache() compares to the ETag. Finding that they match, the method returns a 304 response and immediately exits the script, causing the browser to use the cached version and saving the time it would take to send those bytes.

To demonstate how the ETag is generated let’s modify our response body so that it returns new content for every request (well, every second at least).

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<?php defined('SYSPATH') or die('No direct script access.');
 
  class Controller_Example extends Controller {
 
    public function after () {
      parent::after();
      $this->response->check_cache( null, $this->request );
    }
 
    public function action_index () {
      $this->response->body( 'Hello, world at ' . date( DATE_RSS ) . '!' );
    }
 
  } // Controller_Example

After refreshing we get a new body, and a new ETag, breaking the cache and re-sending the entire page.

Remember, if you implement this, you should try to use an alternate ETag value if you can.

Cache Control

ETags aren’t useful without a Cache-Control header, but you can set that yourself with Response::headers(), just be aware that Response::check_cache() will append must-revalidate to your header value, so don’t add that part yourself.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<?php defined('SYSPATH') or die('No direct script access.');
 
  class Controller_Example extends Controller {
 
    public function after () {
      parent::after();
      $this->response->headers( 'cache-control', 'private' );
      $this->response->check_cache( null, $this->request );
    }
 
    public function action_index () {
      $this->response->body( 'Hello, world at ' . date( DATE_RSS ) . '!' );
    }
 
  } // Controller_Example

Hopefully that clears up how to use the built in browser cache handling in Kohana 3, please leave your own tips or experiences in the comments!

Related Posts

1. Kohana 3 OAuth & Twitter Demo Code
2. Clean Auth module usage in Kohana
3. Simple PHP Caching

1. Thursday Quote: Brian Malow
2. Thursday Quote: Brian Tarbox
3. Thursday Quote: Andrew Wulf

“The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.”

- Edsger W. Dijkstra
The Humble Programmer

Related posts:

1. Thursday Quote: Ed Nather
2. Thursday Quote: Michael Abrash
3. Thursday Quote: Randall Degges

“Playfully doing something difficult, whether useful or not, that is hacking.”

- Richard Stallman
On Hacking

Related posts:

1. Thursday Quote: George Hotz
2. Thursday Quote: Bruce Schneier
3. Thursday Quote: Reg Braithwaite

Quality is a probabilistic function of quantity.

- Dean Simonton
Creation Myth

Related posts:

1. Thursday Quote: Andrew Wulf
2. Thursday Quote: Reg Braithwaite

“A complete commitment to immutability is a commitment to never building anything real.”

- Yaron Minksy
OCaml for the Masses

Related posts:

1. Thursday Quote: Ed Nather
2. Thursday Quote: David Greiner
3. Thursday Quote: _why

I’ve noticed a trend in PHP code to shy away from function application and closures. I understand to some extent. Useful, inline anonymous functions were not available until 5.3.0, which is relatively new. And create_function is an abomination.

Still, I think that PHP programmers just don’t think in this mindset, but it can be very useful.

Here is a rather contrived example, but one I’ve honestly seen an analogue of before.

1
2
3
4
5

$accumulator = array();
foreach( $this->tags->all() as $tags ) {
  $accumulator[] = htmlspecialchars( $tag->name );
}
return implode( ', ',  $accumulator );

Easy to understand, simple, does the job, but it is far more verbose than it needs to be.

Here is a version using array_map.

1
2

$tags = array_map( 'htmlspecialchars', $this->tags->all() );
return implode( ', ', $tags );

The array_map function is doing all the work of the loop, but you don’t have to write it and you don’t have to manage the accumulator.

Doesn’t that feel better?

Next time you mangle an array, first think if array_map or it’s friends array_filter or array_reduce could do it better.

Related posts:

1. Simple BBCode To HTML Function In PHP
2. Handling Forms With PHP
3. PHP/Cron Based IP Tracker

What the hash?

This post is about a fundamental programming concept, hashing.

“A hash function is any algorithm or subroutine that maps large data sets to smaller data sets, called keys.”

Hash function
Wikipedia – 2011/11/15

So, a hashing function can turn this:

Hello, my name is John Hobbs. I like hashes.

into this:

8ced81aa42cd91930024054f26ed92ea

Doesn’t seem super useful, does it? The content looses its semantic value. 8ced81aa42cd91930024054f26ed92ea is not the same as a personal introduction.

That’s okay though, we can find a use for a hash that won’t be seen by a human.

But first, let’s try our hand at writing a hash function.

Mapping to keys

So, the principal function of a hashing algorithm is to reduce a large block of data into a smaller block of data.

Let’s set up some arbitrary limits for developing our function.

For simplicity, let’s work at hashing values into a one byte integer, so eight bits.

Let’s also focus on only hashing ASCII strings, since ASCII characters are conveniently one byte long.

Now that that is established, let me propose a hashing function in pseudo-code:

let hash = 0
for each character in string:
  let hash = hash XOR character

For those of you not familiar with XOR, I mean exclusive or. This means comparing the bytes bit by bit. If the bits are the same, the result is zero. If they are different, the result is one. The common notation for XOR is ^. Here’s an example of 5 ^ 117:

  00000101 =   5
^ 01110101 = 117
----------------
  01110000 = 112

Working at a byte by byte level like this ensures that our hash will not exceed 8 bits in length, or an integer value of 128.

Buckets

With our shiny little hash function in mind, let’s use find a use.

One great use is lookup tables. These are sometimes called dictionaries, associated arrays, hash tables, etc.

Basically, they map a key to a value. So you tell a table that “my name” is “john”, and then later, if you ask it for “my name” is tells you “john”. Great tool for amnesiacs.

Hashing is perfect here for faster lookups, and for giving bound memory performance.

Think of it as a finite line of buckets. When you want to store something you hand it to the walrus, and say “Please keep this fish in a bucket called ‘Salmon’.”

Now, the walrus only has so many buckets, and none of them are labeled “Salmon”.

So, he uses a special walrus method (a hashing function) that deterministically says “Salmon == 5″, and puts the fish in the fifth bucket.

Later, when you come back and ask for the fish from the “Salmon” bucket, he knows to get the fish out of the fifth bucket because he can derive that “Salmon == 5″ using the same method as before.

Unfortunately, he ate the fish already, so there is nothing to give you

Walrus are selfish that way.

Implementation

Now that we have a use case, let’s implement our hashing function.

I chose JavaScript here.

1
2
3
4
5
6
7

function hash ( string ) {
  var c = 0;
  for( var i = 0; i < string.length; i++ ) {
    c = c ^ string[i].charCodeAt();
  }
  return c;
};

Let’s give it a try:

1
2
3
4

console.log( hash( "hello" ) );
// prints "98"
console.log( hash( "goodbye" ) );
// prints "125"

Cool! Now there is just a little more glue to make this complete. We have to have a get and a set for values, and an array to store them in.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

var SimpleHashTable = function () {
 
  // these are the buckets
  this.table = new Array(128);
 
  // hand it to the walrus
  this.set = function ( key, value ) {
    this.table[this.hash( key )] = value;
  };
 
  // get it from the walrus
  this.get = function ( key ) {
    return this.table[this.hash( key )];
  };
 
  // walrus math
  this.hash = function ( string ) {
    var c = 0;
    for( var i = 0; i < string.length; i++ ) {
      c = c ^ string[i].charCodeAt();
    }
    return c;
  };
 
};

The get and set functions are pretty similar. You hash the key, and use that integer as the index to either set, or retrieve a value in the table array.

Let’s give it a try!

var ht = new SimpleHashTable();
ht.set( "salutations", "Hello, good sir." );
console.log( ht.get( "salutations" ) );
// prints "Hello, good sir."

Perfect!

Collision course

Now let’s use our SimpleHashTable to store some stuff!

1
2
3
4
5
6

ht.set( "salutations", "Hello, good sir." );
console.log( ht.get( "salutations" ) );
// prints "Hello, good sir."
ht.set( "q", "BARF!" )
console.log( ht.get( "salutations" ) );
// prints "BARF!"

Wait, what the heck just happened?!

It shouldn’t have printed the value for q, we asked for salutations!

This is known as a collision, and it happens.

What happened is that the hash value we compute for q is the same value as the hash for salutations. It’s bound to happen, we only have 128 possible buckets and an infinite amount of possible keys.

So we have several ways to improve this situation.

One option is more buckets.

We can also improve the hashing function, but this only buys us fewer, more predictable collision rates.

Another common fix is a second level to each bucket keeping the exact key, value tuple, and doing a scan on collisions.

That would be like the walrus putting a sticky note that says “Salmon” on the fish before tossing it into the bucket. Then, when you ask for the “Salmon”, he rifles through the bucket, checking each fish until he finds one labeled “Salmon”.

I’ll leave these as an exercise for the reader, because it’s time we got moving on.

Real hashes

Okay, so now you have an example of a simple hash you could use for table lookups. That’s cool and everything, but in general we should leave details like that to the neckbeards. It’s good to know roughly how it works, but we don’t need to do it ourselves.

What else are hashes good for? Well, there is a specie of hash called a cryptographic hash. These hashes are cryptographic primitives, with strong promises of distribution and collision avoidance.

Examples you may have come across are MD5, or the SHA family of algorithms.

Implementation

These algorithms are considerably more complex than the hash function we created earlier, so I would urge you not to write your own except as an exercise.

Implementations are available in a variety of languages, so Google wisely.

Uses

This class of hash has several every day uses that apply to developer-types.

Data Integrity

Since hashes are deterministic functions, the output can be used as a fingerprint for the input. Passing data and the hash of that data allows for others to quickly check and make sure that the data has not become corrupted or tampered with in transit.

This assumes a secure channel for the hash, so use it wisely.

If you are on a Linux machine, you can quickly get the fingerprint for a file. In this example I’ll use the sha1 algorithm to verify the Redis 2.4.2 source code.

jmhobbs@Cordelia:~/Downloads$ sha1sum redis-2.4.2.tar.gz
d2c9288dcfe16b4718e39a0f8ad7b21f4ffc6de0  redis-2.4.2.tar.gz
jmhobbs@Cordelia:~/Downloads$

Comparing that hash to the Redis website, it looks like my download completed without errors. Huzzah!

Secret Verification

Another common use of hashing is to verify secrets, without storing the secret in the clear.

Say I was writing a system login, and I needed to use usernames and passwords.

I know it’s bad to store passwords in plaintext on my server. What if my server gets hacked and the user table is stolen?

I can’t store my passwords in the clear, but I need to reliably verify them.

Hashes make for a great solution to this problem.

You hash the password and store it in your database. Then, when a user wants to login, you hash the password they provide and compare it to the one in the database.

If they match, you log them in.

Doing this properly is a bit more complicated than that, so if you need to work with passwords like this, I recommend you read up.

Data Authenticity

We can also use hashes to confirm the authenticity of a message.

Say we both know a secret phrase, and no one else does.

If you want to send a message to me, and want it to be provably from you, you can send me the message along with a hash of the message concatenated to the secret.

Then, when I get the message, I also take the hash of the message plus the secret and compare it to your provided hash.

If they match, then I know it’s from you.

This is encapsulated in something called a Hashed Message Authentication Code, or HMAC.

File De-duplication

One final case that I regularly use is file de-duplication.

If you host uploads for a website, it’s a nice trick to store upload meta data (user, original name, upload time, etc.) separate from the file itself.

By de-coupling these you can hash files on upload, and if you already have a copy of that content, discard it and point the meta data at the existing file.

Here is an example in pseudo-code:

let hash = sha1_file( uploaded_file.tmp_path )
file = uploads.find_by_hash( hash )
if not file.loaded():
  file.hash = hash
  file.save()
return file.id

Conclusion

Hashes are useful tools for programmers, so learn the concepts and the applications.

I hope you learned something from this post! How do you use hashes?

Related posts:

1. Replacing Kohana 3 Auth module hashing
2. Premature optimization is the root of all evil. But don’t be stupid.
3. Disappointed In SimpleXML

Six months ago I wrote a little PHP framework as an exercise. I’d been down this path before with the now abandoned Xoket – but this time I decided to go small, and throw away.

Hence Skunk was born.

Skunk is a single file, very lightweight API. Really, it’s just a borrowed router with some sugar wrapped around it.

I was after a Sinatra or Bottle feel, and I think I got something rather nice out of it.

Skunk uses anonymous functions extensively, so use PHP 5.3+ to save yourself the pain of create_function.

Example

Here is a super simple Skunk application:

1
2
3
4
5
6
7
8
9
10
11
12
13

// Initialize
$s = new Skunk();
 
// Add a route
$s->get(
  '/hi(/<name>)',
  function ( &$s, $name = null ) {
    $s->body = 'Hello' . ( ( is_null( $name ) ) ? '' : " $name" ) . '!';
  }
);
 
// Run it!
$s->run();

Let’s tear that apart.

2

$s = new Skunk();

Everything in Skunk revolves around the Skunk object, so we need to set one up.

While it is possible to have multiple Skunk objects, there really isn’t a good use case I can think of. But we won’t restrict your cleverness with a singleton.

5

$s->get(

The two most important functions for Skunk are get and post.

These functions take a route and a function to apply when that route is matched, in a GET request and a POST request respectively.

6

  '/hi(/<name>)',

In this chunk we are setting up a GET request, with the route /hi(/<name>), so that will match /hi, /hi/John, etc.

Note the identifier <name> in the route. This named match will be captured and sent as an argument to the function.

7
8
9
10

  function ( &$s, $name = null ) {
    $s->body = 'Hello' . ( ( is_null( $name ) ) ? '' : " $name" ) . '!';
  }
);

Skunk route functions always need to take a reference to the Skunk object as their first argument. Following that are any other arguments that might be pulled from the route itself.

In this case we are just setting the body of the Skunk response.

13

$s->run();

This kicks off the request process and also renders the response.

Other Tricks

Skunk has some other features too.

You can raise a variety of errors inside of a request:

$s->get(
  '/error/500',
  function ( &$s ) {
    return $s->HTTP_500();
  }
);

You can set headers:

$s->get(
  '/example.json',
  function ( &$s ) {
    $s->header( 'Content-Type', 'application/json' );
    $s->body = json_encode( array( "Example" => TRUE ) );
  }
);

There is even a little hook system, so you can do middleware-ish stuff:

$s->hook(
  'send_head',
  function ( &$s ) {
    $s->header( 'X-Stinky-By', 'Skunk' );
  }
);

Summary

So that is Skunk.

It could use some love, but it’s workable. We’ve run Number Laundry on it with no problems for four months with no problems yet.

For an example of Skunk in action, check out Number Laundry’s source at github.

Related posts:

1. Amarok REST API
2. BlowPass V0.60 – A Major Revision
3. A Simple JavaScript Hooks System

“The only thing engineers like about a deadline is it’s precision.”

- Michael Lopp
@rands

Related posts:

1. Thursday Quote: Michael Lopp (Rands)
2. Thursday Quote: Michael Abrash
3. Thursday Quote: Brian Tarbox

I had previously rooted my Droid2, but it was broken when the OTA Gingerbread came down. I don’t really use root for much now that I don’t really develop for Android anymore, but I found a need today, so I decided to re-root.

I found a nicely packaged exploit on rootzwiki, but when I ran it I hit a snag.

* daemon started successfully *
* Running exploit [part 3 of 3]...
remount succeeded
5524 KB/s (1867568 bytes in 0.330s)
651 KB/s (26264 bytes in 0.039s)
5928 KB/s (196521 bytes in 0.032s)
	pkg: /data/local/tmp/Superuser.apk
Failure [INSTALL_FAILED_ALREADY_EXISTS]
link failed File exists

My old root was blocking my new root. Lame.

Easy to fix after poking around on the file system.

Essentially it boils down to this:

1. Manually remove old Superuser.apk
2. Clean up Superuser.apk data files
3. Install new super user apk
4. Link up new su

Details below, but as always YMMV.

Manually remove old Superuser.apk

Make sure you know the full namespace before you delete this file, you will need it later (i.e. com.noshufou.android.su)

jmhobbs@Cordelia:~/root$ ./adb.linux shell
# cd /system/app
# rm Superuser.apk

Clean up Superuser.apk data files

jmhobbs@Cordelia:~/root$ ./adb.linux uninstall com.noshufou.android.su
Success

Install new super user apk

From here on out you are essentially just finishing up the root script that got skipped.

jmhobbs@Cordelia:~/root$ ./adb.linux install Superuser.apk
3217 KB/s (196521 bytes in 0.059s)
	pkg: /data/local/tmp/Superuser.apk
Success

Link up new su

jmhobbs@Cordelia:~/root$ ./adb.linux shell
# cd /system/bin
# mv su su.orig
# ln -s /system/xbin/su /system/bin/su

All done! You should be rooted again.

No related posts.

Sometimes you have weird requirements to meet. It’s a fact of life.

One of these requirements I came across recently was outputting a result set in alpha order, but with a specific tuple at the end, instead of where it belongs in the sort.

I decided I’d figure out how to do this in SQL instead of gross rendering or a re-sort.

Please note there are other ways to solve this that are less brittle, but this was in a fairly static data set for a one off solution.

Consider this table:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

CREATE TABLE `animals` (
  `id` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
  `name` VARCHAR(64)  NOT NULL,
  `slug` VARCHAR(64)  NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `uniq_name`(`name`(64)),
  UNIQUE KEY `uniq_slug`(`slug`(64))
)
ENGINE = InnoDB DEFAULT CHARSET=utf8;
 
INSERT INTO `animals` ( `id`, `name`, `slug` ) VALUES
  ( NULL, 'Dog', 'dog' ),
  ( NULL, 'Cat', 'cat' ),
  ( NULL, 'Other', 'other' ),
  ( NULL, 'Zebra', 'zebra' );

Now, pull them out in alpha order:

mysql> SELECT *
    -> FROM `animals`
    -> ORDER BY slug ASC;
+----+-------+-------+
| id | name  | slug  |
+----+-------+-------+
|  2 | Cat   | cat   |
|  1 | Dog   | dog   |
|  3 | Other | other |
|  4 | Zebra | zebra |
+----+-------+-------+
4 rows in set (0.00 sec)

But what if we always want “Other” at the end?

Well, our unique indices allow us to guarantee a conditional sort on the slug (or name).

To do this, we will use the CASE statement.

mysql> SELECT *
    -> FROM `animals`
    -> ORDER BY CASE
    -> WHEN `slug` = 'other'
    -> THEN 0
    -> ELSE 1
    -> END DESC, `slug` ASC;
+----+-------+-------+
| id | name  | slug  |
+----+-------+-------+
|  2 | Cat   | cat   |
|  1 | Dog   | dog   |
|  4 | Zebra | zebra |
|  3 | Other | other |
+----+-------+-------+
4 rows in set (0.00 sec)

Here we have two ORDER BY clauses. We first use CASE to return 1 whenever the slug is “other”, and sort on that. Then we sort the sub-groups (“other” and “non-other”) by slug.

CASE is handy for other stuff too, and can handle complex expressions.

mysql> SELECT *, CASE
    -> WHEN `slug` IN ( 'cat', 'dog' )
    -> THEN 'yes'
    -> ELSE 'no'
    -> END AS 'is_good_pet'
    -> FROM `animals`;
+----+-------+-------+-------------+
| id | name  | slug  | is_good_pet |
+----+-------+-------+-------------+
|  1 | Dog   | dog   | yes         |
|  2 | Cat   | cat   | yes         |
|  3 | Other | other | no          |
|  4 | Zebra | zebra | no          |
+----+-------+-------+-------------+
4 rows in set (0.00 sec)

Do you ever use CASE?

Related posts:

1. Replacing MySQL FULLTEXT With JavaScript
2. MySQL & Ruby on Debian (sidux)
3. Simple PHP Caching

“It is the curse of the engineer: if it can be done, it will be done.”

- Juan Reyero
23 visits a day

Related posts:

1. Thursday Quote: Eric S. Raymond

The password hashing in the Auth module provided with Kohana 3.1 is not very good. By default it is a simple sha256 hmac with a global salt.

modules/auth/classes/kohana/auth.php

156
157
158
159
160
161
162

public function hash($str)
{
  if ( ! $this->_config['hash_key'])
    throw new Kohana_Exception('A valid hash key must be set in your auth config.');
 
  return hash_hmac($this->_config['hash_method'], $str, $this->_config['hash_key']);
}

This isn’t strong. If you loose the hashes and the salt it’s just a matter of winding up a GPU.

So how can we fix this? Well, thanks to Kohana’s structure we can easily override the Auth class and tweak it. However, due to Auth’s structure, we can’t drop the global salt. The hash function has to stand alone, so no passing in salts from the database.

That leaves us with key stretching.

Now, I don’t want to deal with a custom key stretching implementation, I’m not a cryptographer. So, let’s find an existing algorithm.

One that pops to mind is PBKDF2. This is a pretty simple algorithm, so it was easy to find and spot check a PHP implementation

We just take some test vectors from RFC 3962 and run them against the code we found.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

<?php
  require_once( 'pbkdf2.php' );
 
  header( 'Content-Type: text/plain' );
 
  $tests = array(
    array(
      'rounds' => 1,
      'bits' => 128,
      'expected' => "cd ed b5 28 1b b2 f8 01 56 5a 11 22 b2 56 35 15"
    ),
    array(
      'rounds' => 1,
      'bits' => 256,
      'expected' => "cd ed b5 28 1b b2 f8 01 56 5a 11 22 b2 56 35 15 0a d1 f7 a0 4b b9 f3 a3 33 ec c0 e2 e1 f7 08 37"
    ),
    array(
      'rounds' => 2,
      'bits' => 128,
      'expected' => "01 db ee 7f 4a 9e 24 3e 98 8b 62 c7 3c da 93 5d"
    ),
    array(
      'rounds' => 2,
      'bits' => 256,
      'expected' => "01 db ee 7f 4a 9e 24 3e 98 8b 62 c7 3c da 93 5d a0 53 78 b9 32 44 ec 8f 48 a9 9e 61 ad 79 9d 86"
    ),
    array(
      'rounds' => 1200,
      'bits' => 128,
      'expected' => "5c 08 eb 61 fd f7 1e 4e 4e c3 cf 6b a1 f5 51 2b"
    ),
    array(
      'rounds' => 1200,
      'bits' => 256,
      'expected' => "5c 08 eb 61 fd f7 1e 4e 4e c3 cf 6b a1 f5 51 2b a7 e5 2d db c5 e5 14 2f 70 8a 31 e2 e6 2b 1e 13"
    ),
  );
 
  foreach( $tests as $test ) {
    print $test['rounds'] . ' rounds at ' . $test['bits'] . ' bits ' . "\n";
    $start = microtime( TRUE );
    $result = trim( preg_replace( '/(..)/', '\1 ', bin2hex( pbkdf2( 'password', 'ATHENA.MIT.EDUraeburn', $test['rounds'], $test['bits']/8, 'sha1' ) ) ) );
    $diff = microtime( TRUE ) - $start;
    print 'Expected: ' . $test['expected'] . "\n";
    print '     Got: ' . $result . "\n";
    if( $result == $test['expected'] ) {
      print "MATCH\n";
    }
    else {
      print "NO MATCH\n";
    }
    print 'Took ' . number_format( $diff, 10 ) . "\n\n";
  }

Run it, and everything checks out:

1 rounds at 128 bits
Expected: cd ed b5 28 1b b2 f8 01 56 5a 11 22 b2 56 35 15
     Got: cd ed b5 28 1b b2 f8 01 56 5a 11 22 b2 56 35 15
MATCH
Took 0.0000329018
 
1 rounds at 256 bits
Expected: cd ed b5 28 1b b2 f8 01 56 5a 11 22 b2 56 35 15 0a d1 f7 a0 4b b9 f3 a3 33 ec c0 e2 e1 f7 08 37
     Got: cd ed b5 28 1b b2 f8 01 56 5a 11 22 b2 56 35 15 0a d1 f7 a0 4b b9 f3 a3 33 ec c0 e2 e1 f7 08 37
MATCH
Took 0.0000190735
 
2 rounds at 128 bits
Expected: 01 db ee 7f 4a 9e 24 3e 98 8b 62 c7 3c da 93 5d
     Got: 01 db ee 7f 4a 9e 24 3e 98 8b 62 c7 3c da 93 5d
MATCH
Took 0.0000147820
 
2 rounds at 256 bits
Expected: 01 db ee 7f 4a 9e 24 3e 98 8b 62 c7 3c da 93 5d a0 53 78 b9 32 44 ec 8f 48 a9 9e 61 ad 79 9d 86
     Got: 01 db ee 7f 4a 9e 24 3e 98 8b 62 c7 3c da 93 5d a0 53 78 b9 32 44 ec 8f 48 a9 9e 61 ad 79 9d 86
MATCH
Took 0.0000200272
 
1200 rounds at 128 bits
Expected: 5c 08 eb 61 fd f7 1e 4e 4e c3 cf 6b a1 f5 51 2b
     Got: 5c 08 eb 61 fd f7 1e 4e 4e c3 cf 6b a1 f5 51 2b
MATCH
Took 0.0019500256
 
1200 rounds at 256 bits
Expected: 5c 08 eb 61 fd f7 1e 4e 4e c3 cf 6b a1 f5 51 2b a7 e5 2d db c5 e5 14 2f 70 8a 31 e2 e6 2b 1e 13
     Got: 5c 08 eb 61 fd f7 1e 4e 4e c3 cf 6b a1 f5 51 2b a7 e5 2d db c5 e5 14 2f 70 8a 31 e2 e6 2b 1e 13
MATCH
Took 0.0144000053

So now all that’s left is to drop it in, which is pretty simple. One thing to note is that I wanted this to stay compatible with the default auth config file, so I just extended that a little bit.

application/classes/auth.php

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

<?php
 
  abstract class Auth extends Kohana_Auth {
 
 
    public function hash ( $str ) {
      if ( ! $this->_config['hash_key'] )
        throw new Kohana_Exception( 'A valid hash key must be set in your auth config.' );
 
      if ( 'pbkdf2' == $this->_config['hash_method'] ) {
        return base64_encode( self::pbkdf2(
          $str,
          $this->_config['hash_key'],
          Arr::get( $this->_config['pbkdf2'], 'rounds', 1000 ),
          Arr::get( $this->_config['pbkdf2'], 'length', 45 ),
          Arr::get( $this->_config['pbkdf2'], 'method', 'sha256' )
        ) );
      }
      else {
        return parent::hash( $str );
      }
    }
 
    /** PBKDF2 Implementation (described in RFC 2898)
     *
     *  @param string p password
     *  @param string s salt
     *  @param int c iteration count (use 1000 or higher)
     *  @param int kl derived key length
     *  @param string a hash algorithm
     *
     *  @return string derived key
     *
     *  @url http://www.itnewb.com/tutorial/Encrypting-Passwords-with-PHP-for-Storage-Using-the-RSA-PBKDF2-StandardL
    */
    public static function pbkdf2 ( $p, $s, $c, $kl, $a = 'sha256' ) {
 
        $hl = strlen(hash($a, null, true)); # Hash length
        $kb = ceil($kl / $hl);              # Key blocks to compute
        $dk = '';                           # Derived key

        # Create key
        for ( $block = 1; $block <= $kb; $block ++ ) {
 
            # Initial hash for this block
            $ib = $b = hash_hmac($a, $s . pack('N', $block), $p, true);
 
            # Perform block iterations
            for ( $i = 1; $i < $c; $i ++ )
 
                # XOR each iterate
                $ib ^= ($b = hash_hmac($a, $b, $p, true));
 
            $dk .= $ib; # Append iterated block
        }
 
        # Return derived key of correct length
        return substr($dk, 0, $kl);
    }
 
  }

application/config/auth.php

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<?php defined('SYSPATH') or die('No direct access allowed.');
 
  return array(
    'driver'       => 'orm',
    'hash_method'  => 'pbkdf2',
    'hash_key'     => 'zomg',
    'lifetime'     => 1209600,
    'session_key'  => 'auth_user',
    'pbkdf2'       => array(
      'method'  => 'sha256',
      'rounds'  => 1000,
      'length'  => 45,
    )
  );

One item to note is that I am packing these with base64_encode. This is to fit into the default field type for the ORM driver. That is also why my length is stunted to 45. If you really want to go all out, alter your table to use a TINYBLOB, up the length to 256 bit and up the rounds.

So that is how I replace weak hashing in K3 with something a bit better.

How do you do it?

Related posts:

1. Clean Auth module usage in Kohana
2. Kohana 3 OAuth & Twitter Demo Code
3. Premature optimization is the root of all evil. But don’t be stupid.

A Problem

“@jmhobbs thanks for your amazonses swiftmailer integration. it works. can’t understand your code tho. way over my head.”

- @rrcatto

“@rrcatto no problem! Neither can I sometimes. :-)”

- @jmhobbs

After this exchange, I thought I would dig back into this project and outline how it works.

Three Core Classes

There are really only two classes in play with this project. The first is Swift_AWSInputByteStream, the second is the transport itself, Swift_AWSTransport. The third, and possibly the most confusing, is ChunkedTransferSocket.

So let’s got over each of them.

Swift_AWSInputByteStream

The purpose of this class is to write out the contents of the message to the provided socket. We have this special class for AWS because the documentation specifies that the message data should be Base64 encoded. One side effect of Base64 is padding on encoding. Because of this, we buffer any excess bytes and encode only on multiples of 3 bytes received.

Here is a documented version of the core function, write:

public function write($bytes) {
 
  // Get the buffer size + new chunk size
  $total_size = strlen( $this->buffer ) + strlen( $bytes );
  // Size of the remainder we will need to buffer
  $excess = $total_size % 3;
 
  // Nothing to write? Return early.
  if( $total_size - $excess == 0 ) { return ++$this->counter; }
 
  // Encode and write bytes to the socket
  $this->socket->write(
    urlencode(
      base64_encode(
        substr(
          $this->buffer . $bytes, // Source is buffer + new chunk
          0,                      // Begin at the beginning
          $total_size - $excess   // Write up to the new buffer 
        )
      )
    )
  );
 
  // If there was excess, store it in the buffer
  if( $excess != 0 ) {
    $this->buffer = substr( $this->buffer . $bytes, -1 * $excess );
  }
  else {
    $this->buffer = '';
  }
 
  return ++$this->counter;
}

Swift_AWSTransport

This class provides the transport for Swiftmailer. It sets up the socket, takes a message, and sends it off to AWS. The core functionality is in _doSend. This function is documented below. I’m not detailing much here, because it’s mostly glue code.

protected function _doSend(
  Swift_Mime_Message $message,
  &$failedRecipients = null
) {
  // Use secret key to generate HMAC used to 
  // authorize the message to AWS
  $date = date( 'D, j F Y H:i:s O' );
  // Use the native extension if available
  if(
    function_exists( 'hash_hmac' ) and
    in_array( 'sha1', hash_algos() )
  ) {
    $hmac = base64_encode(
      hash_hmac( 'sha1', $date, $this->AWSSecretKey, true )
    );
  }
  // Otherwise, fallback to a PHP implementation
  else {
    $hmac = $this->calculate_RFC2104HMAC( $date, $this->AWSSecretKey );
  }
 
  // Now we use that to create the authorization header
  $auth = "AWS3-HTTPS AWSAccessKeyId=" .
          $this->AWSAccessKeyId .
          ", Algorithm=HmacSHA1, Signature=" .
          $hmac;
 
  $host = parse_url( $this->endpoint, PHP_URL_HOST );
  $path = parse_url( $this->endpoint, PHP_URL_PATH );
 
  // Open up a raw SSL socket to the host 
  $fp = fsockopen( 'ssl://' . $host , 443, $errno, $errstr, 30 );
 
  if( ! $fp ) {
    throw new AWSConnectionError( "$errstr ($errno)" );
  }
 
  // Convert that into a chunked "socket"
  $socket = new ChunkedTransferSocket( $fp, $host, $path );
 
  // Add our date and auth headers (generated above)
  $socket->header("Date", $date);
  $socket->header("X-Amzn-Authorization", $auth);
 
  // Write the initial post parameters
  $socket->write("Action=SendRawEmail&RawMessage.Data=");
 
  // Hand it off to an Swift_AWSInputByteStream to write the message
  $ais = new Swift_AWSInputByteStream($socket);
  $message->toByteStream($ais);
  $ais->flushBuffers();
 
  $result = $socket->read();
 
  return $result;
}

Okay, not simple, but fairly straightforward.

ChunkedTransferSocket

This class makes an HTTP request direct on the socket. Since we don’t know the message size before encoding, and it’s memory intensive to encode, buffer, and then send, we do a chunked transfer encoding POST.

It’s actually pretty easy. You send some headers, and then every time you have a chunk to write, you preface it with the number of bytes you are sending before you send them.

Here’s the code for the write function:

public function write ( $chunk ) {
  if( $this->write_finished ) { throw new InvalidOperationException( "Can not write, reading has started." ); }
 
  if( ! $this->write_started ) {
    fwrite( $this->socket, "\r\n" ); // Start message body
    $this->write_started = true;
  }
 
  // Write the length of the chunk, carriage return and new line
  fwrite( $this->socket, sprintf( "%x\r\n", strlen( $chunk ) ) );
  // Write the chunk
  fwrite( $this->socket, $chunk . "\r\n" );
  // Flush the socket to send the data now, not later
  fflush( $this->socket );
}

Pretty simple once you understand how chunked transfer works. The rest of the class is just state keeping.

Conclusion

So, that’s that. Nothing really deep in there, just a collection of fairly simple methods that, glued together, send email to AWS SES.

Hit me up with any questions in the comments section if you have them.

Related posts:

1. Swiftmailer Transport for Amazon Simple Email Service
2. Tail In PHP
3. PHP/Cron Based IP Tracker

“We shouldn’t be doing math in front of the computer, the computer should do the math for us.”

- Josh Reich
BankSimple Demo

Related posts:

1. Thursday Quote: David Thornburg
2. Thursday Quote: Harold Ableson
3. Thursday Quote: Eric Hellman

Today Alex pointed out the new iCloud website had lot’s of fancy effects. One I liked best was the polished metal effect on the login box that shimmered when you moved your mouse.

I went ahead duplicated it as best I could in a short time. There are some obvious differences in approach, but it’s essentially the same.

View The Demo

One thing I did not do was the easing on the mouse move. I really like that, but it would be time consuming to get it running.

Also, I’m not browser compatible. I only tested it in Chrome 14.

Most of the work is done in two functions.

mousemove takes the mouse position and converts it to a degree of rotation.

mousemove: function ( event ) {
  // Use the mouse x coordinate conbined with the window width to 
  // come up with a degree to rotate.  You can make it more responsive
  // by decreasing the reduction.
  var reduction = 200;
  var deg = ( window.innerWidth / 2 - event.clientX ) * -1 / reduction;
  if( deg != Shimmer.current_rotation ) { Shimmer.draw( deg ); }
}

draw rotates the canvas and draws the image onto it.

// Rotate the metal background
draw: function ( deg ) {
  Shimmer.current_rotation = deg;
 
  // Clear the canvas
  Shimmer.context.clearRect( 0, 0, Shimmer.canvas.width, Shimmer.canvas.height );
  Shimmer.context.save();
 
  // Set the rotation point at 50% from left and & 80px from top
  Shimmer.context.translate( Shimmer.center.x, 80 );
 
  // Rotate by degrees (convert to radians)
  Shimmer.context.rotate( deg * Math.PI / 180 );
 
  // Draw metal
  Shimmer.context.drawImage( Shimmer.image, Shimmer.offset.x, Shimmer.offset.y );
 
  // Clear transforms
  Shimmer.context.restore();
}

That’s essentially it. Simple, but visually powerful. The source is embedded in the demo and commented.

Related posts:

1. PanoramAh: A jQuery Panorama Viewer
2. Das BlinkenTechts!
3. Fun with mootools

The Internet seems a bit sparse when looking for good demo code for using the Kohana 3 OAuth module with Twitter.

I think the main issue is that the OAuth module isn’t very well documented, and doesn’t do API requests. For that you need an API implementation, like shadowhand/apis.

Anyway, here is a gist I put together with an example controller for Twitter OAuth:

https://gist.github.com/1267793

Related posts:

1. Clean Auth module usage in Kohana
2. SVN to Twitter
3. Charting Weight Change With Google Visualizations

“There is something about a lack of sleep which makes perl code and regexs seem like a good idea.”

- Holden Karau
Automatic spelling corrections on Github

Related posts:

1. Thursday Quote: Eric S. Raymond
2. Linux Multiple File Search And Replace
3. Thursday Quote: Reg Braithwaite

“A designer who does not write markup and css is not designing for the web, but drawing pictures.”

- Andy Rutledge
Web Design is Product Design

Related posts:

1. Thursday Quote: Antoine de Saint-Exupéry
2. Thursday Quote: Eric S. Raymond
3. Thursday Quote: Scott Belsky

“To me, being a geek is about being allowed to be a child forever. To take passionate joy in your interests.”

- Jessica Mills
When Geeks Become Bullies

Related posts:

1. Thursday Quote: Justin
2. Thursday Quote: Scott Belsky
3. Thursday Quote: Scott Belsky

“If you want to be a game programmer, or for that matter any sort of programmer at all, here’s the secret to success in just two words: Ship it. Finish the product and get it out the door, and you’ll be a hero.”

- Michael Abrash
Quake’s 3-D Engine: The Big Picture

Related posts:

1. Thursday Quote: Michael Lopp (Rands)
2. Thursday Quote: Joel Spolsky
3. Thursday Quote: Ed Nather

The online docs for testing and deploying your Enyo apps to the TouchPad are woefully weak. I finally found how to get the TouchPad into developer mode, and thought I’d share.

1. Start “Device Info”
2. In the menu, pick “Custom Application”
3. Enter “##devmode#”
4. Done!

You’ll need to do a restart before you can install any packages, or at least I did.

No related posts.

“Scope should be restricted by budget, but should never be expanded to consume the budget.”

- Des Traynor
Where to draw the line?

Related posts:

1. Thursday Quote: Derek Sivers
2. Consume Less. Create More.

“..the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code.”

- Eric S. Raymond
Why Python?

Related posts:

1. Thursday Quote: Eric Hellman
2. Thursday Quote: Jonathan Edwards
3. XML, XSL Are Stinkin Powerfull

“Programmers are most effective when they avoid writing code.”

- John D. Cook
Blog Post

Related posts:

1. Thursday Quote: John von Neumann
2. Thursday Quote: Andrew Wulf
3. Thursday Quote: _why

Three months ago we made some cheese:

Today we tasted it:

Turns out we didn’t get it squeezed out enough, so there was some milky leakage.

It’s definitely not Monterrey Jack, but it’s pretty good.

Soft and spreadable, and fairly sour/bitter. It would be good with a nice summer sausage.

Related posts:

1. Sugru To The Rescue!

This weekend I slapped together a little project called in-omaha.com

It’s essentially a little parody on the fact that people discount Omaha, especially people on the coasts. This was born out of an NPR radio bit reporting on Saddle Creek and The Slowdown.

Here’s the choice quote that I took offense to:

“It all started with some young guys and some surprisingly good music.”

As if there is nobody in Omaha who appreciates good music, and definitely no one who can make it.

So if you make cool stuff, live in Omaha because it’s great, and enjoy some chuckles, feel free to play with in-omaha.com!

Related posts:

1. Hall Of Shame
2. Thursday Quote: Remy Porter
3. VCReviews – A MediaWiki Extension

There really are only two acceptable models of development: “think and analyze” or “years and years of testing on thousands of machines”.

- Linus Torvalds
Mailing List Message

Related posts:

1. Thursday Quote: Gina Trapani
2. Thursday Quote: Abelson & Sussman
3. Thursday Quote: Jeff Foster

This weekend Darcy and I went to Omaha’s first “Pop Culture Expo”, Contagion: Outbreak. This is true, for extremely small values of “pop”. Mostly it was “geek culture”, so Star Wars, Firefly, Robots, Zombies, Anime, and lots of Comics.

The big reason we went was Jewel Staite, one of our favorite actresses. We attended the Q&A on Sunday, and it was really neat. She’s just as nice as you would think she is.

Another cool bit was the R2 Builders Club. I’m really considering giving that a shot.

Anyway, here are some more blurry-cam photo’s I took. We’ll probably go again, but if they raise the price next year they better raise the quality of the content and organization, because it wasn’t that great this year, nothing compelling.

No related posts.

“The Cloud” is something idiots call what the rest of us call “The Internet”

- @hipsterhacker
Tweet

Related posts:

1. Thursday Quote: Tim Berners-Lee
2. Thursday Quote: Scott Belsky
3. Thursday Quote: Brian Malow

Yesterday I had a discussion wherein Regular Expression’s were (sort of) jokingly referred to as “magic”. Well, today I had cause to write a clever(ish) one, and I thought I would lay it out real quick.

What I needed to do was break a string at between 200 and 250 characters, but it had to be on a word boundry. So, it could run over if needed, but it had to break on a word boundary.

Now, I know you could do this with a couple of other functions mixed together, but I’m going to do it with preg_replace. No idea how this compares on performance. Poorly for large strings, I would guess.

I’ll ruin the surprise and show you the answer first:

preg_replace( "/(.{200,}?\b).*/s", '\1', $string );

So let’s break this down.

/(.{200,}?\b).*/s

Starting from the left. ( this signifies that I am opening a subpattern. What comes inside of these parentheses are the content I want to save.

The dot, ., is a wildcard that matches any character.

The string {200,} is a repetition marker, allowing for the previous character (the dot, so, anything, remember) to repeat 200 or more times.

The question mark, ? here is really important. By default repetition is “greedy”, or, it will suck up as many characters as it is allowed, with precedence being given out left to right. By putting the ? here we make {200,} un-greedy, so it only grabs as much as required for the match to work.

The \b is an escape sequence for a word boundary. Handy!

After that we close the subpattern, )

Next we have .* which is a greedy repetition matching any character. Since this is greedy and {200,} is not, it will pick up any of the slack.

The last little bit is the s after the pattern delimiter. This is a modifier that tells the dot character to match anything, including newlines.

Again

So, to recap, we say…

Give me any character, 200 or more times, ungreedily, followed by a line break, and then collect any following characters greedily.

Examples

//  T  h  i  s     i  s     a     t  e  s  t     s  t  r  i  n  g  .
// 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21
$string = "This is a test string.";
echo preg_replace( "/(.{5,}?\b).*/s", '\1', $string );
// This 
echo preg_replace( "/(.{7,}?\b).*/s", '\1', $string );
// This is
echo preg_replace( "/(.{9,}?\b).*/s", '\1', $string );
// This is a
echo preg_replace( "/(.{11,}?\b).*/s", '\1', $string );
// This is a test
echo preg_replace( "/(.{25,}?\b).*/s", '\1', $string );
// This is a test string.
 
// I'm removing that ? here, see what it does to it...
echo preg_replace( "/(.{3,}\b).*/s", '\1', $string );
// This is a test string
// It made {3,} gobble everything up to the last word boundary. Greedy bugger!

Got it?

Related posts:

1. Simple BBCode To HTML Function In PHP
2. Recursive Word Count With Bash
3. Premature optimization is the root of all evil. But don’t be stupid.

Original

No related posts.

The key to making programs fast is to make them do practically nothing.

- Mike Haertel
why GNU grep is fast

Related posts:

1. Thursday Quote: Abelson & Sussman
2. Thursday Quote: Scott Belsky
3. Warped Jane Austen Quote

I’ve used a lot of twitter clients over the years, I even wrote one back in ’08. TweetDeck is pretty good, but Air is shoddy on 64-bit Linux, and so it didn’t integrate well and would crash.

I rejected Gwibber and Choqok, just not my style. My old web standby Brizzly seems pretty lame these days too. Then I found Hotot. It’s awesome. It’s like a Linux native TweetDeck, but with a friendlier, more welcoming feel.

If you are looking for a Linux twitter client, you have got to give Hotot a try.

Related posts:

1. Linux On The Desktop
2. DigiPro Tablet And Linux
3. Submitting Assignments In Blackboard With Linux

First make it possible.
Then make it beautiful.
Then make it fast.

- Nathan Marz
Become Efficient or Die

Related posts:

1. Thursday Quote: Brian Tarbox
2. Thursday Quote: _why
3. Thursday Quote: John von Neumann

“you seem to think i’m random, but i’m only psuedorandom. you would be exactly this way, were you seeded at the very same time and place.”

- _why
Twitter

Related posts:

1. Thursday Quote: Jeff Foster
2. Thursday Quote: David Thornburg
3. Thursday Quote: Jeff Atwood

So I wanted a VPN this weekend for working from coffee shops and the like. I found a good recipe for a PPTP install and set it up on an EC2 Micro, which happens to fall in the free tier. Free VPN!

During the process I went ahead and turned it into a script. It only takes a few seconds to run and will set you up for good.

Ther are side effects to this script. If you have made any changes to /etc/rc.local they will get removed, sorry. On a fresh install it will be fine.

You can read though it here: https://gist.github.com/950539 and after logging into your instance, you can use it like so:

$ wget https://gist.github.com/raw/950539/pptpd.sh
$ sudo su
# /bin/bash pptpd.sh

Bonus!

To connect to this VPN from KDE4, I had to do a little config song and dance. Screenshot follows:

KDE4 NM VPN Advanced Settings

Related posts:

1. Swiftmailer Transport for Amazon Simple Email Service
2. WLAN Router Monitor BASH Script
3. HowTo’s

I’ve had trouble compiling native extensions on the Mac before, but I finally found a fix.

You just need the correct ARCHFLAGS environment variable. You can set this in your .bashrc or use it right before python setup.py build

ARCHFLAGS="-arch i386 -arch x86_64"

This works because XCode dropped the PPC compiler in v4, and with that variable we tell the setup script not to bother trying to compile for that arch, just i386 and x86_64.

Much thanks to Y.H. Wong in this Super User thread.

No related posts.

“until programmers stop acting like obfuscation is morally hazardous, they’re not artists, just kids who don’t want their food to touch.”

- _why
Twitter

Related posts:

1. Thursday Quote: Jeff Atwood
2. Thursday Quote: Andrew Wulf
3. Thursday Quote: Ed Nather

Courtesy of @alexpgates

No related posts.

This is a nifty little trick I swiped from a Stack Overflow question.

Basically it adds a command to run any command in a new tab of the same terminal window you are currently in.

For example, this:

$ tabdo cd /tmp

would launch a new tab, then navigate to /tmp.

To get this trick, just put this in your ~/.bashrc

tabdo () {
        /usr/bin/osascript 2>/dev/null <<EOF
activate application "Terminal"
tell application "System Events"
    keystroke "t" using {command down}
end tell
tell application "Terminal"
    repeat with win in windows
        try
            if get frontmost of win is true then
                do script "$@" in (selected tab of win)
            end if
        end try
    end repeat
end tell
EOF
}

Here is a demo video, sorry it goes so fast, YouTube sped it up.

Related posts:

1. GTK Tooltips On Notebook Tab Labels
2. m4a2mp3
3. Bash Line For Constant Monitoring

The Ruby gem for Qt4 bindings, appropriately called qtbindings, do not build by default with brew installed Qt4 libs.

There is a quick fix, but it’s hidden in the issues on github. Essentially you just need to symlink the brew headers to an expected location.

$ for DIR in /usr/local/Cellar/qt/4.7.1/lib/*.framework; do ln -s $DIR/Headers ${DIR%%/lib/*}/include/$(basename $DIR .framework); done

After that is run, gem install qtbindings runs like a champ.

Related posts:

1. MySQL & Ruby on Debian (sidux)
2. Learning Ruby: Unpacking Arrays
3. MarkedUp: The Power of Python

“Ideas are just a multiplier of execution.”

- Derek Sivers
http://sivers.org/multiply

Related posts:

1. Thursday Quote: Scott Belsky
2. Thursday Quote: Paul Graham
3. Thursday Quote: Scott Belsky

“Test-driven development is a coder’s psychological safety net. A bug doesn’t mean the code’s crappy; it’s just missing test coverage.”

- Gina Trapani
@ginatrapani

Related posts:

1. Thursday Quote: Brian Tarbox
2. Thursday Quote: Ed Nather
3. Thursday Quote: C. Scott Andreas