for..in should not be used to iterate over an Array where index order is important. Array indexes are just enumerable properties with integer names and are otherwise identical to general Object properties. There is no guarantee that for...in will return the indexes in any particular order and it will return all enumerable properties, including those with non–integer names and those that are inherited.

That means you can do the following:

var ary = ["foo", "bar", "baz"];
ary.bing = "boom";
Object.prototype.blamo = "bingo";

for (var x in ary) {
  console.log(x);
}

And conceivably get output like this:

0
2
"bing"
1
"blamo"

Fun stuff.

At dinner a friend told me about a new app that let you rent scooters on the fly. I wanted to try it out. I knew it was called Scoot but I didn’t know their website. So naturally I Googled it. What’s wrong with this picture?

Every search result is an article talking about the service, not a single link to the actual service. You won’t find the real link until the very last result. Clearly something is seriously wrong when Google returns 9 results talking about something before the thing being talked about. This got me thinking about how I really relate to the things I’m looking for. I sat down and thought “Hey, I want to look up that scooter thing in the city that my friend Matt told me about tonight”. Shouldn’t I be able to search “Scooter service that Matt likes”? Well that is exactly what Graph Search is for.

I’m not yet in the beta so I can’t say whether or not it would have worked, but I am excited to find out.

Also, Scoot is awesome. Check it out.

It’s that time of year again where the question of New Year’s Resolutions comes up in just about every conversation. The answer I am hearing most often this year seems to be something to do with reading more. This is interesting because the answer I gave last year was “I will read 25 books this year”. Each time I hear this answer, I ask myself “How well did I do on this resolution last year?”.

It turns out that this is a harder question to answer than it seems. I have a distinct feeling that I read more than in recent years but I’m also pretty sure I missed the 25 books mark by a decent bit. Or in other words, I’ve got no idea. Of course I could try to think back and count the books, but memory is tricky and I certainly don’t trust mine. Besides, it wouldn’t really help solidify the feeling that I read more this year generally (which is what I really care about). I don’t know how I would count the short stories or academic papers much less the hundreds of blog posts either. Essentially, from day one I was hopelessly ill-equipped to fulfill my resolution last year.

This always leads me to thinking about just how incredibly high the failure rates are (article about why). In fact, many of the resolution conversations I have with people are essentially tongue-and-cheek in nature because people are already resolved to fail before they even start. This is scary to think about because the resolutions people mention usually map to real, strong, personal desires. Large portions of people have started their year by sarcastically writing off their ability to succeed at bettering themselves in at least one important way. Yikes, if that’s true, you could argue that New Year’s Resolutions have the opposite impact on society as intended.

If New Year’s Resolutions don’t work, then what does? Dr. BJ Fogg, of Stanford Persuasive Tech Lab, makes a strong argument for looking at habit forming like any other skill in that it takes practice to acquire and you need to start small. Dr. Fogg runs a program called Tiny Habits that focuses on how to build the skill of forming habits. In that program, he focuses on simplicity, positive feedback, and “anchoring” new habits to existing ones. I encourage you to join me next week in participating in one of the Tiny Habits sessions.

In addition to simply building raw habit forming skill, it is interesting to be able to bring a ruler to the table when measuring that skill. I strongly believe the old credo “Knowledge is Power” and I think that having the data to calculate progress and understand your state is a powerful tool. At the gym, you can measure exactly how strong you are because you can count the weights you lift. To make a simliar measurement of habit forming skill, you need a similarly easy thing to count. That means data. Luckily, technology has reached the point that simple Quantified Self style measurements take only a few seconds a day using apps like Lift or a simple repeat GMail task list (There are tons of these apps, some crazy ones too).

I plan on starting to build some of this data around some larger habits I have already formed. In the style of Dr. Fogg’s Tiny Habits program, one of my three habits will be: “After my morning shower, I will record the last day’s progress in Lift”. Note that the habits I am recording in Lift are things that I have already established strongly, not habits I am trying to form. The hope being that I can at a future date use the combination of an improved habit forming skill and a data driven base line to start with a leg up on altering those existing larger habits.

So, I guess you could say, my New Year’s Resolution for 2013 is to improve and quantify my habit forming skill…

Before talking about the java.util.concurrent package I want to go over how a couple of the Java keywords effect concurrency in Java 1.5+. In an effort to maintain good performance, when synchronization is not explicitly dictated, the compiler, JVM and cache take serious liberties with reordering memory operations. This can provide significant performance gains but also can cause your programs to function in unexpected ways. The only rule on reordering is that the thread that is currently executing can’t tell the difference. This is called within-thread as-if-serial semantics. Luckily, rules are added with the use of the keywords volatile and final.

The volatile keyword guarantees that reads and writes to those variables are ordered totally across all threads; meaning that neither the compiler or cache can reorder these calls with each other. Furthermore, volatile reads and writes will not be reordered with any other memory operations. So this means that when thread A writes to a volatile variable X, and thread B reads from X, any variables that were visible to thread A at the time X was written will also be visible to B. This means that X can be used to indicate state to other threads. There are also similar guarantees when a thread is started, a thread is joined with another thread and when a thread enters and exits a synchronized block.

The final keyword means just that final, except when it doesn’t… Before Java 1.5, without synchronization another thread could potentially see the default value (null or 0) for a final field and then later see the correct value. Java 1.5+ now guarantees that when a final object’s constructor completes that all final fields are frozen and that any other threads that reference the object will see the correct values from those fields. Like volatile, operations that initialize final fields also will not be reordered with operations that follow the freeze. Beyond this, any variable that can be reached through a final field are also guaranteed to be visible to other threads. The impact of this is that immutable objects are inherently thread-safe.

java.util.concurrent

The concurrency library was added in Java 1.5. The library uses hardware level constructs such as compare and swap (CAS) to create thread safety mechanisms largely without the need for locking at the object level. This generally provides a significant performance increase over using the synchronized keyword to lock critical paths (the performance improvement can become a performance hit in extremely heavy traffic situations where extended periods of waiting are necessary because many of these algorithms poll rather than sleep and therefore waste CPU cycles). It is usually advisable to use the canned implementations from this package rather than implementing your own class. The concurrent package introduces the first truly thread safe data structures to the standard library. You can see the full documentation here. I am going to go over a couple of the most useful and a few of the interesting classes in the package and give some usage examples.

Semaphore

This class is very straight forward. It works primarily through the acquire() and release() methods. This implementation simply keeps an internal count variable that controls access. The primary use case is for restricting access to a limited resource. In the example below we use a semaphore to control a finite number of available db connections.

public static class DbConnectionFactorExample{

    public static final int NUM_CONNECTIONS = 10;
    private static final String DB_URL = "jdbc:mysql://localhost/exampledb";

    public static Connection getConnection(){
        try{
            return LimitedConnectionDecorator.getConnection();
        }catch(InterruptedException e){
            return null;
        }
    }
}

class LimitedConnectionDecorator implements Connection{

    private static final Semaphore connectionAccess = new Semaphore(NUM_CONNECTIONS);
    private Connection dbConnection;

    public LimitedDBConnection(Connection dbConnection){
        this.dbConnection = dbConnection;
    }

    public static Connection getConnection() throws InterruptedException{
        connectionAccess.acquire();
		try{
        	return new LimitedDBConnection(DriverManager.getConnection(DB_URL));
		}catch(SQLException e){
			connectionAccess.release();
			return null;
		}
    }

    @Override
	public void close() throws SQLException{
	    if(!dbConnection.isClosed()){
	        try{
	            dbConnection.close()
	            connectionAccess.release();
	        }catch (SQLException e){
	            //Roll back logic
	            //May release lock anyways depending on db
            }
        }
    }
    //... rest of implementation omitted
}

Exchanger

The exchanger is an interesting convenience class that facilitates the coordinated exchange of a data structure between two threads. Once the first thread finishes processing the object it calls exchange(object) and waits for the other thread to be done as well. Then they swap. The obvious use case here is for having one thread fill and one thread empty that structure. In the example below, I do just that.

public class ExchangerExampler{

    public Exchanger<MagicDataStructure> exchanger = new Exchanger<MagicDataStructure>();

    public MagicDataStructure mds1 = new MagicDataStructure();
    public MagicDataStructure mds2 = new MagicDataStructure();

    public static void main(Strings args[]){
        new Thread(new Filler()).start();
        new Thread(new Emptier()).start();
    }

    class Filler implements Runnable {
        public void run(){
            MagicDataStructure myMds = mds1;
            try{
                while(true){
                    if(myMds.isFull()){
                        myMds = exchanger.exchange(myMds);
                    }
                    myMds.fill(new MagicStuff());
                }
            }catch(InterruptedException e){e.printStackTrace();}
        }
    }

    class Emptier implements Runnable {
        public void run(){
            MagicDataStructure myMds = mds1;
            try{
                while(true){
                    if(myMds.isEmpty()){
                        myMds = exchanger.exchange(myMds);
                    }
                    myMds.empty();
                }
            }catch(InterruptedException e){e.printStackTrace();}
        }
    }

}

FutureTask

The FutureTask class is interesting because it basically combines a Thread with a method. When you spawn a thread you are basically just creating an asynchronous computation. The FutureTask wraps this up and allows that computation to return a result. You create a FutureTask by passing in a Runnable and, optionally, the result to be returned. You now start the FutureTask and then get() can be called some time later in the future. If the task has completed then get() will return the result, otherwise it will block until the result is finished being computed. Below is a simple example implementation.

public class FutureTaskExample{

    public static void main(String args[]){
        //Do you get it? It's a computater...
        FutureTask computater = new FutureTask(new OverclockedPotato());
        computater.run();
        //Do other awesome computing
        //....
        Object frenchFries = computater.get();
    }

}

class OverclockedPotato implements Callable{

    public Object call(){
        potato = new Object();
        //Do stuff to it
        return potato;
    }
}

CopyOnWriteArrayList

The previous examples were all nice concurrency constructs but now let’s talk about some of the interesting data structures provided by the concurrent package. The CopyOnWriteArrayList is a completely thread safe List designed to be used in situations where traversals occur significantly more often than mutation. In this case it is beneficial to be able to iterate over the List without locking. This is a common need when a List is used to hold Listeners such as in AWT or Swing applications. CopyOnWriteArrayList accomplishes lock-less iteration by creating a copy of the List when a mutation occurs. This allows iteration to guarantee that it will return the list in the same state as at the time it was constructed. What this means is that the CopyOnWriteArrayList holds a mutable reference to an immutable array, rather than the other way around. I’m not going to include a code sample here because the use is identical to that of a normal list and it is the use case and data structure implementation that is interesting.

ConcurrentHashMap

The ConcurrentHashMap is an extremely interesting class that provides complete thread safety with minimal locking. It is optimized for the most common operation, retrieving an element that already exists in the map, and can succeed without locking the majority of the time. In a highly parallel system with many threads interacting with the map the performance excels far beyond that of a synchronizedMap or a HashTable. Several neat tricks are used to accomplish this.

First, rather than having a single object wide lock, the ConcurrentHashMap has a lock for each bucket (or however many you want as this value can be set in the constructor). This means that theoretically that many threads can be writing to the Map all at once. While in practice it will be lower, it is still significantly higher than only one.

The second trick takes advantage of the JMM rules on immutability. All of the fields that make up the Map are final except for the entry value fields, which are volatile. What this means is that elements cannot be added or removed from anywhere but the front of the hash chain. Although you can’t know if you are at the head or not, this does guarantee that if you have a reference into the hash chain that the rest of the list won’t change its structure. Since the entry value is volatile, you are also guaranteed to see any updates to the field immediately, thus avoiding the potential for a stale view of memory.

The get() operation begins by getting the head reference in the desired bucket, without locking. It then traverses the chain looking for the value. If it is not found then a bucket lock is acquired, the head pointer is found again and it traverses again. Below is a simplified partial implementation.

public Object get(Object key){
    int hash = hash(key);
    //First pass without locking.
    HashEntry[] tab = table;
    int i = hash & (tab.length-1);
    HashEntry head = tab[i];
    HashEntry e;
    for(e=head; e!=null; e = e.next){
        if(e.hash == hash && equal(key, e.key)){
            Object value = e.value;
            if(value != null){//null value indicates the element has been removed
                return value;
            }else{
                break;
            }
        }
    }
    //Second pass with a lock
    Segment seg = segments[hash & SEGMENT_MASK];
    synchronized(seg){
        tab = table;
        i = hash & (tab.length-1);
        HashEntry newHead = tab[i];
        if(e != null || head != newHead){
            for(e=newHead; e!=null; e=e.next){
                if(e.hash == hash && equal(key, e.key)){
                    return e.value;
                }
            }
        }
        return null;
    }
}

The remove() operation can’t simply remove the element from the chain because it is possible that other threads traversing that chain would see the stale value. Instead removal begins by finding the correct element and sets the value to null. Since the field is volatile, any threads that check this value will immediately see the null value and know that they need to re-traverse the chain with a lock. Then the elements from the head of the chain to the removed element is cloned and joined to the remainder of the chain.

I highly recommend you read through the complete class implementation. It is a really well thought out piece of code that minimized the impact of locks and is cognizant of how the JMM rules works.

I hope this article built on the basic knowledge introduced in Part 1. You should now be familiar with the kind of tools provided by the java.util.concurrent package. It contains a variety of helper class both for building data structures that are thread-safe and for controlling synchronization. When doing multi-threaded programming this should be one of the first places you look for tools to facilitate your needs. You are hopefully also now more confident in your understanding of the effects that using final and volatile keywords has on program execution. In Part 3 of this series I am going to go into some examples of problems that can arise from seemingly harmless code, including incorrect use of the volatile keyword, this references that escape during construction, how to use ThreadLocal to render a stateful class thread-safe and using fork-join to split a workload into multiple tasks.

I am by no means an expert so I welcome any comments, corrections or requests for additional information in this series. I think this is an important and poorly understood topic and I would like to provide a strong introduction here.

What is Concurrency?

Concurrency in programming is the idea of breaking up computation into pieces that can be executed simultaneously. How this actually occurs depends on the underlying hardware but for our purposes we will assume that execution occurs in parallel. One of these pieces of computation is called a Thread. Concurrency becomes especially interesting and complex when the threads interact with each other. This can generally be through shared access to the same data, direct action on each other’s fields or shared need for limited resources. When this is the case it becomes necessary to synchronize the actions of individual threads.

Terminology

There is a lot of terminology involved with concurrency so I am going to go ahead and define the terms I am going to use and what I mean by them. Many of these terms have multiple meanings but unless otherwise stated I am referring to the following meanings.

Lock: A lock is what a thread gains when it enters a block of code (or gains access to a resource) that is synchronized. In general, when one thread has a lock, all other threads are blocked from accessing that resource, object or block of code.

Critical path/section: A critical path is a piece of code that is synchronized.

Condition Variable: A condition variable is a variable that is associated with a lock.

Mutex: A mutex (mutual exclusion) is a construct for creating a lock and is generally used interchangeably with lock.

Semaphore: A semaphore is a mutex that supports the use of a counter to allow multiple threads access to the same block or resource. (A semaphore with a counter of 1 is equivalent to a mutex)

Barrier: A barrier is a holding zone for multiple threads. It acts as a gate that only releases threads when a certain number have reached that point.

Monitor: A monitor is a term that has an inconsistent meaning and I will avoid using it but if you see it elsewhere it will generally be referring to a lock or an implementation of a locking mechanism.

Concurrency in Java

Java is especially interesting because, unlike many programming languages, it has support for multi-threaded execution and synchronization built directly into the language. All objects in Java implicitly have a single condition variable used for locking associated with it. This variable is accessed through the wait() / notify() methods. Prior to Java 5 this was the primary way synchronization was achieved. Concurrency is a very abstract topic so let’s just dive into an example that uses multiple threads.

This first example demonstrates how multiple threads work independently. Here we create three simple thread objects that simply print the given character 100 times.

public class MultiThreadExample {

    public static void main(String args[]) {

        //Create 3 Thread objects
        Thread aThrd = new PrintThread("a");
        Thread bThrd = new PrintThread("b");
        Thread cThrd = new PrintThread("c");

        //Start the threads
        aThrd.start();
        bThrd.start();
        cThrd.start();

        //You might expect this code to simply print 100 a's followed by b's
        //and c's but instead you see the values are interweaved and likely
        //to be different each time you run this file.
    }

    static class PrintThread extends Thread {

        String val;

        public PrintThread(String val){ this.val = val;}

        //Our run method here simply prints the val 100 times
        @Override
        public void run() {
            for(int i=0; i<100; i++){
                System.out.print(val);
            }
        }
    }
}

Example 1 output:

andrew@andrew-Inspiron-1520:~/dev$ java MultiThreadExample
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaabbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

As you can see from running the last example, what we get is our three threads executing in parallel and as a result printing to the console in no specific order. The threads share access to the standard out. We could modify the code to guarantee one thread completes it’s use of the standard out prior to another beginning. That is boring though so let’s instead look at a slightly more complicated example.

In this example we are going to build a Queue implemented with a linked list that is thread safe. Before we do so, let’s look at what could go wrong if we didn’t. Below is the implementation of our queue. Take a second to try and figure out where a problem might arise with using a Linked List if we didn’t include the synchronized keyword.

public class QueueExample {

    class Node {
        public Object data;
        public Node next;

        public Node(Object data){this.data = data;}
    }

    private Node head, tail;

    public synchronized void enqueue(Object data){
        if (null == head){
            head = new Node(data);
            tail = head;
        }else{
            tail.next = new Node(data);
            tail = tail.next;
        }
    }

    public synchronized Object dequeue(){
        Object result = null;
        if(null != head){
            result = head.data;
            front = front.next;
        }
        return result;
    }
}

Figure it out? That’s right the problem occurs when two threads add an element at the same times. Since the act of adding an element is not an atomic operation, there is room for an inconsistent state to develop. Consider the following execution given our queue q and two threads t1, t2. Let’s also assume q already contains elements and the tail is called x.

So what does this leave us with once both threads have finished executing? A broken queue is what. The former end of the list X is now pointing to Z as it’s next node when it should be Y; X is completely lost with nothing referencing it at all; and our queue thinks a null node is the tail element.

Why does that magic word ‘synchronized’ fix the problem? ‘synchronized’ is the Java keyword for requiring a lock or mutex on the object to use it’s method. It means that in order to enter that method a thread must first obtain a lock. Since only one thread can have a lock on that object at a time, our problem is fixed because an element can only be added or removed when no other thread is acting on the queue. So in our execution example above t2 would simply wait for t1 to exit the method and be notified that it was free before it began executing the enqueue() call. To further use our terminology, we would say that the enqueue method was our critical path because it required a lock to use.

What if we have a much more complex method and only a small part of it is a critical path? Naturally we don’t want to synchronize the entire method because it would waste cycles waiting and reduce the level of concurrency. Furthermore, what if we don’t need a lock on the entire object whose method we are using but rather just the small critical path? We instead lock only on what is absolutely required and we use a separate object. An example might be a method that modifies a user’s account given some identifying information about the user. We don’t need to lock on any computation to get the account, but instead only the portion that is making changes to the account.

public void modifyAccount(Object userId){
    //...
    //perform complex steps to get userAccount object
    //...
    synchronized(userAccount){
        //execute critical path
        //make changes to account
    }
    //...
    //finish computation
}

Note that we are locking on the userAccount object and not the object whose method we are executing. Often times the object that you use to lock a critical path is insignificant and used purely to construct the mutex with it’s implicit condition variable.

So now we know how to lock critical code blocks, but what if we have a limited resource that has a finite quantity available. A simple mutex won’t be sufficient because it doesn’t allow us to make full use of our resources. Here is where we use a semaphore and take advantage of the count variable. Below is a simple example implementation of a Semaphore class.

public class Semaphore {

    private int permitCount;

    public Semaphore(int permitCount){
        this.permitCount = permitCount;
    }

    /**
     * Get one resource lock
     */
    public synchronized void acquire(){
        try{
            while(permitCount == 0)
                wait();
            permitCount--;
        }catch(InterruptedException ie){System.err.println(ie.getMessage());}
    }

    /**
     * Release numPermits resource locks
     */
    public synchronized void release(){
        permitCount++;
        notify();
    }
}

We have now successfully created a basic semaphore. It is essentially just a mutex that allows multiple threads to have a lock on the resource. Some good examples of when this is useful is when you have a finite number of database connection available or you have a pool of threads handling messages as they arrive. There are a couple of things worth noting about our implementation that is different from what will be found in the Java 5 library. In our implementation we make no guarantee that threads will receive resources in the order they are requested and we allow threads to release resources independent of whether or not they had acquired one to begin with.

The last basic concurrency construct I am going to discuss is the barrier. A barrier is very simple and is quite often overlooked because its functionality is easily achieved in a variety of ways. It is essentially exactly what it sounds like. An object that holds up a set of threads until a certain number have arrive, then releases them. It can be thought of as the reverse of a semaphore. Below is a simple implementation:

public class Barrier {

    private int numMembers, generation = 0, waitingCount = 0;

    public Barrier(int numMembers){this.numMembers = numMembers;}

    public synchronized void await() throws InterruptedException {
        int currentGeneration = generation;
        waitingCount++;
        if(waitingCount == numMembers){
            waitingCount = 0;
            generation++;
            notifyAll();
        }else{
            while(generation == currentGeneration){
                wait();
            }
        }
    }
}

This simple construct simply forces the first numMembers – 1 threads to wait and then releases them all once enough have arrived. The barrier is then reset and available for use again. The barrier is a very straight forward construct that can be used for a variety of things.

To wrap up we have now been introduced to the basic terminology related to synchronization and concurrency as well as looked at how to implement the basic constructs in the Java programming language. You should hopefully now feel comfortable using threads to facilitate simple computing in parallel. In the next post, I will discuss the most common errors that result from incorrect synchronization, introduce the java.util.concurrent library that provides complete implementation for all of the constructs we created above as well as some more advanced tools. Finally in a third post I will discuss advanced concurrency issues specifically related to the Java programming language.

I am by no means an expert so I welcome any comments, corrections or requests for additional information in this series. I think this is an important and poorly understood topic and I would like to provide a strong introduction here.

Like an endless stream of spam!

I recently switched to T-Mobile from Verizon in order to pick up the Nexus S handset. I’ll discuss my thoughts on the Nexus S and that switch in an up coming post. Today I just want to talk about T-Mobile. From the moment I activated my phone, I started receiving what were essentially spam text messages. From what I can tell, whoever had the number I now have before me has a Yahoo! email address that receives nothing but spam. I know this because that individual also decided these messages were important enough to send text notifications about each time they got an email. I am now the proud recipient of said notifications.

I have called T-Mobile to take care of this twice. Both times, when I was finally able to get a person on the phone, they were extremely friendly and understanding. Unfortunately, they also appear to be wildly incompetent. Each conversation concluded by them telling me that the number I am receiving texts from has been specifically blocked and that all text messaging has been totally blocked on top of that (don’t need it; use Google Voice for everything). I promptly receive a message from T-Mobile saying the changes have been made. Great. Excellent. This is exactly what I wanted. Then, a few hours pass and right when I start to believe I am finally free from the spam, I receive the urgent announcement that “contact@SmallBassShop.com” wants me to know that I can “View Singles in SAN FRANCISCO at Match=2Ecom…”.

This is absurd. There is no excuse for me receiving even the first message after activating my brand new account, but hey sometimes things just happen. I understand and I didn’t get worked up about it. But seriously, making me call you at least 3 times to have a single number correctly blocked is completely unacceptable. If I am not able to permanently fix this when I speak with yet another empathetic, friendly employee tomorrow I will be canceling my account, retailing my Nexus S for a net profit, and buying a damn iPhone on Verizon.

H.264 is a standard for video compression used extensively across the web. Most of the popular video sites including Youtube, Vimeo, Netflix, Hulu etc. use this encoding for their content. Currently, Internet Explorer, Apple Safari, and Google Chrome support H.264 while Mozilla Firefox, Opera, and soon Google Chrome do not. They instead support Ogg Theora and WebM codecs. The main source of conflict is that technology used by H.264 is patented and owned by the company MPEG LA.

With the maturation of the HTML 5 standard and the introduction of the <video> tag, we are seeing a struggle to define what will be the standard encoding used with that tag. For many it is seen as extremely important to make the encoding a free and open standard, like HTML itself. Others feel the most important factor in choosing a standard is finding one that will facilitate heavy adoption. Others still are most interested in choosing the highest quality video compression system available. As you might expect, Mozilla, Opera and Google reside in the first camp that see openness as being the most important factor, while Microsoft and Apple are more concerned with adopting a higher quality, widely used codec.

While you might be tempted to say ‘Who cares what codec we use, video is video!’, the implications of this decision are quite large. If H.264 becomes the defacto standard implied by the <video> tag then a single, for profit entity will essentially have control over one of the fundamental building blocks of content on the web. This is in direct conflict with the aims of the W3C and could result in censorship and monopolization both in the user’s choice of content creators (ie Youtube, Netflix, etc.) and consumption vehicles (Internet Explorer, Chrome, etc.) as a result of unreasonable licensing fees for encoding and decoding video. On the other hand, if WebM or Ogg Theora become the standard, it is possible that the <video> tag may never be adopted widely or simply take much longer to take off. This is due to the fact that only a small portion of Internet video is currently encoded in either of these formats, while as much as 80% of content is already encoded in H.264. Rather than spend the time and money to re-encode or dual encode the content, businesses are likely to never use the <video> tag and instead continue to rely on Flash to deliver video to users. Furthermore, whatever codec is adopted will likely be used for a very long time. H.264 is a better standard that provides slightly better performance and quality all around. Using one of the suggested open codecs will be settling for a slightly inferior tool.

So assuming this is an important issue, what does Google’s decision to drop H.264 support mean? It means Google is throwing its weight behind the WebM standard, which by the way is developed by Google. Google is taking a long term strategy where they are willing to risk curtailing the rapid adoption of Chrome in exchange for the security in knowing they will have a large amount of control over the future of video online. In line with this, it is likely that Google will begin dual encoding the content on Youtube and possibly take the drastic step of removing H.264 encoded content all together. Google is in a unique position to do this because as you have probably noticed they are the creators of the WebM codec, they are the owners of the largest video content site in the world, Youtube, and they are the creators of the fastest growing web browser Chrome (about 10% total market share now).

That is all well and good for Google, but what does that mean for me, the guy who just wants to lay on his sofa and watch cute kittens? At this point, pretty much nothing. As you may have already noted, I said Firefox has never supported H.264 and you don’t hear anyone complaining they can’t watch Youtube. That is because the vast majority of websites deliver video content to users through Flash when the <video> tag won’t work (or more accurately most site don’t use the video tag at all yet). Now Chrome is simply going to be in that boat as well. So even when you are using Firefox or Chrome, you are still watching video encoded with the H.264 codec and then decoded by Adobe Flash for your viewing pleasure. Furthermore, Google has announceed that they will be releasing plug-ins for Internet Explorer and Safari that allow WebM codec use.

This whole scenario probably sounds a lot like a similar one from about 9 months ago but instead of H.264 codec and Google it was Flash plug-in and Apple. That is because it is very similar. In both cases it is a large industry player using its muscle to break the hold of a proprietary technology for the greater good of the busin.. eh .. users.

A long story short, what is likely to happen in the short term is further paralysis on coming to a consensus on a codec for the <video> tag. There is absolutely 0% chance of Mozilla or Opera yielding on this issue and Google’s decision is likely to only anger and further entrench Microsoft and Apple. The web will continue to rely heavily on nonuniform plug-ins to deliver content and the user will continue to suffer as a result. In the long term, we can hope that WebM matures and widespread adoption takes place. One day we can hopefully be free of the need for anything but uniform, open technologies to power a standardized web experience.

For an in depth look at the technical differences between H.264 and WebM look here.

Google’s official statement on dropping H.264.

Statements from MPEG LA about H.264 Licensing.

Update (2011/01/16): It has been brought to my attention that I have over looked the possibly serious consequences on performance in many of today’s devices that would be caused by moving to WebM only. This is a result of most devices lacking hardware acceleration for WebM which they do have for H.264. Have a look for a more in depth explanation.

Tl;dr – App World is slow, unintuitive and actively impedes users trying to purchase apps and leverage what the platform has to offer. This drives customers away and hurts RIM tremendously.

For starters I’m not going to talk about the selection of applications available. Everyone already knows what the deal is there, so let’s not beat a dead horse. Instead I am going to focus on the “App World” application itself and why it is such an impediment to users.

First and foremost, App World is the slowest application on my device in every way. It takes between 5-7 seconds to get from selecting the app icon to a UI that responds to any kind of user input. I say any kind of user input because the UI is never what I would call responsive. Sluggish at best. A static list of my apps (less than 20) doesn’t even scroll fluidly. Everything about the performance of App World makes me hesitant to download another app for fear that it will be as poorly done. When I find an application I would like to read a description of, it of course takes an unacceptable amount of time to load. I cannot stress this point enough; App World is inexcusably slow in every way.

If you are a little more patient with it this may not bother you, especially if you haven’t spent time in the iOS App Store or the Android Marketplace, but the BlackBerry ID Program will. I won’t say I got no notification of the mandatory enrollment prior to having it pop up when I went to update my Foursquare app (the nicest app for the whole platform by the way) because it is likely they sent out some email warning I didn’t see. I will say they didn’t give me any reason not to be peeved about all of the sudden needing another superfluous set of login credentials though. I held off from updating or downloading any applications for over a month because every time I thought Hey, I’ve got 2 minutes. Let’s download a new app. I was hit in the face with “Register for BB ID!” and I said meh, I’ll do this later. When I finally did get around to registering, what new features were awaiting me? Nothing really. That is unless you count being able to download applications again and not losing every app you’ve bought upon upgrading to a new BlackBerry (Thanks but I’m going to probably get a Nexus S). Here’s the real kicker too. You can’t save your login credentials for BBID. Do they really even want anyone to buy applications?

I attended several presentations RIM made for developers that talked a little about BBID. I think they said the two golden words of mobile dev about 100 times in those talks, carrier billing*. BBID was the path to carrier billing and that developers need only to create a “Super Apps” and money baths would be eminent. Well that is all fine and good but, one of the main reasons this is normally true is that carrier billing means users don’t need to waste time registering or  inputing CC/PayPal info when they buy an app. This is completely lost when you instead have to put in the even less familiar BBID credentials every single time.

While the rest of my complaints could be considered nit picking, I think they are important. With the rollout of OS 6 in August came an updated App World where RIM made the poor decision of mixing UI formats throughout the app. The only real new feature of the app, the Top 25 section, employs the new scrolling tabs interface that is featured across most of OS 6. No complaints there, it would be a great UI if the app wasn’t so slow but they neglected to bring the rest of the app up to date. Instead you have 4 different basic UIs depending on what part of the app you are in. This is ludicrous. There is absolutely no flow to the app and it comes off as a jumbled mess. After using the app for 2 years now I am still unsure of how to navigate it. UX is of vital importance and you cannot underestimate the ease at which user will be lost and confused by your app’s layout. I am not a designer but I know better than to do what they did with App World. Not to add injury to insult, but on the Storm series devices, App World does not even correctly implement the user input controls. They neglect to correctly differentiate between a touch and a click (arguably the only interesting feature the Storm Series had going for it and one that I genuinely like).

My last point has more to do with the platform than App World itself but it certainly factors in. Downloading or updating an application requires a full reboot. A full reboot can take upwards of 5 minutes.  That doesn’t seem so bad at first, but when you consider the fact that I only want to spend a total of 5 minutes using most applications I download, this is a major sticking point. That platform issue aside, App World factors in because it is painfully difficult to find the “Full reboot” option if you decide not to do so immediately after installing the app. You can’t find it within the app and have to dig through multiple menus and submenus to find it anywhere on the phone. It would make more sense to have added a “Complete Installation” option in the app menu (and the device home menu).

I feel like RIM is disconnected from how their users interact with their devices. We hate things that are slow, confusing and demanding login credentials. Rather than simply delivering the slightly inferior product we have come to expect, App World is actively alienating users. It serves as a terrible example of what the RIM app store has to offer and is a serious factor in why I will not purchase another BlackBerry. It epitomizes much of what is wrong with RIM OS. The app store experience should be fluid, reliable and as intuitive as anything on the device. It is the key to unlocking the potential of the platform and, by neglecting it, RIM is pushing users that interact heavily with their devices away.

Priceless: While writing this article I had my BB open to App World. I look up from writing a paragraph and what do I see but RIMs version of the Blue Screen of Death. “App Error 523” – my only option, “Reset”, which did nothing and instead required the ever common battery pull.

Note: I own a BlackBerry 9530 Storm and have worked with most BlackBerry models ranging from the 8100 series up to 9800. I also do development work on the Android and J2ME platforms. This post pertains to all devices that are on OS v5 and v6. In my experience, the BB 9800 Torch and the BB 9700 Bold have a slightly better App World experience but still fall well below what I would call highly usable.

*Carrier billing is being able to charge the users phone bill directly through the carrier,. This drastically increases purchase rates for applications and provides a better user experience.

On Sunday 11/28/2010 they began publishing 250k US Embassy Diplomatic Cables, which they are calling Cablgate. Almost immediately they became the target of DoS attacks that brought the site down for several hours.

On Monday 11/29/2010, to counter the DoS attacks, they moved to Amazon EC2 hosting. The EC2 servers are much more resilient to spikes in traffic and is more prepared to deal with DoS attacks. Also Monday a hacker that goes by Th3J35t3r was arrested for conducting the DoS attacks. Ecuador also announced they were prepared to offer Wikileaks founder Julian Assange asylum and a public forum to discuss the Cablegate documents.

On Wednesday 12/01/2010 Amazon, amid pressure from the US government, ceased to host Wikileaks. This resulted in a small amount of down time while Wikileaks relocated the Cablegate data back to servers hosted by Bahnhof.

On Thursday 12/02/2010 EveryDNS, who had been directing traffic to the wikileaks.org domain stopped and in essence revoked Wikileaks .org TLD. Wikileaks came back up on a Swiss domain wikileaks.ch which is owned by the Swiss Pirate Party. The servers for Wikileaks are now hosted by a French company called OVH and the Swedish firm Bahnhof.

Late Friday 12/03/2010 PayPal announced that they will no longer support donations to Wikileaks because their ToS prohibit supporting illegal activity.

(Update) Sunday 12/05/2010 Wikileaks began Mass Mirroring the site, requesting anyone with extra server space to donate to ensure Wikileaks cannot be taken offline..

There are certain to be more of the above happening in the coming days and weeks. I will probably post an update of events in a week or so. I hope this gave a good overview of what has happened and when.

If I missed anything significant let me know and I will add it in.

To get into this habit, I decided a few weeks ago that I would post something every day in December. I read a fair number of articles about procrastination, being efficient, GTD, motivation, etc.  One such article I read today has convinced me to expand from simple blogging every day for 30 days but to also do a couple other things that I need to do and have been putting off. The list of things I want to do is really long but I have settle on 4 things listed below that should take about 2-3 hours.

Daily list:

• Blog
• Work on 2 side projects for >= 30 minutes each
• Exercise
• Wake up by 6:00am

I chose these things because they are all extremely doable, not doing these things is a major source of stress in my life and the results are likely to have a higher impact than many of the other things on my list. Keeping track of my progress is going to be an important motivator so I am tracking these things in a spreadsheet this month but will be transitioning it to a different medium in 2011 (more on that closer to January).

As always I welcome any thoughts or comments.