Developmentables

Joel Spolsky’s Future Trello Posts

June 1, 2012

Trello works on phones! We knew from day 1 we would have to go mobile in order to realize our goal of being a horizontal app for the masses. But as you’ll recall, when we launched Trello, we proudly stated:

There will never be an “installed software” version of Trello.

And we’ve held true to that. Most people out there think you need to build a native application in order to get your app to work on a phone. I’m happy to say, we’ve found a way around this restriction. Using our approach, you no longer have to learn cumbersome proprietary technologies, and you can keep delivering your application updates continuously just like on the web. What’s our secret? Well, we’ve developed a system of rendering special views for our application so that it works on your mobile phone’s browser. Here’s how we did it…

November 12, 2012

It turns out MongoDB totally sucks. We’ve learned this the hard way, but building a web application on top of a NoSQL platform actually causes more problems than it solves. We’ve now ported all of our data into MySQL and our app is much easier to maintain. We’ll share our discoveries with you so you don’t have to experience these headaches for yourself. Here’s how we did it…

March 7, 2013

Node.js is of no practical use whatsoever. Turns out our developers were just performing technical masturbation. We’ve migrated all of our server side code to Rails. Here’s how we did it…

December 17, 2013

Starting in 2014, we will be switching over to paid accounts for Trello. It turns out we built a vertical application after all. The only people using Trello are our friends, and they place high demands on our team to keep delivering new features. Since we’re not competing with anyone else, we can start charging money! But don’t worry, we won’t make the change for a few months so that you have plenty of time to make the switch. Here’s how it will work…

PS: I used to work at Microsoft and I single-handedly made Excel a smashing success. Don’t question me.

AJAX Facebook Like Button

Relating Inventables Products with Hadoop MapReduce

If you’re a frequent user of e-commerce websites, you’re probably familiar with the lists of related products these sites often feature to help customers find what they’re looking for on the store. Amazon, in particular, sports several such lists on every page, including “Frequently Bought Together” and “Customers Who Bought This Item Also Bought.”

If you’ve been to our site lately, or read our announcement, you know that Inventables is now an e-commerce site. As such, we thought it would be a good time to explore adding some of these types of lists to our product pages.

In this post I’ll talk about how we scripted a process that uses Hadoop MapReduce to parse the millions of product views we’ve accumulated over the past year or so, and put it into a format that can be used to display the list “People who viewed this product also viewed” for every product on our site. To see examples of this list in the wild, check out one of our cool products.

Formal Definition of our List

Given two products p1 and p2, we will define p1 related-by-path-to p2 (and vice versa) if there is exists a user-session-specific path P where page-view(p1) and page-view(p2) are both members of P. Additionally, the weight of this relationship between p1 and p2 will be equal to the number of distinct paths for which they are both members. If 10 users viewed both p1 and p2 during a single visit, then those two products will have a path-relation of weight 10.

To display the list for a given product, we will sort the set of related products in descending order of weight. For example, let’s say you’re looking at Rubber Glass. If you see Flexible Metal Brick as the first item in the People who viewed Rubber Glass also viewed list, then that means more people who looked at Rubber Glass looked at Flexible Metal Brick than any other product on the site.

The Raw Data

To build our list we’ll be drawing on a single table from our application database. This table stores an entry for every unique product viewed by a user during a given session. Only two columns from this table are needed for our task: path_id and product_id. We can export this data from our Amazon RDS MySQL instance, using the mysql command line utility like so:

 mysql -u $user --password=$pass --host=$host --batch -e \
      "SELECT path_id, product_id FROM recent_items" \
      $database > /tmp/input/recent_items.tsv

This tab-separated file will constitute the initial input to our MapReduce workflow.

Tool Selection

I selected Hadoop Streaming to drive the MapReduce workflow because of the large volume of data we’re dealing with, and also because we run our application infrastructure on the Amazon AWS platform, which, in the form of their Amazon Elastic MapReduce offering, has built-in support for Hadoop Streaming. For now we can get by with running Hadoop in single node mode, but if our scale grows or we start doing more things with MapReduce, we’ll be able to easily switch to true cluster mode.

Solution Outline

The goal of this exercise was to start with a table of path_id, product_id values and ultimately generate a new table of source_product_id, target_product_id, weight values, where the source and target product are related to each other with the given weight. From this final data format, it is trivial to build a gallery to display the top n related products for a given source product.

To get from the millions of path_id, product_id records to the desired source_product_id, target_product_id, weight format using MapReduce, we take a two-phase approach.

MapReduce Phase 1

This phase creates an output file where each row represents one instance of a product-to-product relationship from a single path. As such, each row will carry a weight of 1.

Map:

The input to this function is already in the desired format of path_id, product_id, so this mapping step is just the identity function.

Reduce:

Hadoop MapReduce guarantees that all rows for a given key emitted by the map step will go to the same reducer. In our case, the key is a path id. So in this first reducer, we iterate over all the rows for a given path, appending each product id to an array. Once we have all the product id’s for a given path, we can emit all of the permutations of product_id, product_id pairs.

For example, let’s say the input included the following rows:

 path1, product1
 path1, product2
 path1, product3

(The user looked at 3 products during his session.)

The reducer would first map path1 to the array [product1, product2, product3]. Then, it would emit six key/value pairs of the form source_product_id_target_product_id, weight:

 product1_product2, 1
 product1_product3, 1
 product2_product1, 1
 product2_product3, 1
 product3_product1, 1
 product3_product2: 1

And the ruby source:

MapReduce Phase 2

The second and final MapReduce will sum the weights associated with every unique product1_product2 key.

Map:

Since the input to the second map phase is already in the desired format of product1_product2, weight, the identity function can also be used here.

Reduce:

The reduce step, being given all the weights for a given product-product combination at a time, simply iterates over each key, sums the weight, and emits one key/value pair with the total.

Ruby source:

Pulling in the data

After the second MapReduce phase completes, we have the data we need. All that’s needed is to import it into our database. We can accomplish this again with the mysql command line utility:

 mysqlimport --local --compress -u $user --host=$host \
      --columns=source_product_id, target_product_id,count \
      --replace $database \
      /tmp/final_output/viewed_products.tsv

Note: the viewed_products table has a unique index on source_product_id, target_product_id.

Stitching it all up

We want our process to be automated so that it can be run on a schedule. This can be accomplished with a simple shell script.

Conclusion

As you can see, it wasn’t hard at all to build our first list of related products using Hadoop MapReduce. We plan to do a lot more in this realm going forward.

—Jeff

How Startups are like Soccer Teams

This summer I watched a lot of World Cup soccer games. Initially, I only watched the U.S. soccer team’s games, but then I slowly expanded my viewing. I started watching all of Spain’s games because I have close friends in Spain. Then I started catching the replays of as many games as I could in the evenings, and I’d sometimes listen to some of the audio of games during work if it was a particularly compelling matchup.

Prior to this World Cup, despite playing in recreational soccer leagues for nine years as a kid, I was pretty ignorant of the game inside the game, especially at the top levels of competition. While watching the games this summer, though, I really started to soak up the details. I learned about different strategies like pushing your players up and attacking vs. playing a defensive game while looking for chances to counter-attack. I learned about tactics like trapping players offsides or taking some corners short to open up the middle of the field.

But the biggest thing that struck me in watching these games was how well the players of different positions worked together within a team. When I was a kid playing in those recreational leagues, players at different positions hardly worked together at all. Forwards would not help defenders. Defenders would not help forwards. Most of the time a team was little more than a group of individuals all playing their own game.

That’s not how it works at the world cup level. Sure, the players fill the same positions we did based on what they do best: strikers are best at moving the ball past defenders and scoring goals; midfielders are best at maintaining possession of the ball and setting up their teammates for goal scoring chances; defenders and goalies are best at neutralizing an attack and recapturing possession of the ball.

But what sets the players apart at the top levels of soccer, other than sheer talent, is that they spend a lot of time doing things that are not the things most often associated with their position. Forwards spend a lot of time helping neutralize an attack, and even defend shots on goal when there is a penalty kick or a corner kick attempt. Midfielders may attack and score goals, or they may fall back and help their defense deal with a threat. Defenders may push the ball up the sidelines on an attack, or come all the way into the box to field a corner and put a shot on goal. Each player’s position is really just a default. It’s what they spend most of their time doing and are primarily responsible for, but in order to make the team effective at an elite level, each player must spend a lot of time outside their core area of expertise.

There are two big corollaries to this model in the startup world. First, there is a high correlation in the assignment of core responsibilities. On the front lines, you have the individuals who are responsible for customer development. They spend most of their time engaging the enemy (the customer)*. If your startup is a soccer team, they are the forwards. On the other side, you have your development team. Their responsibility is to neutralize problems by delivering features that will give the customer development team a chance to pivot and try a fresh angle. In the soccer model, they are the defense. In the space between these two units roam the business analysts. They help the development team convert problems into solvable units and then make sure the completed features lead to data and insights that funnel smoothly back to the customer development team. They are the midfielders.

The second big corollary is that just as with soccer teams, the startup that enforces rigid divisions of labor is doomed to fail. To be truly elite, a startup must emulate the fluid boundaries between roles demonstrated by the elite soccer team. The development team needs to understand the customer’s problems and assist with designing and administering usability tests and developing personas. A developer may even need to propose a business model hypothesis (take a shot on goal). The customer development team needs to help the development team, if not by actually getting into the code and building a feature, then at least by listening to developers and working with them to simplify and distill features down to a size that can be executed consistently and efficiently. Business analysts need to take an active role in customer development by assembling and distributing surveys and questionnaires, conducting formal customer interviews, as well as crafting and administering usability tests. They might have to go further to retrieve insights (the ball) than they would like by building spreadsheets to evaluate usage patterns or crafting and administering A/B tests of a given feature.

What this exploration of the similarities between soccer teams and startups highlights, of course, is the importance of teamwork—a specific kind of highly collaborative and help-based teamwork. If you’re involved in a startup, I hope this discussion serves as a fresh source of inspiration. And I hope your team takes home the next World Cup.

—Written by Jeff

—Illustrations and Title Photo by Billy

If you’d like, you can fork our jerseys on github.

* If you are familiar with Jeffrey Moore’s “Crossing the Chasm”—where penetrating a market is likened to launching a D-Day invasion—you will no doubt be used to thinking of customers as “the enemy”

On Batteries and Brainstorms

“I’m fully charged by this idea.”

I re-read Mark Boulton’s A Practical Guide to Designing for the Web this week and was inspired by a bit of wisdom from his ‘Ideas’ chapter.

In this chapter, Mr. Boulton lays out a very useful strategy for conducting brainstorm meetings that involves ranking generated ideas using what he calls a ‘Passion-O-Meter.’ By his own admission, this is just a “fancy name for using some stickers”, but I was struck by the potential of the idea to offset emotions that often accompany such an exercise.

Have you ever been in meeting where someone got offended because their idea was ill-received? I have.

Because Mr. Boulton did not provide any explicit instructions (or ready-made stickers for that matter), I was required to develop my own version of the ‘Passion-O-Meter.’ I wanted to make the ranking process as fluid, fun, and free of conflict as possible. So the first thing I had to decide was what to actually put on the stickers themselves. In my experience ideas generally fall into 1 of 3 categories:

• Great
• Good
• Not So Good

Since I knew the stickers were going to be physically affixed to the ideas themselves (we write them out on a giant 20ft X 20ft whiteboard) I had to make sure they weren’t egregious in any way. No ‘loser stamps’ allowed. At the same time, I wanted something clever and memorable. For some reason, my mind very quickly jumped to the idea of battery meters. Maybe I was inspired by the errant behavior of my iPhone 3G’s battery since updating to the latest OS.

Who knew how much the fluctuating charge level of an iPhone battery could affect one’s mood?

Inspired by the iPhone’s battery meter icon and Simple Bits’ Charge T-Shirt, I created icons and phrases for three possible rankings:

After creating the artwork, the next step was to import the images into an Avery template, in this case #5160. I dubbed my version of the idea ranking process the ‘Passion-O-Meter Blitz’ as the team was only given 5 minutes to rank about 30 ideas. I wanted there to be a flurry of activity so that each team member’s votes could be more or less anonymous.

Based on the feedback I received after the meeting (and the efficacy of the ranking procedure itself) it would seem that the battery stickers and my interpretation of Mark Boulton’s ‘Passion-O-Meter’ exercise were a success!

If you’d like to use the battery stickers I created in your own brainstorm meetings, please feel free to download the Adobe Illustrator source file from my repository on GitHub:

http://github.com/wwhited/brainstorm_battery_stickers/tree/

—Written by Billy

Diffing Darwin: Visualizing the Future of a Web App

Could there be more reciprocity between the way programmers and designers think than what most in our industry have traditionally believed?

In a planning meeting this week, our development team discovered a surprising similarity in the way our designer, Billy, and our lead developer, Jeff, visualize things.

As our CEO discussed his vision for the company over the next several months, Jeff and Billy were both struggling to get a vivid picture of each of the 3, 6, and 9 month sign posts he was laying out. So we both spoke up about how those sign posts could be made more clear.

First, Billy said he pictured the website as a series of time-lapsed screenshots similar to the way human evolution is often portrayed on the Darwinian scale. He then said that to explain the signposts he would simply pluck an image from the series created in his head and describe it.

While Jeff liked this approach, he said that his picture would center more around the differences between each sign post. He visualized a set of high-level git diffs showing the features that had been added and subtracted between signposts.

At first, the Programmer vs. Designer contrast seemed so cliche and obvious that it was comical. But after a few quick moments of reflection, the core similarities between the two approaches dawned on everyone. It became apparent that if you were familiar enough with the features of a website, a git diff could serve as a blueprint from which you could construct a visual image.

We decided to illustrate our revelation by ‘git diffing’ the ‘visual scale of human evolution’. At Inventables, our designer sits right next to, and works directly with, our programmer. Here’s to hoping that more companies abandon the “over the wall” approach to designer/developer collaboration:

—Written by Jeff and Billy

PS - If you’d like to build a better monkey, the source code is on github.

What Motivates us at Inventables

Lately there have been a lot of headlines about what motivates us in the workplace. In this talk, Dan Pink boils it down to two things:

1. You make enough money so that you’re not worried about money
2. You have the autonomy to direct your own work and make an impact

Dan then gives an example of an amazing place to work where one day a quarter the employees get to work on whatever they want. The only condition is that they have to present their work at an informal, fun type of gathering. This company, the Australian based Atlassian, has reaped a lot of benefits from this approach, including bug fixes and new features that they wouldn’t have otherwise had the time to build.

Based on this data, I’d say Atlassian does sound like a good place to work.

At Inventables, we don’t have one day a quarter to work on whatever we want. We have one day every week. Instead of 4 days a year, we have 50 days a year. 50 days to work on whatever you want. And instead of having to present your work at a meeting, you don’t have to present your work at all. Unless you want to. And that presentation can take any form you want. Maybe you want to have an informal, fun type of gathering. Maybe you just want to send out a link.

Because we build software using an agile approach, we’ve dubbed our one day a week of free exploration “Free point Friday.” And we, too, have reaped a lot of benefits from our approach. Like Atlassian, we’ve fixed some bugs. We’ve built some features. We’ve redesigned the look and feel of our website. We’ve built entirely new applications just for fun. We’ve mined data looking for patterns, caught up on reading, and just played around with new technologies.

As the leader of our technical team, I believe our “Free point Friday” approach is the best way to run our business because if someone has an idea for how to improve things, they don’t have to sell it. They don’t have to set it aside for later. They can just do it, and they’re free to fail. The failure will not be analyzed or filed away and brought up later.

Our business is about discovery and innovation. We believe anything’s possible, and we believe that if you just let people explore, they’ll prove it to you.

—Written by Jeff

Goodnight, IErene.

As of May 2010, Inventables will no longer support Microsoft’s Internet Explorer 6. The reasons for this are numerous and varied and range from the technical, to the methodological, to the ethical, to the practical.

IE6 is a 9 year old piece of software. That’s at least a century in internet years.

At a time when even resource-laden internet powerhouses like Google and Amazon are discontinuing support for the aged, buggy, and categorically unsafe browser, we simply felt it was time to say goodnight.

Still, we acknowledge that there are many users (namely at large corporations with staggering IT infrastructures) that have no choice but to use IE6. We understand the predicament these users find themselves in, but given the small size of our development team and the ever changing nature of our app, we found we could no longer afford to spend valuable development time getting IE6 to comply.

Using Dean Edwards IE9.js, we’ve managed to make IE6 “hang together” for now, with minimal effort. This has made developing for IE in general much easier, and has provided those using IE6 a somewhat usable consolation. Here’s a screenshot of what IE6 users can expect:

Let’s let the web evolve…it’s ready and willing! In this spirit I am reminded of Bob Dylan’s immortal lyrics:

Your old road is / Rapidly agin’ / Please get out of the new one / If you can’t lend your hand / For the times they are a-changin’.

—Written by Billy

Adventures in Typekit

One of the many great things about working at Inventables is that we, the development team, are continually afforded the opportunity to experiment with—and even implement—cutting edge web technologies like CSS3, HTML5, Typekit, and many others. Because we’re small and nimble we are able to adapt quickly to the changing landscape of the modern web. It’s a great atmosphere in which to create.

But while being at the cutting edge is great for our team’s morale, we must balance our excitement with restraint. Using advanced and not-always-fully-supported web technologies is an exercise in trial and error, frustration and reward.

Case in point: Typekit, the web-font hosting service that has been gaining popularity in recent months.

Where Typekit succeeds

Typekit works well as a repository for high quality, professional and legal web-ready (though not always “web-optimized”) fonts. Because Typekit has struck partnerships with several well established foundries they are able to serve up many excellent, complete and otherwise unavailable (due to strict licensing agreements) fonts that free-font sites like fontsquirrel simply cannot. As a result, Typekit takes the guesswork out of choosing custom web fonts. I can browse the site and quickly/easily identify a host of viable fonts without having to worry about EULAs and without being restricted by a limited number of available font weights. Until the WOFF takes off, Typekit and similar font-serving sites like Font Deck provide the best options to those designers looking to use fully-featured, custom fonts on their websites.

The Unfortunate FOUT!

The biggest frustration I currently have with Typekit is with what is known as the “flash of unstyled text,” or FOUT for short, which is an issue that effects websites that use font-linking. The FOUT is an unsightly effect that occurs because of the fact that linked fonts are loaded, like images, after all the other content on a webpage has been loaded. As a result, your back-up fonts (as specified by your CSS font stack) will flash for a few milliseconds before your typekit fonts load. Due to the spatial difference between various typefaces, this produces a distracting “visual hiccup” that can make a site look broken even to the most oblivious of non-designers. To put it bluntly, the FOUT sucks. It is for many who would otherwise commit to a Typekit service plan, an outright deal breaker.

Now unfortunately for the folks at Typekit (who are painfully aware of this problem and taking on much of the blame), the FOUT is an issue that affects everyone using linked fonts whether they employ a third-party solution or host fonts themselves. As far as I’ve been able to tell, it’s a browser based issue. Typekit performance appears to vary, to some degree anyway, between Firefox, Chrome, Safari, and IE.

Nonetheless, much to my disappointment, the only way I found to effectively manage the FOUT was to significantly reduce the file size of my “type kit” to around 75k, or just one font with two weights. So much for my dreams of versatility!

With that said, Typekit is working on improving this issue and they have made some experimental solutions available that seem quite interesting. However, considering the monthly cost of the service, I am really hoping these solutions move beyond the experimental stage in the very near future.

To keep tabs on Typekit’s response to the FOUT, check out these threads:

http://getsatisfaction.com/typekit/topics/avoiding_the_fout

http://getsatisfaction.com/typekit/topics/flash_of_unstyled_content

So, The Empire Strikes Back…you’re gonna proof those fonts right?

Another equally serious issue I’ve had with Typekit surrounds the display of certain custom fonts on Windows (mainly while running XP). Windows uses a system called Cleartype for anti-aliasing fonts which needs to be turned on in order for font smoothing to occur. In Windows 7 and Vista, Cleartype is turned on by default and Typekit fonts look more or less like they should (if we use OS X as a benchmark). But in Windows XP Cleartype is turned off by default which can mutate the appearance of certain fonts and make them look jagged and unreadable.

Again, this is an issue that the folks at Typekit are not directly responsible for as the problem is rooted elsewhere (in this case, both in Windows XP’s native font rendering settings and the actual “hinting” of the offensive fonts). Nonetheless, it is a real problem that can affect a design in truly serious ways. And though Typekit does offer a robust “type tester” feature that does a pretty good job at approximating what a chosen typeface will look like in each browser, I believe Typekit could do more to advise it users on which fonts are best suited for use in Windows and which are not. Perhaps by flagging troublesome or poorly hinted fonts?

For a discussion of this problem, check out these threads:

http://getsatisfaction.com/typekit/topics/typekit_fonts_rendering_horribly_on_windows_based_systems

http://getsatisfaction.com/typekit/topics/only_readable_if_cleartype_is_enabled

Conclusion

For the time being, we are still using Typekit here at Inventables, despite its shortcomings. We believe in the “case for real fonts on the web” and prefer the greater freedom and flexibility that using custom fonts provides. However, despite our hopes for Typekit, we believe it still has some issues to work though before it becomes more widely adopted.

—Written by Billy

Amazon RDS Redux: Tracking down a slow query

In our last post we talked about how Inventables migrated from a MySQL server running locally on an EC2 instance to the Amazon RDS service.

One benefit of using Amazon RDS is that you gain access to a large array of performance statistics provided by the Amazon CloudWatch service without having to do any extra work. We were eager to start viewing these statistics to see how the new platform was doing.

Checking out our stats

The CloudWatch command line tools are a breeze to download and install locally—you don’t have to run them on your EC2 instances. In just a few minutes we were able to view the minimum, maximum and average CPU utilization, available storage space, and number of database connections for any period over any time frame since the launch of our RDS database instance.

By running the following command, we immediately we noticed that our CPU utilization was maxing out in the mid-to-high 90’s every hour:

The numbers seemed too high. Something was amiss.

Tracking down the cause

We knew we had several options for investigating the problem, but since we were new to RDS we decided to use this as an opportunity to improve our familiarity with some of the core features of the system. We developed the following plan of attack:

1. Launch a clone instance of our production RDS database
2. Turn on slow query tracking for the clone database
3. Launch a new, standalone EC2 instance running the same code as our website, but pointed at our clone database
4. Exercise the standalone instance and then check the MySQL slow query logs to see if we had any bad queries

This is why we love AWS: with a few simple steps we can produce a brand new environment that looks and acts just like our production environment at all levels, with the same data. If a query is only taking a long time when there is a lot of data, we’ll find it. And we won’t be tampering with our real site in the process—it will happily hum away, utterly oblivious to our performance experiments.

Attack of the clones

Launching the clone database instance was effortless. As with our initial migration to RDS, we decided to use the ruby client:

Turning on slow query tracking took a little more effort. This required creating a new DB parameter group for our clone database instance. Our production instance uses the default parameter group, so we didn’t need to copy over any custom settings. We just set up our new group to flag any queries executing for more than 1 second on any table with more than 100 rows:

With slow query tracking turned on, we launched and configured our EC2 instance and manually walked through a few user workflows to exercise the most commonly used parts of our site. Then we checked the slow query log on our clone database instance:

select * from mysql.slow_log;

Bingo. The result set included one entry.

We fired up a mysql command line client and ran EXPLAIN on the bad query, which showed that we were doing a full table scan on a table with several hundred thousand rows.

Follow-up

After adding the appropriate index, we kept a close eye on our maximum hourly CPU utilization in our production environment. It immediately dropped to the low 80’s after the fix and has been holding steady at or below that level since.

Catching index problems in production is definitely not our first choice. We’re reviewing our approach for identifying needed indexes at development time and how we can improve on it. If you have any suggestions, let us know!

—Written by Jeff