Alex R. Young

A vim-friendly Mac Keyboard

2012-01-15T00:01:00Z

I suspect the ultimate vim keyboard is the Happy Hacking Professional 2. The Control key placement is good for vim, tmux, and screen, and the removal of uncessary arrow and function keys appeals to me. However, it's $300, so I wondered if it's possible to hack the keyboard I already own into being a little bit more vim-friendly.

Swapping Modifier Keys

The position of Control sucks. Fortunately, Apple gives us the ability to swap modifier keys around with no hacking at all.

Under System Preferences, Keyboard, Modifier Keys..., the Control and Caps Lock keys can be switched:

It only took me a few days to get used to this, and it works well for those of us who type Control a million times more often than Caps Lock.

Use hjkl Instead of Arrows

KeyRemap4Macbook allows any key to be remapped, and key repeats to be accelerated. Of interest to us is the 'Vi mode' setting list, and 'Ubiquitous Vim Bindings'. I selected 'Fn+hjkl to Left/Down/Up/Right' so I can use hjkl to move in any application.

KeyRemap4Macbook requires a reboot after installation, but once it's running the settings can be changed and experimented with at any time.

Using Powerline with Mac OS

2012-01-13T00:01:00Z

vim-powerline by Kim Silkebækken is a great plugin, but it's a little bit tricky to get it looking right in Mac OS. I've tested it in both Snow Leopard and Lion and figured out how to get it working nicely.

Snow Leopard's Terminal application doesn't support 256 colours, so I started using iTerm2 instead. With a little bit of configuration it's functionality equivalent to Terminal, and offers tonnes of extras. Lion's Terminal is fine. Both need to report xterm-256color, so ensure this is set up in the Preferences pane. To get the cool UTF-8 fonts you'll need patched fonts. Nick Quaranto posted some suitable fonts here: vim-powerline patched fonts. I like "Menlo" myself.

Next, read the Powerline documentation. When the author says set nocompatible and set laststatus=2 are required he means it, so add those to ~/.vimrc. I also had to add set t_Co=256 and let g:Powerline_symbols = 'unicode' (Kim notes that setting it to fancy will work as well, and make better use of the patched fonts).

Changing Powerline_symbols requires the cache to be cleared, so rm /tmp/Powerline.cache.

Now your vim should look like Kim's screenshots in the Powerline repository.

Build an IDE with tmux and vim

2011-12-19T00:12:00Z

A friend of mine had visited an office where the employees used tmux and vim to edit Ruby projects. He wanted to know why people would work with the console version of vim, considering the loss of the convenience of mouse input.

I actually find this a good way to work, for several reasons. Originally, using console vim forced me to learn vim's motion commands properly. Combined with touch typing, this opened up a range of powerful ways to jump around files and lines arguably more efficiently than by using a mouse.

I like to tile terminals alongside my editor. My web development work usually requires a console for issuing ad-hoc commands, a database console, and a log viewer. Some of my projects have test suites that run automatically when files change, so I like to be able to see the status of my tests as well.

There are vim plugins that provide integration with these things, but I feel more comfortable with the vim/tmux combination. I think it's a visualisation thing.

By using command-line tools this way, we're effectively building a customised, lightweight IDE. I also find the idea of using Unix commands in my spare tmux console panes appealing, because it's easy to pipe commands together thereby performing complex scriptable operations without the bloat of an IDE.

The difference between this approach and a traditional IDE is this interface adapts around my current task, and is only limited by the thousands of commands and scripting languages that I have installed. Rather than creating a template of some kind, I create tmux panes and vim split windows as required. Despite Eclipse and Xcode offering task-focused interfaces, I still feel like they get in the way too much. Particularly in Xcode, where keyboard shortcuts feel like an afterthought, and my hand is consistently on the mouse.

Task switching is another benefit. I'm a freelancer, so in a typical working day I may switch between three or four projects. By using tmux, I can detach from a session and come back later, keeping me focused on the current task at hand. I think this is really an advantage of console vim over GUI vim and a collection of terminals, because Eclipse and Xcode attempt to save interface state when closing a project (although the latest Xcode seems to close my split panes all the time).

Why People Use `hjkl`

It may seem needlessly awkward or elitist to some, but if you can touch type relatively correctly, vim and tmux (set to using vim's keys) make it easier to keep fingers on home row and off the mouse.

That's the secret of hjkl -- people who like it generally touch type correctly. For those unaccustomed to these keys, try taking it slowly for a few days. Type deliberately and methodically, making full use of the little fingers.

The hjkl argument reminds me of trying to convince novice PC gamers to learn wasd over the arrow keys. At first wasd feels less intuitive, but it makes using the keyboard and mouse together a lot easier. The advantage becomes apparent when actually trying the alternative in earnest.

Configuration Tips

I use this in my .tmux.conf:

set-window-option -g mode-keys vi
bind h select-pane -L
bind j select-pane -D
bind k select-pane -U
bind l select-pane -R

This allows me to move around tmux using vim's movement keys.

If you're struggling to get used to vim's movement keys, try disabling arrow keys for insert mode: Vim: Making those Arrow Keys Work With You.

If you're new to vim, it's also worth remembering that it supports ctags. The TagHighlight plugin can dynamically display a panel of tags in a pane for the current buffer. This is yet another IDE-like feature.

Dotfiles

Does your IDE's configuration sync between multiple computers?

I have a private git repository called dotfiles that has my vim and ctags configuration and plugins. I wrote a little install script to automatically symbolically link configuration files with the files in my local dotfiles repository. When I get a new computer, one of the first things I do is to check out this repository. When I start editing a project, vim and tmux behave exactly as expected.

Copy and Paste

I usually copy and paste console output into vim by using tmux's copy and paste commands. Depending on how tmux is configured, the exact keyboard shortcuts may vary, so it's worth reading man tmux about how this works. The default should be ctrl-b [ to enter copy mode, Space to start making a selection, Enter to copy the selection, then ctrl-b ] to paste.

Motion Commands

To really get efficient with vim, try to always question when a keyboard-heavy operation could have been better served by using the appropriate motion commands. For example, when I first saw ~ (swap case) I thought "that's cute, I'm never going to use that". I've actually used it three times while editing this post.

I started to get the hang of motion commands when I noticed more experienced vim users avoid going into insert mode unless necessary. Keep coming back to vim's help, and you'll be surprised how many ways there are to improve your editing skills.

iOS 5 Redemption: ARC, UIAppearance, Blocks

2011-12-07T00:12:00Z

iOS 5 redeems iOS development in my eyes through several new features. There are Objective-C changes, but also new APIs that make common tasks a lot easier. When Apple released the new SDK, I created a blank project to see what fundamental project organisation changes had been made, and the only major thing I noticed was the addition of Storyboards.

Unfortunately, Storyboards are not compatible with iOS 4. It may be possible to create a fork of a project with iOS 4 support that has a different UI, or to somehow conditionally use Storyboards. However, we all know how much hand-coded interface development is required for real iOS projects, so I can't see myself using Storyboards.

Automatic Reference Counting

Automatic Reference Counting implements automatic memory management for Objective-C objects and blocks, freeing the programmer from the need explicitly insert retains and releases. It does not provide a cycle collector; users must explicitly manage lifetime instead.

From: LLVM: Automatic Reference Counting

When friends tell me they're learning iOS development, the first thing I find myself helping them with is memory management. Learning where to use retain, release, and autorelease isn't too difficult, but it's sometimes hard for people to realise it's a convention rather than something the compiler will automatically help with. That has now all changed with Automatic Reference Counting (ARC).

ARC is a build setting, and once it's enabled properties get new keywords and the compiler will raise errors where the old memory management keywords are used unexpectedly. Now we can alloc and initialize objects without worrying about releasing them. If you work with pure Objective-C (no C data structures) then this is easy and cuts down a lot of code. It feels wrong at first, but after a while it's completely natural.

Be sure to study the notes in What's New In Mac OS X: Automatic Reference Counting because it's not quite as simple as throwing away all retain and release keywords. Everything you need to know about automatic reference counting is also a thorough post on the topic.

In terms of accessibility, ARC makes iOS a lot easier for new developers.

`UIAppearance`

I've always struggled with meeting client expectations in terms of application appearance. Apple's iOS apps tend to have intricately themed interfaces, and while the default widgets and colours are adequate, corporate branding soon rears its ugly head and forces me to write lots of custom UI code.

The first tool in my application skinning arsenal was stretchableImageWithLeftCapWidth:topCapHeight::

UIImage *buttonImage = [[UIImage imageNamed:@"button.png"] stretchableImageWithLeftCapWidth:5.0 topCapHeight:0.0];

This makes it possible to create buttons with gradients, shadows, and rounded corners -- anything you can put in a PNG. iDev Recipes: Stretchable Images and Buttons is a good post on this because it includes a diagram showing how images get stretched.

Skinning buttons isn't enough for most clients. I think every freelance client I've had has demanded a background image in a UIToolbar. Not just a tint, but an image. I typically subclasses UIToolbar and change drawRect to draw an image. This solution goes against the general wisdom that, if possible, subclassing API classes should be avoided in Objective-C. Incidentally, there's an interesting on Stack Overflow about this very topic: Is subclassing in Objective-C a bad practice?

We now have an adequate solution for application-wide skinning through the UIAppearance protocol. It's now possible to skin an entire app with very little code:

UIImage *navBarImage = [UIImage imageNamed:@"images/menubar.png"];
[[UINavigationBar appearance] setBackgroundImage:navBarImage
                                   forBarMetrics:UIBarMetricsDefault];

UIImage *barButton = [UIImage imageNamed:@"images/barbutton.png"];
[[UIBarButtonItem appearance] setBackgroundImage:barButton
                                        forState:UIControlStateNormal
                                      barMetrics:UIBarMetricsDefault];

UIImage *backButton = [UIImage imageNamed:@"images/back.png"];
[[UIBarButtonItem appearance] setBackButtonBackgroundImage:backButton
                                                  forState:UIControlStateNormal
                                                barMetrics:UIBarMetricsDefault];

UIImage *tabBarBackground = [UIImage imageNamed:@"images/tabbar.png"];
[[UITabBar appearance] setBackgroundImage:tabBarBackground];
[[UITabBar appearance] setSelectionIndicatorImage:[UIImage imageNamed:@"images/selection-tab.png"]];

Given suitable high quality 24bit PNGs with subtle shadows and textures, your app will now blow your client's mind.

Blocks

At first I thought blocks were pretty needless, because after years of working with asynchronous APIs in Objective-C I'm comfortable with delegates. After using blocks in a project I'm now a convert. I actually write a lot of JavaScript, so I'm a master manipulator of closures, and I feel like some of the same code organisation techniques transition to Objective-C.

What convinced me was actually something unexpected: Parse. Parse is a service aimed at mobile developers that provides a backend that can be used to sync user data. It can handle user registration, password resets, and arbitrary structured data and files. While iCloud is great, Parse allows us to sync our data across iOS and Android, and potentially more platforms in the future -- all without having to write a web app.

Parse's code samples have a button for displaying code that uses blocks or callbacks. This is an example of block-based code:

[PFUser logInWithUsernameInBackground:@"myname" password:@"mypass"
  block:^(PFUser *user, NSError *error) {
    if (user) {
        // do stuff after successful login.
    } else {
        // the username or password is invalid.
    }
}];

Once the reader understands the block syntax, this code is immediately understandable. Contrast this to the callback-based code:

// First set up a callback.
- (void)handleUserLogin:(PFUser *)user error:(NSError **)error {
    if(user) {
        // do stuff after successful login.
    } else {
        // the username or password is invalid. 
    }
}

// Then, elsewhere in your code...
[PFUser logInWithUsernameInBackground:@"myname" 
                             password:@"mypass" 
                           withTarget:self
                             selector:@selector(handleUserLogin:error:)];

As Objective-C developers we know the second part of this code is in another method, and that's the key. Blocks keep related code together. As long as your block isn't too large, this is easier to follow and therefore potentially easier to maintain.

Blocks are largely intuitive, but __block is a syntactical wrinkle that's worth learning properly. Apple's Blocks and Variables documentation covers this, but I also found Programming with C Blocks on Apple Devices useful as well.

All three of these of these features make iOS development more accessible to newcomers, so if you've been putting off iOS development, now would be a good time to jump aboard.

Changing APIs and Package Managers

2011-10-14T01:10:00Z

When I started programming there was a real ‘if it ain’t broke…’ mentality. At least, I think there was — this was in the 90s and I connected to the community through magazines, computer shops, and public domain disks (that’s disks with a ‘k’, as in floppy diskettes). I learned a lot of BASIC derivatives from magazines, and eventually picked up some books on C. Then I heard about K&R C at university and flipped through a copy at the library. I think one of my friends bought a copy, and I was envious because I couldn’t afford it at the time.

Then later on I started playing with Ruby, and there seemed to be a huge amount of experimentation and creativity within the community. I dimly recall trying to work my way through Japanese documentation and giving up, but I came back to it a few years later when a lot of English–speaking developers had jumped aboard. The creativity in the Ruby community seemed to solidify when _why became well known and cartoon foxes and meta–programming were everywhere.

Then I started working with Rails, and there was a new mentality of constantly refining and changing things, sometimes based purely on aesthetic preferences, but other times to genuinely improve APIs and libraries.

In contrast to this, I’ve dabbled with Objective–C over the years and got paid to write several iPhone apps. I started taking Objective–C seriously partly because the iPhone was such a phenomenon, but also because I like C–like languages. Apple’s APIs don’t change fundamentally, perhaps because they were based on solid principles established by engineers at NeXT in the 80s and 90s. I’ve found I’m extremely happy writing code with events in Objective–C, and data storage is trivial thanks to Core Data.

Meanwhile, I’ve enjoyed building Node apps because I like event–based code, and JavaScript’s first class functions suit me as a Lisp hobbyist who never gets paid to write Lisp. Another reason I enjoy Node is npm’s move away from system–wide library installation, which makes dependency management easy.

Node’s core developers and prominent library and framework authors seem to be acutely aware of how frustrating API changes are. In some ways JavaScript and Objective–C seem to inspire a smaller range of API designs than Ruby — Ruby libraries will come with anything from a DSL to a simple object that can be called with public methods. JavaScript library APIs are usually designed around simple objects or chaining, and Objective–C APIs use a range of design patterns, Apple’s APIs in particular are focused around patterns that suit GUI development.

It can be frustrating to work with Rails when API changes create unexpected problems. The Ruby community has always had a culture of testing, which fits a dynamic language. Tests are generally part of the development process, so code can be updated to reflect API changes with a manageable cost in terms of additional testing and time.

API changes can be mitigated by locking down every library to a specific version. That means mastering a language’s package manager is now a core skill, rather than something that can be picked up on the job. It’s equivalent to the build process in a compiled language. Don’t write your Gemfile or package.json without putting serious thought into how library versions should be locked down. Certain contexts and libraries can generally be on the bleeding edge, others should be locked to a specific version. Also learn the features of your package manager -- I wouldn't like to develop npm modules without knowing about npm link.

Google+, Privacy and Engineering Consent

2011-09-04T01:09:00Z

At dConstruct 2011, Don Norman gave a talk entitled Emotional Design for the World of Objects in which he said Google’s main product is us and the service they sell is advertising. The services Google make aren't just a way of getting eyes on adverts, but a way of gathering information about us to improve advertisement targeting. If this is true, where is the evidence? And what does it mean for social products like Google+?

It’s now possible to link Google accounts together, which can make the awkward necessity of using several Google accounts more manageable. I have Google Apps accounts for work and personal use, as well as an obligatory Gmail account for services the Apps accounts can’t yet access. I’ve linked these together to make it easier to switch between them. Does this mean I’m now represented as one person internally within Google?

Google’s Privacy Policy seems to indicate that this could be possible:

We may combine the information you submit under your account with information from other Google services or third parties in order to provide you with a better experience and to improve the quality of our services.

This doesn’t specifically mention linking data between accounts, but implies information may be shared between services. There’s actually a page called Ads Preferences that shows what Google’s advertising knows about you, based on an anonymous cookie. According to the associated documentation consent is required before collecting personally identifying information:

We will not collect, sell, or share personally identifying information from ad serving cookies without your explicit consent.

This is interesting, because “explicit consent” in our field is something more malleable than it sounds. What it amounts to is agreeing to terms and conditions, which people do on a daily basis without having to read or understand. It’s entirely possible that I’ve already explicitly agreed to linking my personal information with Google's anonymous advertising cookies without even realising it.

This brings to mind two things: The Engineering of Consent and subsequently Manufacturing Consent by Edward S. Herman and Noam Chomsky, and Eli Pariser’s filter bubble.

The Filter Bubble is an ongoing argument against personalisation through technologies like Google’s advertising profiling:

Increasingly we don’t all see the same internet. We see stories and facts that make it through a membrane of personalised algorithms that surround us […] The filter bubble is the personal, unique universe of information that results and that we increasingly live in online.

Digesting all of these ideas around online advertising and privacy ethics finally leads me to this: Your name and Google+ Profiles. Google+ demands real names, and accounts have been deleted for using names other than the name on your birth certificate, driving license, and passport. If Google wanted to be in a position to clearly build accurate profiles of people, account linking and real names would be the way to do it.

References

Designing for Password Managers

2011-08-15T01:08:00Z

1Password works so well because it isn't just a security product, but it changes the way we deal with with website logins. That is, we do things more securely and more easily when 1Password is around.- Looking ahead in security, AgileBits

I currently have 258 unique passwords. I'm not addicted to creating accounts on web services; I have multiple identities as a result of separating personal and professional concerns. Memorising 200+ unique passwords is never going to happen, so naturally I use a password manager.

Password managers are probably best kept offline but they basically all work the same way -- there's some form of local application that integrates with browsers through plugins or the operating system's built-in password management framework. That means password managers generally associate web form fields with credentials.

Using such a high number of accounts quickly reveals the vast differences in form design, and they don't all work well with password managers. For example, Google Apps email has a form specific to the email domain -- it uses "username" but elsewhere Google's forms use "username@domain.com". Should the password manager be designed to cope with this, or should Google simply use a consistent form design?

Some sign in forms contain fields that appear multiple times on the page. Using 1Password to log in could leak the sign in username to the registration form. I've always wondered if this is a potential security issue, but regardless, should pages be designed this way?

I've also found sites that include server-side generated random field names so it's extremely difficult to use my password manager at all. I end up copy and pasting the credentials manually. This has led me to research whether it's possible for browser plugins to silently access the contents of the paste buffer and extract plain-text passwords.

Not to mention banks. Is it more secure to ask me for parts of my password, or let me use my password manager with a high-entropy password that I can easily change regularly?

Password managers also detect when an account has been created or signed into for the first time. Is it practical to support this directly? I've often looked at new accounts in my password manager and found a mess of fields that aren't actually authentication credentials.

Checklist

The next time I design a log in form, I'm going to follow a checklist:

Ensure the field names are commonly used ones: username, email, password
Use these field names consistently
Keep the registration form on another page
Make sure the sign in form is consistent across all entry points
Test registration and sign in with popular password managers

As designers we need to cater for password managers. Hopefully something better will come along, but until then we're in the era of post-human password management (that doesn't mean passwords managed by super-advanced human entities, it just means nobody can be expected to do it properly anymore).

Perhaps conversely password managers should cater for us? AgileBits could publish a set of guidelines for designers that demonstrate both best practices for authentication forms and also what works well with their product.

Lessons From Two Lightweight Markup Parsers

2011-08-01T01:08:00Z

stextile

stextile (License: MIT, npm: stextile) is a Textile parser for JavaScript. The lack of a good Textile implementation held me back from developing Pop. While researching the project I found some amazing Textile implementations. Possibly my favourite is RedCloth, which I remember from back when _why rewrote it with Ragel.

As I was desperate for something that would parse my articles, I created a straightforward parser using regular expressions. I wrote tests as I went so it wasn’t too painful.

This project made me aware of a few interesting uses of replace in JavaScript that I hadn't really thought about before.

Early versions of stextile used callbacks with replace:

string.replace(/expression/, function(match) {
  return match; // Alter the match in some way then return it
});

This form is extremely useful when additional operations are required before replacing values.

I settled on this pattern for replacing links:

var re = /expression/
  , m
  , result;

while ((m = re.exec(text)) !== null) {
  // Process each link
}

This is ideal for conditional matching and replacement. Iterating over each result until every item has been matched makes it easier to deal with things like turning off matching inside pre–formatted tags.

Jadedown

I made Jadedown (License: MIT, npm: jadedown) purely for fun. I was desperately trying to learn Bison, Flex, and Jison to build a better Textile parser, but then I realised I never really liked the syntax anyway. I’m increasingly using Jade for open source and commercial work, so I thought it would be interesting to reuse the selector–based syntax for a lightweight markup language.

Markdown makes certain things very convenient for programmers: raw files read like simple text files, code tags can be inserted with backticks; and the syntax is consistent and easy to parse.

To understand what I mean by easy to parse, consider this Textile markup:

"Markdown":http://daringfireball.net/projects/markdown/ is clean and easy to learn.

"Jadedown":https://github.com/alexyoung/jadedown, although versatile, is currently purely experimental.

It's worth learning "Jade":http://jade-lang.com/.

A simple regular expression is all that’s required for the first example. Things start to get harder when a comma or full stop follows the link. Markdown makes the author’s intent much clearer by surrounding the link text and URL with brackets — a bonus is these brackets don’t usually occur in links.

Generic Tags

I believe lightweight markup languages should offer a convenient, but generic, tag representation. By using Jade’s syntax, I came up with this:

This is some a(href="https://github.com/alexyoung/jadedown")[Jadedown], it's easy to learn and parse!

Tags take the form tagName#id.class1.class2(attributes)[innerHTML]. In a sense it's how tags look from a JavaScript developer's perspective.

Shorthand

Of course, I use links and a few other bits of markup all the time, so I introduced a handful of useful shorthand operators. Links can be written in the form (http://example.com)[Example 1], asterisks and underscores can use used for strong and em, starting a line with an asterisk or hash will generate a list; and backticks are used for code.

Like Jade, starting a line with a recognised tag makes a block–level element. This part of the parser is currently the most underdeveloped. I’d like to add whitespace indentation for code like Markdown, but I haven’t considered the implications of combining that with Jade’s use of whitespace for handling nested tags.

Jison

Although this is intended as a toy language for now, it’s written with Jison. I got a big kick out of writing a Flex-style lexer, and a Bison-inspired parser — I’ve even included a test suite.

An issue I struck early on was I wanted to use capturing groups in the lexer’s regexes, but it seemed like accessing the matches in the grammar file was impossible. I didn’t want to use regular expressions twice — why match text in the lexer and then use a similar regular expression later to extract groups? However, after reading through several Jison projects I discovered this lexer pattern:

<p,b,ol,li>\(([^)]*)\)\[([^\]]*)\] %{
  yytext = {
      tag: 'a'
    , attr: 'href="' + yy.lexer.matches[1] + '"'
    , html: yy.lexer.matches[2]
  }
  return 'LINK';
%}

The regex here uses groups, then the code that gets run by the lexer splits the matches into a structured form that can be used by the parser. Unlocking this pattern drove development of Jadedown from a scrap on my harddrive to a fully–fledged parser.

The discussion that led to the development of Jadedown was from a gist I posted to Twitter about my ideas for a lightweight markup language. I like the idea of using CSS selectors across the entire stack of a project; working with Jade, Stylus, and jQuery makes it feel like everything revolves around them. Maybe Jadedown or something like it can extend this theme into content too?

Pop: A Static Site Builder

2011-07-24T01:07:00Z

Pop (GitHub: alexyoung / pop, License: MIT, npm: pop) is my attempt at building something largely compatible with Jekyll that satisfies my own requirements.

Highlights:

Generates sites quickly
Extensible through CommonJS modules
Can regenerate a single post as I’m working on it (rather than the entire site)
Generates stub posts with sensible defaults
Comes complete with helpers for HTML5 articles, feeds, pagination
Post summaries

Installation

Installing Pop is easy with npm:

npm install -g pop

This makes a binary available called pop. Running pop -h displays usage:

pop is a static site builder.

Usage: pop [command] [options]
new    path           Generates a new site at path/
post   "Post Title"   Writes a new post file
server                Create a server on port 4000 for _site/

-v, --version         Display version and exit
-h, --help            Shows this message

Tutorial

Let’s generate a site:

$ pop new myblog
Site created: myblog

The results should look familiar to Jekyll users:

$ cd myblog/
$ ls
_config.json _includes    _layouts     _lib         _posts       index.jade   stylesheets

If you open the templates you’ll see where Pop starts to diverge from Jekyll. The index page uses helpers to generate paginated posts and pagination controls:

---
layout: default
title: My Site
paginate: true
---

!{paginatedPosts()}
!{paginate}

I’ve retained YAML front–matter. The primary motivation for this was to make posts compatible. Templates are not compatible, but writing templates with Jade and Stylus is extremely simple for those of us versed in JavaScript and CSS selectors.

The resulting HTML can be generated by either running pop or pop server. The server mode will regenerate files when they change, but it can’t currently track dependencies — if you change a post it’ll regenerate the post but not the index page. To use the server you’ll need to have Express installed (I might make this a dependency if this confuses people).

The end result is a nice little HTML5 blog:

Extending Pop

The _lib/ directory may contain two files: helpers.js and filters.js -- both should be CommonJS modules. Helpers are functions available to templates. They're bound to a SiteBuilder object, so site configuration values can be accessed using this.config.

Filters are used to pre–process entire files before they’re put through Textile, Markdown, or Jade. I’ve used this to support the syntax highlighting liquid tags used by Jekyll. I’m planning to use prettyprint with DailyJS instead of Pygments.

Post Stubs

Stub posts can be quickly generated like this:

$ pop post "Awesome Post"
Post created: _posts/2011-07-24-awesome-post.md

It includes example YAML front–matter, in case you don’t like typing it or forget the format.

HTML5

I want people to be able to get writing with Pop in no time at all, so I’ve been researching how to generate a flexible site that can be easily customised.

To this end, a default Pop site generates posts according to the hNews format, and the templates themselves use HTML5. The default stylesheet makes heavy use of CSS3 features, so you get a lot of bang–for–buck without using any images.

The blog at popjs.com uses the default templates and styles.

Tests

Pop includes Expresso tests; run them with make test.

Plans

Pop is optimised for my requirements, which means it might not have the broad appeal that Jekyll has. I’m working hard to make it friendly and extensible, with ideas inspired by popular Node projects like Express.

The biggest thing on the horizon is documentation — I’ve included detailed code comments and the dox output can be found here: Pop API documentation.

Why “Pop”?

I wanted a short, easy to type name that wasn’t taken by a package in npm. It probably isn’t the best name for SEO, but I’m going to be typing pop a lot in the near future!

Bulletproof Deployment: Put Down the Pickaxe

2011-05-17T01:05:00Z

I have an axe to grind with web app deployment. I’ve never been totally happy with any solution, and almost everything I’ve tried has let me down repeatedly.

This is where a programmer typically sees a problem and creates a project to solve it. And this is where it all starts to go wrong.

The need for scripted deployment comes when a combination of factors makes simply uploading files unworkable. Modern applications depend on a complex environment of libraries, daemons, and database schema management. Scripting this process should make it repeatable.

The first tool I used regularly was Capistrano (back when it was SwitchTower). It did the job fairly well for Rails projects, but there were a few things that concerned me. It duplicated lots of functionality from Ruby’s Rake, and it relied on a Ruby SSH library rather than shelling out. Net::SSH is an implementation of the SSH protocol in pure Ruby, and it wasn't particularly portable -- I couldn't deploy from Windows (I had to sometimes work in Windows 5 years ago), and it seemed to have issues communicating with one of the Solaris servers I was deploying to.

Capistrano felt like thousands of lines of code that would be better served by Rake and SSH(1). Then I found Vlad the Deployer.

Vlad is a small project that builds on Rake and SSH(1). It has some recipes that makes deploying to various environments fairly straightforward — using it is a configuration problem more than scripting. Like Capistrano, it uses symbolic links to the project’s directory so rolling back a failed deploy should be simple. However, in reality these “release checkpoints” are better served with tags in the project’s version control system, and messing around with symbolic links feels awkward and can lead to unexpected problems.

The biggest problem with Vlad is a failed command doesn’t return a neat line of shell code; instead it spits out a huge block of several commands making debugging extremely difficult. It can result in situations where the state of a failed deployment isn’t clear.

Back to UNIX

After many, many bad experiences with Vlad, I gave up. However, I’m not just a lazy programmer, I’m also a lazy sysadmin, so I thought it was time to break out my vestigial UNIX skills and script the whole thing myself.

I’m starting to get tired of seeing Rake everywhere. What was wrong with Makefiles? And why are we using version control systems to check out the latest version of a project? Why can’t I run rsync on the code that I'm looking at on my computer?

One reason I get tired of everything being managed by Rake in Rails projects is booting the Rails environment. I don’t want to wait around for Rails to load when I’m doing something that isn’t related to the application’s environment. This is UNIX country!

And running migrations? Seriously, is ssh server cd /app/path && RAILS_ENV=production rake db:migrate really that hard? This stuff is better left to shell scripts!

So I’ve given up on deployment scripts and embraced my UNIX heritage. I sat down and wrote a shell script that uses rsync, ssh, and Bundler to deploy a client’s app. It took me about 30 minutes and worked first time.

Here’s the real script that I’m actually using for that project as a Gist: deploy.sh.

Bonus Features

This simple script actually gives me things I didn’t have before — rsync can be run in dry run mode, which allows me to see what will happen before I commit to deploying. I can tag each deploy in git if I want, then easily switch between versions. It's also easy to see where a deployment fails, because I can see the line in the script and look at the command.

Tools like Bundler make library management on the remote server much easier. I also write package.json files for use with npm when deploying JavaScript projects.

And a friend pointed out that using ssh master channels can help speed things up. The easiest way to do this is through .ssh/config:

Host *
  ControlMaster auto
  ControlPath /tmp/%r@%h:%p

Test–Driven Deployment

One of my static sites is deployed with rsync. That means only changed files are updated each time, so publishing is extremely simple and stress free. When I wrote the script I noticed there were a few things I could accidentally break, so I added a validation stage that could stop deployment:

namespace :remote do
  task :validate do
    puts 'Validating _site/'

    # These files may be lost in a completely new _site
    unless File.exists? '_site/.htaccess'
      puts "[WARN] Copying .htaccess file"
      File.copy '.htaccess', '_site/.htaccess'
    end

    # The jekyll I first started using got this wrong
    unless File.exists? '_site/atom.xml'
      if File.exists? 'atom.xml'
        File.copy 'atom.xml', '_site/atom.xml'
      elsif File.exists? 'atom.html'
        puts "[WARN] An atom.html file has been generated instead of .xml"
        File.mv 'atom.html', 'atom.xml'
        File.copy 'atom.xml', '_site/atom.xml'
      end

    end

    File.copy '_site/atom.xml', '_site/feed.xml'
    puts 'Done.'
  end

  task :deploy do
    puts "Deploying..."
    Rake::Task['tags:generate'].invoke
    Rake::Task['remote:validate'].invoke
    puts `rsync -avz "_site/" dailyjs.com:/var/www/dailyjs.com/`
  end
end

The validation here is just to patch behaviour that I didn’t like in my static site generator. It may have since fixed these issues. However, the concept of validating deployment first stuck with me.

Script Your Own Deployments

Rather than relying on programmers who don’t know any better, let’s just script our deployments. Read about the features of rsync, bash, and ssh to find seriously powerful features, then exploit them instead of hitting the problem with a pickaxe.

There are some simple updates I’d like to add to my bash deployment script:

Validation, perhaps checking things like file permissions before doing anything
Make it possible to run each function in isolation from an option
Configuration options per environment (for deploying to staging or production)

I’ll add these features when I need them!

Previously Ground Axes

I wrote a script to compare Rails dependencies between a local machine and server: depwhack. This is pretty pointless, since Bundler has largely solved gem management.

rsyncdiff uses the changes generated by rsync in dry run mode to show what files will change before a deploy. It's basically a parser for rsync's --itemize-changes output, which isn't human-readable (unless you're a wizard).

References

man ssh
man rsync
man make
Advanced Bash–Scripting Guide
deploy.sh

Alex R. Young

A vim-friendly Mac Keyboard

Swapping Modifier Keys

Use hjkl Instead of Arrows

Using Powerline with Mac OS

Build an IDE with tmux and vim

Why People Use hjkl

Configuration Tips

Dotfiles

Copy and Paste

Motion Commands

iOS 5 Redemption: ARC, UIAppearance, Blocks

Automatic Reference Counting

UIAppearance

Blocks

Changing APIs and Package Managers

Google+, Privacy and Engineering Consent

References

Designing for Password Managers

Checklist

Lessons From Two Lightweight Markup Parsers

stextile

Jadedown

Generic Tags

Shorthand

Jison

Pop: A Static Site Builder

Installation

Tutorial

Extending Pop

Post Stubs

HTML5

Tests

Plans

Why “Pop”?

Bulletproof Deployment: Put Down the Pickaxe

Back to UNIX

Bonus Features

Test–Driven Deployment

Script Your Own Deployments

Previously Ground Axes

References

Why People Use `hjkl`

`UIAppearance`