Fresh Mac Setup

The why

I'm an opinionated prick and I like things a certain way. When I get to a machine with a fresh install of OS X, I like to dive in and setup my base before doing any 'extra' stuff. Since I just got my hands on a new MacBook Pro 15" with Retina Display I figured I'd do the rundown and document it.

UPDATE [ 07/02/12 ]: I should have mentioned that this post is also an expansion of a previous post I did just over a year prior on running homebrew with pkgsrc. More is covered here but that's certainly part of it. Also I should probably share that if you're one of those Mac users who never uses Terminal because you're afraid of the command prompt, this post is probably not going to help you much.

User accounts

I create one admin user (first), then I create my regular user with no admin privileges. This is a safe guard against many possible no-nos, attacks, fat fingerings, etc.. 

Next I add my regular user account (user) to the /etc/sudoers file with a line like: 

    myuser ALL=(ALL) ALL

This ensures that I don't have to switch user accounts to get certain things done from the shell, where I spend about 90% of my time on a mac. 

App Store

While still in the admin account I log into the App Store and start downloading key pieces of software. For me this is XCode, FreeMemory, Better Snap Tool, & Caffeine. 

System Preferences

After making my normal user account (but while still logged in as admin), I go through System Preferences to do some verification and modification of settings.

Security & Privacy

• General - Require password for sleep and screen saver (5 seconds)
• General - Disable Automatic login (checked)
• Firewall - Turn on Firewall
• Privacy - Send diagnostic and usage data to Apple (unchecked)
• Privacy - Enable Location Services (checked)
• Privacy : Advanced - Automatically update safe downloads list (unchecked)

Sharing

• Remote Login (checked)  [ Note: This is for SSH only ]
• Remote Login - Allow access for: Only these users (selected)  [ Add non-admin user ]
• Everything else should be unchecked. 

Universal Access

• Seeing : Zoom : Options - Use scrool wheel with modifier keys to zoom (checked)  [ I use '^' which is the control key ]
• Seeing - Enable access for assistive devices

XBloat XCode

Click to install the main package. Once that is done go to the menu bar and navigate to XCode -> preferences -> Downloads and click to install Command Line Tools

With the above is't now time to login as my regular user and start customizing the environment

Login as User

The first thing I do is remove all of the application from the doc except Finder. Then open up Terminal and create a personal Applications directory in my users home directory: 

    $ mkdir ~/Applications/

then I link the System Applications directory under the one I created: 

    $ ln -s /Applications ~/Applications/SysApps

    $ open ~

I then drag the Applications directory to the doc for easy access. Now when I install applications for my own use (Chrome, Firefox, Adium, etc..) I can just drag them to my Applications directory without polluting the system Applications directory ( I wish there were a way to do this with apps from the App Store). 

List of essential apps

In order I grab the following and set them up: 

• Google Chrome - With:  LastPass, XMarks, Ghostery, ScriptNo
• Mozilla Firefox - With: LastPass, XMarks, Ghostery, NoScript
• Safari Extensions/Add ons: LastPass, XMarks , Ghostery, JavaScript Blocker
• iTerm2
• SparkleShare
• Wuala  Once they switched their model to not allow trading local storage they became useless to me
• DropBox - I currently use it but I'm slowly moving things away as I don't trust them
• CrossOver - Thankfully there are fewer and fewer apps written for Micro$oft platforms that I ever need to consider running. When one crops up, my first line of defense is CrossOver because it's a much more light-weight solution when compared with full virtualization system like VirtualBox, VMWare, Parallels, etc. Plus it's a great way to run Steam games that haven't been released for Mac (like Crayon Physics, which for some reason...holy crap it's now available for the Mac!!!)

 Shell environment

OS X defaults a users shell to BASH (Dear JKH, who the hell made that decision?). Bash annoys me to no end. Luckily OS X comes with KSH93 installed (albeit a VERY old version) as well as ZSH. So first up is installing my profile from http://fossil.secution.com/u/gcw/dotfiles/ . I download the .shrc file and copy it to .shrc in my home directoy, then there is a linking frenzy to make sure all the other shells get it right on startup: 

    $ ln -s ~/.shrc ~/.profile

    $ ln -s ~/.shrc ~/.zshrc

    $ ln -s ~/.shrc ~/.bash_profile

    $ ln -s ~/.shrc ~/.bashrc

    $ ln -s ~/.shrc ~/.kshrc

    $ ln -s ~/.shrc ~/.mkshrc #not installed yet, we'll fix that

Once that is done I can go about changing my shell: 

    $ sudo chsh -s /bin/zsh username

    Password:

    Changing shell for username.

Then I launch a new shell session. Note: my preferred shell is general KSH but because we're about to install pythonbrew and rvm both of which make jack-assy assumptions that you'll be running BASH or ZSH I've decided to use the latter as it's far less irritating than BASH.

PythonBrew and RVM

The instructions may change for RVM and/or Python brew so I'll say only that you should follow whatever those instructions are from their respective sites; PythonBrew Installation & RVM Installation .  These are useful utilities if you work with python and/or ruby much (which I do). It is very unfortunate though that their authors have chosen non-standard (bash) shell code as the implementation (but I digress).

Unix-y tools and packages

There are many options on OSX for installing third party command-line utilities. Being the opinionated prick that I am, I feel pretty strongly about what I use on my machine but you can make up your own damned mind.  That said, here are you major options in summary: 

• MacPorts - Is officially(?) sponsored by Apple, Inc. and (at least conceptually) is based in large part on the FreeBSD ports/package system.

• Fink - Is based on debian's build/packaging system. 

• Homebrew - Created out of a desire to have a better way to create, distribute, install and update applications.

• Pkgsrc - The NetBSD Packages collection. It was built with high portability as a goal from the beginning and has been adapted to run and install applications on Darwin/OS X for quite some time.

I have run them all and decided that no one meets my needs fully. Pkgsrc is rich but doesn't include Aqua apps as they are not of use outside of OS X.  MacPorts and Fink have many packages but still have considerable problems when installed and operated as a non-privileged user. Last I checked fink suffered from apt's limitation of being able to run only one instance at a time because of file locking. In the past there have been MacPorts upgrades that were out-right onerous and often broken requiring a complete do-over. MacPorts, Pkgsrc, and Fink all suffer from the limitation of effectively only supporting one installed version of a given application at a time. Homebrew was built specifically with this consideration in mind but has pretty strict limitations on what gets included for distribution and therefore has far fewer applications available. One could go on like this for days, let's just cut to the chase and say I have chosen to use a combination of pkgsrc and homebrew together.

Installing Homebrew

First we need to prep the area:

    $ sudo mkdir -p /usr/local/src 

    $ sudo chown -R gcw:staff /usr/local/

    $ sudo chmod 775 /usr/local/*

Unfortunately the prescribed method for installing Homebrew no-longer works because they do a brain-dead check to see whether your user is an 'Administrator'. Which is ironic because they stress in the documentation that you should not need any special privileges to install software. In any course here's what to do :

    $ cd /usr/local

    $ mkdir homebrew && curl -L https://github.com/mxcl/homebrew/tarball/master | tar xz --strip 1 -C homebrew

    $ mkdir -p /usr/local/bin/ && chmod 755 /usr/local/bin && ln -s /usr/local/homebrew/bin/brew /usr/local/bin/brew

Now you need to make sure that /usr/local/bin/ is in your path (and possibly /usr/local/sbin/). With my .shrc you can just type:

    $ my_reload

or alternately you could do:

    $ my_pathadd PATH /usr/local/bin/

Note: my_pathadd() was written specifically to avoid adding directories to a path variable if that directory either doesn't exist or is already in said path variable.

Once that is done you'll want to initial your brew base with:

    $ /usr/local/bin/brew update

Then test it with a simple application like most:

    $ brew install most

    $ which most

    $ ls -al /usr/local/bin/most

You're done with the Homebrew piece.

Installing Pkgsrc

There is a git mirror of the pkgsrc repositories which I prefer to use over the CVS system documented at www.pkgsrc.org. Here's how to use it

    $ git clone https://github.com/jsonn/pkgsrc.git /usr/local/pkgsrc 

    $ cd /usr/local/pkgsrc/bootstrap

    $ ./bootstrap --pkgdbdir /usr/local/var/db/pkg --unprivileged --prefix /usr/local/pkg

That will all take a while to run. Once it's complete you will have a directory where things get installed /usr/local/pkg/ and one where you go to do the building and installation /usr/local/pkgsrc/.  You'll want to add /usr/local/pkg/bin and possibly /usr/local/pkg/sbin to your PATH. Let's do a test with most:

    $ cd /usr/local/pkgsrc/misc/most

    $ /usr/local/pkg/bin/bmake install

    $ which -a most

    $ ls -al /usr/local/pkg/bin/most

You're done. Now you can install your apps with either pkgsrc or homebrew as you see fit.

Side notes: Brew is pretty picky about the versions of some utilities it uses. Make sure that the /usr/local/ stuff precedes the /usr/local/pkg* stuff in your PATH variables.  Learn to use brew doctor. 

UPDATE [ 07/02/12 ]: A buddy of a buddy of a friend of mine mentioned that "installing software is superuser duty (in the UNIX world), nothing wrong with requiring root to install stuff". Generally I would not need to disagree. Mac's, however are different. For desktop operating systems, especially, it's  important to avoid mucking thing up. As root, it's simply easier to accidentally do said mucking, whether by fat-fingering a command, doing an errant copy/paste, or running a program that has been compromised. My belief is that you should do as much as you can as a non-privileged user, but there are those who disagree with me.

Essential tools/utilities

The list of stuff I install immediately after getting brew and pkgsrc installed includes:

• fossil - (brew) My version control system of choice (yes even over git and mercurial)
• tmux - (brew) If you use split or GNU screen, you should really upgrade to a terminal multiplexer that not only doesn't suck echidna balls but also rock.
• lua - (brew) If you don't know.... you really need to find out.
• mutt - (brew) The only mail user agent that doesn't make me want to choke people out. [ with the sidbar and pgp-verbose patches ]
• macvim - (brew) already comes with python and ruby extension support I add the --with-lua option
• nvi - (pkgsrc editors/nvi) I love VIM for things like coding but when I want a real vi I use nvi. It's faster and much more light weight.
• gpg2 - (brew) I hope I don't have to explain this one...
• nawk, gawk, mawk, & runawk - (pkg - /lang/) I do quite a bit of scripting in awk and I need to make sure it's portable.
• v8 & Node - (brew) If you're stuck building non-browser apps in javascript these are a must
• msmtp - (brew) sendmail interface to authenticated SMTP for command line programs looking to send mail
• mksh - (pkg - shells/mksh) MirOS Korn Shell is a PDKSH derivative that's much faster and smaller than BASH
• ast-ksh - (pkg - shells/ast-ksh) The Original (still under active Open Source development) Korn Shell. KSH93 has been my shell of choice since about 1997. 
• w3m - (brew) Text-mode browser. It's a must
• openssh - (pkg - security/openssh) The HPN patch makes this one invaluable

Once this is done there's a bunch of configuration, symbolic linking, cursing, and pining for a pint but I'm essentially done with the big stuff once I've hit this point. Good luck.. I hope some of this is helpful to other people.

UPDATE [ 07/02/12 ]: I forgot to mention that much of the linking is of standard config files that I share across machines via SparkleShare. My vimrc, tmux-conf, muttrc are all kept current with SparkleShare running from an encrypted volume on one of my servers. Also after all of that is setup I go into System Preferences and turn on File Vault's full disk encryption.  

My Tinc setup after upgrading from FreeBSD 8.2 to 9.0

The problem that tinc solves

You've got multiple (virtual?) machines on different networks and you want to treat them as if they are all part of the same (private) network. The front-end doesn't matter so much as the backend communication and you want to keep that secure and seamless.

On the light-weight/quick-and-dirty side you can try to establish a point to point connection between all of the hosts with something like PPP, SLIP, or SLIRP but you'd have no encryption. You could try tunneling PPP, SLIP, or SLIRP over SSH but trust me that's a whole down which you do not want to follow the rabbit, even if it does have a shiny pocket watch.

You could go the centralized route and use something like OpenVPN or FreeS/WAN and make every server connect to the centralized server hoping that it doesn't go down. You could even make two or more central servers for redundancy but then you'd have a few heavily laden nodes passing traffic for everyone else. Not to mention that network paths likely would be less than ideal in many cases.

The best solution I've found is to this problem is a secure-mesh. With mesh networking you add and remove nodes as needed. Any node can be connected to as many other nodes as it desires on a peer-to-peer basis. Traffic between nodes not directly connected is handled by either routing through other nodes or having the whole (relevant part of the) mesh act as a bridged network.

Tinc does this for you. It has a very low resource profile, is scalable, and is highly configurable with sensible defaults so you don't have to spend all day reading up on every option. Another key feature is resilience. Once two nodes have accepted each other as peers you let them decide which one will initiate or re-establish the connection to the other (this is great when one is behind a NAT device). One connection between peers is all it takes, you don't have to have one from a to b and then another from b to a.

More information on tinc at http://www.tinc-vpn.org

A bit about my setup

I have multiple machines out there on the intertubes. Some are located in Colorado some in Florida and a few in California and Georgia. I won't detail the setup for all of them. Two should be sufficient to convey the basic idea.

rc.conf for host 1:

hostname="host1.example.com"
ifconfig_bge0="inet 1.2.3.4 netmask 255.255.0"
defaultrouter="1.2.3.1"


###################################
####Loopback Clones and Aliases####
###################################
cloned_interfaces="lo1 lo2 lo3 tap0 tun0"
ifconfig_lo1="inet 192.168.41.1 netmask 255.255.255.0"
ifconfig_lo2="inet 192.168.42.1 netmask 255.255.255.0"

# mesh services 
ifconfig_lo1_alias0="inet 192.168.41.2 netmask 255.255.255.255"

ifconfig_lo1_alias2="inet 192.168.41.3 netmask 255.255.255.255"

rc.conf for host 2:

hostname="host2.example.com"

ifconfig_bge0="inet 2.3.4.5 netmask 255.255.0"

defaultrouter="2.3.4.1"

###################################

####Loopback Clones and Aliases####

###################################

cloned_interfaces="lo1 lo2 lo3 tap0 tun0"

ifconfig_lo1="inet 192.168.51.1 netmask 255.255.255.0"

ifconfig_lo2="inet 192.168.52.1 netmask 255.255.255.0"

# mesh services 

ifconfig_lo1_alias0="inet 192.168.51.2 netmask 255.255.255.255"

ifconfig_lo1_alias2="inet 192.168.51.3 netmask 255.255.255.255"

Note that 1.2.3.4 (the external ip for host1) is on a different network than host 2 which has 2.3.4.5. Also not that I have loopback interfaces with privates IPs. I picked 192.168.[54][0-9] for this example. This is noted here because it's really these interfaces that need to talk to each other in my setup. I have Jails that run on those interfaces and some jails on host1 need to talk to jails on host2 and vice versa.

For the sake of this example let's pretend that we have DNS running on both 192.168.51.2 and 192.168.41.2. We'll say that host1's DNS publishes info for *.host1.example.com and that host2 does likewise for *.host2.example.com. If they are going to resolve entries on the other host they need to be able to connect to that hosts name server and we'll do that over the tinc-vpn.

Tinc-ering

There are many considerations in any network setup. VPN and mesh do not negate these, they compound them. I offer here an example of one possible configuration based on my experience with my own setup. You will have different considerations. I strongly recommend you read through the tinc documentation. It's exceedingly simple and not terribly verbose.

tinc.conf

After installing tinc you'll need to create a directory based on your mesh-network under /usr/local/etc/tinc. You'll also need to keep several files in sync between the hosts. For this example we'll use /usr/local/etc/tinc/example. The main configuration file will be /usr/local/etc/tinc/example/tinc.conf

Host1's tinc.conf:

Name = host1
Device = /dev/tap0
AddressFamily = ipv4
BindToAddress = 1.2.3.4
Forwarding = kernel
ConnectTo = host2
PrivateKeyFile = /usr/local/etc/tinc/example/rsa_key.priv

Host 2's tinc.conf:

Name = host2
Device = /dev/tap0
AddressFamily = ipv4
BindToAddress = 2.3.4.5
Forwarding = kernel
ConnectTo = host1
PrivateKeyFile = /usr/local/etc/tinc/example/rsa_key.priv

Note: here I'm using tap0 because it offers the most flexibility with regard to how you setup your mesh-network. With tap devices you can operate in bridge mode which is basically like being plugged into the same ethernet segment. You then (based on your ip layout and net masks) have the option to choose either routed or bridged operation for the nodes and virtual hosts on your mesh. Alternately you can use tun0 if you don't care to have a bridged network. With tun devices there is less traffic (no ethernet headers).

Note: The 'Forwarding = kernel' line tells tinc to pass packets through the kernel for delivery which means you'll need to have you firewall properly configured if you want to see traffic go from one tinc peer to the other successfully. I will not go over firewall configuration here. I suggest you start with a very open ruleset to verify that the link is working and then do whatever you need to do from there.

Host configuration

In addition to the main config (tinc.conf) there is a host specific configuration file that needs to be on both hosts and located in the hosts/ directory under /usr/local/etc/tinc/example/. 

contents of /usr/local/etc/tinc/example/hosts/host1:

Address = 1.2.3.4

Subnet = 192.168.41.0/24

Subnet = 192.168.42.0/24

Port = 655

-----BEGIN RSA PUBLIC KEY-----

blahblahblahblahblahblahblahblahblah

-----END RSA PUBLIC KEY-----

contents of /usr/local/etc/tinc/example/hosts/host2:

Address = 2.3.4.5

Subnet = 192.168.51.0/24

Subnet = 192.168.52.0/24

Port = 655

-----BEGIN RSA PUBLIC KEY-----

yadayadayadayadayadayadayadayadayada

-----END RSA PUBLIC KEY-----

On host1 the host1 file is used for options that are specific to host1 but are shared with other hosts. As well, the host2 file is used on host1 to make host1 aware of it's specific configuration (e.g. the subnets available through the connection to it: .51.0 and .52.0). Also the public key's are used for authentication in combination with the PrivateKeyFile specified in tinc.conf. The tinc docs cover generating keys.

up/down files

Tinc has a series of scripts that can be executed when a link is established or before shutting a connection down. The main one to worry about is the tinc-up. This file is executed once the two tinc daemons have established a peer relationship. This is usually useful for setting up routes, stopping or starting service, or possibly modifying firewall configurations. 

Host1's /usr/local/etc/tinc/example/tinc-up:

#!/bin/sh

ifconfig $INTERFACE 192.168.40.1 netmask 255.255.255.0

route add -host 192.168.50.1 192.168.40.1

route add -net 192.168.51.0/24 192.168.50.1

route add -net 192.168.52.0/24 192.168.50.1

Host2's /usr/local/etc/tinc/example/tinc-up:

#!/bin/sh

ifconfig $INTERFACE 192.168.50.1 netmask 255.255.255.0

route add -host 192.168.40.1 192.168.50.1

route add -net 192.168.41.0/24 192.168.40.1

route add -net 192.168.42.0/24 192.168.40.1

What these scripts do is assign an ip address to the tap0 interface (or tun0 if that's what you chose to specify in the tinc.conf). The IP used on the $INTERFACE needs to be unique to that node of the mesh and not on any other interface. Next we establish a route to the other end of the vpn-link with the route add -host line(s). The makes sure that your localhost can route to the machine on the other side. The final two lines establish routes to the networks on the two loop back interfaces where the DNS servers live. 

Note: When I was on FreeBSD 8.2 I didn't haven't to explicitly specify the other end of the link. It took me a while to figure out after the upgrade to 9.0 that I need to add a host route. 

Note: With the last to lines, by setting the route for accessing the loopback hosted services to the 'far end' of the link I'm doing direct routing which is ok for a simple setup. When you get more hosts added it's probably wise to set the route to the 'near end' and let the mesh route in a 'most available' manner. 

Troubleshooting

If you're having problems, you should fire up tinc in debug mode on both sides of the link:


/usr/local/sbin/tincd -n example -d5 -D

Try using ping. You should see messages about packets being passed on both nodes at roughly the same time. If not check your firewall rules. First check to make sure that you're not blocking traffic from going out over your $INTERFACE and then make sure that you're allowing traffic to come in.

Bit of irony: Database Driven (Web) Applications

I've been doing web development since 1996 but really started working seriously on it in 1999 when I met Zope and Python. That's when I got serious about it. I did use other technologies over the years, Perl, Ruby, Eiffel, AWK (yes really),  Java (oh the horror), and C/Objective-C (long before I ever owned an apple product), Django, Rails, JRun, and even PHP (*blech*) among others. Most of the applications I built between 2000 and 2007 were in Python usually via Zope and very often with Plone.

One thing that set zope apart from most other web frameworks of its time is the persistance layer. While most other systems were using Relational Databases, Zope used an object database called... wait for it...  Zope Object Database (ZODB).  It took me a while to really understand how to make proper use of zodb as I wasn't, at the time, really crazy about Object Oriented Programing (and almost 12 years later, I'm still not crazy about OOP. I use it. I understand t. I just don't get all the hype).

From 1997 through about 2007 when I built apps with systems other than Zope, persistance was often handled by the likes of mSQL, MySQL, BDB, GDBM, DJB's CDB, or SQLite. At the time I was just gobbling up any technologies I could find, in part because I wanted to be able to do things outside of zope the way I could do them within the confines of zope. The hope was that one of these systems could be leveraged to that end.

In 2007 I started mucking around with Django, Grok, Repoze, and Ruby on Rails. In 2008 I discovered web2py which just seemed right to me and about 90% of my web work has been in web2py ever since, much of it deployed to Google's App Engine. The rest was mostly Lua (Kepler, Sputnik, and LuCI) which I'd also discovered in 2008 with a bit of Rails and Django work thrown here and there.

Odd and unfortunate....

From about 2004 through 2008 I was interviewing for various positions and whenever called in for web development roles the conversations would invariable go something like this:

Inverviewer:		We've reviewed the code you wrote for zyx (some app that I'd built and that was running live on my servers somewhere). We only see the application code where's the database code?
me:		It's right here...
Interviewer:		That looks like Python [sometimes Perl or C]. Where is the SQL?
me:		Well if you read the code you'll see that there is a call here to this function here which is a wrapper to retrieve... and here is a wrapper function to store... this here does...
Interviewer:		But Databases use SQL for doing things.. where is the SQL? Even if you use wrapper functions those functions have to talk to the database. Where is the SQL that those functions use?
me:		This particular app doesn't talk to a relational database so there's no SQL.
Inverviewer:		What do you mean it doesn't use a relational database.. either it's connected to a database or it's not. If it's connected to a database it has to use SQL....

At this point the conversation would generally devolve into a set of emphatic statements about the way applications had to work and the end result would be that I didn't get the job. In a few cases I'd be told that they wanted someone with demonstrated knowledge of SQL, and relational databases to which I'd respond by sending them some of my code that makes extensive use of SQL and relational databases. In most cases I doubt they ever really looked at it because instead of commenting on the quality of the work I'd presented to them I'd be questioned as to why I didn't use relational databases for some of my personal projects. I'd respond that they weren't really needed for those projects and that I didn't want the overhead of Oracle or MySQL. Usually I'd get "well if you really understood databases you'd realize that the overhead of using one is far lower than maintaining your own home-grown database-substitute. You've learned enough SQL to cobble together some libraries but you obviously don't understand databases otherwise you would have used them for project xyz".

A Revelation

It took me a while to really get what was going on. Despite many conversations like the above it didn't really occur to me that I was doing anything wrong. In my mind, I knew about databases. I wasn't an expert but I knew what the hell a database was and I'd used several of different types. They were 'wrong' because they thought database meant a client-server relational database management system like Oracle. To me database meant a system for storing and retrieving data; a system designed to facilitate the storage and retrieval of data; a repository of data; the set of data being stored and/or retrieved. It dawned on me that in some cases my answers to some questions probably sounded petulant which might be ok when you're right and they know it, but it's never ok you're wrong (to them).

Take the classic 'glass half-full v half-empty' question. Naturally (to me) unless in a perfect vacuum the glass is always 100% full. Most people assume that the question is about the liquid, sand, or other 'non-air stuff' that's in the glass. I've come to understand that few people avoid making assumptions like that and that when they encounter someone who does (avoid making assumptions) they often assume the person to be intentionally provocative.  Further if you were to assert that the glass is completely full to someone who has never been taught about matter and gases they would probably look at you like you're completely bonkers.  In any case you (in this case: I) end up looking like a jackass if you're answer is one of the correct wrong answers.

The people I'd been talking to had never seen anything referred to as a database that wasn't a relational database. They had developed an assumption that if it's called a database it has to be 1) relational 2) use SQL and 3) likely uses the client server networked model. For the record all of that is rubbish but it was very common rubbish to encounter from 1995 up through say 2010.  This can be seen all over the interwebs look at this article from 2006 and this question from 2008 on stackoverflow.com.

NoSQL (rdb) vs NoSQL (terribly named 'movement')

I mentioned that I built some web applications with AWK and Perl. One relational database system I used for quite a few quick and dirty hacks was rdb and later NoSQL which was built out of the Shell as a 4GL doctrine. It was great for a unix nerd like myself who wanted a simple and easy to understand system that I could fix on my own. Since web2py made it so easy to develop applications to be run on Google App Engine, I started learning and using Google BigTable ~ 2008 or 2009.

In 2010 (as I recall it) NoSQL fever hit. Not the NoSQL I was familiar with for over a decade but something new. This NoSQL was (apparently) sparked by the white paper released by Google about their BigTable implementation and design. Systems like Hbase, HyperTable, and Cassandra implement very similar systems and are a few among the NoSQL 'movement'.

The original NoSQL was a relational database system built specifically for POSIX/Unix systems using the native tools/facilities of the Operating System. The new poser NoSQL system has been rebranded at least three times as new(?) projects get adopted into the fold.  Depending on who you asked, in 2010 the constellation of systems in the NoSQL universe included Riak, Redis, HTable and maybe CouchDB and some others but specifically did not include BigTable and Dynamo (I guess because they were not Open Source run-on-your-own-system solutions) nor something like ZODB or Berkley DB. Now the latter two are considered 'grand daddy' NoSQL solutions and are part of the flock as it's been realized that much of what people are calling NoSQL is really just a new implementation of existing concepts.

The irony of it all...

Now I've noticed that hiring managers looking for (web) developers will ask about their database experience expecting to hear about a system that's non-relational. If a candidate only mentions systems like Oracle and MySQL it's often assumed that didn't get the memo that 'database' no includes NoSQL options. 

still more of the same

Even so when I'm asked about my experience with Object Databases, NoSQL, or non-relational databases and I mention that I've been using them since the 90's I'm usually told that they didn't exist back then. I've learned though to understand that many assume all of this stuff just popped up between 2008 and 2011. I'm getting there.. one day I'll have it mastered...maybe.

So long Libre.fm.. I will miss you..

Don't panic.. Libre.fm is still around live and well (at least as of today, 05/08/12, they were). I'm saying so long because I won't be using the service any longer. There should be no mistaking the fact that I was at best, an infrequent user of Libre.fm and my lack of use/participation will in no way have any significant effect on their numbers or service. I am taking time though, to express why it is that I've decided to not use them anymore and will likely make the same decision with other service providers on the interwebs. Let me be clear here: I'm not picking on Libre.fm to be malicious. I believe greatly in what they do generally. I am not, here, seeking to get anyone to join me in a boycott, repudiation, demonstration or any other such nonsense. They are just one of many services who has done something that I'm sick of and it happens to be that they are the last straw (for me).

What's the problem...

Idiots! But I digress. Today I got an email message letting me know that libre.fm had decided to protect it's users by being proactive in responding to the potential threat posed by the recent security breach at LinkedIn (6.4 million hashed passwords of linkedin.com users were publicly posted). [UPDATE 05/09/12] Thanks to Mike for pointing out that this was a response to breaches at both LinkedIn and Last.fm, the latter of which is a fairly similar service to Libre and therefore the likelihood that passwords might be shared between the sites is increased. [/UPDATE] Libre.fm decided that since it's very likely that a significant number of their users are also users at LinkedIn.com it's reasonable/acceptable to reset passwords for all of the users at libre.fm. This is because some (hell many) users use the same password on multiple sites. If user theidiot@dumasuser.com is on linkedin and also uses libre there is a chance that the same password was used for both accounts and once the hackers get around to cracking that particular entry in the database they can use password they found on every other known service just to see if it works. 

This is great for those among the populous unwise enough to use the same password on multiple sites but for those of us who know better, it's annoying and time consuming. Many users take the time to develop good strong password practices and don't reuse the same password on multiple sites. Libre.fm is sending a message to both groups of users: 1) If you're silly enough to use the same password for both linkedin and libre we'll take care of you so you don't need to worry about being a better netizen & 2) If you're a user who's taken the time to understand and be aware of the implication of lax security/password practices it doesn't matter here because we treat everyone like they just don't know any better. 

They are not alone...

I've dealt with this sort of thing before. I was notified by one of my financial institutions that I needed to reset my password because my email address showed up in some list of accounts from some compromised site (no it wasn't a phishing scam). They had the same thought as Libre.fm: let's protect our users from their own lax practices...which is bullshit. 

Interestingly in the case of the financial institution I let them know (via email) that there was no way to have the same password on both sites as their policies were completely incompatible. This did not sway them obviously. 

What I'm doing about it...

If a company decides that they want to protect me from the idiotic practices of others, I'm simply going to avoid using that service. 

Why it's a problem...

The approach taken by Libre and other sites in response to compromises on other internet services does not scale well. Since the linkedin compromise at least 15 other sites on the interwebs have admitted to being breached in some way. If every major service required users to reset their passwords every time one of the other major services online got compromised we'd spend more time managing passwords than getting shit done. 

I love you libre.. and I will miss you...