Thoughts On ...

My first clojure macro

I'm finally experimenting with writing macros in clojure. Learning macros is (for me at least) a 4 stage process:

1. Learn to use them (pretty straightforward)
2. Learn to read their implementations (including the quoting)
3. Learning to write them (in progress)
4. Learning when to write them (in progress)

Those last two are iterative; #4 is especially tricky -- the web is full of general considerations ("when a function won't do", "when you want new syntax", "when you need to make decisions at compile time", etc) - but actually making that judgment in practice, takes... well practice.

Hence this exercise. Anyway to the code:

Clojure offers the if-let and when-let macros that allow you to combine a let block with testing the binding for nil:

(when-let [a (some-fn)]  
   (do-something-with a))

(if-let [a (some-fn)]  
   (do-something-with a)  
   (otherwise-fn)) 

I found myself (on some real code) wanting to be able to do something similar with try:

(try-let [a (some-potentially-exceptional-fn)]
  (do-something-with a))

(try-let [a (some-potentially-exceptional-fn)]
  (do-something-with a)
  ArithmeticException ((println (.getMessage e)) 42)
  :else (do-something-by-default-fn)
  :finally (println "always"))

etc.

So I wrote this (non-hygenic) macro that seems to do the job:

(defmacro try-let [let-expr good-expr & {fin-expr :finally else-expr :else :as handlers}]
  (letfn [(wrap [arg] (if (seq? arg) arg (list arg)))]
  `(try (let ~let-expr ~good-expr)
    ~@(map #(apply list 'catch (key %) 'e (wrap (val %))) (dissoc handlers :finally :else))
    ~(if else-expr `(catch Exception ~'e ~else-expr))
    (finally ~(if fin-expr `~fin-expr ())))))

Thing is... I don't if it's a good idea or not. For one thing its not hygienic (it implicitly declares e that can be used in the handler clauses) though this seems the kind of case that sort of thing is for.

For another... I don't know if its correct. It seems to be (I've tested all the scenarios I can think of), but this is kinda like security -- I suspect anyone can write a macro that they themselves can't break, but that doesn't mean its correct.

Some things to note: - e is available to handler expressions
- the local function wrap allows for a complex expression or single value to be spliced in
- any number of handlers can be included
- ':else' (default) handler and ':finally' handlers are optional (as are any others!)

In short: I'm interested in any opinions/feedback that aim at learning steps 3 & 4 (writing and when to write). Fire away!

Where Style Rules Come From

From a larger tutorial on Common Lisp Programming Style, comes a nice list written by Peter Norvig & Kent Pitman surveying "where your 'Style Rules' come from":

• Religion, Good vs. Evil "This way is better."
• Philosophy "This is consistent with other things."
• Robustness, Liability, Safety, Ethics "I'll put in redundant checks to avoid something horrible."
• Legality "Our lawyers say do it this way."
• Personality, Opinion "I like it this way."
• Compatibility "Another tool expects this way."
• Portability "Other compilers prefer this way."
• Cooperation, Convention "It has to be done some uniform way, so we agreed on this one."
• Habit, Tradition "We've always done it this way."
• Ability "My programmers aren't sophisticated enough."
• Memory "Knowing how I would do it means I don't have to remember how I did do it."
• Superstition "I'm scared to do it differently."
• Practicality "This makes other things easier."

( via Common Lisp Tips )

Stand-ups

I've been saying this for a while, now I'm just gonna write it down:

Stand-ups suck. Treat them as a last-resort.

The classic stand-up: each day a group of people stand huddled in a circle chanting: "His Name is Robert Paulson... His Name is Robert Paulson... His Name..." - no wait that's not it...

They stand in a circle bored out of their skulls while each in turn says what they did yesterday and what they are going to do today. A couple of people sorta care what everyone says (usually project managers, or customer reps) everyone else is interested in what maybe one or two people say. No one can go into enough detail to really educate anyone. Everyone's glad when its over.

Sometimes people attend many of these per day (cross-team stand-ups anyone?). A few might even make it a significant chunk of their day & job descriptions; I think of these people as "Meeting Moths" - attracted to meetings like moths to the flame.

STOP. DOING. THIS.

Instead:

1. Figure out who needs status and how often; GO AND GIVE IT TO THEM. It's a good bet they don't need to hear the minutiae of what code was written anyway & you're probably addressing that in existing stand-ups by making sure the programmers don't get too detailed killing any value they might've gotten from talking about what they are doing - dysfunctions beget dysfunctions...


2. Those who are doing actual tasks should be free to talk about them when they need to. In my experience this part of the stand-up gets short shrift when its scheduled and attended by those not doing the work. GET THEM OUT OF THERE. If you do, you might just see spontaneous, short discussions start happening daily among task completors (that's a real stand-up btw, just don't call it that or you'll ruin it by attracting meeting moths)


3. Move people next to each other who need to talk a lot. Conversely, if you already are around each other all day, you probably already know who did what and whose doing what (if not WORK ON COLLABORATING MORE)

If after you are doing all of these things, you still feel like you need a periodic daily meeting, then think for a while! Be creative! Don't just do what some book or consultant said to do, use your brain and solve the problem to everyone's benefit as best you can. Only then, if you *still* feel you need a stand-up, then fine; go ahead and schedule one for the things you still didn't address (but don't cover things already addressed!). The result is much more likely to be useful for everyone involved.

All periodic meetings including stand-ups evolve into wasteful time-sucks; They develop an inertia that supersedes whatever utility they initially might have had. Aggressively question their very existence; make the burden of proof be on keeping them, not ending them. Simpler still, kill em all and see what's not being done and then be creative on how to fix it.

Who knows you might just get away with one less meeting in your day.

Software Patents are Amoral

UPDATE: This article illustrates what I'm getting at very well: Google's position is weak while Apple's is strong - making patents a moral issue actually hurts Google's chances at reforming them. Change the rules, then change your strategy)

Martin Fowler just wrote a great piece on why Software patents are absurd and should be abolished. As Martin is so gifted in doing, he pulls in just enough background and history to educate you on the merits of patents in general, then shows clearly and simply why software patents are harmful to innovation and a pointless tax on established businesses.

In fact, I'd go so far as to say Martin's article (as is so often the case) is almost a perfect short introduction to the rationale for abolishing software patents... until he gets to the end of his argument and ruins is it by admonishing "my fellow programmers" for being "complicit in this tragedy." Martin rounds out this unfortunate paragraph with: "Any programmer who cooperates in getting a baseless patent should be ashamed of themselves."

Now I respect that this is clearly a moral issue for Martin and I'm sure that many (including those reading this) share his belief that it is unprincipled for a programmer to be against patents and yet cooperate in the application process. I also respect that Martin is a man of principle. However, I think he is wrong to hold this one - and ruined an otherwise great argument by introducing it into his reasoning.

And while I think he did so as an attempt to influence those faced with the situation, I think it's likely to have the opposite effect - pushing some to rationalize that patents are a good thing, so they can sleep at night when they choose not to quit their jobs to avoid the shame.

Fortunately, I didn't have this moral hangup when faced with a similar situation, and so I can still sleep at night (I'll just have to hope Martin still respects me after this). Because I don't think patents are inherently immoral - any more (or less) than engaging in running a business is inherently moral.

Now I recognize (and you should too) that what follows could be nothing more than mere rationalizing on my part, that Martin's dig hit a little too close to home and I am simply jumping on my (much smaller) soap box so I can continue to sleep at night. You can judge for yourself. My goal here is to share my experiences with software patents, and why I think they are amoral (while also a mistake that should be abolished forthwith).

Yes, I've signed patent applications. Yes, I'd consider doing it again. I believe (though I don't hold them) that some of them were even awarded. Here's how it went down:

At a critical time in my former employer's growth (For those that don't know my employment history: the business involved manufacturing of software-driven custom-built hardware devices) we were faced with a thorny - yet common - business situation: Namely, we were in competition with other companies to secure the same market.

Further, the nature of this business did not require patents to secure a "moat" around one's market share, but that was certainly a strategy that could be employed - though in my then and current opinion not nearly as effectively as how my employer was going about doing so which was by providing better service, securing better partnerships and scaling up production faster.

However, one of our competitors was attempting to secure a software patent for their machine. This patent (as described by one of my fellow programmers at the time as "patenting client/server") had all the failings that Martin describes (and then some), but our attorneys - to my skepticism - felt our competitor had a good chance of getting it approved with a few tweaks and could then use it against us, so our best strategy was to get our own patents to fend them off.

Without giving away details, a combination of patent application, patent purchase and architectural change was the suggested strategy. We executed that strategy in various ways and thus positioned ourselves appropriately if the (as I believed farfetched) doomsday scenario occurred and our competitor got their patent of client-server.

As I said, I thought the whole thing absurd. No one would allow a patent for client/server, the lawyers were overreacting. Still, I'd seen enough absurd patents to agree we were at risk, and so even though I thought it a colossal distraction for our small team, and overly paranoid, I worked to identify our own "original" inventions and how we could change our architecture to tap dance our way past their absurd patent if it were granted.

Now, despite my skepticism, they did get (a slightly narrower version) of that damn patent awarded, they did rattle their saber, and the changes the suits suggested worked extremely well to make the whole thing a non-event in my employers growth.

So, now you know my "shame"; how I sold my soul to the patent demons. Right? Keep in mind, I was instructed by the highest management of our company to do this and if I held the position that patents were inherently immoral the only principled decision was to resign. However, once I understood the situation, I accepted the charge because it was clear that it was best way to counter the absurd patent. By the way, I didn't hide my opinion - I even shared it with the CEO - and while he recognized the merits of my argument (and even that the whole patent thing should be abolished) he also clearly saw the need (as did I) to defend ourselves based on the reality that we were doing business in.

And this is the key point I'm trying to make: It wasn't an unprincipled decision for me because my employer wasn't doing anything unprincipled. They were simply playing the game correctly based on the rules in front of us all. To ignore this threat to the business would've been wrong; wrong for our success, wrong for our shareholders and wrong for our customers (because we had the better product).

This is not to say one cannot act immorally while playing by the rules - the business world is replete with such things - but the mere existence of patents is just a rule of the game we call business. If there's a moral element - it's in how they are used. It is my belief, that neither of the companies in my example behaved immorally (though our competitor was playing a weaker hand), and if the rules were changed, the outcome would've been the same (just cheaper for all concerned). But since the rules are there, they were used because that's what business is: A competitive game where those who use the rules most effectively win. That the stakes are high, and that many cheat, doesn't change that (though it obscures it for many) and it doesn't change that patents are part of the rule book and should no more be avoided than forgoing castling in chess because one thinks that moving two pieces should never be allowed.

Anyway, I hope for those who are faced with signing a patent application, I've done a little bit to ground your decision on the purpose of that patent, and not its mere existence. The latter is a naive position that fails to recognize the business landscape we all (well most of us) work in as professional software developers.

What we need to do is to change the rule book. What I would love to see is a more active attempt to get patents banned by the software industry - and I see no inherent hypocrisy if they continue to use them while working toward their extinction (not that I really believe they will, unfortunately).

In sum: Making patents inherently a moral question simply distracts and retrenches participants and plays right into the designs of those who have the advantage of the status quo. Martin's shaming of his fellow programmers was respectuflly offbase in my opinion as it ignores the much more important consideration of how patents are used while they exist.

Software is not an asset

Software is not an asset - I know everyone thinks it is, but its not; its a liability - its very existence generates cost. Every minute someone spends coping with it in order to adjust the behavior of hardware (which is an asset) is a net-credit on the balance sheet and a decrease in profitability. A necessary expense in many cases, but a cost nonetheless. Getting rid of it goes directly to the bottom line.

Software isn't a thing - its a form of communication. Just as no one would list meeting minutes, or recordings of the CEO's speech as company assets, software should be seen as stored communications (with hardware). Its value should be viewed in how it changes behavior; its very existence should be seen as creating risk, and its deletion as key to a healthy techno-ecosystem.

This is the essential argument behind why refactoring is worthwhile, and why its value isn't really comparable to writing code. Its also why its hard for non-programmers to see that value - because they think of programmer time as a credit, when really its can also be a debit.

My advice to managers (and everyone else): Want a sure fire way to add value to your organization? Reward and encourage deleting of unused code; fight for and insist that programmers be given time to refactor and otherwise eliminate unused code, and don't account for time spent doing so in the same way you do the creation of code.

My advice to programmers: Delete unused code at *every* opportunity and scale (function, class, file, project, organization); don't compare the effort to do so to writing code; work at all times to have your software feature-set reflect its actual, operational use.

Oil 101

As I watched the Deepwater Horizon oil spill unfold last summer, I found there were many fundamental questions about the oil industry that I didn't understand. So, I started reading. I read many websites. Of these, The Oil Drum was probably the most useful, but I was still struggling with some basic questions like: "How does a country benefit economically from letting an independent company take oil from its shores?" - and the sheer volume of contradictory opinion, punditry and proposed solutions, led me to conclude: "I need to dig deeper."

So, I bought a book, Oil 101 as it got excellent reviews, and seem to cover everything about oil from chemistry & history, to exploration, production & trading. I recently finished this book and while its not a quick read, it is very well-written, interestingly presented and comprehensive. Written from a neutral point of view in that it describes what is, not what ought to be, it gave me just what I wanted: a good grounding that I can now use for further education, and evaluation of where we're headed.

Unfortunately, the book has created a new problem for me: Nearly everything I now see being presented on this topic (again the Oil Drum is a notable exception) seems, simplistic and uniformed, aimed at supporting one political agenda or another and/or just plain out to lunch.

To take just one example: I stumbled across this article about how the Koch Brothers are manipulating the oil market.

Now, I'm no fan of the Koch brothers - particularly their support for the anti-union movements and the big sleight-of-hand trick that is the tea party - but this article presents 'facts' and jargon, not to educate, but to scare.

Take 'contango.' Investopedia defines it as: "When the futures price is above the expected future spot price. Consequently, the [futures] price will decline to the spot price before the delivery date." Got that? Simple, right? Riiight...

To make any sense of Contango, you need to understand the oil futures market (not to mention futures themselves), the spot market, global supply and demand (both now and at the time of the contract) how oil is delivered, etc, etc - all stuff covered in Oil 101, but curiously absent from this and every other article I've read that mentions the term - probably because its taken me months to build even a basic understanding of the topic, so an article can't even come close - even if the article writer understands it. Instead its summarized neatly as: "demand is expected to outstrip supply." Wow, so simple a child could understand it. Clearly those who use the term 'contango', just want to hide how simple it is in order to steal from the poor.

Oh, and the gist of the manipulation? The Koch's are artificially reducing supply by leasing four supertankers and storing oil in them (presumably because they are using Contango against us). Just how much supply have they deprived the world of by doing this? Let's do some math (all figures from Oil 101, which is about 2 years old now, but it'll get the point across):

The largest supertankers (unlikely that's what's being used here, but we'll go with it): are called ULCC's (Ultra Large Crude Carriers) and they can hold up to 560,000 DWT (that's Dead Weight Tonnage). Again, unlikely to be filled to the brim, but we'll use the number as is.

4 ULCC x 560,000 DWT ≈ 2.25 million DWT
7.5 barrels per DWT ≈ 16.8mm barrels

Wow, 16.8 million barrels of oil... that's how many days supply?

Not even one. Not even one for the US! In 2008, the *daily* consumption for the US was 20.9mm barrels per day (bpd). How much did the world consume in a day? 85.7mm bpd.

16.8 million barrels isn't even enough to feed all of 2008's US daily reactor capacity (17.5mm bpd). In other words, the heinous manipulation being described in the article - at worst - is shifting supply by a day.

In truth, its not even close to that, because ULCC's aren't likely to be used for that, not all of the DWT is actually oil, the oil being stored may not service as much demand, etc, etc.

So why is Koch industries spending money to lease tankers to store this oil? One guess is that its a hedge. They run refineries, and certainly buy oil to feed those refineries using futures contracts. By storing this oil, they can hedge those contracts with physical oil (much like one can hedge a stock futures position by holding actual stock). Another, possibility is that they are hedging against a short-term supply shortage for input into their refineries (e.g. maybe a couple weeks worth).

Not so evil sounding, once you put some numbers behind it. huh? Actually, it sounds to me just like the descriptions in the book of how global supply is stored and moved so we don't see occasional spikes of $15.00 gallon gasoline in the summer. In short: The refineries are doing their job.

Now as I said, the Koch brothers seem to have a lot of blood on their hands - but articles like the one linked, make that hard to spot when they simply stir up the rabble with BS. The sheer size of the oil-based economy is incomprehensible, the numbers are enormous. An oil tanker-full sounds like a lot, but it isn't even the proverbial drop in the bucket.

Reading this book has made it impossible for me to see the oil problem in simple terms. Nor are any of the "simple" solutions to it... simple. I'm not even sure 'problem' is even the right word anymore - it's like saying 'life' is a problem to be solved. True I suppose, but not very enlightening. No, oil is deeply entwined into the fabric of our lives, and as its details change, our lives - all of our lives - change with it. No-one - or everyone (take your pick) is to blame for that, it just is. Oil magnates and speculators make such great scapegoats - they're rich, remote - and very likely using their power for their own gain, but hanging them in the square won't fix things. Understanding the facts, so we can make our own decisions about how they are using their power (both right and wrong) will.

We all owe it to ourselves - and each other - to learn as much as we can about how our world works. I've joked that I'm one of only three people who've read this book that aren't in the oil industry (me, the book's editor, and proof reader), if you care about this topic - and want to make your own educated decisions about what's really going on - without resorting to simple solutions like boogeymen. You should join the three of us in reading this book.