Taco Bell Programming

Taco Bell Programming

by Ted Dziuba Thursday October 21 2010

Every item on the menu at Taco Bell is just a different configuration of roughly eight ingredients. With this simple periodic table of meat and produce, the company pulled down $1.9 billion last year.

The more I write code and design systems, the more I understand that many times, you can achieve the desired functionality simply with clever reconfigurations of the basic Unix tool set. After all, functionality is an asset, but code is a liability. This is the opposite of a trend of nonsense called DevOps, where system administrators start writing unit tests and other things to help the developers warm up to them - Taco Bell Programming is about developers knowing enough about Ops (and Unix in general) so that they don't overthink things, and arrive at simple, scalable solutions.

Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ.

The Taco Bell answer? xargs and wget. In the rare case that you saturate the network connection, add some split and rsync. A "distributed crawler" is really only like 10 lines of shell script.

Moving on, once you have these millions of pages (or even tens of millions), how do you process them? Surely, Hadoop MapReduce is necessary, after all, that's what Google uses to parse the web, right?

Pfft, f**k that noise:

find crawl_dir/ -type f -print0 | xargs -n1 -0 -P32 ./process

32 concurrent parallel parsing processes and zero bullshit to manage. Requirement satisfied.

Every time you write code or introduce third-party services, you are introducing the possibility of failure into your system. I have far more faith in xargs than I do in Hadoop. Hell, I trust xargs more than I trust myself to write a simple multithreaded processor. I trust syslog to handle asynchronous message recording far more than I trust a message queue service.

Taco Bell programming is one of the steps on the path to Unix Zen. This is a path that I am personally just beginning, but it's already starting to pay dividends. To really get into it, you need to throw away a lot of your ideas about how systems are designed: I made most of a SOAP server using static files and Apache's mod_rewrite. I could have done the whole thing Taco Bell style if I had only manned up and broken out sed, but I pussied out and wrote some Python.

If you don't want to think of it from a Zen perspective, be capitalist: you are writing software to put food on the table. You can minimize risk by using the well-proven tool set, or you can step into the land of the unknown. It may not get you invited to speak at conferences, but it will get the job done, and help keep your pager from going off at night.

Technical Debt

http://www.linkedin.com/news?viewArticle=&articleID=223249842&gid=90548&type=member&item=32525387&articleURL=http%3A%2F%2Fwww%2Einfoworld%2Ecom%2Fd%2Fdeveloper-world%2Fgartner-technical-debt-will-reach-1-trillion-in-five-years-415&urlhash=7Gx5&goback=%2Egde_90548_member_32525387
OCTOBER 11, 2010

Gartner: 'Technical debt' will reach $1 trillion in five years

In a new push to measure software's true cost, the National Science Foundation and Gartner are drawing attention to the problem of 'IT debt'

Computerworld

The idea that software acquires a "technical debt" that is paid in real dollars is getting new attention and research.

The National Science Foundation approved a $465,000 research grant last year on technical debt, and Gartner has just released its study on this subject, which it calls "IT debt."

Gartner puts the IT debt bill at $500 billion worldwide and says it will double in five years to $1 trillion.

A company that makes software quality tools, Cast, just released a study gleaned from customer software evaluations that puts the technical debt at $2.82 per line of code. It says for the average-sized application of 374,000 lines of code this amounts to just over $1 million in technical debt.

Frank Scavo, president of Computer Economics and management consultant at Strativa, zeroed in on Gartner's estimate, calling it bogus in a blog post, and outlined ways to rationalize an application portfolio.

But the idea that applications acquire a technical debt may have support in the development community.

Among the people who incorporate the idea of technical debt in their development is Gene Baker, the chief architect for the widely used WyStar 401K record keeping platform.

Every time a new piece of software is added to the code base, Baker said, an addition is made to its technical debt. Code is the principal and software maintenance is the interest payment, "so the more code we have out there the bigger our debt, hence the more maintenance, hence the more interest payments."

Gartner defines IT debt as what it would cost to bring an organization to a fully supported environment.

"You can run on unsupported software for a while -- lots of people do this," said Andy Kyte, a Gartner analyst, but if more and more of your portfolio is out of date, as well the components, application servers, databases, compilers, operating systems, "then your portfolio is gradually degrading over time," he said.

This degradation in IT environments accelerated with the recession, and Kyte said IT managers need to determine how much of environment is an "unsupported state or creating a systemic risk" and then let the business know about the problem.

Cast software tools are used to examine the technical quality of an application, developed in-house or externally. Jay Sappidi, director of Cast Research Labs, said part of the idea is to catch problems, such as with security or performance, earlier in the development cycle when they cost less to address.

Sappidi said its research was based on a study of customer code. Users can push maintenance issues aside, but "you cannot get rid of technical debt."

Baker said one method for controlling technical debt is through code reuse.

"The more we can reuse code, then less code there is and then when we have to do testing the less we have to test, and ultimately the less we have to maintain," said Baker, who also uses Cast tools.

The term technical debt may have been coined by Ward Cunningham, a developer, who in a 1992 wrote, in part, "Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite."

Carolyn Seaman, an associate professor of information systems at University of Maryland, Baltimore County and the principal investigator in the NSF-funded project, said they are working on ways to represent technical debt, quantify it, measure it and help make it part of the decision making.

"We have an increasing inventory of old systems that are too big to replace," Seaman said.

Also see Mark Levison's InfoQ article.

http://www.infoq.com/articles/technical-debt-levison?mkt_tok=3RkMMJWWfF9wsRonuaXAZKXonjHpfsX56usrW6%2Bg38431UFwdcjKPmjr1YcFScB0dvycMRAVFZl5nRpdCPOcc45P9PA%3D

Procedures V Bureaucracy

Source BNET:

Bureaucratic organizations are inefficient, expensive, and sluggish. Most government agencies are a testament to that. Nevertheless, organizations need effective processes and procedures to grow.

Finding the right balance and knowing when it’s time for a change are two of the biggest challenges every successful entrepreneur and business leader faces.

For example, when companies are small, decisions can be made on-the-fly with minimal concern. A couple of guys chat in the hallway and, voila, a new product feature is born. And that’s a good thing. The more flexible and nimble a startup is, the better its chance of making it.

But as companies grow, negative repercussions of ad-hoc decision-making begin to appear. For example, growing companies are often plagued by miscommunication, frequent changes in direction (strategy du jour), or the reverse, analysis paralysis. Decision-making is just an example of a broad range of ad-hoc processes that falter as companies grow.

There inevitably comes a time when leaders must introduce structure and procedures to enable their company to scale. If they overdo it, they run the risk of adding unnecessary bureaucracy. But if they do nothing for fear of losing flexibility or screwing up what already works, the problem only gets worse.

Over the years, I’ve worked with and observed dozens of companies that hit a wall because they didn’t recognize the signs of growing pains or failed to take action to improve the company’s scalability. Here’s some guidance that will help you navigate the fine line between chaos and bureaucracy:

First, Learn to recognize the signs. Dysfunctional decision making, miscommunication, and goal misalignment are all signs that a company’s processes are not keeping pace with its growth. Here are just a handful of examples:

• Persistent and disruptive strategy changes without key stakeholders present.
• Constant debate over critical issues with no real resolution or agreed-upon plan.
• Plans are agreed upon but some groups follow them while others don’t.
• Sales says customers want A, marketing requirements say B, product development is designing C, manufacturing is planning for D.
• Product gluts or shortfalls due to inaccurate and unreliable forecasting.
• Miscommunication between management levels, i.e. middle managers are constantly blindsided by “new” information, plans, decisions from above.
• The same as above but in reverse, between geographies, etc.
• Consistent budget misses in either direction.
• Support organizations like HR and IT are constantly overwhelmed.

Second, improve processes without adding bureaucracy. This is where the rubber meets the road and yes, it’s more art than science. Here are five artful tips:

• Don’t fix what isn’t broken. If everything’s humming along just fine, i.e. you’re not seeing the above signs, leave it alone. Take it process by process as opposed to an across-the-board overhaul.
• Make all key stakeholders part of the change process. Don’t impose procedures on key stakeholders without giving them a chance to weigh in and approve the change. We’re not talking lip-service here; we’re talking “integral part of the process.”
• Never add procedures because a consultant or some other disinterested party says you should. Always challenge the risks-benefits potential of any change.
• Think evolutionary change versus revolutionary change. If you keep that concept in mind, it will help you improve your processes without losing the entrepreneurial cultural that got your company to where it is in the first place.
• Change is significant; don’t outsource it. Entrepreneurs need to understand that change is necessary and significant. Don’t be afraid of it and don’t outsource it. Instead, embrace it, respect it, get ahead of it and lead. That doesn’t mean you shouldn’t seek outside expertise, just don’t let it get out of control.