Sunday, January 30, 2011

Infrastructure Knowledge for Developers

As technology is constantly evolving and with the recent market progress in cloud computing the line between hardware are software is getting more obscure every year.

This changes some basic questions. Are you a developer? Are you in Infrastructure? Today, I believe it is unreasonable to expect to be just in one or the other.

Take Microsoft's Azure platform.  Is it hardware?  Is it software?  Well, sort of a little of both.  

If you only know .NET development for instance, how would you accomplish Federated Authentication or even something like OpenID without at least an understanding of Internet Based Authentication?

Authentication has typically been the role of Infrastructure.  Here lies the rub.  A developer is not accustomed to having to decide, "Will I setup to be a "cross country" domain, "worldwide" domain.  These are typically infrastructure questions but need to be answered immediately upon creation of these types of services. 


An example might be a content a Content Delivery Network (CDN).  A CDN can have some fairly complex business rules. Use of a CDN affects marketing, management, system reliability, performance and customer satisfaction. A CDN can also greatly cut down the number of hits to your web site.  There can also be significant costs.

Here is where it gets complicated.  When a development team writes an application and is asked to start moving files to the "Cloud", there is no reason for them to ask about the benefits (or financial implications) of a CDN.  Why would they?  This is historically someone else's consideration.

If a team does all the work without knowing that some day you will be delivering the data over the CDN portion of the storage provider, you may find it almost impossible to do.

The best choice would have been to have development in on the "Infrastructure" part of the discussion.

You may find your team unable to re-write the code to handle the new details in the application.  For instance, some files may be delivered through the CDN and some not.

Most developers are comfortable with the ideas of private and public files, but many will not be familiar with the idea of some public files going through CDN and some not.

Do yourself a favor.  Teach them now about the technology.  Just knowing a tiny bit about it could save you huge expense in the future.

I can think of many such instances where cross-training should be considered and as far as I'm concerned absolutely necessary for the future growth of an internet based company.

The hardware folks need to start sitting in on some developer meetings and developers need to learn about some infrastructure.  This will ultimately help everyone work together and definitely help the teams understand each other's needs.

Sound crazy to you?

Consider a few examples from past experience.

 - .NET developers who are tasked with creating a system of web sites that can intercommunicate, but none of the developers understanding the mechanisms of Public Key Infrastructure, Certificate Authorities, OpenID or any of these authentication technologies typically reserved for Infrastructure People to deal with.  

Clearly this is a large development job and the hardware infrastructure has to support what the development team can do and there is no point of the development team for instance making something requiring VPN's where there is no actual hardware in place (just cloud based servers).

- An SQL server DBA trying to figure out a way to fix a server with so many connections open that it can't even function.  The solution is in a Web Server that is not configured properly in the IIS Connection Pooling settings. 

The SQL DBA doesn't know about the IIS connection pooling because developers handle that and developers don't know about it either because "the hardware people" handle the web servers.

- Developers using the Cache control of a Web Server with no expiry of Cached objects and basically forcing the server to run itself so low on memory that it starts purging memory out of desperation to stay running.

This is clearly a situation where a developer who sat in a Cache discussion about how .NET (in this example) manages the Cache and the fact that a Windows Server does not have unlimited memory resources like a mainframe (just kidding for you mainframe folks out there...I know from past experience that your problems can be worse).

- Consider developers who do not truly understand the impact of making DNS queries and their impact on IP Infrastructure when they make queries to 10,000 servers for DNS records from a large volume email application without any consideration for Caching of DNS requests. 

Any developer who actually saw what happened at the CISCO switches and the upstream DNS servers would cringe at the traffic they created.  Of course, if they only knew.

- The same argument works the other way around.  I have seen a DBA change the password on a SQL server automatically on a rotating schedule without realizing the impact to a Windows Web Service sitting in another building.

It seemed that every week the off-site Web Service crashed and no one could figure out why. It took weeks to determine the cause. 

I have discovered that having developers sit in with the Infrastructure folks occasionally and vice-versa provides tremendous benefits in co-operation between the IT staff.

It's amazing to see a development team wonder if there is a hardware solution to their problem before committing to a month of work.  It's just as amazing to see a hardware person ask if they could just get a developer to make a small tweak to fix a problem and get a positive response just because of a bit of cross-training.

If you're trying to improve productivity, allow the different teams to sit in with each other once in a while.

I have started to bring our developers up to speed with some training on  Caching, PKI (Public Key Infrastructure), Certificate Authorities, CDN's (Content Delivery Networks), different types of Source Code Repositories and basically anything they are interested in reviewing.


Anything the team can learn about together can only benefit everyone.

A Source Code Repository discussion can be equally beneficial for developers (who have to use them) as well as Infrastructure folks who have to plan to make sure they work. Why not let the teams learn this type of technology at the same time in the same location?

Code Repositories are typically a developer item but your Infrastructure folks will greatly appreciate and even help the development team out if they really know why it's so important to development, and maybe even find better ways to  store the data for the team. 

After all, even the best written software still needs proper infrastructure to run it.

I like to schedule the first 1/2 hour of our regular development meeting for members from other teams to join in.  They can see what the team does and how things work.

They are more than welcome to sit in the technical stuff, including table interactions, UI issues, etc. later on. It gives other departments an appreciation for how complicated of a job it can be and definitely helps boost morale.

The point here: Give it a try.  Do a bit of cross-training during your Developer and / or Infrastructure Meetings.  You will be amazed with the results!


Mike Caspar