Saturday, December 17, 2016

What is an API for non-technical people

During conversations over the last few weeks, I have been asked more than once by non-technical people... "What is an API ?"

As technology evolves to have more interconnected systems, many will find the term API becoming a larger part of their lives. This includes both business people and technical people alike.

I thought I would share an analogy to help those that are less technical understand some simple concepts to get you started on your own journey of learning, or just to help you feel better when talking with someone about the topic :->

If you are interested in incremental, multi-team delivery, you will find these concepts important to understand.  

There are other concepts and the following is definitely an over-simplification, but this should get you started.  My goal is not to make you an expert, but to get you ready to have some conversations about this in your environment.

  • Public interface
  • Keep the implementation private
  • Don't make something public unless you really have to

For the purpose of this post, consider an Interface to be .... 

The means by which you interact with something to send and/or receive information or change of state.

An API is an Application Program Interface.  To extend the definition, consider an API to be...  

The means by which you interact with a Program or System to send and/or receive information or change of state.

For the technical people out there.. Yes, I know this is an oversimplification.

The Automobile gas pedal.

Public Interface - To interface is simple. One moves the pedal forward to increase the flow of gas to the engine.  Decrease the distance forward and the opposite happens.

The results are known to us either a - vehicle going faster, or b - engine revving faster, c - vehicle going slower, d - engine revving slower.

It is important to understand, there is no interface through the gas pedal for speed, torque or anything else.  Those are other systems that have their own interfaces.

Keep the implementation private - Imaging for yourself that to know how to push the gas, you needed to know

- How many cyclinders the vehicle has
- Is it gas or electric
- What size engine
- What type of engine computer
- etc.

If you needed these to interact with the gas pedal, it would certainly be more difficult to drive.

Imagine that before you pressed the gas pedal, you had to calculate the force necessary based on engine size, the angle of the hill you were on, figure out which cylinders to turn on in what order to be able to press the gas.

Don't make something public unless you really have to.

These details are referred to as Implementation details. A good interface keeps this information private (not visible to the person or system using it).   

By using this approach, you could for instance, change your engine from one type to another, or get an updated engine computer.  You might have different results as you press the gas pedal with evidence from other interfaces (ie: speedometer), but you didn't need to change anything in the current interface or learn anything new to use it.

Don't make something public unless you really have to - This concept is extremely important for those that are growing systems incrementally or organically. Think about it this way... 

Once you make an interface public in a complex system, it may be almost impossible to remove that interface from future iterations.

This situation will be compounded in systems that take years or generations to change or where interaction from other systems may rely on these interfaces long into the future.

In the example of our car, let's assume that someone said "You know, we could make the gas pedal really awesome if we allowed the operator to press on the right or left of the pedal by angling their feet to give a different acceleration potential".  

This might make sense in your environment (let's say you have a specialised product for race car drivers and that would improve their lives significantly!). Innovation is cool that way.

My goal here is to have you consider this; Would a different interface be appropriate ( a hand control for instance). Could we have a separate control for Normal Acceleration or Faster Acceleration (think about this.. some vehicles have a Sport Mode).

There are two potential considerations should you decide to add this really cool interface Publicly (in your current application).

In a distributed application or system, something or someone will use a public interface! This means that you may need to have that ability in your system forever. You may never take it away (well, not easily at least).

If you realize later that you should have made a different interface (Sport Mode), it will be too late. You have made the interface public.  

  • People may have already created driving schools for the new accelerator system
  • Someone may have created a special shoe to use it
  • Control systems may have been built to control the pedal from a manual control that converts it to suitable pressures on the gas pedal 
  • And so on.
To complete the analogy to software or systems development, now anytime someone wants to use the interface they have to spend time and effort learning and understanding acceleration concepts when all they wanted to do was press the gas pedal.

I hope this helped the non-technical person to be able to have some knowledgeable conversations about Application Program Interfaces or APIs.

There are Interfaces at different levels in your vehicle. Examples include....

- Engine computer
- Fuel pump
- Light switch controller
- Engine itself (think electric)
- Speedometer
- Tachometer

Technically oriented people will be using this term often in the future. APIs and discussions will become more prevalent as technology becomes more pervasive. The Internet of Things (IoT) is mostly about APIs for starters.

Hopefully, I have given you a bit of a non-technical description that helps you out to start your learning journey about this topic.

Actually, even communication and conversation is a form of Interface between humans. That's a topic for another time :->


Tuesday, December 6, 2016

Techpost - The Test-Maintain Loop saved me during a Jenkins Server infrastructure upgrade

Technical Post (just a warning for the non-technical follower).

As some of you know, earlier this year I created a first version of an Ansible Playbook Test framework and put it on Ansible Galaxy.  

Yesterday, this tool saved me from creating a disaster on my Production Jenkins CI instance on AWS (Amazon Web Services). 

I'd like to share that story as an example for others of what is possible if you include testing as part of your Infrastructure as Code strategy.

How I was saved yesterday:

(some details removed for security and simplicity reasons).

Execute the Playbook...
ansible-playbook -i Inventory/CASPAR/staging/ CASPAR_setup_jenkinsservers.yml -u ubuntu --ask-vault-pass --ask-sudo-pass
The result is a new AWS instance server with Java 8 loaded,  appropriate users and groups setup and a base Jenkins Image loaded.  The key here is SETUP (the minimum needed to get a server into MAINTAIN status. The server is in "Staging".

Then, the repetitive playbook is executed (this one runs on a regular basis to keep servers up-to-date in all environments. In my case, I execute for Dev/Staging and Production ( the same playbook is used and applied to all 3 environments ).

ansible-playbook -i Inventory/CASPAR/staging/ CASPAR_maintain_jenkinsservers.yml -u ubuntu --ask-vault-pass --ask-sudo-pass

Note: The only difference in the name is the word "maintain". 

Note: To run the same playbook in Prod or Dev, I simply run the same playbook against /CASPAR/prod/  ( Ansible's dynamic Inventory auto-finds the appropriate machines based on tags )

Then, while the machine is still in Staging, the following test playbook is executed....
ansible-playbook -i Inventory/CASPAR/staging/ CASPAR_test_jenkinsservers.yml -u ubuntu --ask-vault-pass --ask-sudo-pass

The "test" playbook executes all predefined tests to ensure the server is in good shape. 
If there are no errors, all that needs to happen is for the machine to be re-tagged in AWS from Staging to Prod and then the next time the maintain playbook is executed, it will have any appropriate changes using the SAME playbook as before (different Gateway addresses, different database connection string, etc).

Then of course, the TEST playbook is executed again (one final test).. 

The Test playbook can now also serve as a Governance check playbook as well and could be executed by the same team or externally where needed. It provides a means for safer, more comfortable changes, while also providing a built-in governance component if needed.

Yesterday, when I ran the tests in Staging I received an error about missing packages.

ansible-playbook -i Inventory/CASPAR/staging/ CASPAR_test_jenkinsservers.yml -u ubuntu --ask-vault-pass --ask-sudo-pass > test.log
grep "TEST_PASSED" test.log
grep "TEST_FAILED" test.log

I received the message...

 "msg": "TEST_FAILED: package xxxxxx expected present "

(xxxxxx is a hidden package only for this post for security reasons).

If I had converted the host to Production, it would have caused big problems in my production environment.

After doing some research, If found that I had previously requested a newer version of an AMI ( an Amazon Machine Image ). 

Although the entire "setup" and "maintain" playbooks ran flawlessly (with no errors), what I did not know was the newer AMI was missing a critical Operating System package that my environment needed.

I modified the "maintain" playbook to include the missing package, re-ran the "maintain" playbook and then re-ran the Test Playbook.  Everything passed. Now, I know the Staging and Prod machines will always be up-to-date with this package when the "maintain" playbook runs it's continuous loop.

The new Jenkins Server was tagged as "Prod" and then the previous server deleted from AWS. The transition was painless.

By taking this approach and adding new checks to my server first as they become evident, I ensure that I will  not deploy something to production that has not already been determined to be a potential problem. 

I will no longer have this issue or one related to missing this package again.  If an image contains the missing image, no problem.. It will simply pass. Ansible does not re-install packages if they already exist (unless "latest" is specified in the version").

Brief History of the Test/Maintain/Govern Loop

The purpose of creating the Test/Maintain/Govern Loop for Playbooks was to show a Test-First approach to infrastructure delivery to make the transition to Infrastructure as code easier to get accustomed to.

The approach uses knowledge taken from years of insight from the software development world in delivery of complicated environments and applies it to the Infrastructure as Code domain.  

Technical Notes:

Jenkins CI server running in Production on AWS.  

Ansible Playbook uses to Setup/Maintain and Test server(s) in both Staging, and Production. ((how my environment works for build servers.. TODAY).

In AWS, tags are used to determine if a machine is "in production" or "in staging". They are both live in AWS in the same VPC (A VPC is like a private IP range within AWS for my hosts to reach each other).

Playbooks are formatted into YAML (a markup format) to have Dev/Staging/Prod in the same playbook.

A unique matching approach allows the same Playbook to run many times in Dev and Staging. This helps to ensure that when the Playbook runs on the Production machine, it has already executed many times already (and confirmed correct).

An often missing catch with playbooks is that "If" statements can be used to determine of parts of playbooks are executed.  A playbook command can be set to only run a certain instruction only IF a certain environment exists (an example).

When I want to upgrade my Jenkins server or reconfigure a new one, I take an approach of.. "Build a new one, run the setup/maintain and test on it, and IF everything is OK, move it into the Production Tag and then disable the older server. This allows me to ensure all is well before activating a new production change.  

Think of the saying ....  

"All Servers Are Temporary"

A link to the original presentation be found here..... 

If you are so inclined, here's a link to the root repository... 

A sample "test" (also used for governance) playbook is located here...