Friday, January 25, 2019

Scrum, XP, FDD or Kanban? Use all of them!

There are lots of terms in the agile development world; its clear that we still figuring out the best way to make software. Here I illustrate the use of software development methodologies and what I have seen work in past projects. A lot depends on the project, the people available, and what the end goal is. What my intention is to illustrate that one process or methodology isn't the answer; but instead the most effective process depends on your situation. Its the same as using a screwdriver with nails, or a hammer with screws. Be pragmatic and use the tool that is appropriate for the job at hand. The main goals is the same: create quality technology.

Scrum time

How do we go about doing this? Let's firstly understand how Scrum can be used for organizing how people work together to build both projects and products. Scrum can be used to build anything; a bird house, a home renovation, car repair, and software. It's got nothing to do with the actual technology and everything to do with organizing the group of people doing it.

It can work to realize and understand some fundamentals of how we are doing any of these things:
  • where am I in building this?
  • what am I trying to accomplish today?
  • what are the dependencies that I need to get that done?
  • what is preventing me to getting this done?

Scrum the Bad

Scrum can work, but too often its miss-used; and becomes a tracker of time sheets and velocity at the expense of the product the team is actually making. Care about adding value to the product, as no customer is paying the company for time-sheets or velocity. These metrics just don't matter in the end.

Dependencies

For someone managing this process, those last two points are everything. If some piece of work is dependent on other items getting done, the job is to make sure those dependencies are in order and getting done; and if those items are blocked in any way.
In our home reno project; a plumber may be ready to install the bathtub, but to install it the materials need to be there(pipes and a bathtub). So the buying the tub and transporting from the store to the house is a dependency.
For software, its could be a button on the UI that submits a form; you could have your UI developer create that, but if the server API doesn't exist and the logic isn't in place to save the value in the database; then that UI developer is blocked. Ideally these dependencies are in place by the time that front end work starts.

So if Scrum is a general way to organize people, then it needs to be used in conjunction with a methodology that actually builds the technology. These are when terms like XP, FDD, Spiral come in. So, when talking about Scrum, it's really Scrum/XP or Scrum/FDD, or Scrum/RandomConversation. Only with that combination you can have a chance of starting and stopping a project; Scrum helps you organize the participants of a project; but you need a definition of starting and ending. Is Random Conversation a process? Not officially, but many organizations use it!

Product or Project?

Firstly, what is the context of the effort? is it a one-off project for a customer, or a scheduled release that adds to a product? For a one off project you have constraints of time, since you are getting paid based on the time spent and when the project is over there is no more development.
For a product, the constraints are the features in the product. So the measure of success is the amount of value you add to the product, and how it sets you up for the next release. For the sake of analogy; its more like continuously growing a garden that produces vegetables than it is getting one plant growing and produce one vegetable that you give to the person asking for it.

Product

  • increased importance of maintainability since you have to live with this code for the long term. You have the opportunity to get it right over time over a number of releases, but you have to deliver enough value so the product can be used today.
  • High need for innovation as your effort is trying to add value to a product that your competitors don't have. What is the differentiator in what you are making compared to the competition?

Project

This is something that is budgeted with money, so it's a time and materials type of effort. The client is really just wanting a working tool that solves a single problem; so the need for innovation and invention generally isn't part of this. Its important to use established technology and process so that the project has a measure of predictability. This is essential. If the project ends with the technology unfinished, who is going to pay for the remaining time to complete it?
  • you have one chance to get it right, so there is an increased importance in requirements and design, and only building what you have time to do. 

How do to this?

Product

If you are making a product that has no customers, then you are adding features to get to a first version that you can release and get on the road of paying customers. This is where the concepts of MVP and Definition of Done apply. What is the simplest version of the product that fulfills the users goals and solves a real problem? For this, using Scrum/FDD or Scrum/XP has been very effective to constantly iterating to get you to that first release.

What happens after that first release? You need to keep iterating on the product to increase its value for the next release, but you also have customers using it, so you need to respond to them and fix their issues (or not).
For this, split your effort accordingly.

  1. Continue to use the Scrum/whatevermethodology you used to get there, and use the same sprint cadence with people dedicated to that process. This work can be planned, you are in control of how much you take on, how much innovation is attempted and what you are happy with to release.
  2. Use Kanban to organize the unplanned work. As issues come in, have someone dedicated to managing this so that the customer gets a response, and the learnings from that make it as inputs to the next sprint. It may be best to have someone dedicated to support during each sprint so the other members aren't interrupted; but rotate this position as it becomes a bit of a grind)

DO NOT combined planned and unplanned work in the same process; you will end up doing neither well. 

Project

You are making technology to a specification, and that specification needs to be understood and agreed to by the customer and the builders. Getting this correct will make or break your project; so take time to get this right, and get it done. Using iterations during this phase should involve the customer and builder directly, so it's intensive in a people participation aspect, but vital that the expectations and assumptions aren't left to interpretation at the end.

Depending on the project, a number of methodologies are useful here. Waterfall can work here, and the software may be part of a larger effort, so it's important to use a similar overall methodology for the whole project. This is also about predictability; so invention isn't really welcome here, and using established technology is necessary. This may seem less fun at first, but going over time and having to work for free to fulfill the expectations that were agreed upon at the start is a whole lot less fun.

Wrap it up

For Projects
  1. Use established technologies (even if they feel old-school, the client probably isn't paying for innovation) 
  2. Invest time in getting the specification correct, and set expectations for further changes. (Jobs to be Done, and Domain modelling)
  3. Aggressively remove work if its not needed
  4. Identify and resolve dependencies and understand whats needed 
For Products
  1. Use Scrum with a methodology to plan and execute feature sprints. (Personas, Jobs to be Done, Quality Model)
  2. Use Kanban to organize unplanned work so feature work isn't interrupted and customer requests are addressed.
  3. Review each sprint to ensure each release is better than the last.
  4. Innovate and add value so that the product is needed by its users. This is a process that lasts as long as your product is successful; so foster a culture of continuous improvement to get better over time.
It can be done! Projects are similar regardless of the technology; the client needs something to spec since the time and money is finite. So if its a home renovation or a website; use established tech and keep expectations realistic. For projects, you need to continuously add value while being responsive. Doing either well isn't easy, but it can be done by sticking to some sound fundamentals.

Monday, January 21, 2019

Feature Definition with a Quality Model

Defining all that goes into a feature can sometimes stop at the functionality and not take into account the various system attributes that combine to make the feature actually successful for the user. Attributes like Usability, Performance and Security add to or take away from the user experience; so understanding them as a whole is essential.

This post outlines the usage of a Quality Model that takes into account the functional and cross-functional features to provide a template that can be used as a definition of done.

Definition of Done

The quality model is really useful for understanding the scope of what needs to be done for a single feature or a theme of many features together. The application you are building is just a collection of features that work together, and how they work together is what is ultimately how the user experiences the set of features. How fast, how secure, how easy they are to use is what is key. these attributes are sometimes referred to as 'cross-functional' as they affect all functionality in the system. Also the term 'non-functional' is sometimes used, but seems like a miss-leading term. They all do something; and it's hard to convey the importance of working on something that is 'non-functional'.

Where are these defined in working?
These constraints are sometimes included in issues or explained in a specification. It's important to be consistent, and treat these as the requirements they are. So, to that end, use acceptance criteria from the domain requirements to outline the compliance targets for the quality requirements. 

This is applicable from the large to the small; from Product Architecture to Application Architecture to designing a single Feature. This template can be used to fix a bug, define a feature, or an entire product. These constraints provide the definition of done, or feature completeness.
Are you ever done the product? Hopefully not! The quality model give you a high-level glimpse of what it does today. This combines the attributes of the user facing functionality with everything around it; so it provides a much clearer picture of what is required to get that feature to a done state.

in its simplest version, its the basic release of any software.
Functionality: By Theme, use domain research
Usability: Goals and Experience
Performance: general feel and response times
Portability: Dev, staging, production
Serviceability: Observable logging, execptions, query times, dependency health and status.


  • Functional
    • Business/ Human facing attributes to satisfy the business rules/problem domain.
    • Match Personas/Jobs to be done with the Features that get their job done.
    • Domain requirements from subject matter expert
    • The Business value attributes that satisfy the business rules/problem domain. A set of attributes that bear on the existence of a set of functions and their specified properties. The functions are those that satisfy stated or implied needs.
    • Suitability
      • Are there relevant business requirements?
    • Accuracy
      • Is the data shown to the user accurate?
    • Compliance
      • Are there rules/regulations constraining the functionality? are these known and declared?
  • Usable 
    • Is the system usable without a lot of training and previous knowledge? How intuitive are the controls and workflow? How does the User Interface 'look and feel'? This is the domain of user experience and has 
      • How does the app match the job workflow? Is it similar and intuitive as to what I would do?
      • get jtbd matching key problem. set requirements, fulfil solution
    • This is the 'user interface', the 'look and feel'. For print and publications, this is called the 'design'. In software, this is the 'aesthetic design' and is really the tip of the iceberg. You only see about 15-20% of a software system, but its a very important part. Its has to be a comfortable place to get things done efficiently.
      • Survey users
        • UI/UX, HCI research (human computer interaction)
      • Accessibility - the application is a web application, so it needs to be accessible from common browsers and not just desktop browsers, but tablets and phones.
      • How intuitive are the controls and workflow?
      • Is the system usable without a lot of training and previous knowledge?
      • How does the User Interface 'look and feel'?
      • Usability Compliance
        • Accessibility requirements could be legally required.
  • Configurable
    • Configuration for: System (Dependencies), Product (Features), User (Permissions)
      • properties for environment - networking, logging, dependencies
      • properties for domain - domain rules, constraints
      • properties for product - feature flags
    • Permissions: 
      • permissions seem to be the same as feature flags, but they are actually dependent on them. Permissions depend on Role in the system. Features depend on the product. Enabling a feature is decided before determining if a particular user has access
  • Performance
    • This is a measure of how responsive a system is. is it fast? and what does that mean? Do thing take a long time to load, or do some searches take a long time to run? When making a system that does a lot of things, its an equally or larger challenge to make the system do those things in a timely manner.
    • instrument the system, know how long what takes. ask during usability reviews
    • Does it use acceptable amount of memory and does the use of the memory scale linearly with more usage?
    • Set targets to start
      • page loads in 3 second response
      • data ingestion speed
        • high latent sources: 3 days
        • medium latent sources: 4 hours
        • low latent sources: 10 seconds
  • Security 
    • If a user is going to hand over their data, whether its their personal details or business information they don't want to have to worry about the safety of that data. They don't want their account hacked, or have issues where their confidence in the systems integrity comes into question. Its comes down to Trust
    • define a security policy, implement threat mitigation, audit system regularly.
    • Review security aspects of new features, review operational security issues and review results of security testing for Authentication (Identity) and Authorization (Roles and Permissions)
    • No unauthenticated access to client specific data
    • Questions:
      • Is there private user data in the system?
      • How can the different roles access the system?
      • Is it deployed in a secure environment? (Asses the network security and use of standardized technologies)
      • Is the data persisted to a secure environment?
      • Is there any secure data in logs or other outputs?
  • Maintainable 
    • To be able to work on the system with others the code and artifacts need to versioned and the operator needs to be able to quickly diagnose issues. The code base needs to be maintainable to add features without breaking existing functionality
    • Source code should be written in a manner that is consistent, readable, simple in design, and easy to debug. A set of attributes that bear on the effort needed to make specified modifications.
      • Can a user other than the developer run the system? Are the tools to do that useable and coherent?
      • Source code should be written to facilitate test-ability.
      • Does the implementation follow the intention of the design?
        • is there a design?
        • The design of reusable components is encouraged. Component reuse can eliminate redundant development and test activities (i.e. reduce costs).
      • Does the code comply to a code standard?
      • What amount of the system can be verified with tests?
        • How many of those tests pass?
  • Extensible
    • Requirements are expected to evolve over the life of a product. Thus, a system should be developed in an extensible manner (i.e. perturbations in requirements may be managed through local extensions rather than wholesale modifications).
    • Can you change the functionality for future use? Is there a common coding standard
    • Can similar functionality be implemented with the same component?
  • Observability/Serviceability
    • While the system is running, are issues easy to diagnose? If something goes wrong, will someone know immediately? How is it monitored?
    • Can a user other than the developer run and configure the system?
      • Are the tools to do that usable and coherent?
    • Can you change the functionality without rebuilding the application, and can you add to it later
    • Observability capabilities should include:
      • on-demand query of all systems go
      • real time alert of subsystem failure
      • ability to see errors, warnings and info messages
  • Availability/Reliability 
    • The system needs to be able to run for long periods of time without degradation. Memory usage and resource allocations need to be sustainable and system loads predictable.
    • Reliability Compliance
      • Is there a declared/contracted availability?
      • Does it run and not degrade over time?
    • Deployment Execution.
      • How do you know when deployment is done?
      • Are the steps to deploy clearly explained and documented?
  • Portability 
    • The system needs to adhere to standards so that it will be able to run on publicly available 'clouds' without modification. There are many candidates and an agile project cannot be locked into a single vendor relationship.
    • Source code should be portable (i.e. not compiler or linker dependent). A set of attributes that bear on the ability of software to be transferred from one environment to another.
      • continuous integration and automated deployment help here
      • Will the application run on other environments? Other operating systems and networks? Have a developer version and testing version with recent data. 
      • Does the application run on the target environment? Does it only work on the development environment? Is there a process for incident and change management? (ITIL standards are a good start here)
  • Interoperability 
    • Does it play well with other systems?
      • using standard protocols?
    • Can the system survive with its dependencies in a bad state?
    • Are all the dependencies identified? (Assess network connections and other dependency couplings)
  • Efficiency/Scale-ability 
    • The relationship between the level of performance of the software and the amount of resources used, under stated conditions.
    • Scaling. Does it use acceptable amount of memory and does the use of the memory scale linearly with more usage?
    • Can the application share resources, or does it need its own machine/cluster?
    • Time Behavior, Resource Utilization, Resource Allocation
    • Under what conditions does the application leak memory? hog cpu cycles?
    • Efficiency Compliance
      • What is the current performance benchmark? what's the next target?

Application Architecture in the Large

The System satisfies the requirements through its various features. Use the quality model for a specific feature, or a collection of features within a product.
Understand technical debt from a high level. It's quality debt, but like the financials debt can be useful to get you to the goal; it just needs to be managed effectively.

Integrating in the process
All these points encompasses Quality Standards that need to be validated by Quality Engineering

Sign-off
For each quality requirement listed below, get sign-off from owner as part of the architecture review. An understanding of the current benchmark and achievable targets will properly frame expectations.
  • Functionality - Product Manager using backlog stories/acceptance criteria
  • Security - Sec Audit using security guidelines/standards
  • Usability - UI/UX - using usability standards
  • Performance - Performance Engineering using reliability/availability standards
  • Maintainability - Tech Lead using Code Guidelines/Standards
  • Portability - Operations using deploy and configuration Standards
  • Manageability - Operations through monitoring standards
  • Planning - Program Manager - using ADM/Scrum standards
  • Serviceability - Support using support standards

Wrap it up

You need to have some level of design to be successful for a system of any size, but too much architecture can slow down, or stop the implementation. Use consistent models and views that take into account the entire system; not just the UI and some functional stories.

In the end, the design will happen anyway. If it isn't formalized somehow (at least in some words and a diagram), then it's in the heads of whoever wrote the code. I have made some fairly large systems this way; but paid for it in the complexity of conveying that design to more people than myself. In the end the complexity of that mental model overwhelmed me. By using a simple template it becomes easy to account for what is going into the system, and much easier to share as a result.

Start with a lean definition of this for your needs; you don't have to re-invent the wheel here, just use the template.



Friday, January 11, 2019

Software Releases are a lot more than code

We have a release coming up... are we ready? These words are spoken probably on a daily basis in the tech world. It can mean many things; but generally its putting the team and what they are making on a deadline. This is a good thing, you want to show the world (well, your customers) what you have and how it makes their lives better. You have probably committed some features or fixes as well, so the importance of a successful release is paramount.

Like any deadline, people will probably put off many details until late; and in the rush to make a deadline some details will be invariably left out. That isn't intentional, but without a good plan to release it can become a random exercise. I have worked with teams that have zipped up some code and just did a file transfer (with a 'please contact us if any problems') and some others that started release planning on the first day of the project.

What are our goals for a release?


  • Customers get a new version of a product with new features and/or bug fixes
  • Sales gets a new feature to sell
  • Project managers wants to release a well tested version without stopping feature development
  • Operations wants to do less manual work than the previous release
  • Product needs to understand the differences between releases to account for new features and fixed bugs
  • Everyone needs to understand and plan for what the priority is for the next release
Communicate effectively, and put the hard work your team did in the best light possible

Audience

Who is the audience? Well, customers obviously come to mind, but what stage are those customers in? Are they already paying and using the product? Are they new buyers? are they prospects that need a confirmation to buy? The release is for all of these people.  How does it get to those people? Through support, sales and marketing.

Stakeholders

There are many stakeholders that have an interest in the application you are making. It's important to keep contact and update status whenever there will be a change to the outside facing product.

  1. Customer Success needs to know how new features work, and bugs are fixed before the Customer does.
  2. The Product team should know the status of features and the deliverable before the demo to the larger group. They will get questions and shouldn't be put on the spot.
  3. Sales and Marketing should know the status of any features or bug fixes promised so they can communicate to the Prospect or Customer.


Challenges

Things get out of sync with some many pieces that comprise a release. Code, docs, diagrams, how-to, support articles; there is just a lot of things flying around. How to know what is current and what is old?
Deliverable have dependencies. For the product to have accurate documentation, the product needs to be tested and the results documented. You need the software in a good state before you can do that.

Solution: Release everything, release often. Version the code, and everything around it. 

Versions are a great way of indicating when something was last edited, or just approved as still being the most relevant piece of information about the current state of reality.

A Product release strategy : Start in the center with a development release and work your way out. To get to the customer you need to get through the company.



Inner circle: Develop and test to get to a release candidate
Second circle: Company takes the development release and turns that candidate into a product release
Outside circle: Customer and prospects use the release, and the cycle continues.

Development

Lets start in the inner circle, and get that to a state that can be consumed by the 'company' circle. That will provide a solid foundation for the company to release that and everything supporting it for the customers to consume it.

Development releases everything from debug to testing to production, and validate each step

  • use branches to isolate a release candidate, pick fixes for mainline/master. Pick a time for a 'feature freeze' and work on making the master branch solid
    • Version: major.minor.dot = features.tech.fixes
  • Start release notes
    • These are all the issues in the release done lane on your kanban/agile/whatever board
  • Smoke testing try to catch exceptions, crashes
    • automated functional tests speed up this step. Always try to automate more each release
  • Regression testing 
    • Are specific features working from previous releases?
  • End user testing 
    • UX flows with jobs to be done test all the features together

Outputs for Development Release

Internal release notes (audience is product, arch, programmers, engineers)

  1. known issues - in release note
  2. how-to use new features - in help doc
  3. Configuration for deployment
    1. changed properties, added or deleted properties
    2. Dependencies - package code dependencies, client lib for system dependencies
    3. Environment - run certain process or application as a precondition
  4. Architecture quality model, 4C's diagram. 
  5. Programming notes (code standards, patterns, guidelines, review process)
  6. Project: team dependencies
  7. Engineering measurements
    1. feature usage, performance, scale-ability,
    2. cost of operations

Demo all this to the development team with latest version. This step is the final before releasing to the company.

Product

The internal product checkpoint reviews the metrics

  • personas - problems defined and prioritized. What have we learned since our last release?
    • feature usage since last release and collected feedback
    • The key personas using the product
    • The key personas buying the product
  • market needs, competitive analysis
  • ideas, feature requests
  • traffic, conversions,
  • The view from the customer
    • A demo script/playbook with one user persona and one buyer persona represented
    • The external messaging on the marketing website
    • The Product/Major features on the internal pricing list
Demo to stakeholders (internal team, external sales) deploy to demo site. Also, releasing the Story for Sales and Marketing turns out to be really valuable. They had a deadline and had to make decisions on priority for the roadmap; so they had to commit to the priorities for the next 30 days. This aligned the development planning accordingly.

Product release notes

External release notes (audience is buyers and users)
  • summary for buyer (reflect on marketing site)
  • summary for user (user and setup guides)
  • completed features and fixed issues from development release notes
  • one-pager for sales and marketing

Goal setting for next release

Road-maps are part of a release! Define and publish monthly goals, and prioritize top 3 themes for team
  • 30/60/90 road-map of upcoming features and fixes
  • UX prototypes and story map from design sprint
  • publish top 5 priorities for app
  • publish top 5 priorities for team
Review ideas and problems on a regularly scheduled basis, but only talk vision stuff on those days, keep notes between the meetings to keep the focus on fulfilling the vision; and less time spent on rehashing it. Sometimes these big efforts fail because people are just tired of talking about them.

So, have a vision up front, but don't formalize it. Give some compliance guidelines in a quality model (aka non-functional story backlog) and let the design decisions during the iterations do the formalization when it is needed.

The Overall Story shouldn’t change too much; if you are totally re-writing the Story with a major release, that would be a pivot… but changes will happen as more is learned about the market/customer.

Wrap it up

This may seem like a lot, but it is what's required. The first time you do this it will be a lot, but each time after will be substantially easier. The benefits from this organization efficiency will show quickly and get better over time.

Get it out: Each released version just needs to be more coherent than the last. not finished, just better


Thursday, January 10, 2019

Conversational software development does not scale

EDIT Spring 2023: 
The term 'conversational development' I use here is about designing software over a meeting and not writing anything down. The 'conversational programming' with AI tools that are now the rage is a separate concept; this uses AI to help program. 
To that end; using AI as the design assistant, where 'conversational design' with an AI bot will be your documentation and diagramming assistant. That will be a useful companion in the goal of producing a coherent design.

Designing over a zoom meeting.....


When building a small piece of technology, or just dinner; it's easy to just 'talk it out'. If you have skilled people, or you are one yourself, you can understand the design of something small by having a conversation, and building a mental model in your head of what you are going to make.
The smarter the people are in the conversation, the more you can do. You and the people you are working with can grasp the intentions of the conversation, and together have a common understanding of what is going to be made.

Mental models 

This can work up until the mental model that your group are forming becomes too large to have an accurate common understanding of what it is. Mis-understanding form quickly, and can be tackled by more conversations in a meeting, but eventually the complexity will overwhelm the group, no matter how smart the participant are. The smarter the participants are can add to the trouble; people will assume they understand each other because they are used to dealing with large abstract structures.
What can be made successfully with conversational development techniques? Dinner, re-decorating a room, and maybe a birdhouse. Something small, or has been done before by the people involved can be repeated because the situation and the parameters around it are understood and done before.

A bit of Design

While small projects can be made with conversation, what would happen if we defined what we were making in a more formalized way? If we wrote or followed a recipe for dinner, created a blueprint with defined dimensions of the birdhouse or sketched a mock-up of the interior design? The quality of the result would improve substantially.

What about building larger projects? Applying conversational techniques to build a building would result in that building falling down? It's doubtful the building would reach completion at all. Can you imagine that work site? "Ok, Joe, make a wall over there and another over there, and when thats done line up the top of the walls so we have a roof. We can figure out the second floor when we get there". This was how building were made about 300 years ago, and as a result there were no buildings or complex structures; the process just didn't scale. Since then we have developed formalized architecture to make plan we know can be built, and developed engineering techniques to validate that plan and verify that those plans are followed while building the structure.

Design in Software

Building software is an abstract process. You don't get to see it evolve like building architecture, so to understand it you need to describe it; and in doing so you and the people describing it end up making mental models of what it does and 'talk it out'.
The visual aspects of software are catching up to this reality and have evolved to making mock-ups and sometimes even 'flows' and 'experience' design artifacts that explain the visual reality. This is big improvement over conversations but falls short of a full design because of the reality of software. The UI is about 10-15% of what the application does. Some software is more or less, but there can be the case of a well detailed mock-up for the visual 15% and no design at all for the remaining 85%. That's a big gap, and that gap will be noticeable in the final product. It might look good, but it "doesn't work".

Why does this not work as a complete design? The same way a mock-up for decorating a room is not the reference for creating the building itself. There has to be an architect-ed design that is validated by engineering to make the larger structure; even if the complexity involved seems minimal at first. It makes a better structure.

So, how to do this in software? The mental models that you make in your head and collaborate with the group need to be defined and not verbally. Things need to be written down, and measured. How the user moved through the application can be formalized in a diagram. How the containers in the application work together and how the components within them interact to fulfill their responsibilities can be formalized. This technique can scale as it does for skyscrapers.

Write it down

You probably aren't making a skyscraper, but you are probably are building something bigger than a birdhouse. Give your software project a chance to scale by formalizing your design and the intentions of the solution as you understand it yourself, and in the group. Evolve that design, version it and grow it on paper and use conversation to validate and verify that design instead of continually defining it from scratch in your collective heads. The conversion will build on and grow the formalized design to enable that large scale project you all have envisioned in your head.

Tuesday, January 08, 2019

Undisciplined refactoring can create more spaghetti code


When finding a blob of code that is hard to comprehend, it is tempting to just dig in and break apart the blob into an number of seemingly related functions. Functional decomposition is a powerful concept in math and computer science. It breaks down a large and sometimes incomprehensible solution into one that is easier to understand; and as a result provides a more stable solution.

In the heat of the moment this can make a lot of sense by reading the code and breaking it apart for what it currently does and perhaps not what its intended to do. What this can lead to a more convoluted code-base in the name of refactoring.
Its easy to cut these corners when you are under pressure to fix a ticket, and/or tired from a long week. In the rush to fix be careful to see the larger design picture so you don't end up pushing that work further into the future.

So, instead of analysing what the software does at the moment, take a minute to understand what it is supposed to do. This is sometimes referred to as 'first principles' and is key to really understanding the problem; and as a result create a nice solution. How then would we do that in programming? By taking a proven problem solving approach that is used to solve any applied mathematics problem.

How to solve it

Remember our fundamental "How to solve it" steps. Understand, plan, execute, review. For seemingly easy fixes it can seen as overkill to keep to skip the first two steps and understand (design) on the fly while we execute (programming) on the solution. Stick to the more disciplined approach of moving through the steps as a simple checklist so you don't have to keep everything in your head.
This isn't a checklist to go through once, its really a mechanism to break down larger problem and build the solutions accordingly.

A moment to understand the Design

Are the intentions of the feature defined? If not, write down what value this feature is supposed to deliver. The features that implement this are defined by their responsibilities, not necessarily what its doing at the moment.
Take a moment to understand not what the code is doing, but what the responsibilities of the containing object or function has. Understand these responsibilities into an interface or better formed class, and then re-factor to fulfill those responsibilities.
Understanding the actual responsibilities can take some time, but not much; and you do have the time to get this into a coherent structure so you don't have to revisit it later. Draw a diagram and link the dependencies, inputs and the outputs to get a solid understanding of what this is supposed to do.

How it fits in. Whats the plan?

How does this class fit into the bigger picture? A good strategy is to use the 4 c model to see where your class or function fits into the component you are modifying, and if that 'fits in' to the intended functionality. Good architecture fits in, and doesn't stand out.

A Simple Example

Ok, enough talk-talk; lets go through an example. Lets say we are building a feature that lets users login to our fancy web app with social accounts. To 'make it work' and experimenting with the API we might end up with something like:


# using authomatic library and skipping code segments for sake of example
def get_user(network):
    email = None
    if (network == 'linkedin'):
        url = 'https://api.linkedin.com/v1/people/~?format=json'
        response_data = authomatic.login(self, network)
        first_name = response_data.get('firstName')
        last_name = response_data.get('lastName')
        email = response_data.get('emailAddress')
    elif (network == 'twitter'):
        url = 'https://api.twitter.com/1.1/statuses/user.json'
        response_data = authomatic.login(self, network)
        email = response_data.get('email')
    elif (network == 'facebook'):
        url = 'https://api.facebook.com/v2.12/'+result.user.id+'/me?fields=id,name,email,picture'
        response_data = authomatic.login(self, network)
        email = response_data.get('email')

    # we have a user class that requires an email in the constructor
    return User(email)


# 'refactor' attempt to smaller functions
def get_facebook_user():
    email = None
    url = 'https://api.facebook.com/v2.12/'+result.user.id+'/me?fields=id,name,email,picture'
    response_data = authomatic.login(self, network)
    email = response_data.get('email')
    return User(email)

def get_linkedin_user():
    url = 'https://api.linkedin.com/v1/people/~?format=json'
    response_data = authomatic.login(self, network)
    first_name = response_data.get('firstName')
    last_name = response_data.get('lastName')
    email = response_data.get('emailAddress')
    return User(email)

def get_twitter_user():
    url = 'https://api.linkedin.com/v1/people/~?format=json'
    response_data = authomatic.login(self, network)
    first_name = response_data.get('firstName')
    last_name = response_data.get('lastName')
    email = response_data.get('emailAddress')
    return User(email)


def get_user(network)
    if (network == 'linkedin'):
        return get_linkedin_user()
    elif (network == 'twitter'):
        return get_twitter_user()
    elif (network == 'facebook'):
        return get_facebook_user()


This is maybe a bit better, but to modify this to get more data from the api or to add error handling, the functions just expand as you add more logic. This is also only able to work when the app is running, how do I test this?
Let's take a moment and understand what we are trying to do:


  1. The application needs a User object to identify who is logged in
  2. The application can authenticate with an external identity provider
The responsibilities of this object then become: Authenticate, build User, handle errors, so lets refactor to a class that assumes these responsibilities, and no matter the network we use, we end needing these responsibilities fulfilled.

Even better, use an abstract class or interface to define the behavior. Objects have responsibilities and those responsibilities are fulfilled by behavior

The interface defines the behavior, so test and check for exceptions from logical methods that are defined in an interface. This is the main contract that the component has. (should have...of course. So much code doesn't have any structure)


class SocialAuthenticator(object):
    @abc.abstractmethod
    def authenticate(self, user_data, response_data):
        pass 
    @abc.abstractmethod
    def build_user(self, data):
        pass 
    @abc.abstractmethod    
    def get_api_data(self):
        pass

# So for any network you are using, you just extend with its own logic
class TwitterAuthenticator(SocialAuthenticator):

    def authenticate(self):
        return authomatic.login(self, 'twitter')
    
    def build_user(self, result):
        response = self.authenticate()
        email = response.get('emailAddress')
        return User(email)
        
    def get_api_data(self, result):
        url = 'https://api.twitter.com/1.1/statuses/user_timeline.json'
        response = result.provider.access(url, {'count': 5})
        return response

# This structure is more testable, so you can get the response data, and save it as a json file, and use that to test the creation of your users
import unittest
class test_authentication(unittest.TestCase):

    def get_twitter_data():
        #load and return file that would be reponse from twitter

    def test_create_user(self):
        auth = TwitterAuthenticator()
        self.assertNone(auth)
        test_result = get_twitter_data()
        user = auth.build_user(test_result)
        self.assertEquals(user.email, 'mytestaccount@whateves.com')

    #subsequent tests for creating users from different identity providers

Now we have a structure we can test without running the application, and can extend to get more data from each network. Also, we can just make a new class for any new networks we want authentication and user data from.

Wrap it up

By taking a pause and defining the responsibility of the object we were able to better understand the problem we had, which enabled a coherent plan for the execution of the solution. This results in a more coherent design which is much more easily understood, and also testable since we have an interface (or abstract class) to test against.


https://en.wikipedia.org/wiki/Decomposition_(computer_science)