Use functional programming concepts to increase quality when re-factoring

Sometimes when you want to implement a new feature to a system the developer needs to figure out if this is new functionality, or a modification of the existing. How is this going to fit in, or is it new on its own? Chances are you have parts of the functionality there already but it be a mess. There are functions being called from other components and doing mostly the same thing, but in a slightly different way. This is a great opportunity to refactor this behaviour to one abstract type so you can add some automated tests, fix the design issue (without fixing everything) and leaving it in a better place than you found it.

By using some function programming basics your code can read and perform better than when you found it. By recognizing these patterns and the value they bring, you will find yourself recognizing this situation in many places and it will become easier to have smaller refactors that don’t slow you down on your current estimate, but give you the peace of mind that you will save some time down the road.

This will cover (IMHO) the basics of functional programming that benefit any codebase; iterative functions, OOP classes and any ball of mud you run into

Immutability

Immutable parameters don't change the value internally. use this value to create a new one
this is referred to as ‘side effects’

This is essential when doing concurrent programming. The data you put into that thread can’t have a hard reference (or any ref) to the global value.

By default, try to keep inputs immutable. Be efficient at copying the data you need and returning a new set (or reference to that set). You can enforce this in Java with the final keyword. In python, you can use tuples to pass arguments into functions (instead of lists)

Functional Composition

When you have many functions in a file, the callee has to know all of the details of the internals to be able to create the behaviour desired by combining the function calls in the correct sequence.

This really violates basic encapsulation and leaving this will cause some messy coupling between components and pieces of code will call specific functions. Easier to provide an interface that has specific behaviour. There is a parallel here with the Facade pattern in OOP, provide a simplified interface for the behaviour of the component and then refactor all the reference to that functionality to the new interface. This is a great re-factoring strategy, to add some functional design to your classes. Use abstract data types or interfaces to enforce the behaviour of the internals.

Higher-order functions

Using higher order functions in any language that you are using is really important. Why? because you end up making these functions anyway. So, you can create one of your own making with many loops and callbacks, or just use the built-in versions that come with many languages; or are available as a library to extend the languages base functionality.

The most common are using map, reduce, and filter

• Map - apply a function to all elements in the list. The list won’t change, but he values in the like will
• Filter - remove elements from the list that don’t satisfy a certain condition
• Reduce - apply a function to each element, and combine all results into one value

Immutability in these functions. 

When using map, the elements in the list will change, so these are mutable elements. For Filter and Reduce, you are reading the elements and producing a new output. With these immutable values, you could use partition the list, and use concurrency to run filter and reduce in parallel.

Monoid

This has an obscure name; but its really some very fundamental algebraic rules for designing functional behaviour. Monoids have properties and its a good idea to follow these as a guidelines when creating the functions that are used/applied in the higher order functions like map, reduce and filter. 

Closure - types inputs is type output. When you pass in integers, you have integers returned.

Identity - depending on operation, the base value for type

• for addition: 0  (1 + 0  = 1)
• multiplication: is 1  (1 * 5 = 5)
• concatenation is empty: "this" + "" = "this"

Associative

• a + b = b + a
• a * b = b * a

Recursion vs Iteration

Recursion can improve readability, but it comes at a cost of execution. you can exhaust the resources of your stack pretty easily when dealing with large datasets. Use iteration when possible for more predictable linear nature of the execution

Conclusion

By using some concepts from functional programming and being able to see these patterns emerge when refactoring code; the quality and consistency of the refactored result will increase in quality. This is from using tried and true concepts that, like many patterns, will just emerge from your work and add consistency so your teammates will be able to more easily read and maintain the code.

Maintainable codebases increase developer velocity by reducing complexity

Maintainability is a key Architecture Quality attribute of a well-constructed system. Maintainability is the quality attribute that helps other people on your team work on the same project and be able to add value without too much overhead to understand how it all fits together. Collaboration on defining and evolving these quality attributes will significantly affect the quality and velocity of your projects. 

Why is this important?

When you get on a project and see the code base, or just look at code you wrote 5 years ago. What is the feeling you get? Does it feel professional, or does it give you a feeling like nobody cared? 

Professional quality creates momentum in the team and quality practices become good habits. When teams are developing a product with the same codebase, teammates will bring their own opinions on best practices for code formatting and quality. Everyone has a lot of experience and curiosity, so it's good to use that knowledge to implement some simple practices to allow the team to move faster together, with more quality.

The quality outcome is teammates can make changes easily, and enhancements and fixes flow easily

What are the cures?

Code ownership 

It is the professional responsibility of the developer to produce the high-quality code that they can. This is important when developing on your own, but it is essential when developing on a team. The team owns the code and the upkeep of that quality becomes a matter of trust between team members. 

Adhere to common standards

The code has formatting and practices from the language itself. These patterns and practices become familiar and reduce the mental overhead of ‘figuring out’ the code when adding enhancements or fixing bugs.

Trust in teamwork

When you are on a pager-duty call at 3am to debug an incident. If the code is easy to read and understand, then the fix is easier to find. Do it for your teammate who is on call this week. This will also help Pull Request reviews as the code is easier to read, by our own convention. 

How do I do this effectively?

Agree on the context and goals and automate the tools to fulfill them. 

Agree on the concepts and goals, but don’t get lost in the details. If you have been in a code standards meeting, you might run into a situation where a developer is not only advocating to put a squiggly on a new line or leave it at the end. Presenting a higher-level goal of ‘commonly accepted practices’ and getting agreement on those will prevent you and your teammates from getting lost in the details. One thing I have noticed whenever implementing this policy is that the subject didn't return to the conversation. The discussion moved on to higher-level concerns

• Don’t sweat these details, there are more valuable problems to solve. 

Automate with github actions

With these checks running locally in every commit, the same commands are run to ensure your new branch integrates with the work on the main branch. 

Starting with some simple tools that implement maintainability checks, we can automate these actions to really enable development productivity. Automating small problems helps the developer concentrate on the bigger tasks. It always helps productivity; less time and is more consistent than manual checks. 

Automate all of this by adding consistent checks in the CI and CD process

• GitHub actions that do not allow merging until the checks pass
• Using git for doing the tasks
• git is an amazing piece of software engineering for source control and versioning. Also, it will run tasks or 'hooks' to run specific functionality before and/after a commit. 
• run pre commit for everything,
• use the same hook to run on the main branch of the repository when merging pull requests

Maintainable Python

Let us use a Python example to show what maintainable checks can be added to your developer workflow to take care of the maintainability and allow collaboration on a higher level

For this exercise, we are going to upgrade an open-source package with the tooling 

https://github.com/jseller/python-edi

For Python, we will use the following tools:

• black, flake8, mypy, bandit, vulture

1. Run locally with pre-commit
2. Run the same checks in our git host actions
3. Take a look at this PR as an example. https://github.com/jseller/python-edi/tree/upgrade
1. For adding this functionality to a project, I wouldn't do all of this at once, but split this up and run the tools file by file before committing to the whole project

Formatting and Linting with black and flake8

• Common code formatting and structure help team members understand each other's work. This results in a more effective PR that can be reviewed and merged easily.

For the first version, have a code formatting and lint tool. The output is much easier to review when the formatting doesn't need to be understood. Running a linter on your code will always give me more robust code and reduce complexity.

Type safety with mypy

For dynamic languages like Python and javascript, a type-checking action is really handy to increase code quality and make more robust functions and classes. Using type hints, a tool like mypy will check the validity of the variables being used to prevent TypeError exceptions. 

Security checks with bandit and vulture

Security holes and unused code can be scanned. 

Languages will naturally have conventions 

For JavaScript, we have TypeScript now that adds a lot of type safety over top of vanilla js. 

Running tools like Prettier and ES lint on legacy javascript code will bring that quality up to a higher level. 

Frameworks like React and Angular enforce conventions. Agree with that opinion and move on to focus on building value.

Code Structure

The programming language used to implement the system may use some common conventions, but it's still an application that can be decomposed into a set of features

Clean code that is common and familiar, code structured in files and directories that follow common conventions in structure and naming

Package by layer or feature?

When packaging application code for development, there is the application, and then there is all the artifacts around the application. Data files, unit tests, and different configurations are needed to build the app, and then there is the app code

package by feature

Many applications have a directory structure that reflects the implementation pattern of the system architecture and not so much the functionality it has. Directories like "Model, View, Controller" and "API, Data, Logic" or some variation of the components' structure. 

This gets complex as I would have to add and edit files to many different directories to add a feature. The new SearchCatalog feature would need some changes to a file in api, or a new file and so on with the different parts of the pattern. Not all features are implemented with the same pattern, so having a strict structure can cause issues.

Instead, try packaging by the feature itself. In a large system, this looser coupling within the files will help understand the feature as a whole

Let's make a feature called "SearchCatalog" and implement your pattern for the data models, views and logic classes that make the feature work. By better understanding the interfaces, this makes understanding the dependencies a little easier, and can also result in nice shared libraries used by many features.

Packaging and deployment

• Using common packaging has a lot to do with Portability, but the code that does those things needs to have some maintainable characteristics
• package app code
• check versions of dependencies, and automate dependency security checks (GitHub and GitLab do this automatically now)
• update when deprecation warnings are in the logs
• deployment as code with Terraform 
• shell scripting can get out of hand ‘to make it work’

Feature Flags

Using the pattern allows an easier addition of Feature Flags as there are fewer places to add them and all the code for a feature is in its own directory. As a result, provides a more elegant way for new code to live in the application. 

Design

Consistency in the implementation helps, but is there also maintainable design? There has always been an effort to standardize design with diagramming standards like UML and conventions like C4. The textual design has not found a decent standard that has been widely adopted. 

For easier collaboration with QA and documentation (if you have someone doing this or you are doing this yourself), link to specs that created the functionality RFCs and ADRs for design. Not sure that is worth the effort and as long as don't lose your architecture in Jira or an issue tracker

Conclusion

By establishing good habits and automating the application of those standards; a nice safety net is created that allows a flow of increased productivity. 

The team's culture is positively affected as the standards are agreed upon and evolved over time. More time is spent adding value to the product, instead of working the codebase. 

Consistent Software design enables quality and schedule predictabilty

All forms of technology have an architecture. If that design is intentional, it can enable velocity as its understood what is being made. if its organic and coincidental the vague plan will result in confusion, unpredictability and lower quality outcomes

Why software design?

A coherent design gives a clear understanding of the structure and behaviour of the system. How complete does this design need to be? For civil engineering, the blueprints that are required to build a structure are quite detailed. Early software design followed this expectation of detail, but it was found that created detailed designs before implementing the system really slowed the process. This became known as "Big Design Up Front" and was such an impediment to completing a project that the concepts of design were seen as slow and left behind for the sake of velocity.

Where did that lead us? We ended up making big balls of mud really quickly, and then took twice as long to debug this pile and put it into production

There is a happy medium in the middle here; where Just Enough design provides the clarity of the finished product, but doesn't get lost in the details. Its become clear that asking for a clear estimate (When this is done) is really problematic before knowing What is being made and How it gets built.

But we are in a hurry....

When creating products and keeping promises to your customers and yourself when it come to delivering these products. It natural to imagine you can produce more than the time and capabilities allow. Given the rush to deliver, the ‘status’ of the feature that you are delivering becomes the most important question. Whats blocking this from getting done? In many cases, the definition of WHAT you are making isn’t clear. With this not clear there is still more investigation, experimentation, and conversations that need to happen before its known what is being made.

With a clear picture of what is being made, the clarity of HOW that thing gets made is clearer. The dependencies get understood, and the complexities are known, so the planning of making that are known. At this point, the predictability of the plan becomes a little more solid. This can enable a decent estimation of the work.

What is produced?

An understanding of the behaviour and structure that satisfies the simplest requirement. This works on a whiteboard, and I have built many projects with a few people in a room with a whiteboard. This worked as long as the group was small and the deliverable was the groups alone.

As the product scope and resulting dependencies increased, this informal process didn't scale well. There were always a need for a meeting to clear things up. This gets even more unwieldily in the world of remote work and remote meetings. The need to write things down and operate from a common source of truth is clear.

By having a standard for design, the consistency of this will start to enable better decisions as the clarity of the context will become clearer for the implementation. 

Formalizing software design with a standardized template, but keep it really simple. Functional Specs, and ADRs to give the requirements a solid foundation that can be implemented. 

How can design be a lightweight process? By never finishing it, or having the expectation that the design is complete and a deliverable on its own. Its purpose is to clarify what the real deliverable is; the implemented feature that has value for the paying customer.

Who is involved?

You need the group to build something significant. Efficient architecture is the result of efficient group communication. Good communication enables velocity when the outputs are recorded and the context is known. Collaborate as a group. Silence probably denotes confusion, not consent or agreement; so this can take a few tries before people start talking. 

So what works?

Architecture meetups on the scrum team level. 

• This isn't a planning or status meeting, its a discovery meeting. The group needs to discover the value of this feature and the dependencies around it. How does this new piece of functionality fit into to the larger picture?

Working groups on the company level. 

• For the quality attributes that are cross-cutting concerns (Usability, Performance, Security, Maintainability) its a very efficient way to formalize some basic concepts and enable some consistency the technical roadmap.

In general; Writing things down and being objective about goals will align the group. Evolve your template for a technical spec, and keep removing as much detail as possible. Complexity in the documentation works for consulting, but its the enemy of efficient design. Keep it clean and revisit often.

Conclusion

When building a product and operating with velocity, there can be a tendency to not formalize the design for the sake of moving faster.

This can result in early wins for the demo of the implementation; but if the debt incurred does not get addressed the complexity will slow progress of the project.

Communicate and collaborate as group on What you are making. When this becomes clearer, the process of how that gets built will become more predictable. This predictability makes estimation of effort possible, and the next deadline will be less of a guess and more of a plan

Effective Software Architecture meetings

For an effective architecture strategy to develop; there needs to be some alignment when meeting together. In meetings with no agenda or focus it can seem to be a waste of time. It is. if the meeting invite is “we are going to brainstorm on everything” then meeting fatigue will set in and participants will have less engagement.

Its also essential to be clear that this isn’t a status meeting; the status is on the board, backlog and roadmap. Those are the outcomes of these meetings. The status should be visible without a meeting.

The common goal/output of the meeting is some decision on how to move forward with alignment of the participants. To set the theme and get an output of the technical meeting to be an effective one; its essential to understand the context of the meeting, and the participants.

Problems and Solutions

How to enable problem solving and know the valuable problem is being addressed? How to enable decision making so that we have a solution to work with and con move the understanding of the problem and our technical solution forward. Communication is a lot of listening and focusing on context

Focus on Context and Outputs

Meetings and the audience

For any meetings you have; take a moment to understand the Goal of the meeting, and from that it should be understood what you are working with here. Its also nice for any participants to know the context if they have some topics to bring up, or to be a listener in the background; or not participate at all

Discovery

Product discovery meetings are key to understanding user problems for features; and the deeper understanding of the technical constraints for this new feature.

These could be from ideas from business, dev, product, or triaged from a customer issue or suggestion; but the rationale for adding or modifying the architecture should be rooted from something that creates value in the product.

These output to the Roadmap level, product or technical; and sometimes need a prototype or more investigation to qualify the actual goal (and motivation).

Outputs

1. Valuable problems
2. Feature value
3. Market direction
4. Business Impact

Audience

• Product
• Sales
• Support
• Development
• QA

Planning

You have discovered the valuable problems and understood the value solving that problem will bring. Now its time to plan the technical solution; to creating the requirements and design for system features. 

User story mapping is really helpful as this stage as it uncovers the quality attributes to support this new feature. Also at this time trade-offs are made, so be mindful of any debt you are going to put on the books. Debt is useful, but like any debt it will overwhelm you if not kept in check.

As you go a long, you will learn more, so this meeting works well as a regularly scheduled meeting. In agile this is usually a "Backlog grooming"

Output

• Backlog items for functionality
• Technical spec
• Architecture Decision Records (ADR)

Audience

• Development
• QA
• Product

Implementation

The backlog has items for the functionality and the tech spec has the designs and decisions made. Lets build this thing! This is usually called a sprint onboarding, a kick-off meeting and gets the project people involved. This is when estimation can become possible with the details of what is being made now down on paper.

Outputs

• implementation Epics for some definition of done
• interfaces for implementation
• how requirements build acceptance criteria

Audience

• Devs
• QA
• project managers
• product managers

Release

Are we releasing work to support a feature, or is this a new one? Release meetings can take on many forms. and again depends on the context. A demo is an internal release and what is in that demo can be used to form the external release. For the technical architecture, the items can be lost in favour of the functional product notes; but its important to keep the architecture in-step with the release numbers you put on the deliverable

Version the architecture with the release

Output

• in the release notes, link to latest requirements that created that version
• Link docs describing features and acceptance criteria
• diagrams of structure
• mockups of UI and flows

Audience

• product
• QA
• devs
• devops
• Support
• Sales

When the meetings happen

There can be a tendency to get people involved, but having the wrong people can really hurt the output. Keep the focus to the group affected. For most architecture decisions its usually the team making the feature, when they plan to do the work. 

For architecture decisions on cross cutting quality attributes (Usability, Security, Performance, etc) its important to have a working group that focuses on setting the policy (a good checklist) for the functional features to get built on

Conclusion

When making the context clear and involving the relevant participants; architecture meetings can be a reliable way to output relevant decisions. Its essential to know What we are making so we can figure out how we make it and when it happens.

By taking a moment to understand the context and the problem to be addresses; the actual people that are needed to participate will become clearer and the chances of a meaningful solution will increase.

Micro-services pattern. good and bad.

Some patterns get a lot of press, some don’t. Microservices have enjoyed a good run so lets review the good and bad. Like any architecture challenge; using a pattern is really useful in the correct context, but seeing a pattern as a solution for all problems can be really problematic.

the good. 

Microservices can be taken on by a single team, scaled to handle a specific responsibility of the architecture. Services like Auth (auth and accounts), Notifications, Webhooks or other event handlers are usually good candidates to exist as a service

the bad. 

When not being pragmatic, the developer will apply a pattern for the sake of it, in the hopes of getting the benefits. This has led to extremely complex application, where the networking and added infrastructure can lead to slow and buggy applications

take-away. 

Be pragmatic; understand the problem you are solving and the pattern for implementing the solution will become apparent.

Focus on the functionality of the architecture and the other system attributes of performance, security, portability and maintainability will let you know when you want to split out a feature into its own service

A little history

The idea of decoupling functionality into distinct services has been around for about 20 years now. Web Services as a pattern started showing up as the networking and standardization of the protocols started to mature. 

Early networking was done with unique and proprietary protocols, like IIOP and CORBA. Once XML became an standard and the speeds of the infrastructure caught up, the basic setup of XML / HTTP became the common standard. This has now evolved into the JSON / HTTP we use mostly today. 

Smaller web services

Netflix was growing so fast that grouping teams to the functionality and making the deliverables from the teams into distinct services showed a lot promise. It seemed like a team of 5 could handle the functionality for a major software feature, and these big features can be deployed independently and they would communicate to form the product

From this success, the term ‘micro-services’ became a buzzword, and engineering departments felt the need to keep up and adopt this pattern.

Scaling a specific feature

You can always put your monolith app into a kubernetes setup and scale it up. For any service that in the API pattern, where its a stateless processor of requests; this should work fine. 

The trouble can start with other services that have more state and are used differently.

There has been some clear benefits to isolating specific features in the architecture.

Auth (auth and accounts) 

• this is a high traffic service that needs high quality. using a microservice to handle all the details around identity and permissions, and be able to see this scale. I have seen an auth service running in kubernetes for a very large e-commerce site. On the black friday shopping event, this had over 4200 nodes running!

Notifications 

• Sending out notifications, and dealing with the dependencies to do that work requires a lot of configuration and specific behanvior.

Webhook or other event handlers. 

• This type of service is handling a lot of incoming requests and need to have specific configuration and infra.

Too many services

When applying micro-services to any problem, there can be a real problem with the associated complexity. This is sort of the same affect of having many small required libraries in any framework. The whole thing starts to bloat and in the end you could end up wth a monolith of services.

Also, the dependencies between services and the teams developing them will produce a lot of overhead. This can really slow the velocity of an product cycle.

Performance of service based architecture can be improved on the networking side with protobuf and other lower level protocols, but the state management is always going to be a complex problem. How big of a challenge do you need this to be?

Let the problem determine an effective solution. 

Patterns emerge from the architecture, applying patterns for the sake of it just adds needless complexity. Focus on functionality that brings value to your product and as you break down the dependencies and understand your architecture the need to split a specific piece of that functionality into a container will become obvious. 

Being an effective developer in a company

Lessons learned when developing software in a company:

Add value

• Add value. Look to solve problems, not just build stuff. 
• Be Pragmatic.
• Be a professional; ego trips and flame wars have no lasting value, professionalism does

• Learn, there is always a better way.
• while(developing) { Have a personal process and refine it }
• keep a journal/log/todo at all times. keep a running queue or stack (depending on how you organize)

Be a good teammate

• don't try to keep everything in your head, put designs and thoughts in paper to formalize.
• Extremely high rate of return on information sharing. Collaborate.

• Utilize tools so you don't have to make new ones that do the same thing.

Specs

• Implement to spec. If so spec exists, document. The document is now the spec.
• Writing code is part of the solution, after the problem is understood.
• Write to the API before creating it, make it something you would want to use.

Quality

• Time to do it right. If the time doesn't exist now, when will it?

• Are you making good software, or just trying to make some software work? Making a quick patch just pushes the time to do it correctly into the future. Fixing will make the problem go away forever.

• There is always work to do in areas of performance and code clarity/quality. Schedule this as part of the time spent.

Complexity

• Aggressively attack complexity or it will catch up with you.
• Don't be too clever for your own good if clever adds complexity. Be clever in simplifying. Complexity will cause brittle code and crusty developers, avoid needless mental work in code.

What math is needed for software development?

We sometimes see the articles or comments that you don't need math to be a good programmer.
The thing is, software development has its roots in computer science, and cs is applied mathematics. So, it is true that you don't need math to do software development, it just makes it a lot easier if you do. How good do you want your work to be?

In the more broad spectrum of software development there is so much to do. Planning, designing, reviewing, etc. Do they have the design skills to make it look good? Do you have the empathy skills to know what the customer wanted in the first place? These don't seem related to math at all; but in pulling them all together to create a solution you would benefit from the problem solving skills that math provides

What actual math is needed for programming?

Computer Science is Applied Mathematics, so to be a computer scientist you would need a strong mathematical foundation. For making software, you don't have to be a full-blown computer scientist, but first you would have to use and understand the logic and data involved and how these two things work together to create your program.

One cannot do computer science well without being a good programmer, nor can one understand the underpinnings of computer science without a strong background in mathematics. Education must make students fluent speakers of mathematics and programming, and to expose them to both functional and imperative language paradigms so that they can effectively learn computer science.

Early programming courses and discrete mathematics will articulate the strong ties between mathematics and programming. Then the coursework should bridge the gap between the mathematical perspective and the implementation of an algorithm as a sequence of instructions through an imperative language.

I have thought of programming as largely a combination of set theory and predicate logic. Category theory may be a better way of going about the first part, as it is really set theory when combined with functions. I'm not sure if that replaces logic as much as it extends it. I'm really seeing more light in functional approach as the way to glue the concepts together.

How to get there

The first year programming course should not be viewed as computer science in it's entirety. It is a formal language and propositional logic course, which is a foundation aspect to CS, but doesn't represent the entire profession.

• set theory
• predicate logic
• combinatorics
• probability
• number theory
• algorithms
• algebra
• graph theory

• Understand sets and how regular algorithms apply to them
• Functions ,Transcendental functions, including trigonometric functions, logarithmic and exponential. Algebraic vectors. Combinatory logic is the root of lambda calculus 
• Computability and Turing style computer science is a bit at odds with formalism. It's a different philosophy of the nature of mathematics

Logic - first order logic http://en.wikipedia.org/wiki/First-order_theory theory of computation second order logic and computational complexity np complete also need to understand relations: unary, binary, ternary, n-ary important for iterations over sets with algorithms, check out STL style of applying functions. relational data

Numerical methods for solving simultaneous linear equations, roots of equations, eigenvalues and eigenvectors, numerical differentiation and integration, interpolation, solution of ordinary and partial differential equations, and curve fitting.

Wrap it up

Math is the language of technology and the computer science is applied mathematics. The more you know about these foundations allows you to make better software.