Using Pydantic models instead of dictionaries increases maintainability in Python apps

The dictionary is a fundamental data structure in python and the uses of them occur often. We get parameters from a request to an endpoint, responses from APIs and internally have functions and methods to pass data to. The dictionary works in many of these cases and is easy to create. The ease of this creation comes at a cost of maintainability and clarity of the intentions and responsibilities of any data object.
In this post I outline the advantages of using pydantic to represent data internally and when integrating with external sources.

 

Objects have responsibilities

For dictionaries, you have to deal with the mental overhead of what the data is, where it came from and what it is used for.
Improved code readability

Pydantic models are explicitly defined, making it clear what data is expected and what types are required. This improves code readability, as the structure and constraints of the data are evident from the model definition.

Dataclasses and dictionaries can become unwieldy and hard to read, especially when dealing with complex data structures.

Stronger typing and validation

Pydantic models provide strong typing and validation out of the box. When you define a pydantic model, you specify the types of each attribute, and Pydantic will automatically validate the data at runtime. This ensures that your data conforms to the expected structure and types, reducing the likelihood of errors. It is also easier to add complex validation and keep it in the scope of the object

When using mypy for checking type safety and maintainability, the explicit types and nameing allow more coherent code and easier collaboration.

In contrast, dataclasses and dictionaries do not provide built-in validation. While you can add validation manually, it's easy to forget or make mistakes. Pydantic's automatic validation helps! I guess the case can be made that dataclasses and Basemodel are on par with the intention of the responsibilities through the class naming, the easier syntax in pydantic is easier to read

Example:

https://gist.github.com/jseller/65be3a2b30adc749496b6854dee097fc

Conclusion

For quality code that is easier to understand (in your own head and in PRs) try using the BaseModel of a dictionary and see the differences in the quality of your code. Dataclasses provided a big improvement in the developer experience but pydantic has taken this a step further.

Objects have responsibilities and behaviour: using a specific object that fulfils the responsibility of the object. Data objects are responsible for their attributes and validation of them. Full Stop! 

Use logical classes to use these data objects and maintain a cleaner separation of responsibilities. When unit testing the behaviour of using the logical object, the higher quality of the data objects used will show.

Portable code enables a smooth flow to production for the product

What is portability? The application or platform you are currently developing needs to work on your machine but also in the production environment where all the users are using it. The measure of Portability is the ability of the application to behave predictably in all environments. 

In this article, I talk about portable configuration and using code branching effectively with your team when developing software products. By adding some DORA metrics to your reporting, it's easy to quantify the effect portability has on your overall quality

why do I need to do this? 

The reliability of the software increases when it works in production without modification, so this effort starts when developing on your local machine. When developing an application there is going to be a need to build the app locally and deploy it to your production environment for the users to use the app. There could be testing, staging, and other environments along the way. Enabling some good practices for the configuration of the app makes developing and testing a lot easier and reduces the stress of deployment.

Keys to success:

• A new person should be able to pull, run tests, start app with just .local.env, and can hook into secrets in a vault when in dev, staging and production environments
• Keep environment config versions and use the environment to use a specific file.

how do I do this?

The application code shouldn’t change, the configuration of the code is what changes. This configuration is unique to each environment (local, dev, staging, production). Reduce the complexity of moving to production removes barriers and increases velocity

• Collaborative development with PRs
• The sooner the smallest architecture changes are established, the easier the validation in the build process. The maintainability of the codebase is fundamental to portable software
• Configuration of the System. 
• By deploying to the integration environment and testing the system, you can gain confidence that the system will work the same way in production
• use env files and keep secrets safe in cloud storage
• Configuration of the product
• Use Feature Flagging to limit access to incomplete functionality

• Align on the product level for releasing the user-facing feature. Release internally first so docs and support know what the customer is getting. 
• https://jseller.blogspot.com/2019/01/software-releases-are-lot-more-than-code.html

Branches of Code

How to integrate my work with the code from other developer teams? Use Trunk-based development for collaboration and velocity. This process allows integration of PRs and rapid deployment of the outcomes.

What is the trunk? it is the main branch and should match the deployed branch in the production environment. When changes are in the main branch, expect those changes to work immediately in staging and production environments.

• Ensures smaller changes
• features that aren’t ‘ready’, that don’t fulfill the full intention of the feature and don’t fully solve the user problem don’t need to be in a separate branch,
• Use feature flags to only allow users that have the alpha, beta or GA access, but remove them and tie them to the license source of truth for access and usage stats to understand the impact (and usefulness) to the user
• sometimes an alpha version is required to show progress and demoing the functionality through the alpha-beta-GA cycle 
• Prevents over-engineering through smaller changes
• Favours rebasing changes on the main branch instead of large merges. This keeps the history cleaner on the main branch and makes rollbacks much easier if needed. 

Feature Branches

• building a feature in a separate branch sounds like a good idea, but incurs a lot of maintenance effort when keeping the branch up to date
• frequent updating and merging
• merge conflicts

Release branches

• When a deployment to production is complete, the semver of the release is known. It can ease the transition to trunk-based development to create a Release branch to handle any hotfixes made to the production system between releaseses. This allows small changes in prod so users aren’t blocked by using the product while the large feature branches are merged and validated. 
• the same change is applied to the main branch with a test so the main branch has the changes integrated already for the next deploy

More on specific git commands here 

https://jseller.blogspot.com/2020/03/source-control-saves-time-and-complexity.html

Automation

Use automation to validate this portability and test the observability at the same time.

• Use a certified CI/CD for package and dependency validation

Use pipelines in bitbucket or github to automatically deploy your main branch to a development server, for larger production deployments there is Harness and other tools that can deploy to many nodes to reduce the complexity of setting up many production instances.

• Run tests appropriate to the environment

1. local tests and mock data for local development
2. Integration tests and synthetic data for staging
3. Production smoke tests to validate

Use an automated test to add a quality step to your deployment scripts. The deployment is finished when it is validated, not just deployedValidate your observability when validating the deployment. This is essential to certify the system is up to date for deprecation or security fixes

Metrics to measure?

Good portability helps velocity and collaboration, but how does it do that? Use DORA metrics to understand how the portability is affecting deployments and rework

• Change Failure Rate
• Deployment frequency

https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance

Conclusion

No matter how careful you are about the portability of the app; things are going to happen when many people are working on the application. 

Adding quality through Portable patterns like code branching and automation will go a long way to ironing out the issues before deploying to production

Also see:

https://jseller.blogspot.com/2019/01/feature-definition-with-quality-model.html

https://jseller.blogspot.com/2020/03/packaging-code-for-development-and.html

https://dora.dev/capabilities/trunk-based-development/

Use functional programming concepts to increase quality when re-factoring

Sometimes when you want to implement a new feature to a system the developer needs to figure out if this is new functionality, or a modification of the existing. How is this going to fit in, or is it new on its own? Chances are you have parts of the functionality there already but it be a mess. There are functions being called from other components and doing mostly the same thing, but in a slightly different way. This is a great opportunity to refactor this behaviour to one abstract type so you can add some automated tests, fix the design issue (without fixing everything) and leaving it in a better place than you found it.

By using some function programming basics your code can read and perform better than when you found it. By recognizing these patterns and the value they bring, you will find yourself recognizing this situation in many places and it will become easier to have smaller refactors that don’t slow you down on your current estimate, but give you the peace of mind that you will save some time down the road.

This will cover (IMHO) the basics of functional programming that benefit any codebase; iterative functions, OOP classes and any ball of mud you run into

Immutability

Immutable parameters don't change the value internally. use this value to create a new one
this is referred to as ‘side effects’

This is essential when doing concurrent programming. The data you put into that thread can’t have a hard reference (or any ref) to the global value.

By default, try to keep inputs immutable. Be efficient at copying the data you need and returning a new set (or reference to that set). You can enforce this in Java with the final keyword. In python, you can use tuples to pass arguments into functions (instead of lists)

Functional Composition

When you have many functions in a file, the callee has to know all of the details of the internals to be able to create the behaviour desired by combining the function calls in the correct sequence.

This really violates basic encapsulation and leaving this will cause some messy coupling between components and pieces of code will call specific functions. Easier to provide an interface that has specific behaviour. There is a parallel here with the Facade pattern in OOP, provide a simplified interface for the behaviour of the component and then refactor all the reference to that functionality to the new interface. This is a great re-factoring strategy, to add some functional design to your classes. Use abstract data types or interfaces to enforce the behaviour of the internals.

Higher-order functions

Using higher order functions in any language that you are using is really important. Why? because you end up making these functions anyway. So, you can create one of your own making with many loops and callbacks, or just use the built-in versions that come with many languages; or are available as a library to extend the languages base functionality.

The most common are using map, reduce, and filter

• Map - apply a function to all elements in the list. The list won’t change, but he values in the like will
• Filter - remove elements from the list that don’t satisfy a certain condition
• Reduce - apply a function to each element, and combine all results into one value

Immutability in these functions. 

When using map, the elements in the list will change, so these are mutable elements. For Filter and Reduce, you are reading the elements and producing a new output. With these immutable values, you could use partition the list, and use concurrency to run filter and reduce in parallel.

Monoid

This has an obscure name; but its really some very fundamental algebraic rules for designing functional behaviour. Monoids have properties and its a good idea to follow these as a guidelines when creating the functions that are used/applied in the higher order functions like map, reduce and filter. 

Closure - types inputs is type output. When you pass in integers, you have integers returned.

Identity - depending on operation, the base value for type

• for addition: 0  (1 + 0  = 1)
• multiplication: is 1  (1 * 5 = 5)
• concatenation is empty: "this" + "" = "this"

Associative

• a + b = b + a
• a * b = b * a

Recursion vs Iteration

Recursion can improve readability, but it comes at a cost of execution. you can exhaust the resources of your stack pretty easily when dealing with large datasets. Use iteration when possible for more predictable linear nature of the execution

Conclusion

By using some concepts from functional programming and being able to see these patterns emerge when refactoring code; the quality and consistency of the refactored result will increase in quality. This is from using tried and true concepts that, like many patterns, will just emerge from your work and add consistency so your teammates will be able to more easily read and maintain the code.

Maintainable codebases increase developer velocity by reducing complexity

Maintainability is a key Architecture Quality attribute of a well-constructed system. Maintainability is the quality attribute that helps other people on your team work on the same project and be able to add value without too much overhead to understand how it all fits together. Collaboration on defining and evolving these quality attributes will significantly affect the quality and velocity of your projects. 

Why is this important?

When you get on a project and see the code base, or just look at code you wrote 5 years ago. What is the feeling you get? Does it feel professional, or does it give you a feeling like nobody cared? 

Professional quality creates momentum in the team and quality practices become good habits. When teams are developing a product with the same codebase, teammates will bring their own opinions on best practices for code formatting and quality. Everyone has a lot of experience and curiosity, so it's good to use that knowledge to implement some simple practices to allow the team to move faster together, with more quality.

The quality outcome is teammates can make changes easily, and enhancements and fixes flow easily

What are the cures?

Code ownership 

It is the professional responsibility of the developer to produce the high-quality code that they can. This is important when developing on your own, but it is essential when developing on a team. The team owns the code and the upkeep of that quality becomes a matter of trust between team members. 

Adhere to common standards

The code has formatting and practices from the language itself. These patterns and practices become familiar and reduce the mental overhead of ‘figuring out’ the code when adding enhancements or fixing bugs.

Trust in teamwork

When you are on a pager-duty call at 3am to debug an incident. If the code is easy to read and understand, then the fix is easier to find. Do it for your teammate who is on call this week. This will also help Pull Request reviews as the code is easier to read, by our own convention. 

How do I do this effectively?

Agree on the context and goals and automate the tools to fulfill them. 

Agree on the concepts and goals, but don’t get lost in the details. If you have been in a code standards meeting, you might run into a situation where a developer is not only advocating to put a squiggly on a new line or leave it at the end. Presenting a higher-level goal of ‘commonly accepted practices’ and getting agreement on those will prevent you and your teammates from getting lost in the details. One thing I have noticed whenever implementing this policy is that the subject didn't return to the conversation. The discussion moved on to higher-level concerns

• Don’t sweat these details, there are more valuable problems to solve. 

Automate with github actions

With these checks running locally in every commit, the same commands are run to ensure your new branch integrates with the work on the main branch. 

Starting with some simple tools that implement maintainability checks, we can automate these actions to really enable development productivity. Automating small problems helps the developer concentrate on the bigger tasks. It always helps productivity; less time and is more consistent than manual checks. 

Automate all of this by adding consistent checks in the CI and CD process

• GitHub actions that do not allow merging until the checks pass
• Using git for doing the tasks
• git is an amazing piece of software engineering for source control and versioning. Also, it will run tasks or 'hooks' to run specific functionality before and/after a commit. 
• run pre commit for everything,
• use the same hook to run on the main branch of the repository when merging pull requests

Maintainable Python

Let us use a Python example to show what maintainable checks can be added to your developer workflow to take care of the maintainability and allow collaboration on a higher level

For this exercise, we are going to upgrade an open-source package with the tooling 

https://github.com/jseller/python-edi

For Python, we will use the following tools:

• black, flake8, mypy, bandit, vulture

1. Run locally with pre-commit
2. Run the same checks in our git host actions
3. Take a look at this PR as an example. https://github.com/jseller/python-edi/tree/upgrade
1. For adding this functionality to a project, I wouldn't do all of this at once, but split this up and run the tools file by file before committing to the whole project

Formatting and Linting with black and flake8

• Common code formatting and structure help team members understand each other's work. This results in a more effective PR that can be reviewed and merged easily.

For the first version, have a code formatting and lint tool. The output is much easier to review when the formatting doesn't need to be understood. Running a linter on your code will always give me more robust code and reduce complexity.

Type safety with mypy

For dynamic languages like Python and javascript, a type-checking action is really handy to increase code quality and make more robust functions and classes. Using type hints, a tool like mypy will check the validity of the variables being used to prevent TypeError exceptions. 

Security checks with bandit and vulture

Security holes and unused code can be scanned. 

Languages will naturally have conventions 

For JavaScript, we have TypeScript now that adds a lot of type safety over top of vanilla js. 

Running tools like Prettier and ES lint on legacy javascript code will bring that quality up to a higher level. 

Frameworks like React and Angular enforce conventions. Agree with that opinion and move on to focus on building value.

Code Structure

The programming language used to implement the system may use some common conventions, but it's still an application that can be decomposed into a set of features

Clean code that is common and familiar, code structured in files and directories that follow common conventions in structure and naming

Package by layer or feature?

When packaging application code for development, there is the application, and then there is all the artifacts around the application. Data files, unit tests, and different configurations are needed to build the app, and then there is the app code

package by feature

Many applications have a directory structure that reflects the implementation pattern of the system architecture and not so much the functionality it has. Directories like "Model, View, Controller" and "API, Data, Logic" or some variation of the components' structure. 

This gets complex as I would have to add and edit files to many different directories to add a feature. The new SearchCatalog feature would need some changes to a file in api, or a new file and so on with the different parts of the pattern. Not all features are implemented with the same pattern, so having a strict structure can cause issues.

Instead, try packaging by the feature itself. In a large system, this looser coupling within the files will help understand the feature as a whole

Let's make a feature called "SearchCatalog" and implement your pattern for the data models, views and logic classes that make the feature work. By better understanding the interfaces, this makes understanding the dependencies a little easier, and can also result in nice shared libraries used by many features.

Packaging and deployment

• Using common packaging has a lot to do with Portability, but the code that does those things needs to have some maintainable characteristics
• package app code
• check versions of dependencies, and automate dependency security checks (GitHub and GitLab do this automatically now)
• update when deprecation warnings are in the logs
• deployment as code with Terraform 
• shell scripting can get out of hand ‘to make it work’

Feature Flags

Using the pattern allows an easier addition of Feature Flags as there are fewer places to add them and all the code for a feature is in its own directory. As a result, provides a more elegant way for new code to live in the application. 

Design

Consistency in the implementation helps, but is there also maintainable design? There has always been an effort to standardize design with diagramming standards like UML and conventions like C4. The textual design has not found a decent standard that has been widely adopted. 

For easier collaboration with QA and documentation (if you have someone doing this or you are doing this yourself), link to specs that created the functionality RFCs and ADRs for design. Not sure that is worth the effort and as long as don't lose your architecture in Jira or an issue tracker

Conclusion

By establishing good habits and automating the application of those standards; a nice safety net is created that allows a flow of increased productivity. 

The team's culture is positively affected as the standards are agreed upon and evolved over time. More time is spent adding value to the product, instead of working the codebase.