dibiasi /dev/null

PREPRINT
Software Development in Small Teams

15. September 2024

Or: How to tackle big projects with a minimal number of hands

A little bit of background: This article reflects my experience as Tech Lead and de-facto software project manager at a small company. Startups and technology-heavy companies often try to punch above their weight. The company I am part of, is no exception. This article describes my experiences on how to tackle projects with small teams. For more credentials, click here.

Contents

  1. Problem Statement
  2. Proverbs (tl;dr)
  3. Explicit DONT'S
  4. Software Development Practices
  5. The Team
  6. Project Management
  7. Communication
  8. Lean Tech Stack
  9. Tooling
  10. Further Reading

Problem Statement

So you find yourself spearheading software development at a small company. Good job, you have made it. Let's say that you have got up to three other software engineers in your department. Now, one day the CEO of the company wants to introduce a new "feature". Nothing groundbreaking, just little functionality here and a few tweaks there. After some initial research, it turns out that this small feature is, in fact, a standalone service with its own database, integration tests, will be required to hit close to real time latency requirements, has a customer facing frontend, will require its own deployment, et cetera. If this sounds vaguely familiar to you: Congrats, this article might be just for you.

Proverbs (tl;dr)

Want a quick overview of my preferred approaches and got no time for a full article? Just read this.

Your role as "Lead" (whatever your actual job description might be), is to guide the software department through uncharted territories and already well known problematic waters. As you're already aware of, software development does not actually stop once the product is delivered. It needs to continuously be maintained and improved. So, software must be properly thought through and be maintainable for an indefinite amount of time. You never know how long your project will actually be used.

The following proverbs have been proven well in projects I have been involved in. I am strongly convinced that these rough rules form the basis for getting projects done properly:

Sounds simple and logical, right? Well, yes, that's because it is. That said, you've probably already worked with teams that did not abide by these (or similar) proverbs. Did they manage to produce a proper product in an acceptable amount of time, while keeping devs happy?

Explicit DONT'S

The Proverbs stated above are a very rough guideline for projects that are heading in the proper direction. There are also indications if the project or management style is flawed. If one or more points of the following list fit your projects description, you might be in for a bad time:

  • Projects have more non-technical staff than actual devs
  • It takes more time to "plan" (by non-engineers) and "get permission", than is spent on actual software development
  • Have an overly complicated architecture (most applications do not need to consist of more than 3 components [App / DB / UI])
  • You’re using blockchain technology
  • A non-technical person makes technical decisions
  • The team lead / project manager / software architect has not worked on software for many years (or never)
  • Management ignores pleas from technical staff
  • Poor project management
    • Waterfall model
    • Very large tasks (with no real approach on how to solve them)
    • Sequential tasks assigned far into the future
    • Aggressive application of SCRUM (more time spent on the process, as opposed to actual work)
  • No backups plans (staff and data)
  • Hire consultants from big players and think they are going to help you with technical decisions

Software Development Practices

Proverbs on how to guide your team through the software development process have already been stated, but what about some concrete 'hard' software rules? What about style guides for code and the best programming language for the job? As many things, these points highly depend on your team and objective. Some general software development best practices that are programming language agnostic:

  • Use feature branches The Gitlab Flow version control workflow should by read and understood by your team. Also have a look at branching workflows as described in the Pro Git book.
  • Have Development, Staging, Production Environments
  • Prioritize code quality and readability above all else.
  • Tag releases
  • Issue reference in commit messages (GitLab, Github)
  • Do not use a hammer when wanting to drive a screw
    • Use the right tool (language), for the right task (project)
    • But also don't adopt 100 different languages because you think they are the right tool
  • Don't reinvent the wheel, but also DO NOT hop on the latest trends and switch libraries every other month
  • Refactor and cleanup often (gradually build a better product)
  • Test what actually matters, coverage does not say a thing about test quality

The Team

All these guidelines and proverbs are well and good, but won't actually get work done, without the most important part of any project: The Team.

When trying to tackle big projects with small teams, a huge factor that determines your success will be the skills of your team. In stark contrast to large companies with armies of programmers to tackle a problem, you are probably going to have one to five software engineers. Very similar to RPGs, your team should consist of people that are generally well skilled and have niche skills in specific areas.

To once more reiterate the following point: A team without capable members, or members that are unwilling to learn is bound to fail. Having experienced teams where skill was a rarity, or highly held back by process / project management, trust me: If you find yourself in such a team, leave sooner rather than later, for your own sanity.

I am convinced that giving a team member meaningful tasks is key. That means devs must know why this feature is required, even if the feature is annoying or very difficult to implement.

Fields of Expertise

If you're reading this, you're probably a dev or lead of a small company. You will be expected to take on a broader range of responsibilities than engineers in larger companies. Being an ace of all spades is key, but not an easy thing. Make sure to organize your team well (have experts in different fields) and support them if they want to broaden their knowledge. Each developer has their own individual technical strengths and academical knowledge. Embrace their skills (e.g., CI/CD work) while keeping them informed about all other technologies in use.

Every developer should be able to fix major parts of the applications, even if he has not actively worked on this specific part for a while. Sharing project and field-specific knowledge is important to keep the team on the same page. If SHTF and a key team member is on holiday or sick, the team must be able to solve the problem at hand. This can be achieved by creating a knowledge base in the form of wikis (BookStack, GitLab, GitHub), readmes and other forms of knowledge bases (Zettelkasten, Obsidian).

"Free Friday" / "Creative Friday" projects have also proven to be a very effective tool to broaden and deepen the skills of teams. "Free Friday" projects means that the team is encouraged to spend time on non-business-critical projects, on specific days of the week. For example:

  • Team Member A has always wanted to look into FluxCI for GitOps.
  • Team Member B has some ideas on how to implement an embedded controller, to trigger the coffee machine over HTTP.
  • Team Member C wants to get better with Jupyter Notebooks and creates some plots regarding time spent on drinking coffee per project.

These projects might seem silly at first, but have two key positive aspects: Team members are kept happy, because they can work towards their own goal. Team members learn new technologies and are encouraged to present their work to other members at the end of their project.

Project Management

I'll start by saying the following: Project Management is not something you will start out doing. Especially for startups it is very common to have no project management in place at all. This is not inherently bad, as it keeps the initial development cycles quite lean. Eventually though, as more and more people join the team, project management will gain in importance.

Startups and companies of all sizes are often tempted to hop on to the latest trend. So, when buzwwords like SCRUM and Agile are thrown around, entrepreneurs and engineers alike will be tempted. Project management relies heavily on your team and project specifics. You will need different project management strategies when developing software for NASA, than when creating a cross-plattform calculator app.

The main objective is always the same: Teams need structure, projects need to be properly planned. The crucial point to keep in mind is the following: The project management process should not hinder actual work on the project. Take ideas from different approaches, i.e. story points and sprints, discuss them with your team and try working with them. If these ideas don't work for your team, rethink your process don't make your team fit the process.

With that out of the way, the following ideas have worked well for our team: Keep in mind that our team already works very well together. One main reason for this is that I've known my main development partner for a very long time and thus both of us know exactly what the other person is capable of. The chemistry in your team will be different.

  1. Developers Own the Product

Project management can be perfect, but the project will fail if the most essential part of the process is not motivated: the developer. Including the engineering team in the planning process is crucial. They will provide crucial input, that might have otherwise been missed. It also streamlines communication as they are not commanded from top down on what to do, but are actively participating in the planning.

Another crucial point is the following: If an engineer comes up with an idea on how to improve existing projects, or do something completely new, do your due diligence if this idea actually makes sense (see Free Friday projects above). If so, give the engineer authority to implement his idea. This, of course, has to fit into existing time constraints and deadlines, but will result in great software developed by happy engineers.

  2. The best meeting is no meeting

Have as few meetings as possible, period. Whenever meetings are actually necessary, they must be as concise and short as possible (talking shop). Attention spans degrade rapidly during meetings. If the most technical stuff is discussed last, the meeting has probably not been set up or moderated properly.

  3. Keep goals small

Avoid planning for a quarter of the year. If you decide to "sprint" make the sprint goals achievable. In our case, sprints are considered more of a todo list, but there is no rush to finish all tasks until the end of the sprint. We work on multiple wildly different projects at the same time, so having a quick and easy overview of what should be done next is key.

  4. Split / define work packages well

This depends on the seniority and skill of your developers. Defining a "silver bullet" ticket might not be a good idea, for new developers that might not be very familiar with your architecture and guidelines. Even for more experienced engineers, it makes sense to break up big tickets into smaller issues, which are all well-defined (and preferably independently testable).

Defining ticket scopes and estimating the time a ticket might take is a very hard problem to solve. Time estimations will be wrong, scopes will change. That said, plan tickets with field experts / a tech lead. When estimating time and effort / story points, the person that actually works on the issue should provide the most input.

  5. Have a clear big picture

Projects will inevitably get sidetracked, make sure every team member sees the big picture and knows which steps are needed to get there. It is also important to keep track of your issues / tasks. As already mentioned, tasks / issues / features / scopes will constantly change and shift. Make sure to identify mission-critical tasks and roadblocks early and account for them in your planning.

  6. DO NOT reinvent the wheel

If you're in the research phase of a new project and discover that well-established industry-standard software exists, it probably makes sense to go with the existing software (as long as all requirements are met).

  7. Reevaluate decisions

As mentioned numerous times, scopes will change, mistakes are going to happen. Don't be too proud to admit mistakes and / or pull the emergency break on a poorly implemented / planned project.

todo Please check the following parts of a markdown document for spelling and clarity.

Communication

Keep management in the loop. Software development can often be very "elusive" as non-engineering staff has absolutely no idea what the actual Linus developers do. Management most likely does not consist of elite software engineers. Make sure to communicate important issues in a timely manner and regularly reiterate why issue XYZ must be tackled. The following rule of thumb is quite true in my experience:

If you can explain to your mother what you're doing (say building a chat application), management can most likely follow and understand what you're working on. If you are unable to explain it to her (i.e. building a new authorization layer on top of OpenID), management won't understand either.

Software can be a bit similar to atomic radiation; it cannot be heard, touched or tasted. Thus, software is given little thought by non-technical staff. Make sure they know you're there for a reason.

A good communication skill to learn is to first listen, then talk. Listen to your Co-Engineers, higher ups and other staff. Software (and OPs) should make the lives of other employees easier, not overcomplicate them. Do not dismiss input from non-technical staff. They are the ones using your software every day. They are the first to encounter bugs or usability problems.

Un•gustl - Austrian slang for a pretentious despicable person Be humble, just in general. No amount of credentials or experience in any field justifies being a "Ungustl".

Lean Tech Stack

One of the main goals to strive for is simplicity. Yes, I know writing complex code with the latest templating magic, macros, inline loop and of course a turing machine in Regex seems to be alluring, but keep in mind: Complexity kills maintainability. If code is simple, it requires less cognitive resources by the developer that tries to work on it. Less complex code also has the advantage of being less prone to bugs.

As presented by Rob Pike in 2015 at dotGo:

  • Readable code is reliable code.
  • It's easier to understand.
  • It's easier to work on.
  • If it breaks, it's easier to fix.

The tech stack highly depends on the skillset of your engineers. If none of your engineers have proper knowledge on how to run and manage Linux servers, it is probably not the best idea to set up your own cluster on bare metal servers. Some consultant suggests you set up a Kubernetes cluster for a static website, with 100 daily visitors, because it is the hot new thing? Management talks about implementing AI to use in your very sophisticated barcode label printing program? Well, as previously already mentioned: Complexity kills maintainability. Making your tech stack or program artificially more complex will result in a worse end product that nobody wants to work on.

That said, leveraging tools from big players is a must for small teams. Firebase Authentication / Google Identity, GitLab CI/CD & Registries, et cetera, are all products that immensely help projects, without much associated cost. Keep in mind to not lock yourself in. Vendor lock-in Vendor lock-in Describes a situation where the customer is dependent on a vendor for products and unable to use another vendor without substantial switching costs. can pose a very real problem once your application starts to scale. Completely relying on serverless cloud computing code, that cannot be locally debugged, might not be the best idea for your bank transaction application with 10.000 requests a minute.

Let's get down to some actual technical recommendations. It is extremely important to start working towards the goal your application should achieve and not spend most of your time fighting the tooling. For small to medium scale projects (100–500.000 requests per hour) I prefer the following tech stack:

  • traefik (Routing / Load Balancing / TLS)
  • Docker in Swarm Mode (Orchestration + Scaling)
  • Keycloak (Authentication / Authorization)
  • GitLab CI/CD or GitHub Actions (Pipelines and Container Registries)
  • GoLang / .Net / Kotlin (Micro)services
    • Split along logical and application boundaries
    • i.e. User Service handles all user interactions vs Inventory Service handles all things inventory
    • Each service has its own project, CI/CD pipelines and DB / persistance layer
    • Just use a proper server language, interpreted langs have other places
  • Prometheus, Grafana, .. (Monitoring)
  • Jaeger OpenTelemetry (Telemetry, Tracing, Logs)
  • Professional multi instance hosting

The wonders of CI / CD

CI/CD pipelines can not only be used to deploy web services to vServers, but they can also be used for periodical schedules (run BI tasks), upload software to microservices (which in turn provide OTA) capabilities / notify customers about updates, run backups, et cetera.

Learn how to properly facilitate pipelines, and small development teams will cut their time spent on manual overhead at least by 80%. Automate everything, as long at it does not take up more time than it saves.

Automation xkcd 974

The General Problem - XKCD 974

If pipelines are set up properly, developers (with the correct permissions and proper review) can deploy from their IDE and waste less time on manual overhead. Reuse pipeline tasks with templates in other projects to keep overhead minimal.

With increased computational load required by more extensive CI/CD pipelines, premium plans (with more free compute) or self-hosted CI/CD runners will be required. Self-hosting has the additional advantage of being able to equip the runner's host machine with as much power as needed.

(Crazy) tales from the world of pipelines:

  • Test, cross compile ARM software on x64 runners and deploy directly to embedded hardware
  • Long running data mining tasks over night
  • Spin up entire clusters to run integration tests in
  • Set up a pipeline step to register REST routes with your identity provider

Tooling

Teams should have proper tools at their disposal. A carpenter cannot work without a proper hammer. The same goes for a software engineer, the quality of the end product will suffer, if no proper tools are supplied. Developers have their own preferences and will have strong opinions on what tools they want to use.

Automation xkcd 974

Real Programmers - XKCD 378

We've tried many tools and most often than not come back to the following tools:

  • GitLab / GitHub
  • IDE of developers' choice
    • Jetbrains products
    • VSCode
    • neovim
  • Stable Linux distro of choice
  • Docker / Podman
  • Keepass
  • Coffee ☕ ☕ ☕ (Seriously.)

If you're not familiar with Linux and don't know where to start, have a look at my post about software recommendations.

Closing Remarks

If you've made it this far, thanks for reading. I hope this article gave you enough knowledge to tackle big projects with a minimal number of hands. If you would like to comment, feel free to contact me through the channel of your choice. Also, I would like to thank my great team at Hoss Mobility. ♿

Further Reading