Rusting, a quick dabble with the Rust language

I have spent a few spare moments in the last week looking at Rust, a relative new language.

With the kids on Easter break and naturally requiring frequent attention, and a few brief actual paid client work, I have not had as long uninterrupted time focused on Rust as I would have liked. But I have managed to knock together some basic code.

Rustic Pizza

I created github.com/flurdy/rustic-pizza as my playground with rust and various web frameworks. It is very basic code and I expect I will be quite embarrassed by it in just a few months but it is a start. Rustic pizza will eventually contain several very basic web applications each in a different Rust web frameworks, all for ordering pizza. The pizzeria is a concept that I have created many times over in Java and Scala as an example for piece of framework or similar, some of which I have documented on my flurdy.com website.

Experience so far

So far I have found Rust interesting. 

It perhaps use a bit too much of lazy abbreviated names for its keywords, primitives and core methods than I feel is reasonable, but that is me being pedantic. The ownership transfers, lifetimes and everything by reference is a bit of headfuck, but I think I got the hang of it. Rust does also have some of the monadic traits I am comfortable with from the Scala world, though its Futures seems to be somewhat in its infancy at the moment.

Install Rust

Rust can be installed in numerous ways, including via Brew for both Rust and Cargo, its build tool. But I recommend using Rustup, rustup.rs. Rustup lets you toggle between release and nightly version of Rust, and some frameworks require nightly builds.

Learning Rust

The main Rust website, rust-lang.org, is a good place to start to learn about Rust. Especially the "Rust book", https://doc.rust-lang.org/book/README.html, is a great source to learn about basic Rust concepts. Further reading at rustbyexample.com/index.html and aturon.github.io was very useful. As well as various blogs on Rust such as hoverbear.org and hermanradtke.com.

Once you start coding I also recommend the standard library documentation, doc.rust-lang.org/std/, to find out what methods are available, e.g. does option have map, and what is getOrElse called in rust (unwrap) etc.

Web frameworks

As I most of the time create web services or full webapps an important part of my interest is which web frameworks are available. Fortunately the Rust community also recognise the importance of this, so www.arewewebyet.org and github.com/flosse/rust-web-framework-comparison are great information on which frameworks and libraries are available with Rust.

This lead me to take a look at the newest kid on the block Rocket, rocket.rs. So far Rocket have worked very well for me.

For my Rustic Pizza I plan to also take a look at Iron, http://ironframework.io, as it is the most popular Rust web framework on github. Though it has not been updated as much lately compared to Rocket.

Hopefully if time allows I may get to look more in depth into Nickel, Conduit and Rustful as well.

Keep rusting?

Whether I will keep using Rust time will tell. Core Rust concepts such as ownership, references, memory management etc are not aspects I care to manually control and think about too much as I hope a compiler and garbage collector optimises and handles those for me. But I will keep at it so maybe Rust will accompany if not replace Scala as the main hammer in my toolbox some day.

Pair first, pair always, never occasionally

Pair first

Think pair programming first by making it default for all tasks, not the optional occasional pairing which is how most most places fake pairing.

Why pair

I have already ranted why pairing is so beneficial in "Pair up now". I have since worked in pair always teams for another two years and really can’t fathom the waste of not pairing in any future projects.

Don’t do optional pairing

Many places, and I used to work at a few, do optional pairing. They say they pair occasionally as and when the team members think they want to.

I call this fake pairing. As it does not happen very often, and is more like as a second opinion or help when someone gets stuck on a task for a short time. That is not what pairing is. Please read the blog post mentioned above what pairing actually means. (code quality, knowledge share, low bus factor, best practices, egalitarian technology, feature speed)

Optional pairing lets people find excuses to not pair even when they think the tasks may be more suitable as a pairing tasks. Or wait until people are available which is wasteful as well as context switch cost for the other temporary pair member.

Optional pairing is better than no pairing, but only slightly.

Default to pair

By making it default you avoid discussion if one should pair or not. It is less ambiguous and the team just gets on with the tasks straight away. You realise most tasks can be paired on.

You also then start to see the big benefits of everyone pairing over time. You will start to notice higher quality code, that less bugs creep through and the code start to be quite lean as unnecessary code does not even get written. And the flow of tasks is steady as there is less bottlenecks of people with sole knowledge of some part of the system as the knowledge is spread across the team

You will have a team that knows each other better, that really work as a team. A team that has shared their own ideas and niche knowledge of technology. The code base now also looks more homogeneous and evolved as the team has improved their abilities unilaterally.

Non pair tasks

Sure as mentioned in my previous post there are still tasks that does not need to paired on. But there are fewer of these than you think. Mostly research tasks etc can be done alone. Or a pair may not sit together whilst ‘googling’ separately for a bit.

If someone is working solo on a task it is important that these tasks are minimised and time boxed as they often become time sinks and over complicated solutions thought of.

Odd number of team members

Often though through holiday, sickness, meetings, team rotation etc there is an odd number of members in the team making pairing everyone mathematically impossible.

You could triple up on one task. It can be done, though often end up with too many cooks in the kitchen. There may be suitably encapsulated sub tasks that the third triple member can work on his/her own and sync with the others frequently instead.

You could let the odd team member pick up a normal task as an individual. This can also work but I recommend highly against it. Many times I have seen this just result in the problems pairing was meant to solve. They get too entrenched in the task and it takes longer than planned. Or get defensive about it. Or it gets over-engineered. Or they take shortcuts or bad technology choices that would not have happened if paired on.

If there are some obvious non pair tasks to pick up then it might be a suitable filler until pair rotation happens again or a new member becomes available. Tasks such as perhaps some rare support or monitoring, or evaluate some low priority PRs.

Or they can work on some fun but non priority technical debt. Such as automating wall displays, or support scripts or tools to simplify/automate a process. It provides a nice break from the other normal tasks, and also hopefully helps the team in the future and reduces paper cuts.

Don’t chicken out

So do not chicken out by agreeing only to pair when the task's problem is suitable. Instead always plan to pair unless it is a rare hands off research task or similar.

ROC theorem: Readable, Optimised and Correct code. Pick two.

ROC theorem

With database you have the well known CAP theorem. Consistency, Availability and Partition Tolerance. You can only have two. Databases have to make compromises between these pillars. You can not fully have all three.

With code you also have to decide on compromises between readable code, optimised code and correct code (ROC). And you can not have all three.

This often creates arguments between people on soap boxes from the various camps.

Correct code

Correct code, clever, terse, generic code that avoids handling a lot of edge cases. Often functional code that can be very elegant with little to no theoretic side effects. And easily composed as part of other code.

Correct code can be readable and fast, but also sometimes horrible to understand and very costly to train, write and maintain.

Optimised code

Optimised code, fast and scalable. No unnecessary cruft and takes short cuts to achieve the end results so performant from day one.

Scalable code, code you optimise to support horizontal scalable solutions, with little state and restartable.

May discourage typed system e.g. a message based Actor system, multiple layers of caching, or overuse of parallelised futures to avoid bottlenecks.

Optimised code can be "correct" code but often full of unclear and undocumented short cuts and frustratingly slow/buggy to develop.

Readable code

Readable code, simple to read and quick to understand, by people of different levels of skill. Easy to spot bugs and is maintainable by anyone. It is pragmatic in its approach and quick to develop. Can be terse if it is the most readable but often more verbose.

Readable can be "correct" as flow is easier to understand, but often not particularly performant and can be at more at risk of bugs due to more exposed code.

Not mutually exclusive

You can have all 3 pillars for some smaller sets of code. But not for whole code bases and at a cost for how much.

This is more about the priority and focus of the code you write.

• Will others work on the code base, today, next week, next year? Then readable is important.
• Is multiple people working on the same part of the code base? From different teams? With varied experience, or even just different locations? Then readable is important.
• Is the solution used by millions? Does big O make any difference? Then consider optimised code. Note: very, very few companies/products actually need this.
• Is a bottleneck been proven in production? Then optimised code is valid. But not necessarily across the code base.
• Is the product business critical? With heavy integration dependants? Then correct code may be a priority.
• Is the team highly skilled? Not that large, and low churn? Then correct code is an option.
• Do you pair program 100%, from day 1 of onboarding? Then correct code is an option.

With unlimited time to implement and continuously heavy training, and therefore a lot money, you may achieve higher levels of 2 of them or even all 3. But that is not realistic. Pick your priorities. These are not mutually exclusive but they are at a cost of each other.

All the ROCs

You may detect my preferences. I prefer very simple and readable code, that is functional, and scalable. In that order.

I like that anyone can pick up and work on a task for most parts of a code base. I subscribe to the idea of frequent pair rotation across tasks and systems to make sure multiple people is aware of and had an input into any part of an architecture. That leads towards readable code so the overhead of swapping is low.

That a new member of our team or from another team in 6 months time can easily contribute to "our" code base for a small pull request without learning "our take" on category theory is valuable.

I prefer functional programming, with proper type checking, using functors and monads for composition. I like terse code that I can trust, but it must still be readable and maintainable by others than the original author(s).

So some overuse of higher kinded types, free monads, etc adds too much cruft for me, and risky recruitment demands. (At the moment, I am prone to evolve and may have completely changed my mind by the time you read this...)

Horizontally scalable, concurrent code is in the back of my mind of most of solutions. No state, using futures, REST principles etc are core to all my code.

But I detest premature optimisation. Only occasionally in my career have I had to modify any code to support some optimised flow. I do not work for Facebook/Twitter/Google (yet) but I have worked for financial, telcos, and games companies with enormous traffic, and still this was rarely a problem at code level.

I often spot potential optimisations and consciously say no, not yet, if it is not also the most readable and correct alternative. I even avoid parallelisation of futures if there is not yet any obvious need especially if it makes the code less readable.

Readable, Optimised and Correct code. ROC. Pick two. Or just rank them.

Contracting 101

Seems a large number of people I meet are interested in becoming a contractor/consultant/freelancer. So I thought I better write a up what I did and recommend. Note this is very UK / London centric.

Information 

For general information, tools, conversations and advice on contracting in the UK contractoruk.com and contractorcalculator.co.uk are very good sites. And a very dodgy contractor calculator... grid.flurdy.com/contractCalc

Forums

The forums at forums.contractoruk.com have been a great help to me. But prepare to be condescended and shot down if asking noob questions. 

Association

IPSE, formerly known as PCG, the major contractor and freelancer association in the UK. ipse.co.uk 

They will represent you if you have disputes with the HMRC of which they have a great track record. And they offer a range of services in general that will help you sleep at night. And discounts with Apple etc. Membership is a no brainer. You can use my referral code of IPSEMGM0921 if you want.

Insurance

You will need indemnity and public liability insurance and optionally others such as IR35 cover. QDos came recommended in the forums. Don’t under insure yourself but also don’t over insure. qdoscontractor.com

Umbrella

To soften the move to contracting world I joined an umbrella company initially. Basically the only paperwork you need to worry about then is the contract renewal. I was with contractorumbrella.com and they were great. 

But eventually I took the next step to become a limited company as Umbrella taxes do add up.

Accountants

As a limited company you need an accountant. I joined Crunch as they have web site that automate almost everything on top of friendly advice. crunch.co.uk So far they have been great. Please use referral eraybyfl to receive Amazon vouchers.

Finding contracts

To find contracts is the major challenge. If you have a great network where you turn down work, then marvelous. Network at meetups and conferences. Verbally sell yourself to former colleagues as they may not be aware you could help them with a problem they have at work. Get business cards from moo.com

Otherwise global project sites such as Elance/ODesk/Upwork can turn up some work, just remember to not bankrupt yourself. I have actually had the best responses when I quoted a lot higher than anyone else. 

Job sites such as jobserve.com, jobsite.co.uk, etc is the normal way to find contracts. Be aware a large chunk is simply cv fishing and not real job ads. 

Contract Recruiters

If you can go direct then great, but most of the time you will have to deal with recruiters, and contract recruiters are much worse than normal recruiters, which does say something. Especially in a numbers game location such as London. 

The major difference is for permie roles the recruiter will be better off if you agree a higher salary, whilst with contracts the recruiter/agency will be better off the lower they can get your rate. 

And never ever believe them requiring two references. That is only so that they can extend their contact list. 

Office Space

If you are not at a client I recommend using shared workspaces. I frequently use the 4 days a month for free IPSE members receive with Club Workspace club.workspacegroup.co.uk. I have also used neardesk.com and the more corporate regus.co.uk. 



Good luck


Ivar Abrahamsen
Benevolent Dictator / Technical Architect Consultant

Eray by Flurdy Ltd 
eray.uk | flurdy.com

Pair up now

I fully believe we can deliver faster projects, with better code, exposing less risk to the company and in the end be more profitable if team members pair program.

The world was fine before

The world and businesses worked for thousands of years, even decades since computers was invented before agile methodologies took over how projects and companies was run. Projects was delivered and companies didn’t go bust.

Workable products got shipped before testing was done thoroughly, before unit testing was common and TDD was even heard of.

At the same time many teams worked well, developers wrote good code, meeting deadlines and shipping products before pair programming was a thing.

Sort of..

Were we also fine before cars replaced horses, computers replaced pen and paper, modern medicine replaced witch doctors?

Before Agile projects was delivered often late, not meeting the updated requirements, and all the other benefits with Agile. But some was delivered on time with features as desired and to budget. Just fewer and a lot slower.

And products were shipped with many bugs over and over again, with costly fixes over and over again.

Why not pair?

Double the resource cost

Many companies and managers can not see past the issue of using two people on one problem. Basically doubling the resourcing cost per story/task/whatever.

And some (not many) are aware of the teachings of the Mythical Man Month that adding more resources to a problem is not a linear reduction in time to finish the problem.

My space - my own thing

Many people and developers are also not interested in working with others on a task, especially all the time. They feel their private space violated and interfering with their work habits. I/they/most people do like to think and experiment and solve a problem on their own. And be trusted to do so.

Being forced to sit and watch, to talk to another person about every step of solving a problem is a big change in work habits and social skill requirements.

So why pair program?

3 reasons: Code Quality, Knowledge Share and Feature Speed.

Code quality

Mini plan

If two people sit together to work on a task they are quite likely to discuss the issue and basically perform a mini up-front architecture analysis of the problem. They might even apply TDD approach to solving it. In essence it should deliver a slightly more thought out solution than simply jumping in the deep end and hacking out a solution.

Human rubber duck

Explaining the problem and solution to a human rubber duck also unveils the root cause and the better solution much quicker.

Less rabbit holes

Two people are less likely though not guaranteed to go down a completely wrong rabbit hole in solving the problem. And at least quicker to realise that they are and pull out out sooner.

No short cuts

If you sit with someone you are less likely to cut corners and take short cuts in your approach and implementation. This is a very good reason for why pair programming results in better code quality. You are for example less likely to skip writing tests if someone sits with you.

Less mistakes

Simple mistakes and syntax errors are quickly spotted and corrected. Human compiler and typo spotter is often what happens but that is not really a bad thing, just don’t be offended by it.

Clean readable code

By having two people agree on the implementation then the readability and approach to the code should be cleaner and more maintainable as you both have to agree and understand the delivered implementation. This basically negates the need for extensive code review process later on as covered in my "Pair with people you like and code reviews with people you don't like" blog post

Ninja code

A lone developer often so called ninjas may also try to write smart code. Code that none else can understand nor maintain when that person leaves. A pair would prevent writing code that is unnecessarily smart. And share understanding of elegant code that is actually useful.


In the end per finished feature the code is on average more likely to be of higher quality, with more tests, more maintainable and properly solves the initial problem.

Knowledge share

One obvious benefit of pair programming is that now two people know everything about that task. And two or more now know about the product / system that the task involves.

No single point of failure

This reduces project, product and company risks and costs a lot. If one person goes on holiday or is sick there is no stoppage in working on that task or using that system as the other pair half can continue on his own.

And worse if a person leaves the company or is hit by a bus there is no panic as multiple people know about that system or feature. Ideally if a person is off for longer period or permanently a new pair is formed quickly to remove the new temporary single point of failure threat.

Best practices and showing scars

Another good knowledge share benefit of pairing up is that the two people start to share best practices in approaches to implementation or even develop new ones. They share stories of previous scars of bad practices. This should make them both better developers.

Rotate pairs - spread knowledge

If you also continuously or with some frequency rotate the members of each pair you gain even more knowledge share.

Product and system knowledge is now shared across the entire team. The risk of loosing knowledge is reduced further.

Best practices are spread around the team. Members pick up new skills and enjoy sharing their experiences. The knowledge and quality increases, and team morale should also increase.

Feature speed

Initially the feature tasks and project speed may be slower if your team members pair program. In a mathematical sense it could be halved as it ties up two members per task. And the benefits of pair programming are often more longer term.

But some implementations will go quicker as less likely to waste time on the wrong rabbit holes etc.

Speed per feature != initial implementation

In the longer term a great increase in benefit is visible. If you count bugs, planning, etc as part of the actual total feature speed, a pair programmed feature is much more likely to be of higher quality and less likely to come back over and over again with bugs. And it is bug fixes that take time.

The original implementation is often a low fraction of the total effort and time cost per feature. As Olaf describes in his "Pair programming Economics" post, implementation is perhaps only 10% of the time spent on a task.

Clean code

In addition with good clean implementation the code is more maintainable making future extension and fixes much easier and quicker.

Less procrastination

There will also be less procrastination with dealing with issues, with talking to other people, etc as two people should drive each other on to finish a task.

Less wasted time 

When working on your own you may often spend a frustrating long time trying to figure out what is the actual problem, and how and where you should solve it. Sharing that with another person often cuts that waste down as it is quite likely one knows where to look already.

Quicker ramp up and less resource bottlenecks

If knowledge of a system of feature is already known by several team members if not all then starting a new related task can be ramped up quickly compared to starting on a feature or system you have never touched before. Shared knowledge also means there is no virtual bottleneck waiting for a person to be available to work on that issue if he is the only one that knows about it.

Not religious but no tricks 

Fake agile

Many companies and teams say they are Agile but in reality they are not. “Being Agile” is not a black and white thing, it is a grey scale from non agile black to an unreachable white. Most projects are unfortunately quick dark grey even if they think and say they are Agile. Simply just using Atlassian’s Jira, having a daily standup, even some sort of iteration does not make you Agile. It is a lot more of fully adopting the practices and continuously try to come more agile.

Fake pair programming

The same goes with pair programming. Many say they do some sort of pair programming. Simple occasionally sitting together for a little while is not enough when most of the time they work by themselves.

You need to commit to pair programming being the default convention for every tasks, for everyone. Set adaptable conventions on how to organise pairing. And when exceptions are acceptable, not the other way round.

Shared responsibility

Agile processes means not delegating and assigning tasks to individuals, but in the end team members pull tasks from the queue and make themselves responsible for that tasks. If they pair program that assignment is for both in the pair not one person. If it goes well, both get praised, if there is a problem both stand responsible. Similar to how the team shares praise and criticism.

Personal space and time

Another important step is to allow for personal space and time away from the other half of the pair.  I don’t believe 100% pairing from 9am to 5pm is a good thing, no need to be in each others arm pits all day. That would cause friction and stress on the team. There is no need to be religious about it. Its a convention not a rule.

Let people take breaks from each other. Much as Pomodoro Technique allows frequent breaks from work, pairs should take breaks from each other.

They could spend that time on researching the tasks, perform mini hacky experiments. Watching another person google around for an hour is not that productive nor fun.  Allowing people to catch up with IM chats and emails, make personal calls etc will allow for a more happy and productive pair when they pair up again.

Non pair tasks

Some tasks does not need to be paired up for. Simple research, monitoring, tiny typo fixes can be done by just one person. Having a nice balance of the majority of the time is spent on important paired tasks with a few smaller non paired tasks is probably a good idea.

Effective onboarding

Taking on new team members, recruiting junior members still have a great benefit from pair programming. Sarah Mei write a great article on the benefits of "Pairing with Junior Developers". Leaving new team members to plod along fixing bugs is definitely the wrong way to onboard them.

Who to pair

Probably out of scope and can cover an entire blog post but a tip is to mix pairs. Senior with juniors. Front end with back end focused skills. Testers with developers if appropriate. And also seniors with seniors.

Remote challenge

If the team is distributed pair programming does become a challenge. However it can still be achieved. Pragmatic Bookshelf have a great book on “Remote pairing” by Joe Kutner. And a lot of tools and advice are available.

Not for everyone but for most

As I have shown here, I believe the pair programming is essential in a modern project. Sure, not pairing has worked for a long time and will still work. And for some specific teams and people it is still fine not to pair.

But pairing works better with most teams. It reduces risk, it builds team culture, increases actual velocity, makes the team enjoy their work and in the end deliver much better products.

Just approach pair programming with some common sense. It is not a factory, but don’t trick yourself into a halfway house either.

Evolve or wither slowly

3 years ago (wow that went fast!) I wrote a blog post called Do not rewrite. The main points in it was to never rewrite whole applications. Never. Instead gradually rewrite one feature at the time whilst still delivering new business value.

I also wrote a more recent post about Paper cuts and broken windows. Which emphasises the importance of fixing problems straight away and not let them fester.

I want to extend these posts to include you should rewrite all the time, even when not immediately necessary.

Recap

Extracting the main points in the previous blogs why big rewrites are bad and continual rewrites are good, and to also sum the obvious why no writes at all are very bad.

Big rewrite disadvantages

• Cost
• No business feature - no value
• Risk
• Big deployments
• Forgotten features
• Likely to never finish
• Not fully replacing old system
• Support yet another system

Continual rewrite benefits

• Reduced risk
• Smaller delta
• Quicker feedback
• Modularising
• Faster - less bottlenecks
• Leaner
• Remove features and bloat
• Actually finishing

No rewrite disadvantages

• Death by thousand paper cuts
• Even the smallest change becomes slow and painful
• Broken windows
• People do not care if they introduce bugs or break other things
• Staff exodus
• Few wants to work with painful systems

Less obvious benefits

There are other perhaps unexpected but important benefits not raised in the previous post. 

Business domain knowledge

Every time a refactor or minor rewrite takes place the knowledge of that part is refreshed and stay current in the company. If a system is not touched for a long while the knowledge of it may be forgotten especially if staff involved has moved on. 

If a critical bug or urgent feature needs to be added to that system then the turn around is exponentially different between a recently updated system and one forgotten.

Technical knowledge

This is also true for the technical part of a system, the how, not just the what and why it does something. An old COBOL or complicated legacy custom build Java application will take a long time to work out compared to a contemporary tech stack application.  You might even have to hire new staff or expensive contractors to fix it.

With up to date knowledge it will be a lot less risk that they might not do it properly.

Quick turnaround

If a new feature needs to be added then to any system then getting people ramped up and delivering it will be much quicker if it is on a contemporary and well known tech stack. If a new system needs to be integrated with existing systems then the API integration will be smoother and a lot quicker.

Technical migration

If a mass migration of a technology stack is needed, perhaps migrating from in house data warehouse to a public cloud based provider, or moving from monolithic to horizontally auto scaled instances, then having most systems on a contemporary and probably quite similar technology stack will speed any urgent migrations. This will avoid/reduce the need to for many deep cave explorations of old customised mystery legacy applications.

Staff retention

A very important reason to rewrite applications and features and in general keep technology up to date is staff retention. (Obviously a million other reasons exists for staff churn as well).

If you avoid death by a thousand paper cuts and broken windows staff will not mind working on the products. A negative culture will be less likely.

If you migrate to newer technologies and let people frequently learn new stuff they will be much more interested in the work and less likely to want to move elsewhere. (refer to my blog post Peak interest - the learning and sharing curve).

Recruitment

If word spreads that you keep an up to date and interesting technology stack then hiring new members of staff will be a lot easier, and you will hopefully attract more qualified candidates. On boarding will also be quick and people will be up to speed quicker with newer well known technology.

Lean technology - lean organisation

If the applications are continually refactored and evolved (as long as it is not done in a hacky ninja rock star development fashion) the architectures will become modularised, scalable and leaner in general. 

Not necessarily a micro service architecture but a lean architecture that are more likely to be adaptable for the future.

Hopefully a lean architecture and highly skilled retained staff will also lead to / need a lean organisation so the business is more likely to succeed.

Rewrite when not needed

I would also suggest people and organisations refactor and rewrite when they do not see the obvious reasons for it.

I do not entirely mean rewriting applications on a whim when there is no features to be added or bugs to be fixed, after all I do emphasise the need for delivering business value along with all rewrites. 

But I do think even when the obvious pain is not there that you should try to refactor and update applications. For example if the application is just one or two version behind the latest but still seems fine then still update it. It will be less painful than when it next time is 3-4 version behind with a bigger delta of change.

Rewrite encapsulation

One type of no business value rewrite I would suggest is a good idea is to encapsulate legacy systems API as soon as you come across it.

Even if you are not changing a legacy system but merely reading data from for example, then adding a contemporary facade to it straight away will be of value in the future when you do need to update it.

Experiment and rewrite experiments

Another important staff retention technique on top generally keep up to date with technology is to let them experiment with new technology. 

I would avoid core systems, especially customer facing systems, but internal tools is prime candidates for field testing technology.

This will also lead to discovering technology that you can use in other applications if proven useful that otherwise would have been missed or delayed.

Naturally these experimental applications should also not be abandonware and be rewritten as often as other systems. Though they will probably always be good candidates for next set of technology experimentations.

Rewrite exploration and euthanasia

Certain system never comes involved in new features so will not be included in a normal rewrite. These will consciously have to be found and rewritten without business value. Though probably just as a contemporary facade initially. 

Leaving these as abandonware is not a good idea. Either kill them or update them.

Summary

The main points was already covered in my previous blogs, but I hope people see the value of rewriting frequently to especially keep staff, organisational architecture and company in general up to date for whatever happens in the future.

A utopian expectation of all applications being up to date is too much to hope for. At the other end where companies that rarely and/or minimally update their applications, they are doomed to fail. The grey scale in between decide whether companies will wither and eventually die or survive. 

Don't download the internet. Share Maven and Ivy repositories with Docker containers

Downloading the internet. A term commonly used when building Maven projects. It is due to downloading your applications dependencies and their transitive dependencies. With a normal application those dependencies can add up to quite a few jars and a lot of megabytes to download. Especially on a 'clean' machine without cached dependencies in your local repository.

Running a new container in Docker sometimes feels the same as every image layer it is built on top of also has to be downloaded. Eventually these are also cached in the Docker cache.

If you then have a Maven based application in Docker you have the perfect storm of bandwidth hogging. Especially as your Docker image's Maven configuration will by default not reuse any cached dependencies and instead always download everything from Maven Central. And for each new build or launch it will re-download the internet as it knows of no local cached dependencies.

Work around, not solution

My work around is to mount my local host repositories for Maven, and for its derivative Ivy, as data volumes for the Docker container. This then avoids re-downloading the internet on every further build and launch and will also reuse dependencies I most likely have already downloaded on the host.

It is not the best solution as your images should ideally not be influenced by what you have on your host machine, but it saves a lot of time, and a lot of grief.

Vagrant

When I run my docker containers inside a Vagrant instance I first mount my host repositories by adding these 2 lines to the Vagrantfile:

config.vm.synced_folder "/Users/myusername/.m2", "/home/vagrant/.m2"

config.vm.synced_folder "/Users/myusername/.ivy2", "/home/vagrant/.ivy2"

Boot2docker

Since Docker 1.3 the OSX /Users path has been by default shared with the boot2docker VM on the same path. So to share the maven repositories you need to link the /Users path to your docker user home folder.

   boot2docker ssh;

   ln -s /Users/mysername/.m2 /home/docker/.m2;

   ln -s /Users/mysername/.ivy2 /home/docker/.ivy2

Data volume container

For easy sharing these folders between many docker containers I mount them as a data volume container and naming it ‘maven’.

   docker run -d -P --name maven \

   -v ~/.m2:/root/.m2:rw \
   -v ~/.ivy2:/root/.ivy2:rw ubuntu

I am basing it on the basic Ubuntu image, and it will stop immediately as no process is running. That is fine, the mounted volumes will still work.

Alternatively instead of mounting the host folders you can have a persistent container with these folders exposed as VOLUME in the Dockerfile so it is shared in a similar manner.

Launch container 

   docker run -d --volumes-from maven \
   yourapplicationimage

Obviously you probably have other options on your docker launch, but the important bit here is the ‘--volumes-from maven’ which maps all the volumes from the data volume container called maven into this container.

Now in theory all containers with Maven and Ivy based build tools such as SBT should first look at the mounted Maven and Ivy repositories for their dependencies.

Build problem

Unfortunately if you build new images frequently you will have the same problem still as the above solution is only for running containers not building. During the build stage Docker does not allow you mount any volumes. This is so it is reproducible anywhere.

In your Dockerfile you can ADD or COPY whole repositories into your image, but that will make it very bloated with a lot of irrelevant dependencies unless you somehow construct and maintain a perfect repository. (Ps. don’t do this).

Maven repository manager

One solution that makes sense in general is to add you companies Maven repository manager’s public Maven Central mirror to the image. That way at least you will only download the intranet, not the internet.

For a Maven build add a settings.xml file to your image folder with at least these settings if using Nexus:

<mirrors>
   <mirror>
      <id>Nexus</id>
      <name>Nexus Public Mirror</name>
      <url>http://nexusmachine/nexus/content/groups/public</url>
      <mirrorOf>central</mirrorOf>
   </mirror>
</mirrors>

                

Then add this to your Dockerfile:

   ADD settings.xml /root/.m2/settings.xml

Note this will be overwritten if you later during the run stage mount .m2 folder on top of it, but for most that is fine as the .m2 folder usually contain a relevant settings.xml file anyway.

For a SBT build add this repositories file to your image folder with this content:

[repositories]

local 

maven-local

company-repo: http://nexusmachine/nexus/content/groups/public

scala-tools-releases

maven-central

Then add this to your Dockerfile:

   ADD repositories /root/.sbt/repositories

Repository manager container

A further solution is to run a repository manager as another container and have all Maven/SBT builds refer to it instead. This avoids the involvement of the host computer yet saves any network traffic as everything is localhost.

Pull down something like https://registry.hub.docker.com/u/mattgruter/artifactory/ .

    docker pull mattgruter/artifactory

Configure then run and name it:

    docker run -d --name artifactory \
    mattgruter/artifactory

Modify the above settings.xml and repositories to refer to you locally linked artifactory name as url instead, e.g this one liner:

   company-ivy-repo: http://artifactory/artifactory/repo/,
   [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
   company-repo: http://artifactory/artifactory/repo/

(The example above added it as an Ivy repository which Artifactory supports).

Hopefully these tips will avoid downloading the internet too often and save a few grey hairs.

Learning whilst writing, and relearn later

Martin Fowler recently made a comment that he writes books so that he can learn about a subject.

That resonated well with me, as whilst I don’t write books I do write technical howto documents on my website and my blog. And it is true I mostly do write these documents as I am learning the topic myself.

The main reason I write and share these documents is still to help others. Hopefully someone else will be able to stand on my shoulders and have an easier time learning the same thing. I certainly use a mixture of other’s docs as I learn stuff and hopefully I reference them appropriately in my docs.

Focus and relearn

But I write howtos also to help myself. By documenting each step I keep focus on learning it properly. It also encourages me to achieve certain meaningful levels before I bounce on to the next shiny thing.

An additional unintentional benefit to me is that in e.g. 6 months, 2 years or even just a few weeks later when I need to use/learn the same technology, I have ready made revision notes for me to get up to speed instantly.

Accidental expert

People mistakenly/naïvely think I am expert on the domain of each howto I write.

On some topics covered in a few older docs I may be an experienced fountain of knowledge…, but with most I only scratched the surface. Quite a few I never touched again and have more or less completely wiped any knowledge from my little mind.

So writing these docs can be a little bit of a curse. Of my top five most visited blog posts three of them are related to Hibernate from years ago. I am not an Hibernate expert nor do I want to be, but this could trick people into assuming that I am. My most visited (by a country mile) howto doc is about setting up an email server. I know a bit about the subject but it is not my job nor interest, I merely wrote a good howto in 2003.

But in general sharing is worth it. The amount of thank notes I have received and general good feeling I get from these are gold.

Evangelical

Another benefit to myself is that I now have reference points when I evangelise about a certain topic. I can refer to my own work and words instead of soon to be forgotten spoken words.

I also starting to be confident enough to present, submit CFPs and in general talk about subjects which are often built on documents and blog posts I have previously written.

Dockerise it all - containerised addiction

TL;DR: CDD - Container Driven Development 

(CDL - Container Driven Life)

Occasionally you come across new interesting technology that is easy to use, and suddenly a veil is unveiled and you become aware of so many opportunities and potential. Maybe it was the same the first time you discovered programming, or OO or FP programming, message queues or NoSQL databases, or distributed source control, or moving from IDE only to proper reproducible build tools (Ant, Maven, Gradle, SBT etc). With certain tools it becomes your favourite hammer that you carry everywhere searching for nails to hit it with.

Docker is such a revelation, at least for me. And I am hammering a wide variety of nails with it. And with the hammer-nail pattern/anti-pattern some of those nails really didn’t need hammering but it was fun.

Docker history

Docker was only revealed in March 2013 by Solomon Hykes and other people at dotCloud . It got a lot of publicity in the hacker news section of the world as people could see the possibly potential in this new tangent of virtualisation. It baked and matured through 2013 as people occasionally showed examples of applying it to their workflows, but still it was too rough for most to use.

But during 2014 especially since the summer the  publicity and widespread use of Docker has exploded. Downloads from Docker’s registry hub of ready made images has exponentially rocketed sine the summer of 2014 by 1387%! Now the majority of people I talk to or follow on twitter mention Docker and show examples of how they use it. Conferences and tech news are flooded with new ways to use Docker. And projects that extend or are based on Docker are multiplying all the time.

As someone who has always been interested in virtualisation, automation, reproducibility, build tools and provisioning, introducing containers like Docker has been a welcomed evolution and a revelation. Being already heavily invested in the use of Vagrant for development and systems the migration to/ combination with Docker has been smooth.

Docker addiction step 1. Exploration

First steps are usually to try the easy hello-world-ish examples. Downloading an Ubuntu image and launching bash inside a Docker container. Basically exploring Docker and its commands, which is what I cover in my how to install, basic use patterns and many handy tips with Docker howto.

Docker addiction step 2. Imagination

My howto document also takes you into step 2 which is more about creating your own basic images. Creating a simple Apache or Nginx container with a simple volumised website. Creating images ready with a Java SDK or node.js etc. I have made some simple Dockerfiles of these type of images available at github.com/flurdy/Dockerfiles.

Then you usually also wrap a simple application of your own in an image and running it on other computers. You start to understand how and where Docker can be used.

Docker addiction step 3. Diversity

Once you got one application inside a container you start to experiment with other applications. Might take a look at other types of applications and more likely support applications such as databases inside a container.

You might start to link containers to each other such as an application to a database container and some of the other handy Docker tips in my howto.

Docker addiction step 4. Proliferation

Now you have tried a mixture of applications with Docker you might spread the use to even more core applications and more. You start to link multiple containerised applications to each other.

Replacing third party integration and old legacy applications with a container image, wiring up and switching between different versions of an application via container linking, etc.

As you start to basically recreate environments using Docker containers you might start to facilitate promotion between staging environments only using container images. And eventually production is containerised.

You are probably already sharing and discovering images at work and on the internet. Using the public Docker registry hub and/or internally with tools such as Quay.io

Docker addiction step 5. Eccentricity

When comfortable with writing Dockerfiles and fully aware of the Docker way of layers and process then you start to experiment with more unconventional Docker images.

You might create Android SDK in an image, ready made SSH tunnels to apps or databases in another. You add tools such as a local maven repository manager, local DNS servers, mail servers, etc. into containers.  You realise desktop applications can actually be containerised.

Eventually tools to facilitate easier and more manageable Docker life such as Fig, Panamax, Shipyard, Dokku, etc. are maybe among your toolset.

Docker addiction step 6. Ubiquitous

By now all your core applications are run/runnable inside a container.

They link to databases, message queues, 3rd party adapters, and other smaller applications that are all also inside containers.

Your testing environments, staging environments, production environment, development environment and REPL are all running inside containers.

When you create a new application or touch an old legacy application your first step is to write a Dockerfile and containerise the application. You design applications and processes from the ground up to be containerised.

This is what I call CDD - Container Driven Development.

Possibly by now you are convinced everything belongs inside a container, so a better name is perhaps CDL- Container Driven Life.

Peak Interest: The learning and sharing curve

The interest curve of learning and sharing at work and its eventual peak are partly responsible for higher turnover of staff and reduced productivity if not managed well. 

When you learn and share what you know at work you enjoy it. You also contribute to a very productive environment. However when you no longer learn nor share then you are not enjoying it anymore and become negative and a liability. And then leave. 

About a year ago I read an article that explains everything in this blog post in a much clearer and scientifically researched way with references to specific companies. I haven’t been able to find it again, but when I do I will add a link and note to read it instead.


The problem

When you start a new role you have lots to learn. (Although sometimes you may have too much to learn and you are simply out of your depth and not able to keep you head above water).

At the same time you may have a lot experience and new thinking to share with the others in your team and company.

This learning and sharing is usually very enjoyable and rewarding, and you quickly become more and more productive in your role.

This productivity over time can simplistically be viewed in a graph like this:

However after a while in a role you learn less and less. You have absorbed the majority of the domain knowledge, you know all the people involved, and you are familiar with most of the technology stack (if in a similar line of work as me). 

Your team members and colleagues have already heard most of your experiences and you are starting to even repeat yourself. The rewards for sharing are reduced. So your productivity could be viewed as flattening out:

However what tends to happen is that you enjoy your role less due to not learning and not sharing as much as before, and over time you start to not like your role and become quite negative.

This in turn start to have a negative effect on your productivity over time:

This is what I call peak interest.



The consequence

With this negative productivity trend you may start to become negative person at work. This might then spread to your colleague and the whole team starts to suffer.

In the end you probably will change jobs. So the company will have to recommence the expensive recruitment process to replace you and will loose all your knowledge.

Even if you leave quickly it affects the remaining employees negatively. Naturally they might be upset if you leave on a personal level, but also upset with the company for letting you leave, jealous if you leave for a better package elsewhere, depressed that your knowledge and capacity is gone and they have to figure out how to compensate. If many employees leave quickly all the time then the company will have a bad negative culture.



Known problem

This is in fact a known problem, and whilst I can not refer to any scientific research, I have read about this issue many times. 

Major companies are fully aware of these peaks and take many actions to delay and counteract this curve for their employees.



Not applicable to all

The magnitude and curve gradient are definitely different for everyone and for every situation. And not quite as smooth. Some are quicker learners, some in a difficult role, some join a company where there is little to share, etc.

And then there are some sets of people that this hardly applies to at all. The "lifers". Not meant as a derogatory word but I fail to find a better description of this common characteristics of a set of people. 

These are the people that rarely changes jobs, that are not that interested in learning anything new, nor as keen share what they know. They just get on with their job, and stay there for a long while.

Their interest curve whilst perhaps not entirely linear are certainly very much less curved. And may never peak. However my unscientific opinion is that they also make a much less productive contribution to the company over all. 

Every company will have some lifers. There are many roles where this is a good trait: in jobs that are very 9-17 and never really change. 

However it can be frustrating if you mix lifers and “peakers” in the same team. Developers, artists, any creative or knowledge workers need to work together, learn together and share together. If some are then learning and producing at different speeds you will have conflicts.

That being said, mixing peakers with "ninjas" is a recipe for disaster as well. Ninjas and rockstars do not have an interest curve, they have an interest seismograph. But that is for a different blog post.

Solutions

Companies do try to resist and reduce the effect of these peaks. They try to stretch out the length of the peak and create further peaks.

They can ensure the work environment is nice and comfortable. That bureaucracies are not frustrating, latest tools are available and most of all avoid stressful tasks, overtime and other counter productive situations. This will make the peak last longer and post peak decline slower.

Even then the major peak will eventually happen. Companies can then create further peaks by changing team focus, changing project, changing people, changing teams or changing role type completely. Basically avoid status quo for too long.

What they can then achieve is multiple peaks that keep you at a company for much longer.

So this avoids you having to leave the company to find another enjoyable role. However I do think eventually that undulated curve will start to fade and you might leave your role in the end.

The critical bit is then though that instead of leaving after for example 1 to 2 years you might stay for 5 or 8 years. (Time being relative to personality, industry, culture, etc).



Solution faults

Even if the company is aware of interest peaks, and want to mitigate their effects they might not always be able to.

In a small company there might not be any other team or project to move to. In a big company you might be insignificant or just too many to cater for so you are not moved around as often as needed.

Due to deadlines, business restrictions etc there might not be possible to move at the appropriate times.

Different people will have different gradient of curve and peak and companies will struggle to individually predict this.

These might lead to the negative cycle setting root and you resign.


Other issues are of course that changing everything around all the time is not appropriate either. 

It may not be good to do it too often, to do it to too many people at once, to unevenly change roles for some compared to others.  

People and teams do need some sort of stability so constant change is not good either.



Personal experience

I know that I go through these interest curves, and I have especially been taking notice of mine and others lately.

Whilst I do not like to change projects too often as I like to get a thorough knowledge of the domain and project, I know I also need new challenges once I have mastered a domain, technology and/or team to a sufficient level.

When I have worked with startups the curve has always been very steep, the peaks very high, but also the peaks come quickly and the need to change status quo come sooner.

When I worked with large enterprises the gradient was less steep as there was many things that took a long while to do, the technology was less challenging and pace slower. However I also reached the peaks later. But I think the productivity magnitude was much less.

As a consultant these peaks are not so much of an issue due to the normal change of clients and assignments. Some long term clients have even been helpful by moving me to new projects with them in the same assignment although perhaps less on purpose for my benefit, more for their business reasons. 

My beautiful better half reminded my last year when I was moaning about being bored at work. She pointed out that exactly a year before I had told her that I really need to step up my game as there was so much to learn in my new role and that I was loving it. I had clearly reached and passed my peak for that role. 



Conclusion

Peak interest happens to most people and companies. It is nothing to be ashamed about. It is in fact a nice and beneficial trait which needs to be managed.

Company need to embrace interest peaks. Let people move around the company. No questions asked. Encourage it. Better than haemorrhaging good employees, or let a bad vibe fester.

Avoid status quo. For instance make sure technology stack and skills keep continuously evolving even if not always great business value. Staff will stay interested and not need to leave find new challenges.

As an individual if you start to get bored in role, quickly request a change in your role. New project, new technology, new people, new role, whatever it is that will be a nice new challenge.



Disclaimer

Of course productivity and reasons people leave jobs are very much affected by other factors. I do believe however that peak interest of learning and sharing does contribute to both enough for it to be a significant factor.

I do not have any specific scientific research (atm), just a personal opinion backed by reading over time of similar material. If you agree, disagree or have have any research to back it up then feel free to comment or link back.