Want to hear some "behind-the-team" stories about software projects?
I hope these software project stories show you how unpredictable work in the software industry can be. You'll notice relationships between people impact software projects much more than technology. I'll give you some background on my role, the industry of the product, the technologies our team used, and any politics I ran into.
I've also included links in each story to episodes from my YouTube channel, Healthy Software Developer that I did about some of them.
In my second year of going to college, a teacher of mine knew I was interested in web development and asked me out to dinner. He told me he knew a real estate company that wanted to get their properties listed online to grow their business. And he offered me a short side job to build them a basic website.
This was my first "real" software project before I landed a salaried position, but I learned something valuable that I wouldn't *really* appreciate until I got back into consulting years later. Working closely with your customer (or client) AS you're building their solution is far superior to any up-front design or planning you can do. Though the client had ideas of what they wanted - ultimately it took a lot of phone calls and changes to satisfy them. This experience was in stark contrast to some of the future projects I'd work on where plans were rigid, and the way the products were built wasn't even remotely "agile".
I completed this little project in my spare time over 3 weeks. And got paid for the first time for coding! 😄
Towards the end of my college education, my mom met someone at church who knew a Software Development Manager at Rockwell Automation (at the time called Rockwell Software). I interviewed and accepted a position as a software tester.
During my early time at Rockwell I sought out older mentors and learned quite a bit about manufacturing software. One particular man showed me quite a bit about system programming and how to do things to a computer's operating system. Another showed me a lot about how to test software.
My biggest influence at the time however, was my boss. He wore a wallet chain, biker boots, and kept his hair back in a ponytail. He was a risk taker and an innovator, and he inspired me almost immediately with his energy and enthusiasm. He saw something in me early and encouraged me with almost everything I did.
When people manufacture products on a large scale, they would often use expensive hardware called PLCs (Programmable Logic Controllers). This company I worked for was disrupting the industry, and had built a software package that did everything a PLC could but for way less money.
And one of the other products this company offered was a set of user interface controls that were used to build "dashboards" that people operating machines in manufacturing plants could use to monitor things. These dashboard controls were mostly used in Visual Basic, which was a popular language for building user interfaces in the late 90's.
So my job on this project was to a) test that the dashboard controls worked properly, and b) write code samples showing customers how to write programs that interacted with the controls.
This project gave me a lot of valuable experience with testing, because I had to learn how to write a test plan, test cases, and think up things developers hadn't tested (but that our customers might do) to make sure the product was high quality.
It also helped me understand how to learn programming languages quicker because I created code samples that did the same thing in Visual Basic, Java, and C++. So I learned quickly what was common across languages, what was different, and how the API for our controls should be designed to make it easy for customers to use no matter what language they were using.
While my wife was pregnant with my second son, she would go to bed at 7:00 PM most nights. With me not feeling good about going out with her being pregnant, I would stay home and read computer books for hours after she went to bed. It was during this time that I learned Java, COM, and early interactive web technologies.
Over the years Rockwell had acquired several other companies. Unfortunately, this caused our customers to have to learn and use five or so different complicated programs to do their work.
While I was at home learning, I had been noticing at the time how web sites and "windowed applications" looked completely different. For fun, I began creating a "web app" that looked somewhat like Rockwell's existing windowed applications, but it was built using early versions of the web technologies we know and love today.
The other thing I did with this side project was come up with a user interface that brought elements of the other products together. This was in an attempt to make it easier to use and simpler.
Somehow I ended up telling my boss about it and he asked me to give him a runnable demo on a CD. Soon afterwards I found out he met with the then VP of Rockwell who called it "the most strategically important project in our portfolio". We were given permission to pick 12 of the best software people across the company, and within a few months we had a team.
Everyone was excited to work with Java. I was asked by Microsoft to be an MVP for Java and COM integration. We patented an XML-RPC protocol that was essentially the precursor to SOAP. I contributed Sun Microsystem's "Netbeans" Java IDE's first plugin, on that ran the ant build system. I got several promotions, eventually to "Application Architect" by the time I was 23. I thought my career was off to a great start.
But half of my team didn't like me because they had been at the company longer, and were much older. They were distrustful of me, and it didn't help that I had no idea how to build relationships with them or help calm things down when they disagreed.
Within a short period, things came to a boiling point between I and another man on my team and he complained about me being unfit to lead the project to HR. When others on my team were asked to confirm his complaints, they disagreed and he was fired. However during the time he was there, he had sewn seeds of discontent. The CTO out of another big city in the country at the time began to feel threatened that this "team in West Allis, Wisconsin" was doing so much to push the envelope of technology and teamwork outside of his influence.
I was asked to take a trip to Vancouver to meet with a company that might partner with us on Java technology. When I arrived at the airport with my boss, we met someone who he shortly told me privately to "look out for". I signed a non-disclosure agreement at our first meeting with this potential partner. Within hours my new friend we met at the airport had pissed our client off so bad the partnership didn't have a prayer of happening. It almost felt like he wanted to sabotage the relationship.
When I arrived back home, the Director who was above my boss called us into his office. He told us he realized the non-compete we signed was drafted by Rockwell, and barred us from working on any project with Java technology for 5 years. Somehow the CTO had maneuvered us out of innovating. This was my first exposure to higher level politics in a company, and it disgusted me!
Shortly thereafter the company asked us to build a new portal product using the then-emerging Microsoft .NET framework. Our customers felt like it was too hard to keep up with our products since they all were updated at different times. Because of this our company mandated that all products needed to ship a new version and release on the same day.
Unfortunately due to more politics, the release was delayed and my boss was let go.
My boss ended up at a new company that sold watering products to the animal industry. He started building a new team and in short order hired me and the rest of the people from Rockwell who liked working together. Working at this company proved to be one of the weirdest software project stories I've been a character in.
We were brought in to build a web application that would run on touch screen computers, mounted on the walls of research facilities. This was several years prior to the iPhone and most of the touch screen devices we have today. It was an exciting opportunity to try and figure out how to design the user interface, as the Product Manager for the product would fly to Japan and try and observe how they were using touch screens. At the time Japan was significantly ahead of the United States with touch screen adoption.
We were reading about agile methodologies like scrum, which were still very new at the time. To use them on our projects, we created a small portal site that we used to manage our work. It was crude but it gave us experience with trying to write user stories, estimate tasks, many of the other typical things scrum teams do. We really didn't know what we were doing though.
Soon there was even more political drama, and my boss was let go again. This time due to being blamed for a mistake by someone else at the company who was going though a personal challenge. I again saw corporate politics put my boss in the crosshairs, due in no part to his actions.
Next my boss ended up at a company that was struggling to ship a .NET product to market. He was able to hire me and a few other people from the prior debacle to help.
The main product of this company was a Manufacturing Execution System (MES). It allowed factory floor workers to operate the software with a touch screen.
My boss and I first identified issues with the software development process. We then built some SOAP Web Services that used the OAGIS standard. This standard was useful at the time, because it allowed our MES product to then connect to many Enterprise Resource Planning (ERP) software packages that our customers used.
I also prototyped a new application for them in ruby on rails. I left the company before this project moved forward to move out of state.
My family and I fixed up our aging house from the 1970's in Wisconsin and sold it in 2007. We moved to Austin, Texas where family from my dad's side lives.
Upon arriving in Austin, I spoke with several recruiters and hiring managers about positions. One night I received a call from a recruiter for a small (at the time) consulting agency in town, Catapult Systems. I had heard some bad things about consulting, so I was hesitant.
However the interview changed my mind completely. When I interviewed, I met several people who would come to be my friends to this day. When they asked me about my story, and told me about working there - it was immediately obvious that there was an understanding of the software business here that I hadn't come across before. They convinced me to join them and I started a few weeks later.
The names of the products, companies, and people in the software project stories I share about the clients I worked with at Catapult have been changed to protect their confidentiality. This is an unfortunate byproduct of working in the consulting industry for a firm, you're not allowed to advertise who you've done business with!
On my first project at Catapult I met a guy who taught me many things over the 10 years I worked there. He led a project I started on where we were replacing a DOS (!!!) app with a Windows Presentation Foundation (WPF) app.
It was in the legal market, the client offered legal funds for low income citizens that need representation. It was a pretty cool niche, with a bunch of business problems I'd never run into before.
One of the most fascinating things though, was that one of the women who would use the new app had memorized a huge number of keyboard shortcuts that worked with the DOS version of the app. So even though the redesigned application used all the latest (for the time) user experience interactions one would expect, we had to still support keyboard hotkeys mapped to what she knew from the old DOS app!
This project was also interesting because the client didn't have a lot of money. So we never built them a proper "installer" for it. We would release a build, burn it to a CD, and drive it over to their offices downtown! Talk about quick and dirty :).
They were a great client to work with though, they treated us well other than occasional concerns about costs. I learned on this project what was different about working as a consultant instead of an employee. I was there to provide my client with good service - not show off how much I know about technology.
On my second project with Catapult, a client they had built a significant application for in the student housing space wanted a new version. The old version was looking stale, and it didn't handle many of the requirements their business needed to handle.
I was joined up with a guy who had been on the original project many years before, and he knew the contact at the client who we worked with. When I met there to kickoff the project, the client became irritated with me because I didn't know everything my buddy from Catapult did.
Though I felt like it was unfair that he expected me to know everything considering we told him I was new to the company, ultimately "the customer is always right". So my colleague from Catapult and I discussed how he would get my questions answered without upsetting our client.
Over the next couple weeks, my new friend helped me get the information I needed, and I was able to start on the code. But I quickly became overwhelmed with some complex math needed to calculate leasing costs for a variety of edge cases. After struggling for a few days, I reached out to my friend again.
He was happy to help - and I found out he was a math wiz! I never would have finished this project without him. Even though I already had 12 years of experience under my belt (and felt like it was silly to need help with something that SEEMED like it should be easy), it wasn't enough for this one. I was super grateful for my friend's willingness to help me, and to teach me what he did about consulting on that project.
My agency started a project with a large client in the oil and gas energy market. Their primary product collected data about drilling oil rigs. They wanted to offer a reporting product that would "bolt on" to their primary product to drive sales.
The client had agreed to use an agile delivery process, until a week before starting. When they demanded a more formal "waterfall" approach, we made a big mistake and didn't re-negotiate the contract. You can probably guess what happened next.
Over the course of 9 months, we lost money building the client a 300 page specification for a product they never ended up building. Though we tried to advise the client to use a different design, they were hell bent on having us design what was essentially Microsoft Excel again for them. Though we warned them it would be too expensive, they insisted on designing it. The project was a failure.
One of the next sizable projects I worked on was for a major non-profit that provides funding to help improve education. This was my first experience working for a non-profit, and it was an eye-opener to see how much money could be spent at a company who's business model is to give it away.
The building we worked in was brand new and beautiful. The team was great, I met a friend I'm still very close with from Catapult that made the project fun. We learned that we both liked to play music, and had much of the same perspective on life, family, and our work.
Though we made great progress on the project, there were politics. The client's head of the project was gunning for other people's jobs and we were often caught in the middle of it. Before the project ended, 3 people at the client were fired. Eventually we finished the project and handed it over to their very-capable developers at that point.
The agency didn't have a large project for me at this point, and one of our clients was upset with a poor job done by one of our consultants out of a different office. I was asked to help this client get what they needed, and to turn around their opinion of my company.
I knew I wouldn't lie, cover up for, or excuse the mistakes that were done by the other consultant. But I also knew this was not a project to show off, or take any risks whatsoever. The client's solution was a set of Microsoft project management SaaS products that were bolted together to provide a solution for managing a large number of energy improvement projects. I needed to get in and figure out what was necessary to resolve the issues, and nothing more. I didn't have time to comprehend the entire solution!
I came in, and in two weeks fixed all the issues they had. They were very pleased, and I won an award from Catapult for my service.
I had heard in the hallways of my agency that there was a particular client project we were having problems with. I hadn't heard the details until one day I was asked to meet with a colleague who had been "fired" from the client - I was to replace him.
Upon meeting the colleague, I was told that the client liked to argue, re-write work we'd done, and blame us for their bad decisions. I was walking into a fire fight.
This client's product provided tracking and behavioral data to huge companies. They collected millions of data points per day, and were having problems meeting the scale of their user base.
Within a few days of working with the client, another colleague of mine and I were asked to create a prototype. There were too young men from the east coast who had started the company, and they were our main technical reports. They insisted that moving the data into Hadoop would solve their problems.
We tried to explain that their product allowed customers to slice and dice the data in some specific ways that at that time, were not efficient in hadoop. They demanded we continue the prototype, so we built a cluster in both raw hbase and cassandra and did some testing. Eventually they realized we had provided them with good advice, and we began suggesting that one of our Microsoft SQL Server experts meet them.
In the time of our attempting to convince these two young men to start looking at the database more seriously, they began to re-write everything we would build for them each week over the weekend. They were convinced we didn't know what we were doing, so they actually put a webcam on our team of 7 developers so they could make sure we were working! To say we were upset would be an understatement. Luckily after 3 days of this silliness, they turned it off.
Our SQL expert did an analysis on their production system. What he found still blows my mind to this day. When you install Microsoft SQL Server, the database they were using in production, by default it is set to "auto-grow". What this means is when the file containing the data gets too big, the server automatically resizes the file to get bigger. This resizing process stops everything for a bit, resizes the file, then continues.
The "auto-grow" can be set to a size, and our SQL expert found this was set to 1MB! The database was basically resizing itself every few seconds - which would explain why they were using several hundred THOUSAND dollars worth of hardware to handle database I/O. Let's just say we weren't called back when they realized this. Software project stories about clients being careful what they wish for don't get much better than this one.
Next I joined a small team to help a client deliver a website and platform that was to be a clone of mint.com, but for customers with high net worth. The company was based out of another state, so all of the work was remote.
Most of the team I worked with from the client was great, but you could tell almost right away they had no idea what they were building. There was confusion around the vision, the design, and what customers wanted.
Though we did our best to build the product this client wanted, in the end they scrapped it. One of the things that killed it was that the product needed to pull in large amounts of data from banks and other financial institutions. They chose to use a "drag and drop" tool from Microsoft at the time to build these data-import components. Unfortunately the technology was buggy, hard to use, and didn't deliver on the performance we'd hoped.
I was happy to get the opportunity to work downtown for a large international grocer. They had a product that was having trouble getting released. I was brought in to coach them on Scrum.
The team was fantastic to work with, and they were very cooperative. This was a surprise coming from my prior clients who were very resistant to me as a consultant and had a hard time taking any advice contrary to theirs.
After coaching the team over a few sprints, I pulled back the sheets on their deployment process. It was severely lacking. Production deployments were 100% manual, done on the weekends and only a few times a year.
I began reading the book on Continuous Delivery, and immediately started selling the benefits to my client. While I was there, I and another colleague from my firm began using Microsoft Windows PowerShell to automate deployment and release of their product through the typical quality and environment gates you'll find in a Continuous Delivery deployment pipeline. At the time there were no tools for this on the Microsoft platform, so I had to build my own.
The next large project I worked on was for a client who helps college students find housing. The publicly traded, profitable company was hitting issues with scaling. To server their customers, IT had to manage 6 large applications that were written separately and were now tightly connected to each other.
I was asked to come in and do a few things. First, I was supposed to evaluate the maturity of their software delivery process. Second, I was to prototype the use of an Enterprise Service Bus (or Egregious Spaghetti Bus if you go by Martin Fowler).
I essentially ended up building microservices on top of each of the systems that would publish and subscribe to messages from the bus. The architecture was well received, but the company didn't have the time, money, or priority to pursue a broad implementation just yet. I used the same interviews and maturity evaluation I had at the prior client.
An episode from Healthy Software Developer that has 3 embarrassing software project stories tells the fateful tale where I made a fool of myself when trying to quit smoking on this project. It's the last story of the episode. 😳
Over the next year, another colleague of mine was working at a major computer hardware manufacturer in town and began to tell them about the single page application work I'd done. I arrived and gave a short presentation and demo, and the company began using these technologies on projects.
Eventually, I was asked to help the team. When I got there, my colleagues had significantly advanced the technology - bringing in other modern frameworks at the time like backbone and knockout. There was a "religious war" between our team and another at the company as to whether the products should be moved to angular. I completed my work on the project before the decision was made.
Over the previous year, my agency began investing in offering business intelligence and "big data" solutions to customers. I had been given opportunities to work on them, but was too busy with other clients at the time.
I jumped at the chance to work on a team with a colleague who we hired just for the project. He brought a SaaS product he built for rapidly creating data warehouses and years of experience with BI. I had found a mentor.
The client was in the health provider space, and they had hired a prior vendor to build some dashboards to help their executives have more data points to inform their decisions. Unfortunately, after spending a huge amount of cash and having nothing to show for it, they fired the vendor.
So when we started this gig, the client was very distrustful. They did their best to stay open, but we could feel with most interactions that they were still very much hurt and on the defense to have it happen again - understandably so!
Through the use of a platform that was created by a colleague at the time who then went on to expand his business, we were able to deliver the project in 9 months. We accomplished this through a combination of the LeapFrogBI platform for data transformation, and powerdelivery for automated deployment. Within a few weeks we began doing intra-day releases, often up to 10 times a day to production!
Following the success of our prior big project for the health care provider above, we found a local client that offers health insurance brokerage and analysis services that needed help. This client's primary problem was that their staff were experts in one source of health insurance data (of which they had 13 at the time) and so training new people could take as long as 9 months!
We found significant "dirty data" in their product and helped them prototype the use of several data quality and data cleansing products to get more accurate results. We also used master data management solutions to deduplicate and match common records across different health provider's data. In the end, their training needs went down from 9 months to 1 week! Though we ran into some delivery problems due to staff changing in the middle of the project, it was ultimately a big success for the client.
I was asked to lead a team that would create the software to run on an embedded device for charging cell phones in bars and other entertainment locations. The product was built with WPF, Caliburn.Micro, Azure, and powerdelivery.
As with the prior project, we began doing intra-day releases to the physical machines to get rapid feedback. Unfortunately, the client did not have the budget to complete the project and so it was canceled. Of all the software project stories I've told, this one still takes my breath away. I haven't had the guts to make an episode about it - it's almost too hard for anyone to believe.
A project that was in trouble at my agency needed my assistance to deliver. The technology stack was a combination of AngularJS, Phonegap, and ASP.NET Web API. This project provided me with some experience with promises, and the bundling of an HTML app to deliver on mobile. It also used bootstrap for styling.
I'd been spending a lot of time trying to help Catapult Systems learn more about Continuous Delivery, and the general manager of the Austin practice at that time heard of a project offered to us through Microsoft.
The client was a large tax preparation software company with multiple locations. They were headquartered in one state, with another team that was acquired out of state. They were having trouble getting the newly acquired team to integrate with their software development process.
Me and another guy from my consulting firm flew up with a set of interview questions prepared. These covered everything from development process, automated deployment and versioning, and general operational topics.
When we arrived, we spent a week interviewing the first team. Everything seemed to go well at first, and we were collecting quite a bit of data. But when we visited the team in the other state, we realized they were using kanban, though the first team was using scrum. After some time, we realized it was because the second team had to push last minute tax code changes from the government through their process quickly! It made perfect sense.
When we presented our findings to the client, they were happy with how comprehensive our evaluation was. But they were disappointed that they didn't hear what they'd hoped. They wished we would just tell the other team "use scrum!" and be done with it. It was a good project, but definitely political. The client had a hidden agenda.
A State and Local Government client based in Austin needed a solution for ingesting, cleansing, transforming, and reporting on large quantities of data for use by several government agencies. They put out an RFP and my agency won the bid.
The project was one of the first in the world to use Microsoft's Cortana Intelligence Suite (in beta at the time) - AND on the "private" government cloud. Though we did not use every feature of Cortana's platform, the project was a success. Once initial patterns were in place, I rolled off before it was complete to help with the project below.
My next client had an extensive SaaS web application that provided document management solutions for their clients in the education space. Their competition offered iPad applications, and so they chose to hire my agency to design and deliver one for their product as well. The technology stack was Xamarin, iOS, ASP.NET Web API, Identity Server 3 for authentication, and PDFTron for PDF document editing and annotation.
During the project, I was the sole developer and began releasing builds to their beta users several times a day after 2 weeks of development. I met a brilliant woman at this client who built the server API after I designed it's specification with swagger. She built a full test suite that was able to verify correct operation of the API in under 1 minute!
Though we encountered challenges with how Xamarin was managing memory that eventually caused us to use a support ticket - the project was a success. In the 10 months following their first production release of the app, it only crashed once!
In April of 2017 I went on my own as I described in the story on my about page. This led to several independent consulting contracts where I helped companies on my own while working on building up my YouTube channel and software development coaching practice.
My wife has been taking yoga for years, and one of her favorite studios is in north Austin. I started going there around 2015, and at the time the price for both of us to go there was getting expensive. The owner had an existing website, but every time she needed to change it she had to pay an independent contractor. So I suggested she consider WordPress.
Over a period of about 3 weeks, I worked with a designer she hired to setup the site. It let her staff book appointments, post videos and announcements, and she loved it. I didn't tell her I had used bootstrap so it was responsive until it was done.
When the site was getting ready to go live, she checked it out on her phone and was amazed! The site looked great on all devices, and she was one of the first studios in Austin with this capability. WordPress really is an amazing platform that I don't think enough people use. Before you dive straight into an MVC framework, or build a single page application - consider your client's budget, market size, and features. You might be surprised how often WordPress won't only make do - it will be the best decision!
A friend I met through LinkedIn that was newer to Austin helped me find this project. A software consultancy that offered data mining and transformation capabilities was having trouble delivering a solution for their client. I was brought in to make progress and help figure out why things were in trouble.
When I got there, there were only three employees working on the solution. One was in a DevOps role, and we hit it off right away. The other two men were very stressed out. I had to find out quickly what happened.
After about a week of talking with the owner and these employees, I realized the two men who were stressed out hadn't had a break in a long time and were working crazy amounts of overtime. It was a prime situation for serious burnout.
Additionally, the computer manufacturer that was their client had a software product that they used from another company that was going out of business. My client had won a bid to replace it, but there was no source code available! So it was a reverse engineering effort. In these situations estimation is INCREDIBLY difficult because the people who know what the software does "behind the screens" were gone!
I spent a significant amount of time just trying to get the two men who were stressed to even talk to each other. They were angry at each other and playing the blame game. I also tried to get the owner of the consultancy to understand how urgently they needed rest - but he was also in a very difficult situation with his client.
After a couple months, I was able to show some progress on a major component, the two gentlemen started working together better, and the owner brought in another person who really added some fuel to the fire to help reel in the timelines. This let the two men dial back their hours a bit.
I was making a pretty sweet bill rate for this project, so once things stabilized the owner asked me to roll off. It was totally understandable. This is how consulting goes sometimes! I was just happy to have helped the team ease some of their pain, and learned quite a bit in the process. It was the first project I was able to use some of the Go programming language on.
This same friend I had met through LinkedIn told me about two consulting companies that were trying to help their client rescue a troubled project. Their client had paid a vendor in Asia to deliver a product, but hadn't received a new build of the software for 9 months! So I was brought in again to help figure out what was wrong, and to set a path forward.
One of the consulting companies specialized in mobile development, and the other specialized in API development. Between myself and these two companies, we spent several months simply doing discovery. We found developers from the vendor that was fired had branches with changes that hadn't been merged for months! The automated deployment pipeline in Windows Azure was a mess. The requirements were full of holes. And the configuration of third party services like Google Maps etc. were in bad shape - keys were being reused in the wrong places.
The client was initially very frustrated because they wanted to just start building more features. But we had to help them understand major architectural changes and rework were needed to get the project back on track.
So we began some spikes to remedy the easy things we could find to fix. At the same time, one gentleman on the API consultancy team wanted to rewrite the entire back end of APIs. I didn't find out about this until it was underway and it created some friction in our relationship. I tried to find solutions that would be a compromise but he was pretty set on doing it his way.
We refactored AWAY from microservices because it didn't make sense for this project. And we got the deployment pipeline completely rewritten. But ultimately I decided it was time to leave the project. There were politics that began making it hard to feel like I was being effective. And I wasn't getting the support I needed.
It was difficult because I really liked almost everyone I worked with on the project. It was an amicable split, but I was disappointed that the project didn't go better.