I’ve been working my first large software project for almost a year now and I’d like to share my learnings from this project. The things I’ve learned, discovered, and realized during this time might be interesting to some, might be old news to others, and might even shed some light on how certain things are done in certain types of projects, but all in all I hope this proves to be an interesting read.

First things first, where have I been working? I’m currently a software developer at the SURI project, the current largest government software project for the Puerto Rico government under the arm of the Treasury Department, also known as Hacienda. This is the software that is going to handle all the new tax related activities for the treasury and is already being used for corporate tax returns, import declarations, and other tax activities. Next year the plan is to rollout the individual tax returns. We already handle individuals in the system, but only show informative and W2s and not their tax returns yet. I’m here as a contractor on behalf of my employer.

Here are some of the things I’ve learned while working here and some of the things I’ve discovered when comparing to other jobs, positions, and projects I’ve been in or have compared to.

Automated Deploys

Once you’re done developing and testing your components in a large project like this, it is very important that you have a setup a reliable automated deployment framework. In the SURI project we have such a system that’s setup by Fast Enterprises FCR product. Once final testing in done we create a migration script using the internal tools that takes all the component over to the production environment. The same goes for our testing and staging environments. I don’t think we could be as productive as we are without this. Having to manually move things around is a pain and having an automated system to this is really quite helpful. Now, our teams have specific times of day and during the week when things are deployed. Deployment to the testing and staging environment happen every four hours or so on a schedule and production happens at the end of day every day for urgent items and every Wednesday for everything else. One thing to note tho, in our system we still choose which components or configuration to move manually. It’s something that I personally dislike, at least in terms of components, but it works well and I can see the advantages of this method. I’d personally like to see truly automated deploys to the testing and staging environments,but that would require an additional and I believe necessary change in order to this effectively. Which brings me to the following…

Testing and The Meat Cloud

In SURI, being a government project, there’s a lot of people involved. Almost of a quarter of those people are testers. These testers do… well exactly what you’d think they do, they test the software. The way we test in SURI is that developers write manual test scenarios once we’re done with development and then the scenarios are assigned out to testers. This way of testing while necessary at some point, like in QA, just doesn’t sit well with me for several reasons. First reason being that these testers are human and humans forget things or don’t always do things in a reproducible manner. Now granted, we have some great testers in the team and they do an amazing job, but I think that for many of the tests we do we could do a lot better. These testers are also a phenomenon I like to call the meat cloud. The meat cloud is just a bunch of people you hire to do certain tasks instead of doing them with computers. I don’t dislike the meat cloud, it’s definitely something that’s okay for instances where the automation is not there yet, but in the instance of testing and especially unit testing, we could do better.

Automated unit testing in a project of this size would not only be beneficial, but I’d argue necessary to ensure a level of quality and productivity that is required. The kind of software we write here is something that I would consider a great fit for automated testing and even TDD. As and example, most of the components deal with processing returns, flat files, and simple business rules, many of which are perfect candidates for automated unit testing of the “given x then y expect z” nature. One of the types of components our framework uses is called a Document, this document has a very simple structure, it’s just a bunch of fields with excel-like rules and functions. These fields simply take input and produce certain output and validate that input against a simple set of rules. One could easily test these using automated unit tests as it would just be a matter of giving it certain input and expecting a certain output or end state. For the other types of components, what we in SURI call the business objects, testing would not be as simple, but still would of a similar nature. Many of the business objects could be tested using similar unit testing techniques including BDD techniques. The advantage of such a method of automated testing would be that 1. the code and way of usage would be self documenting as the tests would be the design or be heavily influenced by the design and requirements, 2. you would be made aware quickly of any breaking changes you’ve introduced while fixing the software or adding new features, and 3. if you had a set of dependable, automated, and clear unit tests in a test suite, you could be more confident in your ability to deploy code that works well and has the least amount of bugs. But what if you miss something? What if you forget to test a specific scenario? Well, you can always just add the test later on. Wouldn’t you then need tests for the tests? First of all, no, ideally your tests are small and concise and need no tests for themselves as they are very small. Secondly this is very common rebuttal to unit testing and TDD, or at the very least misconception, but it’s totally false.

So why else have automated testing, well that ties in to the first thing I mentioned. Which is that it allows for better more reliable automated deploys. It only takes the system to test before it deploys to choose which components go up. I don’t know about you, but if I have to do one less thing at work that isn’t related to developing I think that’s a win-win. Because again, as a human, I can miss things, having this would make sure I miss less things.

The Importance of Code Review

One of the old adages of software development is that the more eyes on a project, the more bugs are found and eliminated, and the better the software becomes overall. This still holds true and will continue to be. This why code review is so important and in SURI this is no exception. Once we’ve finished our work in progress tasks se send it off to be code reviewed by one of the team leads. These leads, having work on the project for a longer time or been involved with it’s initial development, review the code (duh) and give the go-ahead for testing, migration, and finally deployment to production. They’re job in this role is to make sure that we haven’t miss anything important and to suggest better fixes, better ways of doing things, and that our code and changes are in line with the standards we use throughout the project. This is a step that is necessary in all software projects after a certain size, and I would argue that if you’re team gets bigger than one, then you need code review in your pipeline.

Previously I’ve worked in projects that haven’t had code reviews and they suffered in quality as a result. The lack of useful eyes on a piece of software leads to often software that lacks in many things, most of all in quality.

Databases and Data Duplication

Data duplication is okay. Let me say that again, duplicating data is okay. Not only is it okay, it is often necessary for software to be responsive and to offer a good user experience. We duplicate data all the time in our systems and it’s myth that you should not duplicate data at all. You’d often read in common database class textbooks and lectures that one of the reasons we use databases is to eliminate data duplication and that we should avoid it. While that is good advice for some, it is absolutely the worst piece of advice for big systems. Often one place where we duplicate data is when we want to provide different types of queries or we want faster queries, so we duplicate the data in the format we’re going to query it, effectively denormalizing the data. This is okay, because it’s a tradeoff we make to ensure the system is fast. One thing should be clear, tho, the flow of data is still one way. Data is only written in one way and everything else trickles down from that flow. When a return document is submitted for example, only one system in charge of processing that return, once that return is processed events are triggered that might take that return document and do several things with it, like mapping certain fields to a denormalized table, create transactions off of it, produce letters, etc. This process gets triggered all over again when a new version is crated such as with and amendment, so even tho we duplicate data, the data is eventually always consistent because we do not modify the duplicated data, instead it is derived from the original source.

What does this give us? Well let’s say you need to figure out how many people issued a return for more than $200,000 in business volume, with this setup you could just have a mapping table for the fields you’re interested in that gets updated whenever the document changes and you’d query that for your report, the alternative would be to open each and every document, load it into memory, and you can see how complicated and constraining it will get. Think if it as recalculating the results in a way.

So you can see that duplicating the data isn’t such a big problem. Sure you will end up in scenarios where the data is updated, but this is mitigated against by having only one place that is in charge of changing it and therefore you isolate that transaction there.

Business

More than anything we write software for people, and that’s even more true in the case of SURI. It is important then to have a good relationship with the people familiar with the business, because they have all the expert knowledge you need to write the software for the people who will eventually use it. When you write software, you’re writing for people and the business. The real job of a programmer is to solve problems using a computer, the computer part is secondary to the job. What tools you use, how you do it, and all that other stuff is important, but not more important than solving the problems of the business. We’re engineers of a different breed, ones that work with logic, switches, instructions, and processes, instead of with cogs, gears, bridges, or cement, but in the end we all build for people.

The way we interact with the business in the SURI project is very interesting. On site we have several people who manage the project who are experts in specific areas, such as corporate returns and declarations, audits, levies, etc. These experts give us specifications for new features, bug fixes, and new versions of existing features. The way it works is that they raise new issue (we call them SQRs) with descriptions of what needs to be done. Sometimes their descriptions are accurate enough to be able to do the work without asking them for further clarification, but often we seek them out in a short face to face meeting to get a better picture of the issue to be solved in their words, and our jobs is to translate that into code or fixes to the software. Without these people, we would literally have no job.

Conclusions

So I’ve learned a lot while working here and it’s only been about a year (maybe more by the time you read this), and a lot of it has been very valuable to my future development in my career. I consider that I’ll be more valuable after exiting this project and I think you’d agree. Anyways, I really don’t know how to end this blog post other than maybe asking what you might have learned during your current job or better yet your first job experience in a big project. I’m also open to any questions or further clarification about what I’ve written up here. This is in no way meant as a rant or as a critique, as in a bad review, of my current position, but as an observation of what I’ve been doing and learning over the past year. So with that, let me know what you think and I look forward to writing the next one.