Auditing Agile Projects: Things to Consider

This article is one of a 7-part series on “Technology Auditing Strategies” co-authored by Roberto Calderon, Eric Stewart, and Dr. Paul Eder.

Many new technology projects in government are being implemented via Agile principles. Agile development has proven to be effective through its support of continuous review and feedback, communication, and flexibility. Agile software development provides a special set of considerations that auditors must take into account when planning and carrying out an audit.

First, Agile is a set of principles and values, rather than any specific methodology. The principles guide and help technology development teams to establish an effective and repeatable cycle. Agile places value on:

• Individuals and interactions over processes and tools
• Customer collaboration over contract negotiation
• Working software over comprehensive documentation
• Responding to change over following a plan

These values at first seem at odds with traditional types of documentation that auditors would want to request However, choosing Agile does not replace organizational or Government-wide requirements. Agile development does not negate the need to capture information needed to ensure continuity, consistency, and repeatability when team members turn over.

Agile’s purpose is to establish a different working path that combines best practices and methodologies by placing greater focus in the interaction of all members of the project team, clients, and users; receiving continuous feedback; and allowing changes in all phases of software development. Agile offers an alternative to the Waterfall Model for software development that focuses on a comprehensive plan leading to a complete end product after a certain period of time. The twelve key principles of Agile can be found in the Agile Manifesto.

When conducting audits for agile projects in general, auditors not only should consider the twelve principles and value emphasis of Agile, but also the following characteristics of an Agile project:

1. User Story and Backlog: Commonly emphasized when working under the Scrum model, user stories are created to record , usually with details on what the products should accomplish. They should be written to the most granular level possible. User stories could be used to get requirements and feedback but are not the same as just requirements. Let’s remember that requirements are a FORMAL description of a need, while user stories are usually INFORMAL. A backlog is where the stories, acceptance criteria for success are stored for reference and continuously updated and organized. The Backlog identifies what will be delivered and the sequence in which it should be delivered.
2. Iteration: Agile relies on repetition and ongoing of testing the target product. In some models, these iterations are called sprints. In Agile, portions of the software are developed, tested, and presented to the client in sprints (2-4 week development cycles). This way, requirements can be re-evaluated for every iteration. Requirements in Agile projects are presented through user stories that define a desired functionality or feature. The team can reference these user stories to modify, edit, create, or continue the current piece of software until the whole project is completed.
3. Adaptivity: Agile follows a rolling wave approach for planning, which identifies targets and objectives for each iteration or sprint, but still allows flexibility throughout work progress, thus allowing targets themselves to change based on updated requirements. User stories can be logically grouped into “Epics” to accommodate shifting changes in the organizations. Agile teams should have strong change management skills and be ready for changes in all phases of development or auditing.
4. Retrospection: Agile projects are characterized by constant and continuous reviews of previous work in order to look at what worked well and what did not. Retrospective meetings are critical to continuous internal process improvement and should lead to measurable objectives that can be immediately implemented into the next cycle.

Besides knowing the general characteristics of an Agile project, Auditors should be also aware of the professional environment in which audits will be conducted. In the federal sector, auditors should consider that:

• Federal process is slow. Despite the fact that Agile projects require constant feedback and review, it is important to consider that sometimes it takes longer when working on federal projects due to external factors outside the team’s control.

• Agile is “document light” not “document free.” Auditors will find a bit difficult to find a lot of complex documentation when auditing federal agile projects, due to its value emphasis on working software over documentation. However, choosing to align with Agile does not negate a program’s responsibility for maintaining adequate system and process documentation.

• Documentation shall, at minimum, include user stories, testing documents, and design and deployment documents. Combined with a slow federal process, difficulty to obtain documentation might create a delay in audit deadlines.

• User stories should be maintained in an accessible database. Jira is probably the most common. User stories should also follow the format of “As __________ I want __________ so that __________”, showing the business need and the intended audience (for instance: “As a system user I want a Log-in Screen to I can verify user credentials”).

• Testing documents show proof that the user stories function is a non-production environment. Examples of testing documents can include automated testing outputs or screenshots of before and after states.

• A security review of the design should be conducted and documented. The design document helps security personnel to understand the impact of the code on the system as a whole.

• Auditable deployment approval must exist, which gives the green light for the newly developed code to be implemented in the live production system. In addition, a Deployment/Operations team would be assigned for this task. This team is separate from the development team.

• The primary stakeholder, known as the “Product Owner” (e.g., the Federal program manager) should set the order of the backlog.

Overall, Agile provides some risks to traditional auditing preferences (e.g., the need for documentation of processes and plans). However, this does not mean that processes and plans should not be confirmed through direct documentation or other means. Auditor judgment becomes key in assessing whether the program owners have adequately applied Agile principles, while still adhering to the principles of good project governance. Constant communication with the Agile team becomes necessary in order to obtain all documentation and feedback.