Engineering SaaS: An Agile Software Approach Q&A

For context, please make sure you've attended the webinar, presented by David Patterson.

Q: What should someone do in order to get familiar with SaaS or even get to work on it? Are there any courses/certifications or master's degrees in this field?
A: Not as far as I know. You can get a certificate for taking our courses from EdX (edx.org), CS169.1x and CS169.2x. More info here.

Q: Is this talk part of your SaaS course at coursera.org?
A: We did our first course there, but then switched to EdX a while ago.

Q: How do you account for the real first step in Waterfall, good requirements definition?
A: If the question is, what do you do in Agile instead of an elaborate requirements plan of Waterfall, the answer is User Stories.

Q: Which of those studies is the CHAOS study (which is widely disbelieved)?
A: It is Johnson 1995. I find it easy to believe the results since: 1) It is consistent with the other surveys I quote. 2) Johnson has done the CHAOS study every 2 years since 1995. 3) He is interviewed in magazines about the results and is considered a something of a pundit in the field.

Q: Is Part 2 available now?
A: If you mean CS169.2X, yes. The official next start date is sometime this summer (we haven't set the date yet), but I think you can take it on your own without ability to get a certificate now, kind of as a self-paced course.

Q: Is agile software development time-consuming due to a lot of iterations?
A: No. The idea is to keep iterating until customer is happy or you use up the budget. When you work with the customer, the customer gets the best bang for the bucks.

Q: Could you suggest any good first online course in programming [to prepare for] CS169.1x? How proficient should someone be? Is a CS graduate ok?
A: Being a CS grad gives you plenty preparation. We say mastery of an object-oriented language like Java. Juniors take the Berkeley course, if that helps.

Q: Could you comment on the "new" critical success factors that will substantially improve the results (e.g., as per the survey results, budget being met, relevant applicability of the business process being solved). The techniques have been around for decades.
A: The Agile Manifesto is basically rejecting the careful planning/business-aware process and replacing it with a quick iteration process with the customer continuously involved. It is a very different approach, and less likely for developers to do wasted work, and makes it more likely to implement the most important features first. The CHAOS surveys (see above) suggest better results from Agile processes.

Q: Could you comment on the "new" critical success factors that will substantially improve the results (e.g., as per the survey results, budget being met, relevant applicability of the business process being solved). The techniques have been around for decades.
A: The Agile Manifesto is basically rejecting the careful planning/business-aware process and replacing it with a quick iteration process with the customer continuously involved. It is a very different approach, and less likely for developers to do wasted work, and makes it more likely to implement the most important features first. The CHAOS surveys (see above) suggest better results from Agile processes.

Q: What are the impacts of the "popular" crowd-sourcing approaches, or the open-sourcing development approaches?
A: Agile is very popular today with open source projects, if that is the question. A recent survey of 66 large, distributed software projects found that the majority used Agile.

Q: So, the classic use case does contain several stories? How many?
A: Yes. 20 to 80 is a typical start. As I said, depending on project, can be 100s.

Q: Should productivity include a priority multiplier as well as a difficulty measure, since you can work on low priority hard things and seem to have good velocity?
A: Priority is the job of the Product Owner with Pivotal Tracker. He/she places scenarios (user stories) in the work queue in priority order, so that people do important tasks first.

Q: Should productivity include a priority multiplier as well as a difficulty measure, since you can work on low priority hard things and seem to have good velocity?
A: Priority is the job of the Product Owner with Pivotal Tracker. He/she places scenarios (user stories) in the work queue in priority order, so that people do important tasks first.

Q: Two related questions. Are you establishing a new set of "best practices" replacing prior best practices? What are the characteristics of the types of applications expected over the next 3-5 years that this programming style needs to address?
A: Software as a Service apps in general. Not for tasks that are under regulatory control or safety critical. A software engineering textbook asks 10 questions (below). "No" suggests Agile, "Yes "suggests Plan and Document:

  1. Is a specification required?
  2. Are customers unavailable?
  3. Is the system to be built large?
  4. Is the system to be built complex (e.g., real time)?
  5. Will it have a long product lifetime?
  6. Are you using poor software tools?
  7. Is the project team geographically distributed?
  8. Is team part of a documentation-oriented culture?
  9. Does the team have poor programming skills?
  10. Is the system to be built subject to regulation?

I am not sure I agree with all the questions (e.g., the Estler study above had geographically distributed teams), but you get the idea from the questions of applicability.

Q: When will the next iteration of CS169.2x be offered on edX?
A: The official next start date is sometime this summer (we haven't set the date yet), but I think you can take it on your own without ability to get a certificate now, kind of as a self-paced course.

Q: Do you have any experience applying this methodology with existing (legacy) code?
A: Yes. I think I answered that one during the webinar, but surprisingly, Agile does work well with legacy code. The first step is to establish tests without making changes for code going to modify. Chapter 9 of Engineering Software as Service is exactly about this topic.

Q: When do you do the estimating of points? You mention doing this during a standup, but you're only supposed to cover 3 questions there, aren't you?
A: Good point. Answer comes from friends at Pivotal labs: "We have a one-hour meeting each week called the Iteration Planning Meeting. Prior to that meeting, the Scrum Project Manager (PM) should have the stories for the next iteration in the backlog, prioritized appropriately. During that meeting, which is run by the PM, the team goes through every story in the next iteration, makes sure that the meaning of the story is clear, and points the story."

Q: Regarding scrum, do you recommend (especially) that the product owner rotate through roles? What did the last line of that slide say?
A: The suggestion is to rotate the roles so that different people on the team play them. Particularly important for multiple people to act as Product Owner.

Q: Since the scope of the project is so prone to change, do you have to constantly change the cost? If so, do costumers react OK if you are constantly updating the project cost?
A: I think I answered this one, but surprisingly customers go along with a pay-as-yougo model. We'll make the best effort to implement what you want for N weeks, which costs $X. (Section 7.4 of Engineering Software as Service covers Agile cost estimation.) I think part of the reason for their willingness is contract-based projects are often late, so you need to renegotiate after the contract ends, but the software doesn't have sufficient functionality.

Q: Why did you choose Agile to do SaaS?
A: We wanted students to develop apps for SaaS as part of a research project, which eventually led to this course.

Q: How do you treat bugs that are not noticed during an iteration?
A: Answer comes from friends at Pivotal labs: "Just create a bug story and add it to the backlog according to the Scrum Product Manager's (PM's) judgment of priority. If a bug was introduced by code written during the project, for which points were previously credited, then that bug gets no points. If a bug was extant in the code prior to the project starting (and, thus, no points have previously been credited for the code that caused it), then that bug gets points. In fact, many people would create such a pre-existing bug as a feature story, not as a bug story."

Q: What do you use for BDD/TDD in Python?
A: Django.

Q: You were introduced as academics. Can you please outline your experience with these processes and tools in industry? Thanks!
A: Armando has his own company building software for theaters, where he has (painfully) learned the value of Agile and eventually switched over. But as I said, we interact a lot with industry at UC-Berkeley, and even went to a half-dozen leading companies to get input on what to teach. To see the rationale, take a look at a paper that appeared in CACM.

Q: Is there a PHP-based Agile PM tool among those you mentioned?
A: Zend.

Q: Could you use these techniques for Global Software Development in Multicultural settings?
A: Yes, surprisingly. Take a look at this paper.

Q: How to apply Agile to a research team or special tasks such as Build Management, Configuration Management?
A: It's perfect for research. We are even trying to do it with hardware projects.

Q: If not using Ruby on Rails, can you still use open source tools?
A: Yes. Here is list from Figure 1.1 in Engineering Software as Service of SaaS programming frames works and corresponding programming languages:

  1. Active Server Pages (ASP.NET), Common Language Runtime (CLR)
  2. Django, Python
  3. Enterprise Java Beans (EJB), Java
  4. JavaServer Pages (JSP), Java
  5. Rails, Ruby
  6. Sinatra, Ruby
  7. Spring, Java
  8. Zend, PHP

Q: Does Agile methodology work better or worse than conventional approaches when dealing with remote teams, i.e., teams spread out geographically?
A: It appears to work fine, although it's better if teams are local. See this paper.

Q: Is the DSL implemented in Cucumber or does it have a more general significance?
A: I would say that Cucumber is a DSL implemented in Ruby.

Q: Does this apply equally across the spectrum of software complexity, criticality, etc.? Any recognized limitations?
A: I answered this above, but I can say that one of the instructors of the beta testing group for the book with lots of industrial experience said she disagreed that Agile wasn't good for safety-critical tasks, since it struck her that Agile was resulting in a much higher quality of software than she saw in industry in safety critical tasks.

Q: Hi, David and Armando, Thanks for the presentation. Is there a session on SaaS development itself?
A: CS169.1x and CS169.2x does deployment of SaaS apps as part of the course, using Agile techniques to develop the software.

Q: In QA, we're trying to prove the software works as intended (like TDD), but we also have to prove the software works well when the user doesn't do what is expected. Should TDD consider negative testing, boundary conditions, and the like?
A: Yes. There is some of that in BDD as well as TDD, but there could be a million user stories that don't work, so emphasis is on "happy cases" rather than "sad cases"

Q: How does cucumber compare to similar tools, e.g., Robot Framework?
A: Not familiar with that one, sorry.

Q: How does TDD work for scientific software? Where there's lots of floating point?
A: Yes. The difference is that the customer understands science, so you want user stories to be in language the customer and developers are comfortable with, which should be the science. So it's fine in equations, floating point in user story.

Q: OK, this might work for imperative programs. With the current trend to go functional, what do you think are the added benefits of these techniques for Declarative (Functional, Logic/Relational) languages?
A:  Ruby includes functional. Agile iterations with customers and BDD/TDD are good ideas for programming no matter what the language style.

Q: You didn't touch on build servers as part of the release pipeline. I was curious if you guys use build servers to compile and run all the unit tests or if you have someother solution? Thanks. Great presentation.
A: Continuous integration (CI) testing naturally fits with Agile. Every time commit, runs all the scenarios/user stories to see if you broke anything (become red). In the fall version of the Berkeley course (which will be recorded and made available as a MOOC in September) we will be using Travis for CI.

Q: Which programming language/framework should be preferred for SaaS (web applications/web services)?
A: We think Ruby on Rails has the best tools for SaaS.

Q: Getting developers who are new to Agile to focus on short productive scrums is particularly difficult. How do you overcome this?
A: As far as we can tell, you have to tell them to suspend their disbelief and try it a while to see if it works. I also point out the impressive failure rates of traditional approaches; if the old approach was better, why are 80% to 90% of projects late or terminated?

Q: How to break down user stories into smaller stories and estimate everything in the product backlog when you don't know all the details yet, and not fall into the Waterfall approach of doing all the analysis first?
A: Start with customer and make lots of user stories, then prioritize which ones to do first. The idea is there should not be a lot of effort in doing the user stories, as you just need a good starting point, and then as you get that working, figure out which of the many remaining ones to do next.

Q: Not a question but a possible helpful tool to pass along. This is essentially Cucumber for the .NET guys out there: http://www.specflow.org/specflownew/.
A: Thanks.

Q: In terms of new business, do you think Agile could put together a software that could arrange partnerships via asking questions and be paid direct by government for their patents and copyrights immediately versus waiting for legal arrangements?
A: Agile can develop all kinds of software. If this project is feasible, then you could use Agile to build it. Not sure it's feasible.

Q: One of the important functions of stories, BDD, and sprints is to create a contract among Product Owners, Developers, and Testers for what work is to be done and that is ALL the work to be done, preventing deadly scope creep.
A: Good point.

Q: It is not clear to us (I have students with me) the relationship of the methodology (which looks great) with "software as a service." Could you elaborate on that?
A: You can certainly do Agile for non-SaaS apps. It's just that SaaS involves rapid software evolution by its nature (a single copy of software is so much easier to change than shrink-wrap software), and so you need a software development methodology that embraces change versus careful planning and documentation ⇒ Agile.

Q: For some projects, it is really difficult to find time to sit down with end-users. [Is this] the "Achilles Heel" for the Agile method, or is there a way to accommodate such circumstances?
A: You need to have someone act in that role for Agile to work: A more exaggerated version of a Scrum customer advocate.

Q: How agile can you be when working with a vendor?
A: Answer comes from friends at Pivotal labs: "If we are integrating with an external vendor with an existing, mature, functional API, then that should be pretty much arms-length, and not much coordination is necessary. If, on the other hand, the vendor needs to do work on their side to enable us to do work on our side, we would always seek to have them co-located with the product team during that integration work. We don't necessarily want to pair with them on their work, but we do want them physically close to us during the integration, to minimize communication latency during that tight integration period."

Q: How to use TDD with integration tests, integrating modules developed independently?
A: TDD and BDD go together. The idea is BDD stories typically help with integration tests rather than TDD, which is typically unit tests. (Section 8.8 of Engineering Software as Service covers Unit vs. Integration tests.)

Q: I have customers that "don't have time" to spend evaluating software every two weeks.
A: (It’s a weird perspective if they want software they like.) Then they need to have someone act in that role for Agile to work: A more exaggerated version of a Scrum customer advocate.

Q: Statement, not question: I've worked extensively with geographically-distributed teams with pair-programming. I've found they need to be in close time zones; in my experience no more than ~2 hours.
A: Makes sense. Thanks for the note.

Q: Any publications on Cucumber?
A: There is a book for "Cuke," as it is known, by the authors of the tool: http://pragprog.com/book/hwcuc/the-cucumber-book.

Q: Startups: [How] do Agile programming methodologies interact with the Lean Startup method of rapid customer iterations?
A: As far as I know, Lean Startup method is based on Agile ideas

Q: What if developers need more time to think how to implement a user story? Isn't Agile a bad approach then? Since we are in a fixed time.
A: If they need more than 1 or 2 weeks to implement a user story, it should be broken down into smaller, simpler stories. The philosophy of Agile is that to avoid wasted work (since the customer may change their mind once they see the prototype), go from working prototype to working prototype with customer review.

Q: What is the Agile/Scrum answer for work with project/features that are usually totally new to developers and they cannot estimate them in a satisfying way?
A: At Pivotal Labs they also have a daily standup for whole company (~100 to ~200 people), where people can ask questions like "Does anyone have experience with…?" or "We are having problems with X, does anyone have advice…?"

Q: How about using Agile methods to develop SaaS versus/plus using SaaS tools to improve Agile methods?
A: I don't quite follow the question. They were developed independently as ideas, but SaaS tools pretty much support the Agile approach to development now.

Q: Would you recommend to have an Agile/Scrum-like project with a small team? Let's say 2 or 3 developers.
A: Absolutely. Get together every day at the same time to go over questions. If it's really just 2, why not pair-program?

Q: Is it practical to rotate the job of the product owner? This person must have the ability and authority to prioritize user stories and make product decisions. Shouldn't the PO be from the business, and empowered with decision-making power?
A: Sorry, it was confusing. Scrum has a "product advocate" (called confusingly the Product Manager within the team) who at daily scrums is supposed to represent the customer's perspective and signs off on user story completion. You still meet with real customer in each iteration to see if they are happy with what is being done, and what should be next step.

Q: Statement not comment: Kent Beck in his book, Extreme Programming Explained, says that the "customer" should be an actual user of the system.
A: See previous answer. There is confusion about the "product advocate" role in daily Scrum vs. meeting with the real customer as part of Agile iteration.

Due to the volume of questions received during the May 8, 2013 ACM Learning Webinar, not all audience questions were answered during the live event. Presenter David Patterson answered the questions listed here offline.

OLIVER WENDELL HOLMES

One's mind, once stretched by a new idea, never regains its original dimensions.

OLIVER WENDELL HOLMES

Safari Books Online                         Books 24x7