Wednesday, July 13, 2011

Documenting Key End-to-End Deployment Scenarios

 

The IIS Web Deployment Tool (Web Deploy) and the deployment features introduced in Visual Studio 2010 enable you to automate many deployment tasks, but we have heard you all that many common scenarios not yet documented fully. We are addressing this need by creating step-by-step walkthroughs that will guide you from beginning to end through scenarios that address common real-world needs.

UPDATE:  We just published the Enterprise Deployment Series which can be found at:

Interestingly, remaining of this blog post is not the documentation of the solution but actual articulation of problem statements. This post presents the first set of scenarios that we have identified and solicit your feedback to help us determine they are representative enough or not. If you have any feedback as usual you can post them here as comments or feel free to send me an email at Vishal.Joshi@Microsoft.com.

Scenario 1: Enterprise Deployment with Continuous Integration

In this scenario, a solution that includes multiple web application projects is deployed to test, staging, and production environments, using a continuous integration process for staging and production.

Target Environments

The TESTING environment consists of a server that runs IIS 7.5 and a server that runs SQL Server 2008 R2. The developer machine has a network connection to the test servers, and the developer uses one-click publish to deploy to testing environment.

STAGING consists of a web farm (2 servers running IIS 7.5) and a database server that runs SQL Server 2008 R2. The developer machine has network access to a TFS server that acts as a source code repository, and the TFS server has network access to the staging servers. (The developer machine does not have direct access to the staging environment, and the developer does not have administrative rights on the staging servers.) Team Build builds the Visual Studio solution, runs unit tests, and publishes to staging. Each time that Team Build performs build and deployment, it simultaneously creates a deployment package (web deploy .zip file) for use in deploying to production.

The PRODUCTION environment mirrors staging except that a firewall (or perhaps even different domains) prevents direct access for publishing from the TFS server to production. When a build is approved for production, the IT department uses the package created when TFS publishes to staging to deploy to the production servers.

The diagram below illustrates this scenario:

Enterprise_Scenario_Diagram

Enterprise scenarios may have a QA environment set up in a manner similar to staging; however, for the purposes of demonstrating how to set up deployment it's not necessary to include that here, because the process would be similar to the process for setting up staging.

Visual Studio Solution

The Visual Studio solution to be deployed consists of multiple web application projects, a class library project, and a unit test project. Deployment must take into account the following considerations:

  • One of the projects uses ASP.NET membership functionality, and the membership database must be deployed. Account information can be deployed to test but not to staging or production.
  • One of the projects uses a SQL Server database that is accessed using the Entity Framework (Database First, using an .edmx file). On initial deployment to any environment, only the structure (schema) should be deployed. For any database deployment after the initial deployment, data already entered online in that environment must be preserved.
  • The class library project creates an assembly for a custom control that is used in one of the projects. This assembly needs to be installed in the GAC as part of the deployment process.
  • The custom control gets a default value from the registry. The registry value needs to be different in each environment and needs to be updated when the solution is deployed. (This particular use of registry settings is not common, but updating the registry is a common need, and this provides a simple way of integrating a registry update into the scenario.)
  • The Web.config file contains settings that must be different for debug vs. release builds, and settings that must be different for different target environments.
  • One of the web projects includes a folder for log files. The deployment process must not copy files in this folder from source to destination and must not delete files from the folder on the target server.
  • IIS settings for error handling and authentication must be set up on the target server during deployment. For the test environment these can be the same as the settings on the developer machine, but for staging and production the settings are different.

Some additional deployment considerations apply only to the automated deployment from TFS for staging and production:

  • Deployment should occur only if the unit tests are successful.
  • The web projects need to be precompiled before deployment.
  • The IIS settings for staging and production are taken from IIS on the TFS server. (This is a limitation of the current release of Visual Studio and Web Deploy; when the walkthroughs are updated for the next release of the software, hopefully IIS will not be required on the TFS server (keeping fingers crossed Smile)
  • App_offline.htm must be set up at the start of deployment and removed at the end.
  • Deployment activities should be logged. When deployment completes or fails, email notifications should be sent to designated recipients.
  • If deployment fails, the previous deployment's package should be redeployed, or the current deployment should be retried.
Tasks Illustrated

The walkthroughs for this scenario would guide you through the following tasks"

  • Downloading the Visual Studio solution to be deployed.
  • Setting up the test server.
  • Using one-click publish to deploy to testing servers:
    • Initial deployment.
    • Redeployment without a database change (for example, an update to code in a web page).
    • Redeployment after making a database schema change.
  • Setting up staging and production servers.
  • Setting up the build server.
  • Three deployments to staging (initial, web page change, database change).
  • Three deployments to production (initial, web page change, database change).

For the Visual Studio 2010 version of the walkthrough, database updates will involve running custom SQL scripts as part of the deployment. The scripts will be created manually; tools such as TSData and Red Gate can be used to generate such scripts, but those tools will not be covered in these walkthroughs. Eventually we will look at smoothing this flow as well.

Scenario 2: Enterprise Deployment for MVC and Entity Framework Code First

This is a variant of the first scenario that differs from it in the following ways:

  • The web projects are MVC instead of Web Forms.
  • Entity Framework Code First is used instead of Database First (no .edmx file).
  • TeamCity is used instead of TFS.

Scenario 3: Enterprise Deployment for Web Site Projects

This is another variant of the first scenario that differs from it in the following ways:

  • The web projects are web site projects instead of web application projects or MVC.
  • Web Deployment Projects (WDP 2010) are used.
  • One-click publish is not available for web site projects, so a web deployment package is used for deploying to test.

For those of you who work in enterprise environments, do these scenarios adequately represent the kinds of challenges you face in deploying ASP.NET web applications? Are any key pieces missing? We cannot answer every question in these walkthroughs, but if there are other issues commonly faced by your team, we can add solutions for them to the walkthroughs as well.

Your thoughts and feedback are welcome.  Also I want to thank Tom Dykstra, Bilal Aslam & Sayed Hashimi on our team who will be helping on putting together a lot of this content for you.

FEB 2012 UPDATE:  The work on these scenario documentation has started happening.  The tutorials are still being written but the sample app with an initial draft of the first part of the tutorials is available on MSDN:

http://code.msdn.microsoft.com/ASPNET-Enterprise-Web-6b2ad7cf

(The tutorials are in Word docs in a folder in the sample project.)

They’ll be published on the ASP.NET site most likely in the next couple of months.

Thanks for reading!!

-Vishal

22 comments:

Joseph Daigle said...

Hi Vishal,

I think it's great that you're going to publish some real documentation for these extremely common, yet non-trivial, deployment scenarios.

I have another one. Consider a solution with multiple ASP.NET web applications and multiple WCF service implementations (hosted using ASP.NET web projects to get the benefit of Web Deploy). In both staging and production the normal ASP.NET web applications are deployed to 2 or more front-end servers, and the WCF services are deployed to 2 or more back-end servers. The back-end servers are typically updated before the front-end servers are updated.

Jason Cox said...

These are comprehensive scenarios and I'm particularly interested in your MVC deployment scenario.

Vishal R Joshi said...

Bart Verkoeijen: What if we don't have a TFS server? Our source code is stored in a Mercurial repository. Used to be in Subversion.

Vishal R Joshi said...

Bart, eventually we will cover different source code controls as well, we will just start with TFS and keep moving to the next ones.

Vishal R Joshi said...

Jason, thx. Btw, do note that everything covered for Web Application Projects apply to MVC projects as well. MVC projects under the roof are Web Application Projects for these purposes :-)
-Vishal

Vishal R Joshi said...

Hi Joseph,
thanks for bringing up your WCF scenario. If your WCF services are IIS hosted then the exact same process will be applicable to them as well. But if you see any gaps along the way in the scenarios we outline then do not hesitate to reach out.
Thanks
Vishal

Jason Stangroome said...

Supporting database schema and core-data (eg country list) updates expressed as migrations in T-SQL are an important criteria for environments with strict DBAs. There is typically no opportunity to perform the schema diff to generate the change script in this scenario and it is important that the schema change script is in a language the DBA is comfortable with.

David Batten said...

I agree with Jason S. I'm in an environment that has strict rules around database deployments that forces us to manually provide database scripts for review and rollout. Additionally, I would like to see some documentation around "rollback" scenarios, if possible. Using your previous blog posts, I have been able to create all the CI/packaging scenarios you show here with custom templates, and adding rollback to that would be fantastic.

Anonymous said...

Looking forward to this documentation. Documenting integration/system testing before succesful production deploy would be nice to see.

Vishal R Joshi said...

Re: There is typically no opportunity to perform the schema diff to generate the change script in this scenario.
Hi Jason,
The schema diff can be generated from a mirror of production database and only the final SQL script can be sent out in the package. Is that not acceptable. In that case the DBA gets the SQL Script anyways. Thoughts?

Vishal R Joshi said...

Re: I'm in an environment that has strict rules around database deployments that forces us to manually provide database scripts for review and rollout.
Hi David,
In scenarios where DB deployments are strict do you think it is even feasible to provide a single .zip file which might contain that .sql file and then an integrated deployment can be initiated? Coz if that is really not even feasible then the option is to exclude DB deployment part of the flow and continue doing it manually as per the policies of the organization. Do you expect Microsoft can provide a solution here which could be generically usable?
-Vishal

Vishal R Joshi said...

Re: Using your previous blog posts, I have been able to create all the CI/packaging scenarios you show here with custom templates, and adding rollback to that would be fantastic.
Hi David,
The VS 2010 Rollback story is essentially archiving older .zip packages and re-running them. I know this is not ideal and we are looking at potential solutions but as of now the best way to accomplish this is via re-deploying older versions of the package. Note that Web Deploy will do the incremental deployment so this should work. If you face any issues on this then do write back.
thanks
Vishal

Vishal R Joshi said...

Re: Documenting integration/system testing before succesful production deploy would be nice to see.
Hi Anonymous,
thanks for the suggestion, we will look into it.
Thanks
Vishal

David Batten said...

RE: In scenarios where DB deployments are strict do you think it is even feasible to provide a single .zip file which might contain that .sql file and then an integrated deployment can be initiated?
One main problem I see to automated DB deployment is that the process executing the script must have substantial permissions on the target instance, due to the amount of DDL contained in most scripts. I don't have an answer for that. Works fine in a lab/test environment, but many admins get shaky when thinking of those rights in production. So, for now we do database deployments manually. Your answer to Jason on providing the final diff script in the package seems the most appealing in my situation, because I can provide that automatically generated script to the DBA for review. Some good documentation surrounding that procedure would be great, so I could use the docs in a proposed solution to the DBA's.

David Batten said...

RE: The VS 2010 Rollback story is essentially archiving older .zip packages and re-running them.
Yeah, that's what we currently do. It works, but we're always looking for solutions that require fewer "touches" by individuals, to reduce the chance of error and also to speed things up. Nothing sucks worse than a late night deployment that takes longer than intended.

Anonymous said...

This is very close to what we will be trying to achieve shortly on a new project with new stack for us. We also use WCF services, hosted in IIS and a data-tier application project.

This will be very useful information!

Thanks, Ben.

Carlos Cubas said...

I think you might have covered this in one of your points. But, in an environment where developers and operations(deployers)are segregated. It would be useful to have some documentation geared towards the operations people. ie. Managing the manual steps of the process from their perspective. Such as config changes from qa to prod etc.

Anonymous said...

What about versioning? I would like to see how versioning can be performed and the packages be altered to include this version information. Currently, my CI process does versioning (before any deployment steps, during the packaging step) in a really simplistic manner. We don't actually update assembly manifest's with the versions, instead we just drop a simple version.txt file into the applications root directory upon successful build/packaging steps of the CI process. Where would versioning fit in, in the local/developer scheme of things?

Robert Stackhouse said...

I've recently moved from a DVCS/Distributed Build environment (bzr/buildbot) into a centralized VCS/Build environment (TFS).

I second David Batten's point about "we're always looking for solutions that require fewer 'touches' by individuals."

This already exists, and it is called the automated gatekeeper: http://doc.bazaar.canonical.com/latest/en/user-guide/using_gatekeepers.html. One such gatekeeper is PQM (DISCLAIMER: I've never used PQM):https://launchpad.net/pqm.

I've previously set up an automated gatekeeper using bzr and buildbot. There was a bit of a learning curve (getting buildbot on windows was a bit of a challenge), but the docs for buildbot and bzr are some of the very best I've ever seen.

Part of the pain of doing software development is the human element. People are generally not boneheaded, but they do boneheaded things on ocassion. Regressions happen. Developers forget to run tests before committing to version control. Sometimes they willfully refuse to run tests in a "long running" test scenario. Having categories of tests (long-running, smoke, functional, integration, unit, etc), and an automated gatekeeper, solves the previous problem.

I think the "automated gatekeeper" workflow just works and that Microsoft should seriously investigate it. Gatekeepers—human and automated—and networks of trust give you the centralization of control that centralized source control alone promises but never quite delivers on without the bottleneck that centralized source control creates.

Branches should be local and cheap to create and discard. Making things difficult and/or painful is the fastest way to ensure that they won't be done. I know of at least one place that knows that they should be doing feature branches, but doesn't because of the overhead of maintaining multiple branches on a centralized store.

Branches aren't things that are meant to be stored and maintained in perpetuity. Branching is meant to offer a safe and fast way to allow people to revert to a known working state if things go sideways. Sometimes features get dropped for reasons of techincal complexity, adjusted priorities, or poor fitness within the existing product. The workflow should support this.

Here's what I'd like to do in TFS:
1) A programmer commits a changeset
2) A build slave (believe build agent in TFS parlance) receives a signal to a) locally deploy to IIS b) run the full battery of tests (including web tests—Selenium Webdriver flavor.
3) Build slave sends signal to build master:
PASS: a) changeset gets promoted to "higher" branch b) changeset gets automatically deployed to QA environment
FAIL: a) changeset gets reverted b) offending developer gets notified by email, system tray notification, scrolling LED marquee in break room, or any or all of the above.
4) Regardless of passing or failing condition, post reports (SpecFlow, MSTest, Opencover, or whatever) to a centrally available information radiator (buildbot calls this the waterfall. Google "buildbot waterfall).

This seems conceptually simple. After years of using things like NAnt, MSBuild, NUnit, MSTest, CruiseControl.NET, and buildbot, I know that although the concept may be straightforward, it is not easy to implement. All I would like is some support on how to make this happen using MSBuild/MSDeploy/MSTest/whatever. This could be in the form of a "how-to" or actual application support. I don't care. IMHO, this is the right way to do it. For the love of everything, please help make doing the right thing easy.

Unknown said...

Vishal, Have the steps to implement these scenarios been published yet? I'd love to give them a read!

Anonymous said...

A partially completed initial draft of the tutorial is available here with a sample application:

http://code.msdn.microsoft.com/ASPNET-Enterprise-Web-6b2ad7cf

Projected publication date for the completed tutorial on the ASP.NET site is within the next couple of months.

Daniel said...

It would interesting to include also a SSRS project. In many organizations this is the final outcome: a report.