Sunday, May 02, 2010

Xml Document Transforms (XDT) for any XML file in your project

There have been several requests floating around to be able to use XDTs (the technology behind Web.Debug.Config/Web.Release.Config) with other XML files within the project…  To make that feasible I wrote a XmlDocumentTransform.targets  file which can generically transform any XML file using the standard Web.Config Transformation syntax introduced with VS 2010…

Learn more about XDT & Web.Config Transformation here…

Now to get started first download XmlDocumentTransform.targets file from my Skydive…

Follow the below simple steps to get transformation working for any well formed XML file in your project…

  • Step 1: Save the downloaded XmlDocumentTransform.targets to %ProgramFiles%\MSBuild\Microsoft\VisualStudio\v10.0\Web\XmlDocumentTransform.targets

image

NOTE: I would highly encourage you to make a copy of the Microsoft.WebApplication.targets file as backup before you do Step 2 below, as if this file is modified incorrectly then your VS 2010 instances might start showing funny problems which will be virtually impossible to debug and the only option left with you will be to repair/uninstall-install VS 2010… (i.e. proceed at your own risk :-))

  •  Step 2: Put following line of code in Microsoft.WebApplication.targets file just before closing of the Project node i.e. before </Project>...  
    <Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\XmlDocumentTransform.targets" Condition="Exists('$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\XmlDocumentTransform.targets')" />
    The Microsoft.WebApplication.targets file is located at %ProgramFiles%\MSBuild\Microsoft\VisualStudio\v10.0\WebApplications... 

image  
NOTE: Changing the above targets file will allow you to use this functionality with all the Web Application Projects (WAPs), if you just want to change this for the current project then you can put the same Import node in .csproj or .vbproj as well...  If you use per project model then you can also check in this file into source code control and have your team use it seamlessly…

  • Step 3: Open your .csproj/vbproj file and insert the below property in <PropertyGroup> section  <AllXmlsToTransform>Settings.xml;app.config</AllXmlsToTransform>

image

 NOTE: The Settings.xml or app.config can be replaced with the name of Xml files you want to transform…

  • Step 4: Insert <OnAfterTransformWebConfig>TransformXml;</OnAfterTransformWebConfig> similar to #3 above but do not modify the TransformXml; text here…  This is the actual hook which ties in your project to this generic XDT system...  After making the Step 3 & Step 4 changes your project file should have below content…

image

  • Step 5: Create Setting.Debug.Xml or similar files and put XDT syntax in them... You need to make sure in the .csproj/.vbproj file of yours you have the DependentUpon property is set like the example below:
        <Content Include="Configuration\Settings.xml" />
        <Content Include="Configuration\Settings.Debug.xml">
          <DependentUpon>Configuration\Settings.xml</DependentUpon>
        </Content>
        <Content Include="Configuration\Settings.Release.xml">
          <DependentUpon>Configuration\Settings.xml</DependentUpon>
        </Content>

image

NOTE: The above will allow your solution to look pretty, i.e. just like web.debug.config and web.release.config files show nested under web.config, your *.$(configuration).* files will show nested under your parent file too…

With the above 5 steps you are all set to use XDT with any of the deployment models covered in the Overview of Web Deployment Post…  In a way above steps harness the power of Web Publishing Pipeline (WPP) extensibility model and you can do several other extensions like above if you are familiar with MsBuild sytax…

SAMPLE:   The remainder of the post is just showing you the steps to test whether the changes you made worked or not (i.e. the remaining half of the post is just playing with what you already accomplished in the first half)… :-)

  • To test the above target file I created the below MVC 2.0 project structure:

image

  • Once you put DependentUpon node in your project file you will have to click the  “Show All Files” icon on the solution explorer for VB Projects to see Settings.Debug.Xml
  • My Settings.xml file looked as below:

image

  • My Settings.Debug.xml file looked as below:

image

  • To test out I tried simple “file system” publish (Right click on project and say Publish)… The new WAP Publish dialog for me looked as below:

image

  • After publishing my C:\TestPublish folder looked as below:

image

  • Note that Settings.Debug.xml was removed from my final publish location as it is not required for my web to function and the content of Settings.xml file were transformed and looked as below:

image

TeamBuild/Commandline Approach: You can also use your new transformations from TeamBuild/MsBuild by using the below command (from VS 2010 Command prompt if you are trying locally):

MsBuild MyXDTTestProject.csproj /t:TransformXml

The output of the command line transform should look as below:

image

As specified above your transformed XML will be stored in obj\$(configuration)\Settings.xml… 

As such feel free to open the XmlDocumentTransform.targets file which you download, I have tried to put as much comments as I could to make it readable…  If you go through it I am sure you will be able to do many other cool things out of it…

-Vishal

48 comments:

Matt D. said...

I found a problem in the XmlDocumentTransform.Targets file:

line 100 should read:
...Condition="'$(AllXmlsToTransform)'==''" ...

So the property is not overridden every time.

Good stuff though, thanks.

Anonymous said...

Is there a way to do this that doesn't require "Microsoft.WebApplication.targets" to be changed... As in could you make the change just in the proj file for the project you want to do it in and reference your target file relatively?

Anonymous said...

In trying to place the target file relative to my project I have changed:




...
logging.config;web.config
TransformXml;
...

...


...


Then in the XmlDocumentTransform.Targets file I have changed:




Now I'm new to MSBuild so there could be something that I am doing wrong here. Unfortunately, its not really working (the web.config is still transforming correctly), it just copies up all the logging.*.config's and does no transform.

Anonymous said...

Does this work for XBAP projects?

Anonymous said...

Are there any plans for supporting this functionality stand alone or through prior versions of Visual Studio? I'm working in an environment where an upgrade from VS2008 -> VS2010 is not feasible in the short term. This project requires a large number of different environment configs, so given some solution is necessary to manage them I'd prefer to match the syntax of VS2010's web.config solution for future compatibility. However, I'd also prefer not having to write a complex temp parser to do merging until we move up to VS2010. Are there any options for this?

Anonymous said...

When you don't want to depend on vs, you can use XmlPreprocess from Codeplex

ipad case said...

wow, that is really detailed. might take me awhile , i'll go through it. thanks for sharing!

Ben Adderson said...

Many thanks for the post!

I've been trying to get this to work on one of my projects, with a degree of success. We have a /Config directory off the root of our solution, which contains the .config files I'd like to transform. The transforms are working OK, but unfortunately the transform files themselves are present in the output.

If I move the config files (and their transforms) into the root of the solution without making any other changes, then only the transformed config file is present in the output.

Any idea why this happens, and what I can do to get around it?

Cheers!

Anonymous said...

Many thanks for the post!

I've been trying to get this to work on one of my projects, with a degree of success. We have a /Config directory off the root of our solution, which contains the .config files I'd like to transform. The transforms are working OK, but unfortunately the transform files themselves are present in the output.

If I move the config files (and their transforms) into the root of the solution without making any other changes, then only the transformed config file is present in the output.

Any idea why this happens, and what I can do to get around it?

Cheers!

Vishal R Joshi said...

To exclude transform files or any other files from deployment you should set the "Build Action" to "None" in the Properties of the file.. i.e. select the file, hit F4, when you get the properties grid change the "Build Action" to none... This will tell the Web Publishing Pipeline (WPP) to exclude the file...
Thx
Vishal

Ben Adderson said...

Setting the build action to none certainly does prevent the transform files from being deployed, but it also prevents the transform from being applied as well.

Is there no way to have the transform applied whilst keeping the transform file out of the output, without resorting to having all config files in the root of the solution?

Vishal R Joshi said...

Hi Ben,
Can you share your desired directory structure i.e. Solution --> Project --> Config.xml & Config.Debug.xml or does it look different?
Essentially somewhere in the Web Publishing Pipeline (WPP) we will need to mark your transform files for exclusion... Similar to what you can find if you search for "OperatingFileCollection Condition="$(ExcludeTransformAssistFilesFromPublish"
within the XmlDocumentTransform.targets file that you downloaded...
You can perhaps add similar lines of code to PostTransformXml target after the transformation is complete...
Hope this helps...
-Vishal

Ben Adderson said...

Sure, our desired directory structure is
Solution
--> Project --> Config (directory) --> ConnectionStrings.config & ConnectionStrings.Debug.config & ConnectionStrings.Release.config

As mentioned previously, if I take these three files out of the Config folder and drop them into the root of the project, it works entirely as expected.

Vishal R Joshi said...

This is interesting coz the same structure with following enteries in the project file works for me...

Content Include="Configuration\Settings.xml"
Content Include="Configuration\Settings.Debug.xml"
DependentUpon Configuration\Settings.xml DependentUpon
/Content

Can you confirm you have something similar in your project file too...
Thx
Vishal

Ben Adderson said...

Yep, I have the following in my .csproj file:

Content Include="Config\ConnectionStrings.config"
Content Include="Config\ConnectionStrings.Debug.config"
DependentUpon ConnectionStrings.config DependentUpon
Content
Content Include="Config\ConnectionStrings.Release.config"
DependentUpon ConnectionStrings.config DependentUpon
Content

Vishal R Joshi said...

Ben,
In project --> Properies --> Package/Publish Web can you make sure "Only files needed to run this application" is set in the dropdown... It seems that is the only real difference that I can think of... Otherwise it seems you might have to make copy of your project with the config files etc and send it to me at Vishal.Joshi@Microsoft.com so that I can try and debug it better...
Thanks
Vishal

Ben Adderson said...

OK, I've put together a simple example solution that exhibits the problem, and emailed it to you. Thanks in advance for your help!

Ben

Anonymous said...

After running your MSBuild (I tried to import it like in your blog entry and directly in csproj I was working with) i get an exeption:
Error 1 The "TransformXml" task failed unexpectedly.
System.UriFormatException: Invalid URI: The URI is empty.
at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)...[Stack trace]

Project type was WCF Service.

I tried to run this task manually from command line and I get the same exception. Any ideas why this is happening?

PS: Relevant output from command prompt:
Build started 2010-07-23 15:00:52.
Project "D:\Projekty\Tests\ola1\ola1\ola1.csproj" on node 1 (TransformXml target(s)).
ValidateGlobalPackageSetting:
$(PackageAsSingleFile) is True
$(PackageFileName) is obj\Debug\Package\ola1.zip. Validating...
GenerateTargetFrameworkMonikerAttribute:
Skipping target "GenerateTargetFrameworkMonikerAttribute" because all output files are up-to-date with respect to the i
nput files.
CoreCompile:
Skipping target "CoreCompile" because all output files are up-to-date with respect to the input files.
CopyFilesToOutputDirectory:
ola1 -> D:\Projekty\Tests\ola1\ola1\bin\ola1.dll
CollectFilesFromIntermediateAssembly:
Gather all files from Project items @(IntermediateAssembly). Adding:
bin\ola1.dll to bin\ola1.dll
bin\ola1.pdb to bin\ola1.pdb
CollectFilesFromContent:
Gather all files from Project items @(Content). Adding:
Settings.Debug.xml;Service1.svc;Web.config;Web.Debug.config;Web.Release.config;Settings.xml
CollectFilesFromIntermediateSatelliteAssembliesWithTargetPath:
Gather all files from Project output (IntermediateSatelliteAssembliesWithTargetPath). Adding:
CollectFilesFromReference:
Gather all files from Project items @(ReferenceCopyLocalPaths,ReferenceComWrappersToCopyLocal,ResolvedIsolatedComModu
les,_DeploymentLooseManifestFile,NativeReferenceFile).
CollectFilesFromAllExtraReferenceFiles:
Gather all files from Project items @(AllExtraReferenceFiles). Adding:
PipelineCollectFilesPhase:
Publish Pipeline Collect Files Phase
CollectXmlsToTransform:
Found Settings.xml
PreTransformXml:
Transform input: Settings.xml
apply: Settings.Debug.xml
output: obj\Debug\Settings.xml
TransformXmlCore:
Transforming Source File: Settings.xml
Transformation succeeded
D:\Projekty\Tests\ola1\ola1\XmlDocumentTransform1.targets(257,5): error MSB4018: The "TransformXml" task f
ailed unexpectedly.\r [D:\Projekty\Tests\ola1\ola1\ola1.csproj]
D:\Projekty\Tests\ola1\ola1\XmlDocumentTransform1.targets(257,5): error MSB4018: System.UriFormatException
: Invalid URI: The URI is empty.\r [D:\Projekty\Tests\ola1\ola1\ola1.csproj]

James said...

Does this only work with web projects?
More specifically, I'm thinking I'd like to have a separate config project/assembly that will be reused by several types of projects (including a web app, an NUnit/WatiN web acceptance test project, and an NUnit integration test project) to get their environment-specific config values.

Basically, I'd like to keep it DRY when I have to change a web service URL, etc. -- I don't want to have to go into all the other projects that consume my assembly that contains the actual web reference.

Vishal R Joshi said...

Hi James,
Check out the post http://vishaljoshi.blogspot.com/2010/05/applying-xdt-magic-to-appconfig.html, I think you will have to do something similar to what I did there in your special project...
Thanks
Vishal

James said...

Thanks for the reply, Vishal, but unfortunately, the article you pointed me to does not get me all the way there, and now I'm questioning whether it's even possible...
I've now spent the last 8 hours trying to adapt the "applying xdt magic to appconfig" approach to my project, and I've got nothing.

The problem that I can't seem to overcome is that when I use the xdt magic to transform an xml file in a class library (let's call it TestUtils), I can't seem to make that xml file end up in the output directory of the consuming project/application (let's call this one MyTestProject).

I have no problem getting the xml file to transform properly and end up in the "TestUtils" class library's output directory.
The problem is that it never ends up in the output directory of "MyTestProject"

Any ideas? Or should I just go in a completely different direction with this?

Vishal R Joshi said...

Hi James,
You need to hook up an after build/XmlTransform task in the project file of your class library... Essentially you modify your project file to add the copy task as mentioned in http://msdn.microsoft.com/en-us/library/3e54c37h.aspx and then pick up the generated XML file and copy it to a location of your MyTestProject...
If the build order of your MyTestProject is after your class library project (which I think is the case) then you can put the copy task as a post build step in your MyTestProject (you can do that in the project properties I think)... That will pick up the generated XML file and copy it where you want it to go...
If you have got to a point of generating the XML file then I do not think it would be nearly as difficult to get it copied to the right location...
Hope this helps...
-Vishal

James said...

I had been trying to get it to deploy to MyTestProject (and to any other consuming projects) by trying to include my transformed xml file as an Item (content, initially) inside the PostTransformAppConfig target to make use of the CopyToOutputDirectory property. Is that simply not possible?

Vishal R Joshi said...

Hi James,
I thought that the transformed XMLs are already getting copied to your output directory, is that not right?
The consuming projects might not be configured to pick up everything from the classlibrary's output directory so that step might have to be explicit...
Thanks
Vishal

James said...

If we had a static content file in the class library with CopyToOutputDirectory = Always, it will end up in the consuming project every time.
In this case, we're generating a file dynamically, so the same approach isn't working.
I think the order of the targets' execution is such that static content files are getting copied to the output directory before the transform happens.
I'm afraid making the last step explicit (shifting the burden of finding all the desired files in the various libraries' output folders onto the consumer of the libraries) is a bit too heavy handed for my comfort (as it requires MSBuild project file modifications in several places rather than one centralized spot).

I was hoping to find a way to make it "push" instead of "pull"

Sayed Hashimi said...

Hi James, I work with Vishal.
I made a post about how you can use the TransformXml task outside of the WPP on my blog at http://sedodream.com/2010/04/26/ConfigTransformationsOutsideOfWebAppBuilds.aspx. Can you take a look at let us know if this will work with you?

Sayed Ibrahim Hashimi

James said...

Thanks, Sayed, I'll take a look

James said...

Sayed, I looked over your blog post, and I'm afraid I don't see anything that isn't already covered by Vishal here or in his "applying xdt magic to app.config" post. Am I missing something subtle here?

I think I'm probably just trying to do more than the TransformXml task is equipped to do.

James said...

Oops, I see the difference now... I'll see what I can do with it
And whether or not it works, thanks to both of you for helping!

James said...

Sayed, I tried to incorporate your approach, and while I much prefer yours (because of the decoupling from the WPP), I keep hitting the same wall. I'm just not very well versed in the black art of MSBuild, and I think I'll be forced to take a different, probably less elegant, approach which is more familiar to me.
Thank you both for your help.

Sayed Hashimi said...

Hi, I do not fully understand your road block. Can you put together a sample showing what you are trying to do and send it to me at sayedha [AT] [Microsoft] __DOT__ (com)?

Rob said...

Hi. Im following your post above, and step 5 doesnt seem to be working if my xml files are in a subfolder, but does work if they are at the root level.

Eg Settings.Xml does have the dependancies listed like Web.Config does, however, if i had Config/Castle.xml, this does not work?

Is this a known error or am i missing something?

Vishal R Joshi said...

Hi Rob,
On line 100 of the XmlDocumentTransform.Targets file there was a bug it should have an attribute Condition="'$(AllXmlsToTransform)'==''"... Due to that missing line your castle.xml was always getting replaced by the defaults Settings.xml and app.config which I had in the code... I have uploaded the new file at the same location which you can download and that should fix your issue...
thanks
Vishal

Rob said...

Hi Vishal

Thanks for that. After downloading the updated targets file, i was able to transform my castle.xml. One small point tho, how do i stop the castle.Debug.xml and castle.Release.xml from being included in the output when i do a publish?

One other point, when editing the csproj file to add the DependentUpon behaviour, if your xml or config file is in a subfolder, you need to include the folder name in the Content Include, but NOT in the DependantUpon. See below for example, noting that the DependentUpon is not included the folder location.



castle.xml


castle.xml


Ta

Rob

Rob said...

Hmm, the config example hasnt been displayed properly above, see below

Content Include="Configuration\castle.xml" />
Content Include="Configuration\castle.Debug.xml">
DependentUpon>castle.xml
/Content>
Content Include="Configuration\castle.Release.xml">
DependentUpon>castle.xml
/Content>

Rob said...

Hi guys

Does anyone know how to stop all the transform files themselves being present in the output after a publish.

I have a structure similliar to Bens (previous comment) in that i have a config folder, but all the transofm files are also being output.... I see Ben and Vishal took the descussion offline. Did you get an answer to his?

Rob

Ben Adderson said...

Hi Rob,

I put together a sample solution that exhibited the problem and emailed it to Vishal (that was the 7th July), but as yet I haven't heard back from him.

Ben

Vishal R Joshi said...

Rob & Ben,
Sorry for the delay in response, I think Ben's email got missed in my inbox... Try the below:

Unload your project and edit your project file and set the property ExcludeFilesFromDeployment to
Configs\ConnectionStrings.Debug.config

Let me know if it works...
Thanks
Vishal

Rob said...

Hi Vishal

That worked. Thank you very much.

Your blog has really helped in getting this setup and working

Rob

Anonymous said...

I can't download the targets file? I get an error on SkyDrive saying "This item might not exist or is no longer available"

Darrell Sveistrup said...

I had same issue as Rob with the transforms being copied when in a sub folder.

I used exclude also but with *, so that did not need to specify each config

config\AppSettings.*.config;/ExcludeFilesFromDeployment>

Is there any other way?
as I have multiple config and xml to transform.

Vishal R Joshi said...

Hi Darrell,
I do not think ExcludeFilesFromDeployment will perhaps take wild cards, can you confirm whether this still does not work if you specify all the files?
Thanks
Vishal

Vishal R Joshi said...

Hi Darrell,
Can you check the recommendation that Sayed has for Excluding using wild cards at http://sedodream.com/2010/08/15/WebDeploymentToolMSDeployHowToExcludeFilesFromPackageBasedOnConfiguration.aspx
Thanks
Vishal

Anonymous said...

Works great Vishal. Thanks!
My only issue is that I'm not seeing the transform files as nested if my xml and its transforms are in a subfolder. When I move the same files (and transform files) to the root of solution, it shows up nested in the solution explorer.
The transformations work just fine in both the root and subfolder.
Bhavdeep.

Anonymous said...

Now there is a an easy way: just use a Visual Studio 2010 addon http://mcaf.ee/40zbh made by Sayed Ibrahim Hashimi

Anonymous said...

Hi Vishal

I have a setup that looks like the following:
web.config with transformation-files, appSettings.config with transformation files. The appSettings.config is set to be DependentUpon web.config, and all appSettings-transformation files is set to be DependentUpon appSettings.config.

If I set Build Action for web.config to be None, the appSettings.config file is transformed and published, but if I set the Build Action for web.config to be Content, only web.config gets transformed and published.

Is there any way to set up the config-files with this structure as described above and get both web.config and appSettings.config to be transformed and published, or do I have to have all .config-files on the root of the application?

Vishal R Joshi said...

Hi Anonymous,
This feels like a strange behavior. The transform for app.config should not really be hampered by the BuildAction of Web.config file. Can you possibly send me a simple reproducible project which I can try to analyze locally. I can then try to check what might be going on or whether this is a bug.
Thx
Vishal

Anonymous said...

Hmm, I will certainly try to set up a basic project and check if it works there. If it do not work and you have to opportunity to take a look at it, it would be great! I will get back to you:)

Thanks a lot.