Wednesday, July 15, 2009

Hi all

I did a follow up to my old post about how to loop around the elements of a message a couple of days ago.

As you can se from the comments to my followup, Thiago mentions that there isn’t any performance hit because everything is done inside an atomic scope. Now, Thiago is naturally correct when he states, that no persistence occurs inside an atomic scope. But as you can also see from the comments, I wondered about what to do about using a request-response port – so I just had to test it.

For BizTalk 2009, just as the previous versions, you CAN’T have both the send shape and corresponding receive shape used for a request-response port in the same atomic scope. At compile time you will get this error: “an atomic scope may not contain or call a service or scope that contains both the send and the corresponding receive of a requestresponse operation on a 'uses' port or servicelink”.

This may seem odd at first glance, but it is due to the fact, that in order to make sure the ACID properties are respected at runtime the orchestration engine submits messages that are sent out to the messagebox, but it doesn’t release them for the messagebox to route until the atomic scope has finished. This way, if an error occurs inside the atomic scope, no messages have actually been sent, and therefore, BizTalk can guarantee a consistent state.

So basically, if the message isn’t released until AFTER the atomic scope, then there is no way you can receive the reply inside the same atomic scope. Makes sense.

SO, if you want to loop around elements in a message by calling a receive pipeline inside your orchestration, you can’t use request-response ports, as you can with other patterns. Sorry, but that is the way it is.

Conclusion:
If you want to loop around elements inside an orhestration, use a receive pipeline if possible since this is done in an atomic scope and therefore no persistence points occur. If not possible, use the pattern described here.

Thanks, Thiago for pointing out that my post needed some more details…

--
eliasen

Wednesday, July 15, 2009 7:45:03 PM (Romance Daylight Time, UTC+02:00)  #    Comments [0]  | 

Hi all

Today I faced a very peculiar problem. I was helping a customer split a BizTalk project into two projects, because some of the schemas that were in the original project needed to be common for other projects.

So this is what i started out with:

OriginalProject

which is: Two schemas and a map between them.

And this is what I would like:

OriginalAndNewProject

So basically, the common schema has been transferred to a common project. Now, naturally, I need to reference the CommonProject from the OriginalProject and then reload the schema in my map to make it work. But this is where the weird part starts. After compiling the NewCommonProject and adding the reference, this is what showed up in the schema browser when I wanted to reload the schema in the map:

SchemaBrowser

Basically, my new project didn’t have any schemas inside it. After much troubleshooting, thinking that the issue was something with the newly compiled dll didn’t get copied to the bin folder of the OriginalProject and trying lots of stuff with that, I finally figured it out.

I just happened to notice that the file size of the NewCommonProject.dll was only 5kb which seemed to small. So I browsed the dll in the object browser and it was totally empty. Weird. Then I went and looked at the properties of the schema, and didn’t see anything unusual. So I added a new schema to the project just to see what would happen with that. This new schema showed up in the schema browser. Even weirder. But then I went and looked at the properties of the schemas again and that’s when this new property suddenly showed up:

NewProperty

I changed the build action on the schemas to be “BtsCompile” and this did the trick. Now all the schemas showed up in the schema browser.

Now, another strange side to this story is, that by now, the Build Action property has disappeared again… I haven’t quite figured out when it appears and when it does not.

BUT, if you ever run into issues where your schemas just don’t show up anywhere… this might be your issue.

Hope this helps someone.

--
eliasen

Wednesday, July 15, 2009 7:36:26 PM (Romance Daylight Time, UTC+02:00)  #    Comments [0]  | 
Wednesday, July 8, 2009

Hi all

2 and a half year ago, I wrote a blog post about looping around elements, which can be found here: http://blog.eliasen.dk/2006/11/05/LoopingAroundElementsOfAMessage.aspx. In this post, I mention two ways to do something per element in a message. One is to use the XML (or flat file) disassembler to split the message into lots of messages and have them handled individually, and the other is to do it inside an orchestration.

Since then, BizTalk has been updated such that you can now call a receive pipeline from within your orchestration, which gives you the another way of looping through elements inside an orchestration.

Often times, in the forums, users are suggested to loop through elements of a message inside an orchestration. It is a fairly common approach if you want the subelements of a message handled in the correct order, for instance.

This post is to make sure people know that looping around a message inside an orchestration can have a serious performance impact.

Say you receive a large message – 100MB. You start looping around it inside an orchestration and split it up into, say, 1000 messages of 100k each. Then, for each message you call a web service or send out the message to some system. What will happen is, that the orchestration will persist its state after each send shape, meaning that the 100BM is written 1000 times to the messagebox. Persistence points are something you always need to be aware of when designing your orchestrations, and with messages of 100MB, even more so.

SO, don’t just loop around a message inside an orchestration because you can… do it when you absolutely have to.

Thanks

--
eliasen

Wednesday, July 8, 2009 10:07:07 PM (Romance Daylight Time, UTC+02:00)  #    Comments [2]  | 

Hi all

Upon reading a forum post today, I stopped to think about the usage of custom XSLT to completely replace the mapper, custom scripting functoids and the built-in functoids.

Some people can’t be bothered using the mapper and go straight to custom xslt whenever a mapping is to be created. Others use the built-in functoids whenever possible. And the rest go for some path in the middle, with functoids and the occasional scripting functoid whenever the built-in functoids either can’t get the job done or seem to be too troublesome to use.

The advantages of custom XSLT

  1. Performance. It seems rather obvious that automatically generated XSLT can’t perform as well as custom XSLT. So if you really need high performance, use your own XSLT.
  2. Maintainability. I will be using this argument against custom XSLT as well :-) but in essence, some people think that having a map with perhaps 200 functoids is too difficult to maintain. Sometimes you need lots of functoids in order to complete some mapping task that can be easily done in XSLT. In that case, the XSLT can be more readable and hence maintainable than using functoids.

The advantages of using the built-in functoids

  1. Overview. Using the functoids can give a developer an overview of what functionality a map provides, whereas custom xslt needs to be gone through and “parsed” by a developer before he can understand what is happening.
  2. Maintainability. (See, there it is again – in the opposite direction). A BizTalk developer knows about maps and functoids. He does snot necessarily know about xslt – and even if he does, he is not guaranteed to very proficient in it. By staying with the functoids you give a BizTalk developer the best possible opportunity to understand a new map he is going to maintain.

The advantages of using functoids and the scripting functoids

  1. Everything is possible :-). Basically you can do what you want. Use built-in functoids as much as you prefer and throw in scripting functoids if you think the solution either can’t be done with the built-in functoids or is too troublesome to do.

My preference

Well, those that read my blog regularly wont be surprised when I say that my preference is to use the built-in functoids whenever possible. I turn to XSLT only when I cannot solve my problem with the built-in functoids. And when I need XSLT, I always use custom scripting functoids to supplement the other functoids, so I let as much of the job as possible be handles by built-in functoids. I never use custom XSLT to replace the map. My main reasons for this are:

  1. Maintainability. I truly believe that it is easier for a new BizTalk developer to troubleshoot and/or change an existing map if only built-in functoids are used.
  2. Overview. If a BizTalk developer opens up a map and sees lots of scripting functoids, he needs to open them all up and examine the code inside it in order to know what the map does. And after looking at 5 different scripting functoids, he needs to go back and look at the first again. Really annoying!

If I am doing a rally complex map where the number of custom scripting functoids is larger than the number of built-in functoids, I might consider using a custom XSLT script… but this haven’t happened yet :-) And also, if i run into really serious performance issues, I will consider it as well, off course.

Just my thoughts.

--
eliasen

Wednesday, July 8, 2009 9:36:39 PM (Romance Daylight Time, UTC+02:00)  #    Comments [2]  | 
Tuesday, July 7, 2009

Hi all

When trying to help the guy I wrote about in http://blog.eliasen.dk/2009/07/07/DistinguishedFieldNotWorking.aspx getting his distinguished field working, another guy suggested that he just used he xpath function instead of trying to get the distinguished field working. Now, he actually ended up using the xpath function because he had a reserved word in the xpath statement to the field he needed to access the field, but I thought I’d just write a post about why the xpath function should be avoided and perhaps generalize the post to the usage of promoted properties, distinguished fields and the xpath function. When to use what and why…

Why use distinguished fields over the xpath function

  1. Readability. Inside an expression shape, it is much easier for a BizTalk developer to look at “Message.MyElement.MyField” than it is to look at “xpath(Message, “string(/*[local-name()=’MyElement’ and namespace-uri()=’http://mynamespace.com/something/somethingmore’]/*[local-name()=’Myfield’ and namespace-uri()=’’])”)”
  2. Maintainability. If you need to change a schema at some point, the distinguished field is automatically updated to correspond to the new xpath expression that points to the relevant field (if you are using the BizTalk Editor, that is). If you use the xpath function, you need to find ALL occurrences of the xpath function in your entire solution that has an xpath expression that needs to be updated.
  3. Performance. If you use the xpath expression to get values form a message, then the entire message needs to be loaded from the database and the xpath expression is then evaluated. Distinguished fields, on the other hand, are kept in the context of the message and is therefore loaded quickly at runtime.

Why use promoted properties over distinguished fields

  1. Routing. Only promoted properties can be used for routing in the internal publish/subscribe engine if you need to route based on the content of a message.
  2. Correlation. If you need to correlate a received message into a specific instance of your orchestrations, then you need to do this using promoted properties, as these are the only properties that can be added to a correlation type. This makes sense when you think of it, because correlation is really just runtime routing - setting up the correct subscriptions at runtime that will make the message hit the correct orchestration instance. And routing can only be done with promoted properties, hence these are needed for correlation.
  3. Tracking. You cannot track distinguished fields – for tracking, you need promoted properties.

Why use the xpath function

  1. Distinguished fields cannot be used because there is a reserved word in the path of the xpath expression
  2. Distinguished fields cannot be used because you need to set or read a value from a reoccurring element

Why use distinguished fields over promoted properties

  1. Performance. You want to keep the number of promoted properties as low as possible, since the messaging engine really needs as few promoted properties as possible to evaluate when finding out which subscriptions match an incoming message that is published to the MessageBox.

So, to sum up: Use distinguished fields whenever possible, switching only to xpath function or promoted properties if really needed.

--
eliasen

Tuesday, July 7, 2009 11:36:10 PM (Romance Daylight Time, UTC+02:00)  #    Comments [0]  | 

Hi all

A guy on the forums posted a very small schema, in which he had promoted an element as a distinguished field.

His schema was this:

<?xml version="1.0" encoding="utf-16" ?>
<xsd:schema xmlns:b="http://schemas.microsoft.com/BizTalk/2003" attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <xsd:element name="outbound_task">
    <xsd:annotation>
      <xsd:appinfo>
        <properties xmlns="http://schemas.microsoft.com/BizTalk/2003">
          <property distinguished="true" xpath="/*[local-name()='outbound_task' and namespace-uri()='']/*[local-name()='task' and namespace-uri()='']/*[local-name()='task_id' and namespace-uri()='']" />
        </properties>
      </xsd:appinfo>
    </xsd:annotation>
    <xsd:complexType>
      <xsd:sequence>
        <xsd:element name="task">
          <xsd:complexType>
            <xsd:sequence>
              <xsd:element minOccurs="0" name="task_id" type="xsd:decimal" />
            </xsd:sequence>
          </xsd:complexType>
        </xsd:element>
      </xsd:sequence>
    </xsd:complexType>
  </xsd:element>
</xsd:schema>

His issue was, that when he tried using the distinguished field in an orchestration, the task_id field just didn’t show up in intellisense. And if he just entered the complete value in the expression shape (like this: Message.task.task_id) then he got a compile time error.

I messed around with it a lot, trying all sorts of stuff, but ended up with a quite simple solution: the word “task” is a reserved word. For a complete list of the reserved words, take a look at http://msdn.microsoft.com/en-us/library/aa547020.aspx. So, basically, renaming the “task” element to “Task” or something completely different (but still avoiding any reserved words) will work.

Hope this helps.

--
eliasen

Tuesday, July 7, 2009 10:27:16 PM (Romance Daylight Time, UTC+02:00)  #    Comments [0]  | 
Monday, July 6, 2009

Hi all

A long time ago, I had a post about the toolbox in Visual Studio keeping old functoids on it and crashing and stuff.

The post can be found here: http://blog.eliasen.dk/2006/12/05/RemovingFunctoidFromToolbox.aspx (Note the comments).

With this post I just wanted all to know that the path to the toolbox* file that should be deleted on a Windows Server 2008 with visual Studio 2008 is C:\Users\<User>\AppData\Local\Microsoft\VisualStudio\9.0

Hope it helps

--
eliasen

Monday, July 6, 2009 9:11:22 PM (Romance Daylight Time, UTC+02:00)  #    Comments [0]  | 

Hi all

So, yesterday I had to loop through all properties on a message inside a pipeline component.

When you program a general pipeline component, you get an Execute method that looks like this:

public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)

Now, in order to loop through the properties, the common solution is like this (and you can find this in many many blog posts around the great internet):

for (int i = 0; i <= pInMsg.CountProperties; i++)
{
  string name;
  string ns;
  object value = pInMsg.Context.ReadAt(i, out name, out ns);
  // Do something with the vale, name and namespace
}

Now, this is quite all right as it is, but there is one small pitfall that I don’t think most people realize; The CountProperties property is a uint, meaning that it can contain values up to 2^32 = 4294967296. Unfortunately, the first parameter to the ReadAt method is an int, which only holds values up to (2^31) – 1 = 2147483647. So, if there are, say 3000000000 properties, then the CountProperties property will return the correct number of properties, but there is no way of calling the ReadAt method to get the property.

This is kind of silly.

Now, to be fair, I think that the most properties I have seen on a message ever may have been around 40-50 properties. So there is a looooong way to 2 billion properties :-) So in real life, I don’t expect anyone to run into this limitation – if you do, I’ll buy you a beer! :-)

BUT, just to be sure, what you should do to be absolutely correct in your code is this:

uint counter = pInMsg.CountProperties;
string name;
string ns;
if (counter > int.MaxValue)
{
  counter = int.MaxValue;
}
for (int i = 0; i < counter; i++)
{
  object value = pInMsg.Context.ReadAt(i, out name, out ns);
  // Do something with the vale, name and namespace
}

And then, off course, you should somehow notify someone that there were more properties than you could handle – like a Debug statement, an error in the eventlog or the such… perhaps even throw an exception BEFORE trying to enumerate the properties, sine it would be useless, the result you get.

Anyway… hope this is somehow a help to someone… not sure how, though :-)

--
eliasen

Monday, July 6, 2009 7:20:41 PM (Romance Daylight Time, UTC+02:00)  #    Comments [0]  | 
Friday, July 3, 2009

Hi all

During my years on the forums, I have seen plenty, plenty and plenty more posts about the SQL Server Adapter schema generator closing unexpectedly, without any errors or any artifacts created.

Today I ran into it myself for the very first time with a BizTalk 2009 installation. I did some searching and found this post: http://support.microsoft.com/kb/917847 – which only applies to BizTalk 2004 and BizTalk 2006.

This blog post is just to let everyone know that it applies to BizTalk 2009 as well.

Good luck out there…

--
eliasen

Friday, July 3, 2009 10:24:56 PM (Romance Daylight Time, UTC+02:00)  #    Comments [0]  | 
Thursday, July 2, 2009

Hi all.

So, I am very proud to be able to publish that I have been awarded the MVP title again as of July 1’st. This, naturally is a great honor and I am rally happy about it.

Not sure I am as happy about it as Randal van Splunteren, though – because he promised some weeks ago, that if I didn’t get reawarded he would eat his shorts, shoes, hat or something – can’t remember which one, but it is in my log! :-)

The MVP title has lots of advantages, naturally – the title, the prestige, the gifts, the monthly webcasts and what not, but mainly I am happy that I do not have to bother removing the MVP logo from my blog, web site, business cards, et cetera .-)

Anyway, this is my third MVP title – going for the fourth in a year! :-)

--
eliasen

Thursday, July 2, 2009 6:35:50 AM (Romance Daylight Time, UTC+02:00)  #    Comments [7]  | 

Theme design by Jelle Druyts