FIM, System.DirectoryServices and a memory leak

It all started with a slow synchronisation.  Further inspection yielded several elements to the solution that strayed from recommended practice.  One aspect that was definitely playing a part in the slow synchronisation was several uniqueness checks against Active Directory Domain Services (AD DS) within the rules extension(s).  The check was required – to call-out or not to call-out is an argument for another time – but some of the locations were wrong.  My initial recommendation was to pull the uniqueness attributes, e.g. the sAMAccountName attribute, of all users into a Dictionary<String,String> within IMVSynchronization.Initialize and then use this local dictionary object for uniqueness checks (adding unique values as they are consumed).  There were some other recommendations as well but I won’t bore you with those.

Jump forward two or three months and we’ve got a critical situation – a severity A Premier Support case.  Over the course of a full synchronisation the FIM Synchronization Service exhausts available RAM and (gracefully, I might add) exits.  The host in question is a VM; 8GB RAM, 60GB HDD, 12GB page file.  Customer takes some traces and dumps and provides them to CSS and David quickly identifies that wldap32 is using all the RAM – 7GB in the data we’re looking at (while everything is still running).  The data shows ~65,000 125KB allocations.  Of 8.12GB of outstanding allocations 7.54GB from this stack alone!  FIM is doing this.  We prove it’s the rules extension and not FIM itself by running it in it’s own process and now mmsscrpt.exe (the process that runs rules extension code marked as “run out of process”) is using the RAM and not miiserver.exe (FIM Synchronization Service).

wldap32 is the Windows LDAP APIActive Directory Services Interfaces (ADSI) sits on top of this.  And System.DirectoryServices (S.DS) sits on top of ADSI.  What is the rules extension using?  Right.  We have a memory leak coming from S.DS.

If we whip out the bible the issue is obvious.  Several of the classes within S.DS have to be explicitly disposed.  Let’s look at some documented remarks:

SearchResultCollection Class

Due to implementation restrictions, the SearchResultCollection class cannot release all of its unmanaged resources when it is garbage collected. To prevent a memory leak, you must call the Dispose method when the SearchResultCollection object is no longer needed.

Furthermore, Joe and Ryan have a sidebar in the bible that discusses Close() and Dispose().  Let me quote some of the text:

Both Close and Dispose are intended to be used to clean up the underlying COM object.  The primary difference between the two is that Dispose also suppresses .NET finalization, and Close does not.  Supressing finalization means that the garbage collector will not bother to run the Finalize method on the object because we have signalled that we have already cleaned up the underlying resource that the finalizer needed to take care of.  Objects that need to be finalized are automatically promoted one garbage collection generation, so the tend to hang around in memory longer, which is something we probably want to avoid if possible.

And:

In summary, use Dispose.  It does everything that Close does, and it takes care of the finalization.

Better yet, use the built-in language features to ensure that objects are Disposed properly.  In C# always use the using construct:

Later in the book they call out the SearchResultCollection class too:

This collection of SearchResult instances represents the result set as accumulated by the server for a specific query.  This class internally holds references to unmanaged resources, and as such should always explicitly be disposed using the Dispose method, just like DirectoryEntry and DirectorySearcher.

The crux of the matter here is this: when using S.DS in a process that runs in a loop it is imperative that the objects are disposed.  In the case of FIM the loop is synchronisation, i.e. calling the rules extension code over and over again (per managed object).

The customer quickly fixed the utility class that was using DirectorySearcher to return a SearchResultCollection to another method that was consuming it by properly calling Dispose in a couple of places and deployed the code.  The result was different but the same.  Different allocation sizes but still a leak.  Bring in the heavy artillery and the ninjas identify the actual method, in the customer rules extension, that is making the call(s).  A look at the revised code quickly identifies the issue.  The S.DS.DE and S.DS.DS objects are being disposed but the SearchResultCollection isn’t – probably because of the way it is being used, i.e. the object is returned by a method.  A second flaw is also identified –not the root of the problem but something worth fixing– all attributes are being returned.  We provide some recommendations around rejigging the method and properly disposing and also favouring a using statement in many places.

The next result was…problem solved.  While we see increased memory usage during synchronisation five minutes after synchronisation and it’s freed up (expected, as in-process rules extensions are unloaded after five minutes of inactivity).

Before I finish this post I’ll state some observations.  Many of us know these values but I cannot see that they’re actually publically documented.

  • A rules extension that runs in-process is unloaded after five minutes of inactivity.  You should expect to see all memory used by your rules extension DLLs returned to the OS five minutes after synchronisation completes.
  • A rules extension that runs out-of-process is unloaded after fifteen minutes of inactivity.
  • Unmanaged memory and/or memory that is leaked is not returned to the host after IMVSynchronization.Terminate and/or IMASynchronization.Terminate is called.  The leaked memory is only cleaned up when the hosting or parent process is recycled.  For out-of-process rules extensions this means the memory is returned after fifteen minutes when FIM kills mmsscrpt.exe.  For in-process rules extensions the memory won’t be returned until the FIM Synchronization Service (miiserver.exe) is recycled.

It was an interesting case and I learned some very valuable information.  Hopefully you’ll find this interesting and helpful too.

p.s. Kudos to David, Steve and Rob Lane, without these clever folk we would not have got to the bottom of the problem!

Advertisements

About Paul Williams

IT consultant working for Microsoft specialising in Identity Management and Directory Services.
This entry was posted in FIM, Programming, Troubleshooting and tagged , , , , , , , , . Bookmark the permalink.

8 Responses to FIM, System.DirectoryServices and a memory leak

  1. Paul – you may have noticed I’ve been posting your blogs on both Twitter (#FIM2010) and LinkedIn (3 groups there – Microsoft Forefront Identity Manager, Microsoft Identity Management Experts (MIIS, ILM, FIM), and Microsoft Identity Management Solutions. I’m doing this because I think it’s important that blog posts like yours reach the widest audience and help build the wider FIM/Microsoft IAM community. Can you please do me a favour and start posting your posts here yourself if you can? I find it increases traffic and promotes discussion. Your last post on IaaS got quite a bit of attention. Keep it up Paul.

    • Hi Bob, I had not directly noticed as I logon to Linked in about three times a year and have only recently joined twitter and only really follow wine tweets. 🙂

      I’m behind the times with this social media stuff and will try and get up to speed faster. I appreciate you doing this, I had noticed in spike in the stats this week.

  2. Steve Kradel says:

    Lessons: Don’t use S.DS, S.DS.AM, etc. at all, unless you have a really compelling reason not to use S.DS.P. Always wrap IDisposables with “using”. Don’t request all attributes when LDAP searching.

    • Right! I thought I was being a dork when I kept struggling at S.DS.P but I really do think it’s worth the extra effort – as well as being a little more fun and satisfying. Not requesting all attributes is an obvious one but one that crops up again and again. The IDisposable stuff was new to me. I’d not paid much attention to that in the past but at scale it really did rear it’s head. I’m going to make a conscious effort to utilise using statements a lot more and am going to pay more attention to the MSDN documentation… 🙂

    • Steve, I myself rarely get in touch with S.DS, S.DS.AD and S.DS.P. From here (http://msdn.microsoft.com/en-us/library/bb332056.aspx ) I can see on the picture that they are more or less alternatives for each other S.DS/S.DS.AD vs S.DS.P I’m really interested as to why S.DS.P is “the way to go”. Could you elaborate a bit more?

      • Steve Kradel says:

        Well, S.DS and its close relatives are basically wrappers around the old ADSI / COM classes, which have some drawbacks like difficulty working with custom attributes, memory issues, and more complex network interaction via RPC. The union of .NET and COM is often very complex, and can set the stage for bizarre bugs at scale. On the other hand, S.DS.P is much closer to plain LDAP–lightweight, interoperable with a variety of DSAs, provides access to a wide range of LDAP request controls, async mode, etc., and just more “manageable” for the CLR.

  3. Shane Day says:

    Those libraries are notorious for leaking. Using blocks are vitally important for any disposable resource – using a tool like ReSharper will help you understand where they should be used. Even more difficult is knowing the scope of objects when you are passing them through different methods and objects – who owns them then? Best to be VERY explicit on every method entry point who owns what IDisposable resources.

  4. Michael Liben says:

    S.DS is COM and garbage collection seems to come only at the end of the entire process. S.DS.P uses native C calls. the underlying library is extrememly memory friendly. I have run S.DS and seen allocated memory exceed 1000 MB. S.DS.P almost never exceeds 20 MB, even with large results sets.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s