Perl threads slowly consume memory

Your Perl is 4 and half years old. Upgrade to 5.12. Just search the 5.12 build notes and look at the shit ton of major threads improvements it has that might just magically fix your nebulous problem:

• 5.12:: @_ and $_ no longer leak under threads (RT #34342 and #41138, also #70602, #70974)'
• 5.10:: Under ithreads, the regex in PL_reg_curpm is now reference counted. This eliminates a lot of hackish workarounds to cope with it not being reference counted.
• 5.9:: threads: Several fixes, for example for join() problems and memory leaks. In some platforms (like Linux) that use glibc the minimum memory footprint of one ithread has been reduced by several hundred kilobytes.
• 5.9:: threads::shared Many memory leaks have been fixed.

I mean the list just goes on when you speak of four years of thead development and the wide range of stuff that can cause this problem, check out the modern changelog of threads::shared

I commented on your post, this would be my next set of suggestions: If you're not using glibc and you're using perl malloc (default), you will never release memory to the OS. The process size will represent the maximum size perl has every taken up. Try rebuilding with glibc malloc (requires recompilation) see if that provides for a different memory profile. Other than that, it will be time to show code.

Finally nailed the leak. First, I'd like to show you the improvement. You shouldn't look at the actual numbers because the user base has increased since the first graph was taken, just look at the difference in slope.

The before memory usage graph: http://i32.tinypic.com/311nc08.jpg
The after memory usage graph: http://i51.tinypic.com/29goill.jpg

For many months I was stuck restarting the server every couple days, but over the last 14 hours, there has been no increase in memory usage.

Every tutorial, example, presentation, and book that I used to help develop the server, all omitted one very very important fact regarding IO::Socket::SSL. And everyone using that module within a threaded app better listen up. Nobody ever emphasized one of the last lines in IO::Socket::SSL's documentation, which caused me to very foolishly assume that any socket I create, like almost any other data structure, will be freed once it goes out of scope (And yes, I do know the exceptions to that rule). I thought I would do everyone a favor and call out the line I am referring to.

...IO::Socket::SSL sockets will stay open until the program ends or you explicitly close them. This is due to the fact that a circular reference is required to make IO::Socket::SSL sockets act simultaneously like objects and glob references.
http://search.cpan.org/dist/IO-Socket-SSL/SSL.pm#LIMITATIONS

I was never made aware of the fact that these sockets had a circular reference in them, and if I didn't know about them, then every one's blogs and books I read don't know either (hence the call-out).

So as you can imagine, this had a very simple fix. Within my thread's work loop where a socket gets created every iteration, I simply placed a eval { close $socket; };undef $socket; at the bottom to ensure it gets closed before processing the next client. I started up my server and waited and watched as the memory usage was solid as you can see in my second graph. So after two months of troubleshooting this issue on and off, I finally came to a solution. I hope this provides some insight to any other amateurs programming with sockets. Thanks to everyone who provided answers/comments/suggestions, every bit helped, and it helped to have a place to bounce ideas off of.

Try upgrading threads.pm and threads::shared. Upgrading to perl 5.12.1 is also a good idea.