Log entire SOAP request or only parameters in it?
From my point of view, the solution depends on whether you want to store the whole message or just tracing the calls and a few other data.
If you do not have to store the whole message, I would definitely suggest you to install Microsoft AppFabric. AppFabric is a set of integrated technologies that makes it easier to perform a lot of things including monitoring WCF services hosted on IIS, and it’s free!. AppFabric setup is pretty straightforward and will add new functionalities/icons to your IIS.
We’re using AppFabric + SQL Server Express on production environment to trace some WCF calls. Turning on tracing is pretty simple; you can set the tracing level, target DB, how much history you want to keep, size limits and so on… Moreover there’s a pretty cool user interface that lets you query all the traces stored (it shows you how much calls have been traced, how much fails, ...). The nice thing is that you may have the error description in case of faulted calls. It’s also possible to add UserDefined data in AppFabric traces. More information here.
Now, if you need to store the whole message, as @Aron said, I would chose to go with NoSQL and more particularly with logstash. As written on their site:
logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.
logstash is based on elasticsearch.
The last thing you have to find out is defining the right time/place to store the message, probably with the use of custom WCF behavior.
Hope that helps!
The easiest way to compress XML is to store multiple documents at once and apply any general purpose compression algorithm of your choice to it. The key is to compress multiple messages at once so that the compressor can exploit the extreme redundancies present in repeated XML structure.
This works so well that dedicated XML compression often is unnecessary. Common algorithms like "gzip/deflate" or more powerful ones like LZMA (7zip) are very, very good at exploiting this. All they do is combine repeated substrings (like they are present in XML).
So you could buffer all XML messages for 10 seconds and save them in one binary blob somewhere.
Okay. It is pretty simple to setup WCF to log everything. An example is shown here. You want to use the IMessageInspector interface.
As for the second part of the question. Using large amount of data storage. I have two answers for you. You WILL use up a large amount of data storage because
- Each message is large (XML is not know for efficiency of data storage).
- You are storing lots of messages.
Hence you want to reduce the contribution from each part.
The first is as you rightly surmised. You could reduce the payload size by compressing the data. From simple information theory we know that how compressable the data is depends on the nature of the data. In this case XML, which can easily be compressed. Depending on how much of the schema is known before hand you can compress it more or less.
Secondly you can reduce the number of payloads. By simply truncating the number of stored requests regularly you can limit the storage.
However I would finally want to point out that I would most definitely NOT use SQLite as the storage mechanism. From my experience the P/Invoke overhead will severely limit performance of your server. Also SQLite has a very poor concurrency model, limiting the number of concurrent request on your server as well.
To be frank, I have to ask if you need anything over using just flat files for storage, or perhaps I might suggest to switch to a XML NoSQL solution, if you need querying.
An added bonus to using a XML NoSQL database is that you can put it off box and query it even when the application server goes down.
Now, as for whether you should extract the data out of the XML instead or store it in an incompatible format...I would argue that it would be counter productive. By storing the actual request you can write a small app to resend the request to regression test.