The SimCorp Message Queue is the central method for importing almost any data. We will assume that everyone is generally familiar with the concept and will go straight into some tips around its use.
There are several ways to get data into the Message Queue; the most common are the communication server or some sort of batch setup. But for testing purposes, there are some alternative options. The most used are the ‘Import’ and ‘Import to Message Queue’ from the function’s menu of the Data Format Setup. Let’s instead look at two alternatives.
The Process Message Queue from File single window allows you to run an import of a file to a Data Format Setup (or a filter) without inserting the record in the message queue. This provides a simple and easy way to directly test the filter with a file, and see the resulting errors, if any, without the need to view the message queue, have a service running, or even have a message queue defined.
The Transfer File to Message Queue allows for more complex testing of 3 different import types. In the first option ‘File is ready for import’ the file you import must have the Base Filter/Data Format Setup or Filter ID specified at the beginning of each record. The second option ‘Use ‘Configure Data to Filter’’ calls the window of the same name. In this window, you specify a search criterion of the records in the import file that will return which Filter or Data Format Setup to apply. This feature is especially useful when you receive a 3rd party file that you do not want to change prior to the import and the record in the file needs to be routed to different Data Format Setups and/or Filters. The last option called ‘Filter/Data Format Setup’ acts in the same way as the ‘Import to Message Queue’ function’s menu of the Data Format Setup.
We mentioned above the need for running a Message Queue service to process the message queue. There is another way to quickly process a message queue without having to start a service in the Service Platform. In the Process Message Queue, you define the message queue you want to process and press the Execute button. This will act as if a service was started and will import data from the specified Message Queue. Once all messages have been processed it will sit idle until the next message is inserted into the Message Queue. Note that this function will slow down your SimCorp Dimension session and at times renders it non-responsive, so if you do use it, it is best to have it run on a separate SCD session (starting up a separate SCD using the parameter -type=* is a good idea).
Turning our focus to the View Message Queue, there are a couple of tricks that can help you better manage the process. First, I suggest you add 2 fields from the select fields menu: Message No and Created Date. Why? The Created Date is useful to search for messages imported today by entering today’s date. And Message No is useful to track a specific message. For example, if you are searching for messages in the status ‘error’ from today, as soon as you fix an underlying issue for 1 record, a refresh (press ctrl + e) of this record will make the error record disappear as it should hopefully have been fixed. In other words, as soon as you have today’s errors identified, copy the list of Message Numbers and search on these in the View Message Queue; that way they stay fixed when you refresh.
Second, in the options menu, you can toggle the ‘Enable Message Content’. This will open the content of the records to allow you to manually change it directly in the Message Queue. For example, if the currency was wrongly imported as UDS, you can change this to USD, switch the message status from error to Queued, save the record and it will be reprocessed by the service. This can be done directly in the grid or from the ‘Edit Message…’ options when you right-click. I should quickly mention that this option can be controlled by the authorization profiles in Tasks and Commands, so you don't have this option in production.
The last tip in the View Message Queue is the ‘Queue Information’ found under the function’s menu. This shows the status and statistics of a particular Message Queue selected/highlighted from the main grid window, for example, if it is active when it was last processed, and how many errors and queued records exist.
In some cases, if an imported message fails, you might want the system to retry the import a little later; there could for example be a dependency on another file that timing wise might not always arrive at the time of this import. When you set up the message queue in the Message Queue window, under select fields you can add two fields: ‘Retry after’ and ‘Maximum no of Retries’. If you set these fields to 2 and 0:05 respectively, when importing a message that normally would go on ‘Error’ it would set the message status to ‘Waiting’ and the message would be re-processed after waiting 5 minutes; a descriptive reason will also appear in the ‘Error message’ field stating that the processing of the message failed, that it will retry again at a specific time and provide the error it returned. Additionally, the fields ‘Number of retries’ and ‘Wait to’ in the View Message Queue window will be updated.
The last tip I want to mention is the cleanup of records in the message queue. In the Message Queue window, you can specify a ‘Maximum age’. Any successfully imported message will be deleted by the Message Queue services running on the Service Platform after the number of days specified in this field. If you would like to delete messages in other statuses you can do so in the Cleanup Message Queue window. Select a message queue and Message status below will show you how many records per status there are ready to delete. You can further filter the data using the period and message content fields. It might be a good idea to check the option at the bottom of the window called ‘Inspect messages before deleting’ to get a popup list with details where you can select individual records (or everything) prior to deleting them.
We hope you found these tips useful. If you have any tips on this topic you would like to share that we did not mention, please leave a comment below or reach out to us. We are always happy to hear from our community!