Using Message Cowboy

By | August 4, 2014
0 Comment

In this second article about Message Cowboy, the tool to perform scheduled file moves and message transfers, I will talk a bit about how to use the program.

Configuration Properties

Using the configuration properties, we can configure which database will be used, whether to start an embedded ActiveMQ broker and some additional aspects of database storage and the embedded JMS broker.
The file “message-cowboy-configuration.properties” contains the following properties:

Property Name

Description
DATASOURCE_DRIVER_CLASS_NAME JDBC driver class name.
Note that the library containing the JDBC driver must be added as a dependency and the program must be re-built if it is not already present.
DATASOURCE_URL URL used to connect to the database.
If the embedded database is used, the port in the URL must match that specified by the DATABASE_PORT property and the database name in the URL must match that specified by the DATABASE_DBNAME property.
DATASOURCE_USER_NAME Database user used to connect to database.
DATASOURCE_PASSWORD Database password used to connect to database.
DATABASE_USE_EMBEDDED_FLAG Flag indicating whether to use an embedded HSQLDB database server. Possible values: true, false
DATABASE_DIRECTORY_PATH Directory in which the embedded HSQLDB database will store data.
Only relevant if the embedded HSQLDB database is used.
DATABASE_FILENAME File name used for the embedded HSQLDB database files.
Only relevant if the embedded HSQLDB database is used.
DATABASE_DBNAME Database name for the embedded HSQLDB database. Must match the database name specified in the DATASOURCE_URL property.
Only relevant if the embedded HSQLDB database is used.
DATABASE_PORT Port on which to connect to the embedded database.
Only relevant if the embedded HSQLDB database is used.
ACTIVEMQ_USE_EMBEDDED_FLAG Flag indicating whether to use an embedded ActiveMQ JMS broker. Possible values: true, false
ACTIVEMQ_URI URI at which to connect to the embedded ActiveMQ JMS broker.
ACTIVEMQ_JMX_ENABLED_FLAG Expose JMX beans for the embedded ActiveMQ JMS broker. Possible values: true, false
Only relevant if the embedded ActiveMQ JMS broker is enabled.
ACTIVEMQ_PERSISTENCE_ENABLED_FLAG Persistence enabled flag for the embedded ActiveMQ JMS broker. Possible values: true, false
Only relevant if the embedded ActiveMQ JMS broker is enabled.

The astute reader may wonder why there are no configuration properties for the intervals at which task schedule and transport service configurations are refreshed. As for now, the cron-expressions for both these tasks are set in the Spring configuration class ProductionPropertyOverrides.

Running Message Cowboy

Application Startup

The Message Cowboy can be run as a regular, standalone, Java application:

java -jar message-cowboy-1.0.0-SNAPSHOT.jar

 Alternatively you can run it in Eclipse by right-clicking the MessageCowboy class and selecting Run As → Java Application.
In this section I will use the second approach. In addition, I will assume that the default configuration is used; using the embedded HSQLDB database and an embedded ActiveMQ JMS broker.

  • In src/main/java, locate the class MessageCowboy, right-click it and select Run As → Java Application.
    In the console there will be output which last few rows should be similar to these rows:

2014-07-28 18:54:48,322 INFO  se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Message Cowboy started
2014-07-28 18:54:48,329 INFO  org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup - Starting beans in phase 2147483647
2014-07-28 18:54:48,330 INFO  org.springframework.scheduling.quartz.SchedulerFactoryBean - Starting Quartz Scheduler now
2014-07-28 18:54:48,330 DEBUG se.ivankrizsan.messagecowboy.MessageCowboy - Spring context loaded.
2014-07-28 18:54:48,331 INFO  se.ivankrizsan.messagecowboy.MessageCowboy - Hit [ENTER] in console to stop Message Cowboy.

 Note the last message, “Hit [ENTER] in console to stop Message Cowboy” – this is the preferred way to perform a clean shut down of Message Cowboy.

Next we’ll look at how transport service configuration refresh and scheduled task refresh looks like in the log. These two types of refreshes occur at regular interval and not only during start up.

Transport Service Configuration Refresh

If we observe the console for a while, the following will occur at an interval:

2014-07-28 19:01:30,002 INFO  se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Started executing task MessageCowboyTransportServiceRefreshTask in group MessageCowboySystemTasks
2014-07-28 19:01:30,002 DEBUG se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Invoking the method refreshConnectors on object of the type se.ivankrizsan.messagecowboy.services.transport.MuleTransportService with the parameters []
2014-07-28 19:01:30,003 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Found 2 connector configuration files using the pattern file:production-configurations/connectors/*.xml
2014-07-28 19:01:30,004 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Found 0 connector configuration files using the pattern file:production-configurations/transport-service-configurations/*.xml
2014-07-28 19:01:30,004 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - No changes in configuration resources, skips refresh
2014-07-28 19:01:30,004 INFO  se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Successfully completed executing task MessageCowboyTransportServiceRefreshTask in group MessageCowboySystemTasks

This is the transport service configuration refresh that I have mentioned earlier. We can see that the transport service found two connector configuration files, that no changes were detected and refresh was skipped.

Scheduled Task Refresh

Continuing our observations, we will also see the following occurring regularly:

2014-07-28 19:09:19,999 INFO  se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Started executing task MessageCowboyReschedulingTask in group MessageCowboySystemTasks
2014-07-28 19:09:19,999 DEBUG se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Invoking the method scheduleTasks on object of the type se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl with the parameters []
2014-07-28 19:09:20,000 INFO  se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Starting to (re)schedule Message Cowboy tasks
2014-07-28 19:09:20,000 INFO  se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Existing tasks unscheduled
[EL Fine]: sql: 2014-07-28 19:09:20.105--ServerSession(261255839)--Connection(623423818)--SELECT NAME, CRONEXPRESSION, ENDDATE, INBOUNDENDPOINTURI, INBOUNDTIMEOUT, OUTBOUNDENDPOINTURI, STARTDATE, TASKENABLEDFLAG, TASKGROUPNAME FROM SchedulableTaskConfigurations
2014-07-28 19:09:20,107 DEBUG se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Found 0 number of tasks
2014-07-28 19:09:20,107 INFO  se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Successfully (re)scheduled Message Cowboy tasks
2014-07-28 19:09:20,107 INFO  se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Successfully completed executing task MessageCowboyReschedulingTask in group MessageCowboySystemTasks

 This is the starter service periodical refresh of scheduled message transfer tasks. Since the table containing these tasks is empty, no tasks will be (re)-scheduled.

Scheduled Tasks

At the heart of the Message Cowboy is the scheduled task. A task that, at certain times, polls an inbound endpoint and distributes any message received to an outbound endpoint.
The inbound endpoints may be file directories, JMS queues, HTTP URLs etc – any kind of endpoint from which Mule can receive a message using the MuleClient.request method.
In a similar fashion outbound endpoints can be an endpoint of any type to which a message can be sent using the MuleClient.dispatch method.

Scheduled Tasks in the Database

The SCHEDULABLETASKCONFIGURATIONS table contains one row per scheduled task. The different columns contain the following information:

Column Name

Description
NAME Name of the schedulable task. Must be unique.
TASKGROUPNAME Name of the group of schedulable tasks the task belongs to.
STARTDATE Date on which the task will start to execute according to the cron expression.
ENDDATE Date after which the task will no longer be executed.
CRONEXPRESSION The cron expression decides when the task will be executed. More information at: http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html.
INBOUNDENDPOINTURI URI of endpoint that will be polled for a message when the task executes.
Additional information available at: https://docs.mulesoft.com/mule-user-guide/v/3.8/mule-endpoint-uris
INBOUNDTIMEOUT Time in milliseconds that Message Cowboy will wait for a message when polling the inbound endpoint before giving up.
OUTBOUNDENDPOINTURI URI of endpoint to which messages obtained when polling the inbound endpoint will be delivered.
Additional information available at: https://docs.mulesoft.com/mule-user-guide/v/3.8/mule-endpoint-uris
TASKENABLEDFLAG Allows for disabling message transfer tasks without having to delete their configuration in the database. True means task enabled.

Add a Message Transfer Task

Since Message Cowboy does not currently have a GUI, message transfer tasks are added, modified and deleted by editing the database table mentioned above.

We will now add a message transfer task that moves a file from one directory to another directory once every 30 seconds and note in what way you are informed about the new task having been discovered.

  • Using your favourite database client, like SQuirreL or the Data Source Explorer in Eclipse, connect to the Message Cowboy database using the following parameters:
    Database URL: jdbc:hsqldb:hsql://localhost:9001/mc-db
    User name: sa
    Password: (empty string)
    Database: mc-db
    The connection parameters can be found, and modified, in the “message-cowboy-configuration.properties” file.
    If you are using the Eclipse Data Source Explorer, use the SQL scrap-book rather than the table editor, as I have had problems entering boolean values in the table editor.

  • The database should be empty and we add a new row using the SQL statement below.
    Note that since there are multiple file connectors in the Message Cowboy transport service configuration, we need to specify explicitly which file connector to use in the endpoint URIs.

INSERT INTO SCHEDULABLETASKCONFIGURATIONS (
  NAME, TASKGROUPNAME, CRONEXPRESSION, STARTDATE,
  ENDDATE, INBOUNDENDPOINTURI, INBOUNDTIMEOUT,
  OUTBOUNDENDPOINTURI, TASKENABLEDFLAG)
VALUES (
  'FileMover', 'DemoGroup', '*/30 * * * * ?',
  '2014-01-01', '2015-12-31',
  'file://INBOX?connector=streamingFileConnectorInbound', '500',
  'file://OUTBOX?connector=streamingFileConnectorOutbound', 'true')
  •  Observe the log in the console.
    As the scheduled tasks are refreshed, the new task is found and scheduled. Note the task name and task group name.
[EL Fine]: sql: 2014-07-28 19:20:40.115--ServerSession(1062205019)--Connection(1572688128)--SELECT NAME, CRONEXPRESSION, ENDDATE, INBOUNDENDPOINTURI, INBOUNDTIMEOUT, OUTBOUNDENDPOINTURI, STARTDATE, TASKENABLEDFLAG, TASKGROUPNAME FROM SchedulableTaskConfigurations
2014-07-28 19:20:40,115 DEBUG se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Found 1 number of tasks
2014-07-28 19:20:40,115 DEBUG se.ivankrizsan.messagecowboy.services.scheduling.QuartzSchedulerHelper - Scheduled task FileMover in group DemoGroup
2014-07-28 19:20:40,115 DEBUG se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Scheduled task FileMover in group DemoGroup
2014-07-28 19:20:40,115 INFO  se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Successfully (re)scheduled Message Cowboy tasks

Before trying out the new message transfer task, we’ll have a look at transport service configuration.

Transport Service Configuration

Transport service configuration files are, in this incarnation of Message Cowboy, Mule XML configuration files. Such files may contain, among other things:

  • Connectors.
    Enabling polling and delivery to different types of endpoints and/or endpoints with different configurations.
  • Flow constructs.
    Allowing for, for instance, processing of messages prior to being delivered to the destination endpoint.

Location of Transport Service Configuration Files

Currently, there are two directories in which Mule XML configuration files can be put, in order for Message Cowboy to discover transport service configurations:

  • production-configurations/connectors/

  • production-configurations/transport-service-configurations/

Files in the above directories need to have the .xml suffix in order to be loaded by the transport service.

The transport service makes no difference between configuration files from the different directories – they are just a convenience for the user of the application allowing him/her to categorize configurations.
The number and location of these transport service configuration directories can be configured in the MessageCowboyConfiguration Spring configuration class.

Note!

It is important to remember that all the transport service configuration files are loaded into one and the same Mule context. Thus there must not be components with the same name in the different files, as this will cause an error.

Add a Transport Service Configuration File

In order to see what happens when a new transport service configuration file is added, the following Mule configuration file will be placed in one of the directories that contain transport service configuration files:

<?xml version="1.0" encoding="UTF-8"?>
<mule
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns="http://www.mulesoft.org/schema/mule/core"
    xmlns:http="http://www.mulesoft.org/schema/mule/http"
    xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd">

    <flow name="testflow">
        <http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8081"/>
        <set-payload value="Howdy from the Message Cowboy!" />
    </flow>
</mule>
  •  In the “”production-configurations” directory, create a director named “transport-service-configurations” if it does not already exist.

  • In the “transport-service-configurations” directory, create a new file named “example-config.xml” and paste the above Mule configuration into that file.

  • Observe the console.
    After some time there will be a quite some output, as the embedded Mule instance is stopped and restarted, starting with these lines:

2014-07-28 18:16:30,005 INFO  se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Started executing task MessageCowboyTransportServiceRefreshTask in group MessageCowboySystemTasks
2014-07-28 18:16:30,005 DEBUG se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Invoking the method refreshConnectors on object of the type se.ivankrizsan.messagecowboy.services.transport.MuleTransportService with the parameters []
2014-07-28 18:16:30,008 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Found 2 connector configuration files using the pattern file:production-configurations/connectors/*.xml
2014-07-28 18:16:30,011 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Found 1 connector configuration files using the pattern file:production-configurations/transport-service-configurations/*.xml
2014-07-28 18:16:30,011 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Detected change in configuration resources, refreshing
2014-07-28 18:16:30,013 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Found 2 connector configuration files using the pattern file:production-configurations/connectors/*.xml
2014-07-28 18:16:30,015 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Found 1 connector configuration files using the pattern file:production-configurations/transport-service-configurations/*.xml
2014-07-28 18:16:30,022 INFO  org.mule.module.client.MuleClient - Initializing Mule...
  •  In a web-browser of your choice, paste the URL http://localhost:8081
    There should be a greeting from the Message Cowboy in the browser window.

Note that this example configuration would probably be more suitable to be deployed to a standalone Mule instance rather than in the Message Cowboy, since it does not involve any scheduled activity.

  • Delete the file “example-config.xml” that you created earlier.

  • Observe the console.
    After some time you should see that the modification is discovered and the embedded Mule client is restarted once more.

Example File Transfer

With the above configurations in place and Message Cowboy up and running, we will now look at a message being transferred by the scheduled task created earlier:

  • In Eclipse, create a folder named “INBOX” in the root directory of the Message Cowboy project.

  • Again in Eclipse, create another folder named “OUTBOX” also in the root directory of the project.

  • Copy and paste arbitrary file into the INBOX directory.

  • Wait at least 30 seconds.

  • Refresh the project in Eclipse (F5 or right-click on the project and select Refresh).

  • Observe the OUTBOX directory.

If everything worked as expected, the file should have been transferred from the INBOX directory to the OUTBOX directory. The file-transfer should generate log similar to this in the console:

2014-07-28 18:54:20,115 DEBUG se.ivankrizsan.messagecowboy.domain.entities.impl.QuartzMuleTaskJob - Started executing job FileMover in group DemoGroup
2014-07-28 18:54:20,115 DEBUG se.ivankrizsan.messagecowboy.domain.entities.impl.QuartzMuleTaskJob - Executing mover task job FileMover
2014-07-28 18:54:20,130 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Received message: 
org.mule.DefaultMuleMessage
{
  id=61ae6aef-162c-11e4-86aa-553bdf28c94e
  payload=org.mule.transport.AbstractConnector$7
  correlationId=<not set>
  correlationGroup=-1
  correlationSeq=-1
  encoding=UTF-8
  exceptionPayload=<not set>

Message properties:
  INVOCATION scoped properties:
  INBOUND scoped properties:
    directory=/Users/ivan/MyWorkspace/message-cowboy-master/INBOX
    fileSize=35120
    originalFilename=myFile.xml
    timestamp=1234567890123
  OUTBOUND scoped properties:
    MULE_ENCODING=UTF-8
    originalFilename=myFile.xml
  SESSION scoped properties:
}
2014-07-28 18:54:20,161 DEBUG se.ivankrizsan.messagecowboy.domain.entities.impl.QuartzMuleTaskJob - Message received from file://INBOX?connector=streamingFileConnectorInbound: se.ivankrizsan.messagecowboy.domain.entities.impl.MuleMoverMessage@5e8c1f43
2014-07-28 18:54:20,161 DEBUG se.ivankrizsan.messagecowboy.domain.entities.impl.QuartzMuleTaskJob - Dispatching message to file://OUTBOX?connector=streamingFileConnectorOutbound
2014-07-28 18:54:20,193 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Sent message: se.ivankrizsan.messagecowboy.domain.entities.impl.MuleMoverMessage@5e8c1f43
2014-07-28 18:54:20,193 INFO  org.mule.transport.ConnectableLifecycleManager - Initialising: 'streamingFileConnectorOutbound.dispatcher.1193542816'. Object is: FileMessageDispatcher
2014-07-28 18:54:20,193 INFO  org.mule.transport.ConnectableLifecycleManager - Starting: 'streamingFileConnectorOutbound.dispatcher.1193542816'. Object is: FileMessageDispatcher
2014-07-28 18:54:20,193 INFO  org.mule.transport.file.FileConnector - Writing file to: /Users/ivan/MyWorkspace/message-cowboy-master/OUTBOX/myFile.xml

 The contents of the file found in the OUTBOX directory should be identical to that of the original file.

Logging

Throughout the above examples we have had debug log enabled for the log generated by Message Cowboy. At times one may wish to reduce the amount of log or to increase the amount of log for certain packages etc.
This can be accomplished by creating a Log4J configuration file and, when launching Message Cowboy, setting the appropriate system property pointing at the Log4J configuration file.

In this example I have created a custom Log4J configuration file named mylog4j.xml:

java -Dlog4j.configuration=mylog4j.xml -jar message-cowboy-1.0.0-SNAPSHOT.jar

Future

Some thoughts I have about the future of Message Cowboy are, in no particular order:

  • A GUI for configuration of scheduled tasks.
    This to make integrations truly configurable, without the need for programming, and less error-prone.
  • An alternative implementation of the transport service using Apache Camel.
  • Adding the ability to specify a set of properties associated with a scheduled task that are passed to the transport service when requesting and dispatching messages.
    This would make it possible to develop more generic transport service configurations.
  • An alternative implementation of the scheduling service.
    If I some day find some scheduling framework that is a serious, free, alternative to Quartz.
  • Enclosing some basic, commonly used, transport service configurations.
    For instance, a ready-to-use configuration for guaranteed delivery of messages.

Suggestions and requests are welcome – please place such requests as an issue on GitHub: https://github.com/krizsan/message-cowboy/issues

Leave a Reply

Your email address will not be published. Required fields are marked *