In this second article about Message Cowboy, the tool to perform scheduled file moves and message transfers, I will talk a bit about how to use the program.
Configuration Properties
Using the configuration properties, we can configure which database will be used, whether to start an embedded ActiveMQ broker and some additional aspects of database storage and the embedded JMS broker.
The file “message-cowboy-configuration.properties” contains the following properties:
Property Name |
Description |
DATASOURCE_DRIVER_CLASS_NAME | JDBC driver class name. Note that the library containing the JDBC driver must be added as a dependency and the program must be re-built if it is not already present. |
DATASOURCE_URL | URL used to connect to the database. If the embedded database is used, the port in the URL must match that specified by the DATABASE_PORT property and the database name in the URL must match that specified by the DATABASE_DBNAME property. |
DATASOURCE_USER_NAME | Database user used to connect to database. |
DATASOURCE_PASSWORD | Database password used to connect to database. |
DATABASE_USE_EMBEDDED_FLAG | Flag indicating whether to use an embedded HSQLDB database server. Possible values: true, false |
DATABASE_DIRECTORY_PATH | Directory in which the embedded HSQLDB database will store data. Only relevant if the embedded HSQLDB database is used. |
DATABASE_FILENAME | File name used for the embedded HSQLDB database files. Only relevant if the embedded HSQLDB database is used. |
DATABASE_DBNAME | Database name for the embedded HSQLDB database. Must match the database name specified in the DATASOURCE_URL property. Only relevant if the embedded HSQLDB database is used. |
DATABASE_PORT | Port on which to connect to the embedded database. Only relevant if the embedded HSQLDB database is used. |
ACTIVEMQ_USE_EMBEDDED_FLAG | Flag indicating whether to use an embedded ActiveMQ JMS broker. Possible values: true, false |
ACTIVEMQ_URI | URI at which to connect to the embedded ActiveMQ JMS broker. |
ACTIVEMQ_JMX_ENABLED_FLAG | Expose JMX beans for the embedded ActiveMQ JMS broker. Possible values: true, false Only relevant if the embedded ActiveMQ JMS broker is enabled. |
ACTIVEMQ_PERSISTENCE_ENABLED_FLAG | Persistence enabled flag for the embedded ActiveMQ JMS broker. Possible values: true, false Only relevant if the embedded ActiveMQ JMS broker is enabled. |
The astute reader may wonder why there are no configuration properties for the intervals at which task schedule and transport service configurations are refreshed. As for now, the cron-expressions for both these tasks are set in the Spring configuration class ProductionPropertyOverrides.
Running Message Cowboy
Application Startup
The Message Cowboy can be run as a regular, standalone, Java application:
java -jar message-cowboy-1.0.0-SNAPSHOT.jar
Alternatively you can run it in Eclipse by right-clicking the MessageCowboy class and selecting Run As → Java Application.
In this section I will use the second approach. In addition, I will assume that the default configuration is used; using the embedded HSQLDB database and an embedded ActiveMQ JMS broker.
-
In src/main/java, locate the class MessageCowboy, right-click it and select Run As → Java Application.
In the console there will be output which last few rows should be similar to these rows:
2014-07-28 18:54:48,322 INFO se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Message Cowboy started 2014-07-28 18:54:48,329 INFO org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup - Starting beans in phase 2147483647 2014-07-28 18:54:48,330 INFO org.springframework.scheduling.quartz.SchedulerFactoryBean - Starting Quartz Scheduler now 2014-07-28 18:54:48,330 DEBUG se.ivankrizsan.messagecowboy.MessageCowboy - Spring context loaded. 2014-07-28 18:54:48,331 INFO se.ivankrizsan.messagecowboy.MessageCowboy - Hit [ENTER] in console to stop Message Cowboy.
Note the last message, “Hit [ENTER] in console to stop Message Cowboy” – this is the preferred way to perform a clean shut down of Message Cowboy.
Next we’ll look at how transport service configuration refresh and scheduled task refresh looks like in the log. These two types of refreshes occur at regular interval and not only during start up.
Transport Service Configuration Refresh
If we observe the console for a while, the following will occur at an interval:
2014-07-28 19:01:30,002 INFO se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Started executing task MessageCowboyTransportServiceRefreshTask in group MessageCowboySystemTasks 2014-07-28 19:01:30,002 DEBUG se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Invoking the method refreshConnectors on object of the type se.ivankrizsan.messagecowboy.services.transport.MuleTransportService with the parameters [] 2014-07-28 19:01:30,003 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Found 2 connector configuration files using the pattern file:production-configurations/connectors/*.xml 2014-07-28 19:01:30,004 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Found 0 connector configuration files using the pattern file:production-configurations/transport-service-configurations/*.xml 2014-07-28 19:01:30,004 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - No changes in configuration resources, skips refresh 2014-07-28 19:01:30,004 INFO se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Successfully completed executing task MessageCowboyTransportServiceRefreshTask in group MessageCowboySystemTasks
This is the transport service configuration refresh that I have mentioned earlier. We can see that the transport service found two connector configuration files, that no changes were detected and refresh was skipped.
Scheduled Task Refresh
Continuing our observations, we will also see the following occurring regularly:
2014-07-28 19:09:19,999 INFO se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Started executing task MessageCowboyReschedulingTask in group MessageCowboySystemTasks 2014-07-28 19:09:19,999 DEBUG se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Invoking the method scheduleTasks on object of the type se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl with the parameters [] 2014-07-28 19:09:20,000 INFO se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Starting to (re)schedule Message Cowboy tasks 2014-07-28 19:09:20,000 INFO se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Existing tasks unscheduled [EL Fine]: sql: 2014-07-28 19:09:20.105--ServerSession(261255839)--Connection(623423818)--SELECT NAME, CRONEXPRESSION, ENDDATE, INBOUNDENDPOINTURI, INBOUNDTIMEOUT, OUTBOUNDENDPOINTURI, STARTDATE, TASKENABLEDFLAG, TASKGROUPNAME FROM SchedulableTaskConfigurations 2014-07-28 19:09:20,107 DEBUG se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Found 0 number of tasks 2014-07-28 19:09:20,107 INFO se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Successfully (re)scheduled Message Cowboy tasks 2014-07-28 19:09:20,107 INFO se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Successfully completed executing task MessageCowboyReschedulingTask in group MessageCowboySystemTasks
This is the starter service periodical refresh of scheduled message transfer tasks. Since the table containing these tasks is empty, no tasks will be (re)-scheduled.
Scheduled Tasks
At the heart of the Message Cowboy is the scheduled task. A task that, at certain times, polls an inbound endpoint and distributes any message received to an outbound endpoint.
The inbound endpoints may be file directories, JMS queues, HTTP URLs etc – any kind of endpoint from which Mule can receive a message using the MuleClient.request method.
In a similar fashion outbound endpoints can be an endpoint of any type to which a message can be sent using the MuleClient.dispatch method.
Scheduled Tasks in the Database
The SCHEDULABLETASKCONFIGURATIONS table contains one row per scheduled task. The different columns contain the following information:
Column Name |
Description |
NAME | Name of the schedulable task. Must be unique. |
TASKGROUPNAME | Name of the group of schedulable tasks the task belongs to. |
STARTDATE | Date on which the task will start to execute according to the cron expression. |
ENDDATE | Date after which the task will no longer be executed. |
CRONEXPRESSION | The cron expression decides when the task will be executed. More information at: http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html. |
INBOUNDENDPOINTURI | URI of endpoint that will be polled for a message when the task executes. Additional information available at: https://docs.mulesoft.com/mule-user-guide/v/3.8/mule-endpoint-uris |
INBOUNDTIMEOUT | Time in milliseconds that Message Cowboy will wait for a message when polling the inbound endpoint before giving up. |
OUTBOUNDENDPOINTURI | URI of endpoint to which messages obtained when polling the inbound endpoint will be delivered. Additional information available at: https://docs.mulesoft.com/mule-user-guide/v/3.8/mule-endpoint-uris |
TASKENABLEDFLAG | Allows for disabling message transfer tasks without having to delete their configuration in the database. True means task enabled. |
Add a Message Transfer Task
Since Message Cowboy does not currently have a GUI, message transfer tasks are added, modified and deleted by editing the database table mentioned above.
We will now add a message transfer task that moves a file from one directory to another directory once every 30 seconds and note in what way you are informed about the new task having been discovered.
-
Using your favourite database client, like SQuirreL or the Data Source Explorer in Eclipse, connect to the Message Cowboy database using the following parameters:
Database URL: jdbc:hsqldb:hsql://localhost:9001/mc-db
User name: sa
Password: (empty string)
Database: mc-db
The connection parameters can be found, and modified, in the “message-cowboy-configuration.properties” file.
If you are using the Eclipse Data Source Explorer, use the SQL scrap-book rather than the table editor, as I have had problems entering boolean values in the table editor. -
The database should be empty and we add a new row using the SQL statement below.
Note that since there are multiple file connectors in the Message Cowboy transport service configuration, we need to specify explicitly which file connector to use in the endpoint URIs.
INSERT INTO SCHEDULABLETASKCONFIGURATIONS ( NAME, TASKGROUPNAME, CRONEXPRESSION, STARTDATE, ENDDATE, INBOUNDENDPOINTURI, INBOUNDTIMEOUT, OUTBOUNDENDPOINTURI, TASKENABLEDFLAG) VALUES ( 'FileMover', 'DemoGroup', '*/30 * * * * ?', '2014-01-01', '2015-12-31', 'file://INBOX?connector=streamingFileConnectorInbound', '500', 'file://OUTBOX?connector=streamingFileConnectorOutbound', 'true')
- Observe the log in the console.
As the scheduled tasks are refreshed, the new task is found and scheduled. Note the task name and task group name.
[EL Fine]: sql: 2014-07-28 19:20:40.115--ServerSession(1062205019)--Connection(1572688128)--SELECT NAME, CRONEXPRESSION, ENDDATE, INBOUNDENDPOINTURI, INBOUNDTIMEOUT, OUTBOUNDENDPOINTURI, STARTDATE, TASKENABLEDFLAG, TASKGROUPNAME FROM SchedulableTaskConfigurations 2014-07-28 19:20:40,115 DEBUG se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Found 1 number of tasks 2014-07-28 19:20:40,115 DEBUG se.ivankrizsan.messagecowboy.services.scheduling.QuartzSchedulerHelper - Scheduled task FileMover in group DemoGroup 2014-07-28 19:20:40,115 DEBUG se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Scheduled task FileMover in group DemoGroup 2014-07-28 19:20:40,115 INFO se.ivankrizsan.messagecowboy.services.starter.MessageCowboyStarterServiceImpl - Successfully (re)scheduled Message Cowboy tasks
Before trying out the new message transfer task, we’ll have a look at transport service configuration.
Transport Service Configuration
Transport service configuration files are, in this incarnation of Message Cowboy, Mule XML configuration files. Such files may contain, among other things:
- Connectors.
Enabling polling and delivery to different types of endpoints and/or endpoints with different configurations. - Flow constructs.
Allowing for, for instance, processing of messages prior to being delivered to the destination endpoint.
Location of Transport Service Configuration Files
Currently, there are two directories in which Mule XML configuration files can be put, in order for Message Cowboy to discover transport service configurations:
-
production-configurations/connectors/
-
production-configurations/transport-service-configurations/
Files in the above directories need to have the .xml suffix in order to be loaded by the transport service.
The transport service makes no difference between configuration files from the different directories – they are just a convenience for the user of the application allowing him/her to categorize configurations.
The number and location of these transport service configuration directories can be configured in the MessageCowboyConfiguration Spring configuration class.
Note!
It is important to remember that all the transport service configuration files are loaded into one and the same Mule context. Thus there must not be components with the same name in the different files, as this will cause an error.
Add a Transport Service Configuration File
In order to see what happens when a new transport service configuration file is added, the following Mule configuration file will be placed in one of the directories that contain transport service configuration files:
<?xml version="1.0" encoding="UTF-8"?> <mule xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:http="http://www.mulesoft.org/schema/mule/http" xsi:schemaLocation=" http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd"> <flow name="testflow"> <http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8081"/> <set-payload value="Howdy from the Message Cowboy!" /> </flow> </mule>
- In the “”production-configurations” directory, create a director named “transport-service-configurations” if it does not already exist.
-
In the “transport-service-configurations” directory, create a new file named “example-config.xml” and paste the above Mule configuration into that file.
-
Observe the console.
After some time there will be a quite some output, as the embedded Mule instance is stopped and restarted, starting with these lines:
2014-07-28 18:16:30,005 INFO se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Started executing task MessageCowboyTransportServiceRefreshTask in group MessageCowboySystemTasks 2014-07-28 18:16:30,005 DEBUG se.ivankrizsan.messagecowboy.services.scheduling.MethodInvokingJob - Invoking the method refreshConnectors on object of the type se.ivankrizsan.messagecowboy.services.transport.MuleTransportService with the parameters [] 2014-07-28 18:16:30,008 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Found 2 connector configuration files using the pattern file:production-configurations/connectors/*.xml 2014-07-28 18:16:30,011 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Found 1 connector configuration files using the pattern file:production-configurations/transport-service-configurations/*.xml 2014-07-28 18:16:30,011 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Detected change in configuration resources, refreshing 2014-07-28 18:16:30,013 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Found 2 connector configuration files using the pattern file:production-configurations/connectors/*.xml 2014-07-28 18:16:30,015 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Found 1 connector configuration files using the pattern file:production-configurations/transport-service-configurations/*.xml 2014-07-28 18:16:30,022 INFO org.mule.module.client.MuleClient - Initializing Mule...
- In a web-browser of your choice, paste the URL http://localhost:8081
There should be a greeting from the Message Cowboy in the browser window.
Note that this example configuration would probably be more suitable to be deployed to a standalone Mule instance rather than in the Message Cowboy, since it does not involve any scheduled activity.
-
Delete the file “example-config.xml” that you created earlier.
-
Observe the console.
After some time you should see that the modification is discovered and the embedded Mule client is restarted once more.
Example File Transfer
With the above configurations in place and Message Cowboy up and running, we will now look at a message being transferred by the scheduled task created earlier:
-
In Eclipse, create a folder named “INBOX” in the root directory of the Message Cowboy project.
-
Again in Eclipse, create another folder named “OUTBOX” also in the root directory of the project.
-
Copy and paste arbitrary file into the INBOX directory.
-
Wait at least 30 seconds.
-
Refresh the project in Eclipse (F5 or right-click on the project and select Refresh).
-
Observe the OUTBOX directory.
If everything worked as expected, the file should have been transferred from the INBOX directory to the OUTBOX directory. The file-transfer should generate log similar to this in the console:
2014-07-28 18:54:20,115 DEBUG se.ivankrizsan.messagecowboy.domain.entities.impl.QuartzMuleTaskJob - Started executing job FileMover in group DemoGroup 2014-07-28 18:54:20,115 DEBUG se.ivankrizsan.messagecowboy.domain.entities.impl.QuartzMuleTaskJob - Executing mover task job FileMover 2014-07-28 18:54:20,130 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Received message: org.mule.DefaultMuleMessage { id=61ae6aef-162c-11e4-86aa-553bdf28c94e payload=org.mule.transport.AbstractConnector$7 correlationId=<not set> correlationGroup=-1 correlationSeq=-1 encoding=UTF-8 exceptionPayload=<not set> Message properties: INVOCATION scoped properties: INBOUND scoped properties: directory=/Users/ivan/MyWorkspace/message-cowboy-master/INBOX fileSize=35120 originalFilename=myFile.xml timestamp=1234567890123 OUTBOUND scoped properties: MULE_ENCODING=UTF-8 originalFilename=myFile.xml SESSION scoped properties: } 2014-07-28 18:54:20,161 DEBUG se.ivankrizsan.messagecowboy.domain.entities.impl.QuartzMuleTaskJob - Message received from file://INBOX?connector=streamingFileConnectorInbound: se.ivankrizsan.messagecowboy.domain.entities.impl.MuleMoverMessage@5e8c1f43 2014-07-28 18:54:20,161 DEBUG se.ivankrizsan.messagecowboy.domain.entities.impl.QuartzMuleTaskJob - Dispatching message to file://OUTBOX?connector=streamingFileConnectorOutbound 2014-07-28 18:54:20,193 DEBUG se.ivankrizsan.messagecowboy.services.transport.MuleTransportService - Sent message: se.ivankrizsan.messagecowboy.domain.entities.impl.MuleMoverMessage@5e8c1f43 2014-07-28 18:54:20,193 INFO org.mule.transport.ConnectableLifecycleManager - Initialising: 'streamingFileConnectorOutbound.dispatcher.1193542816'. Object is: FileMessageDispatcher 2014-07-28 18:54:20,193 INFO org.mule.transport.ConnectableLifecycleManager - Starting: 'streamingFileConnectorOutbound.dispatcher.1193542816'. Object is: FileMessageDispatcher 2014-07-28 18:54:20,193 INFO org.mule.transport.file.FileConnector - Writing file to: /Users/ivan/MyWorkspace/message-cowboy-master/OUTBOX/myFile.xml
The contents of the file found in the OUTBOX directory should be identical to that of the original file.
Logging
Throughout the above examples we have had debug log enabled for the log generated by Message Cowboy. At times one may wish to reduce the amount of log or to increase the amount of log for certain packages etc.
This can be accomplished by creating a Log4J configuration file and, when launching Message Cowboy, setting the appropriate system property pointing at the Log4J configuration file.
In this example I have created a custom Log4J configuration file named mylog4j.xml:
java -Dlog4j.configuration=mylog4j.xml -jar message-cowboy-1.0.0-SNAPSHOT.jar
Future
Some thoughts I have about the future of Message Cowboy are, in no particular order:
- A GUI for configuration of scheduled tasks.
This to make integrations truly configurable, without the need for programming, and less error-prone. - An alternative implementation of the transport service using Apache Camel.
- Adding the ability to specify a set of properties associated with a scheduled task that are passed to the transport service when requesting and dispatching messages.
This would make it possible to develop more generic transport service configurations. - An alternative implementation of the scheduling service.
If I some day find some scheduling framework that is a serious, free, alternative to Quartz. - Enclosing some basic, commonly used, transport service configurations.
For instance, a ready-to-use configuration for guaranteed delivery of messages.
Suggestions and requests are welcome – please place such requests as an issue on GitHub: https://github.com/krizsan/message-cowboy/issues