5 Things to do with your ATG Data Warehouse

2 01 2009

ATG Customer Intelligence offers a multitude of great features for business reporting and analytics but the real prize isn’t with all the tools it ships with but with the information that lives in your ATG data warehouse. There are so many things that you could be doing with that data. Here’s just a few ideas to get you thinking this year.

Build a Better Dashboard

Unless you’re like me you can’t really expect that everyone will look at 20 different reports on a daily or weekly basis to get an understanding of how their business is doing. So put together a dashboard that combines all the pertinent information you want to know about your business on a daily/weekly basis. ATG Customer Intelligence contains all the tools you need to build such a dashboard. See the following  post for more details about building a better dashboard.

Performance Management

The data in the ATG data warehouse contains a lot of information about whats happening on your eCommerce site.  But how do you really know if your merchandising or marketing strategy is reaching their goals. That’s where performance management comes in. Performance management allows you to measure the business outcomes against stated goals.  All you need to do is supply the targets and then use ATG Customer Intelligence to track your progress against these goals.

Market Basket Analysis

I’ve posted about this before so I won’t go into too much detail about it here. Market basket analysis allows you to get a better understanding of what products sell well together.  It allows you to understand that the “Camel Pea Coat” sells extremely well with the “Wool Hat” and “Wool Mittens”. This technique is a great way to understand what items might cross-sell together. And it’s all back by concrete data and not just intuition.

Mine for Gold

If you’re a real data nut then you might try your hand at applying some data mining techniques to mine information from the data in the data warehouse. The market basket analysis technique mentioned above is actually one such technique that can be used to mine associations rules between products but many other techniques exist that allow you discover other types of information. For instance knowledge discovery techniques exist that will allow you segment your customer base. You could then turn around an use this information to build better customer segments and help improve your personalization efforts.

Analyze This!

Query studio is a tool that ships with ATG Customer Intelligence that allows to perform ad-hoc queries against data in the ATG data warehouse. This tool is great for those times that you want to explore a little. You might uncover something interesting during your explorations. I know I’ve found some really interesting facts using this tool such as gift boxes tend to sell more during the holidays. That’s probably not surprising but they were purchased by males 10x more then females on your typical clothing retailer site.  This is the type of stuff that you probably don’t have a report built to tell you this and just shows you how exploring a little can get you to better know your customer.


GenericServletService and GenericFilterService Classes

4 05 2008

For anyone doing development in J2EE and ATG I wrote an extremely useful servlet and filter base class developers might be interested in. These classes were used as base classes for many of the framework components that were used as part of ATG Portals but they can be used in any standard J2EE application. 

I think the javadoc explains them nicely. Here’s the javadoc for the GenericFilterService which implements the javax.servlet.Filter interface. Similarly there’s a class called the GenericServletService which implements the javax.servlet.Servlet interface. See the ATG documentation for more information on these classes.

Read the rest of this entry »

Market Basket Analysis

9 04 2008

Market basket analysis is a technique that discovers relationships between pairs of products purchased together. The technique can be used to uncover interesting cross-sells and related products. In this post I’ll show you how simple market basket analysis can be and how to use the data in the ATG data warehouse to do your own market basket analysis.

The idea behind market basket analysis is simple. Simply examine your orders for products have been purchased together. For example using market basket analysis you might uncover the fact that customers tend to buy hot dogs and buns together. Using this information you might organize the store so that hot dogs and buns are next to each other. In an e-commerce environment you might create a cross-sell rule to offer the shopper buns whenever they place hot dogs in their shopping cart.

There are a couple of measures we use when doing market basket analysis and they are described here. The first measure is the frequency. The frequency is defined as the number of times two products were purchased together. If hot dogs and buns were found together in 820 baskets this would be its frequency.

Frequency by itself doesn’t tell the whole story. For instance if I told you hot dogs and buns were purchased 820 times together you wouldn’t know if that was relevant or not. Therefore we introduce two other measures called support and confidence to help with the analysis.

If you divide the frequency by the total number of orders you get the percentage of order containing the pair. This is called the support. Another way to thinking about support is as the probability of the pair being purchased. Now if 820 hot dogs and buns were purchased together and your store took 1000 orders the support for this would be calculated as (820 / 1000) = 82.0% .

We can extend this even further by defining a calculation called confidence. Confidence compares the number of times the pair was purchased to the number of times one of the items in the pair was purchased. In probability terms this is referred to as the conditional probability of the pair. So going back to our hot dogs example if hot dogs were purchases 900 times and out of those 900 purchases 820 contained buns we would have a confidence of (820 / 900) = 91.1%.

Now that we’ve defined frequency, support and confidence we can talk a little about what a market basket analysis report might look like. The report would have the user select the product they are interested in performing the analysis on (ie hot dogs). Then it would list all the products that were purchased together with the selected products ranked by it frequency. It might look something like the following

Market Basket Analysis: Hot Dogs

Product Frequency Support Confidence
Buns 820 82.0% 91.1%
Ketchup 800 80.0% 23.2%
Mustard 750 75.0% 34.5%
Jello 321 32.1% 45.2%

As a merchandiser I tend to look for pairs with a high confidence. A confidence of 100% means that the products are always purchased together. The higher the confidence means that there is probably a strong relationship between the products. When looking at reports like this you’ll find that some of the obvious pairs have the highest confidence (like hot dogs and buns!) but as you keep looking down the list you should discover some of the more interesting pairings you might not have thought of.

After examining the confidence I next look at the support. A low support means that the pair isn’t purchased a lot. This doesn’t mean it would make a bad cross sell just that they aren’t ordered together very frequently. Use your own judgment. Obviously if you see something with both a high confidence and high support you’ve found something interesting.

You can create a report to perform market basket analysis with ATG Customer Intelligence. See the attached xml for the report definition. To install this report just copy the report definition xml to your clipboard. Next start Report Studio with a blank report. Finally use the Tools->Open Report from Clipboard option and save the report to your environment.

download sample report : market-basket-analysis

Open Source ATG Projects

27 02 2008

I’m looking at compiling a list of open source projects that develop on the ATG platform. I know of a few interesting projects. I’ve listed some of them below but if you know of any others please let me know so I can update the list.

Also if you have an idea for an interesting project let me know. Perhaps we can drum up some engineers that would support it. 


Developing with ATG: Part V

9 01 2008

In this edition of the Developing with ATG series we’ll discuss the component life-cycle and scoping. We’ll first discuss the life-cycle of a component in nucleus and follow up with a discussion on component scopes.

The life-cycle of a nucleus component is pretty simple: the component is started and the component is stopped. If a component needs to do any work when started or stopped then it should implement the atg.nucleus.ServiceListenerinterface. A ServiceListener must implement the startService() method and the stopService() method.  When the startService() method is called it is passed an instance of atg.nucleus.ServiceEvent. The ServiceEventcontains information about the components configuration as well as a reference to the instance of atg.nucleus.Nucleus which started the component. These are often great opportunities for the component to create/release required resources. 

So now lets introduce you to scope. Nucleus has three scopes a component can belong to: request, session or global. Up to this point we’ve been discussing globally scoped components but in this edition we introduce request and session scoped components.

Globally scoped components are created when the component is resolved by name and destroyed when nucleus shuts down. There are a lot of important globally scoped components in nucleus. Many of them were described in Developing with ATG: Part III. Globally scoped component live the longest of all the scopes.

A session scoped component differs from a globally scoped component in that these components are created in the context of an HTTP session. When the component is resolved by name it will be associated with the current HTTP session. When the HTTP session is destroyed the component is stopped and taken out of service.

One example of a session scoped component is the /atg/userprofiling/Profile component. This component represents the user associated with the session. If one needs to reference the current user then all one needs to do is resolve this component by name using a name resolver. The following code demonstrates how one can accomplish this.

Profile profile = 
(profile) nameResolver.resolveName("/atg/userprofiling/Profile"); 
String login = 

You can even reference a session scoped component from another session (or request) scoped components property file. The following example demonstrates how you would configure a custom component that a reference the session scoped Profile component.


A request scope component has the shortest lifecycle of any of the scopes. A request scoped component only lives as long as the HTTP request it is being served in. Once the request has be serviced all of the request scoped components are destroyed for that request are destroyed.  The /OriginatingRequest component is an example of a request scoped component. This component implements the javax.servlet.http.HttpServletRequest interface and represents the current request being serviced.

To specify the scope of a component set the $scope metadata property in your components property file. The value of this property can be either global, session or request. If left unspecified the components scope will be global. 

Finally here are some rules you need to keep in mind when creating application that mix components of various scopes. Globally scoped component can only reference other globally scoped components. Session scoped components can only reference other session scoped components (in the same session) or globally scoped components. Finally request scoped components can only reference other request scoped components (in the same request) or sessions scoped components (in the same session as the request) or globally scoped components.  If you think about this for a little its actually fairly sensible.

By the way Nucleus is one of the only component models I am aware of that has support for request and session scoped components. The only other component model I know of that does something like this is Spring 2.0. If you know of any other drop me a line. 

2008 Predictions

1 01 2008

Happy New Year everyone! Since I seem to be seeing many posts these days about 2008 predictions I thought I might try my hand at this.  Here is a short list of trends and predictions I see for the year 2008.

Web 2.0

Web 2.0 technology will hit its stride this year with more and more web sites adopting various technology. The number of rich Internet applications on the net will increase this year. Many of them will employ AJAX technology but you’ll find many of the leading edge sites will employ flash based solutions. Also you’ll find many site providing RESTful web services for machine based interactions.

Web Analytics

You’ll see more companies looking at gathering and analyzing web analytics data. Most will try out Google Analytics because its free. But you’ll find others looking at commercial solutions from Coremetrics and Omniture. After about three months of use of these solutions you’ll find that many companies will either 1) not have the skill set to use the information effectively or 2) will be unsatisfied with the limited results. As a result you’ll also see companies asking the web analytics vendors for access to the raw data so that they can integrate it with offline and other marketing data from corporate data warehouses.  


Sun will release a new version of its popular programming language Java to the world. Sun will celebrate Javas’ 11th year in service! Engineers young and old alike wonder how long this can go on for and start to wonder if they’ll be programing in Java until retirement. Javas mascot Duke starting to show some grey hairs.

Red Sox

This one’s a sure thing. The Boston Red Sox win the World Series for a second straight year in a row.

Happy New Year!

Developing with ATG: Part IV

18 12 2007

In this edition of Developing with ATG: Part IV we discuss layered configuration. Many component models have some method for configuring a component. As you’ve seen in previous articles one can use a property file to configure a nucleus component. This isn’t unheard of as many other component models do exactly this to satisfy the configuration requirements of the model. What makes nucleus different from many other component models is the addition of configuration layering. Configuration layering is best exhibited with an example.

Say Joe developer has just developed a component that receives data from an RSS feed. This feed is accessed using a a feed URL. His component contains a property called feedURL, amongst many others, that is used to represent the URL used to access the feed.  Joe deploys this component to his configuration layer using a property file and sets the feedURL property to http://feeds.feedburner.com/MetaTheory. Joe then publishes this component so others can use it. Joes property file looks something like the following:

# /acme/MyFeed.properties
schedule=calendar * * * 22 0

Anne finds out about the component Joe just wrote and thinks it’s really cool. Anne wants to use it for her project but wants to change the URL to access a different feed instead. Now in other systems you would need to copy the entire configuration file to your project and then change only the properties that needed to be changes. But in nucleus all Anne would need to do is create a property file with the same name as Joes component in her projects configuration layer and include only the properties she needs to override. All of the other properties are inherited from Joes configuration layer. Annes property file would look like the following:

# /acme/MyFeed.properties

Lets take this example one step further. Say Joe revises his RSS feed component adding a new property to control the maximum number of items returned by the feed. Joe adds a property called maxItems to his RSS Feed component and updates the property file in his configuration layer setting the value to 10. Anne is unaffected by this change Joe has made as she automatically inherits the update to this component from Joes configuration layer. If later Anne wanted to change the value of maxItems she could always override the value in here configuration layer.

So how does nucleus handle this? When nucleus starts up it constructs what is referred to as the configuration path. The configuration path is similar to the class path but works in reverse order. It contains a list of directories to search for configuration information in. Directories at the end of the list override or append to configuration in the directories at the beginning of the list. The <DYNAMO_HOME>/localconfig path is usually used the final path in the list.

When nucleus configures a component it searches all the configuration paths for any property files which make up the configuration for the component. Then it combines them into a single configuration object using the override/append rule described above. This configuration object is then used to configure the component after instantiation.

Nucleus takes the idea of configuration layering beyond just properties files. XML files can be deployed to configuration layers and similar rules are used to combine these files into a single XML document. A classic example of this is used when configuring repository definitions. See the ATG documentation for information about XML combine for more information on this.

So that was a brief introduction to configuration layering.  Its used in many different ways within ATG products from configuring hostname and port numbers to enabling/disabling debug output.  In my experience nucleus is the only component model which I’ve seen that works has this feature but if you know of any others drop me a line. I would love to hear about any others.