iDatamining.org

Thursday, July 24, 2014

Asgard 1.5 Config.groovy file format

I tried to setup Asgard 1.5 today and find its initializing process has been changed a little bit. I remember I will be asked for aws credential information when I launched asGard first time. But, for Asgard 1.5 (ASGard 1.4.2 has same situation), I got error message like "com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain". After searching online, I make it work by setting up environment variable AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

In fact, we can just directly create the asgard configuration file ~/.asgard/Config.groovy. The format of the file is as below.


grails {
        awsAccounts=['YourAccountIDasDigitalNumber']
        awsAccountNames=['YourAccountIDasDigitalNumber':'YourAccountName']
}
secret {
        accessId='YourAccessID'
        secretKey='YourAccessSecretKey'
}
cloud {
        accountName='YourAccountName'
        publicResourceAccounts=['amazon']
}

Hope above tip will save others time. In fact, I solve the problem after reading this discussion on google asgard group.

Monday, April 7, 2014

several methods collecting for time series analysis

It is normal for data analysis processing have missing data for reasons. Once the data is missing. It is nature for people ask how the gap on the plot can be "fixed/smoothed". Also, it is very often to be asked to predict next several points ased on previous 7 to 8 points. For example, given history stock level, I am asked to predict stock level for beverage market data analysis.
With the asspumption of whatever has occured in the past will continue to happen in the future, we can treat these questions as time series analysis. Below are list of four methods to do forcecasting for a short time series.

  1. Moving averages
    install.packages("zoo")
    library(zoo)
    ma = rollmean(ts, k)
    
  2. For data without any systematic trend or seasonal components, we can use exponential smoothing
  3. Accounting for data seasonality using Winter's smoothing
  4. Accounting for data trend using Holt's smoothing
  5. install.packages("forecast")
    library(forecast)
    x = c(1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5 )
    x = ts(x, frequency = 5)
    plot(hw(x, 5), byt = "1")
    
Below a few collected videos about exponential smoothing and Holt-Winter forecasting method

1.11 Time Series- exponential smoothing


1.12 Time Series- moving averages