How to Train My own Model using Mahout

Mahout Training A Model

       Mahout is a scalable machine learning algorithms focused primarily in the areas of collaborative filtering. Follow step by step process to train your own model based on your own data. Here in this example we are giving training on 20news-all data folder.

 

1. Create a working directory and copy all the data into this directory, make a copy of duplicate, so that your original data will not lost.

2.    Create Sequence files from Training data using following command:

a.    Command:
./bin/mahout  seqdirectory   -­i 20news-all -o 20news-seq

3.    We have to convert Sequence files to Vectores, mahout will use vector files only for all algorithms, its the basic need to convert data file to sequence and then sequence files to vector files.

a.    Command:
./bin/mahout seq2sparse ­i 20news­seq ­o 20news-vectors ­lnorm ­nv -wt tfidf -wt(--weight) =  TF  or  TFIDF

4.    In practice, training examples are typically divided into two parts. One part, known as the training data, consists of 80 –90 percent of the available data. The training data is used in training to produce the model. A second part, called the test data, is then given to the model without telling it the desired answers, although they’re known. This is done in order to compare the output that results from classifying, using the desired output. Here, we split the data set randomly as 80% training data and 20% test data. The value 20 after the “ randomSelectionPct” parameter indicates that 20% of the data will be selected randomly for the test data set

a.    Command:
./bin/mahout  split ­i 20news­vectors/tfidf-vectors --trainingOutput  20news-train-vectors -­testOutput 20news­test­vectors -­randomSelectionPct 20 -­overwrite -­sequenceFiles -xm sequential

5.    Training Naive Bayes model

a.    Command:
./bin/mahout trainnb -i 20news-train-vectors -el -o model -li labelindex –ow

6.    Self testing on training set: You have ready with your own model now, but we need to test the accuracy of the model by using training set itself.

a.    Command:
./bin/mahout testnb -i 20news­train­vectors -m model -l labelindex -ow -o 20news-testing

b.    Output:

                                          i.    Kappa                                       0.4955

                                         ii.    Accuracy                                    99.103%

                                        iii.    Reliability                                99.0858%

                                       iv.    Reliability (standard deviation)            0.0952

7.    Testing on holdout set, Here we are using holdout set to test model accuracy.

a.    Command:
./bin/mahout testnb -i 20news­test­vectors -m model -l labelindex -ow -o 20news-testing

b.    Output:

                                          i.    Accuracy                                    90.093% 

This is the way to train a model in Apache mahout. Next blog will be how to use this model to get sentiment of a comment.

Serialize and Deserialize java Object

To Serialize and De-serialize java Object

Serialization is nothing but converting an object into a sequence of bytes(here we are encrypting using BASE64Encoder), which can be persisted database or save in a file on disc. And Creating Java object from sequence of bytes which is save in the file is called Deserialize

Your class must be implement Serializable interface in order to Serialize or Deserialize.
Serializable is a marker interface that adds serializable behaviour to the class implementing it.


Here We are going to serialize with sun.misc.BASE64Encoder to secure encrypt, So create encrypt and decrypt objects in your main class;

    static private BASE64Encoder encode = new BASE64Encoder();
    static private BASE64Decoder decode = new BASE64Decoder();



Call objectToSerialize method to serialize your any object which is implemented Serializable interface.

    static public String objectToSerialize(Object obj) {
        String out = null;
        if (obj != null) {
            try {
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                ObjectOutputStream oos = new ObjectOutputStream(baos);
                oos.writeObject(obj);
                out = encode.encode(baos.toByteArray());
            } catch (IOException e) {
                e.printStackTrace();
                return null;
            }
        }       
        return out;
    }                     

Call derializeToObject method to Deserialize your String to again Java Object.

    static public Object SToO(String str) {
        Object out = null;
        if (str != null) {
            try {
                ByteArrayInputStream bios = new ByteArrayInputStream(decode.decodeBuffer(str));
                ObjectInputStream ois = new ObjectInputStream(bios);
                out = ois.readObject();
            } catch (IOException e) {
                e.printStackTrace();
                return null;
            } catch (ClassNotFoundException e) {
                e.printStackTrace();
                return null;
            }
        }      
        return out;
    }

Thats it:

Struts2 File upload using Ajax & Jquery uploadify

Struts2 File upload  using Ajax & Jquery uploadify

         The Struts2 framework provides built-in support for file uploads using "Form-based File Upload". When a file is uploaded it will be stored in a temporary directory and they should be moved by your Action class to a permanent place or directory to ensure the file is not lost.

   You can download jquery.uploadify.min.js file from uploadify.org, include both js and css files to your form

         File uploading in Struts is possible through a pre-defined interceptor called FileUpload interceptor which is available (org.apache.struts2.interceptor.FileUploadInterceptor ).

Add below code snippet to your Jsp file(Multiple is optional):

<script src="js/jquery.uploadify.min.js" type="text/javascript"></script>
<link rel="stylesheet" type="text/css" href="css/uploadify.css" />
 ....
<form ....action="uploadfiles.do">
 <input id="file_upload" name="myFile" type="file" multiple="multiple" />
 <div id="queue"></div>
.....
</form>

<script type="text/javascript">
    $('#file_upload').uploadify({
        'formData': {hoardingid: $("#hoardingid").val()},
        'swf'     : 'js/uploadify.swf',
        'uploader': 'hrdimgs.do',
        buttonText:'Add Photos',
        buttonClass:'uploadimg',
        height:22,
        width:100,
        fileSizeLimit:"2MB",
        fileTypeDesc:'Images',
        fileTypeExts:'*.gif;*.png;*.jpg',
        removeCompleted:false,
        fileObjName:'myFile'
    });
</script>

Add the following code in struts.xml file:

    <action name="uploadfiles" class="com.struts2.uploadFileAction">
       <interceptor-ref name="basicStack">
       <interceptor-ref name="fileUpload">
           <param name="allowedTypes">image/jpeg,image/gif</param>
       </interceptor-ref>      
   </action>

 
Now we are ready to implement you action class, Add the following code in your action to save the file in destPath before its loss:
   Properties in your action with setter and getter methods:

    private File myFile;
    private String myFileContentType;
    private String myFileFileName;
    private String destPath = Constants.IMGSAVEPATH;
.................

Method need to write, Here you can not use same session from normal action and uploadify.swf , so you need to put the saved file path in your servlet context or you should save in your database immediately.

public String uploadImages(){
        try {           
              File destFile  = new File(destPath, myFileFileName);
             FileUtils.copyFile(myFile, destFile);
             //putting files in context, to use after submit the form.
             HttpServletRequest request = ServletActionContext.getRequest();
             List<File> files = (List<File>)request.getServletContext().getAttribute("uploadedfiles");
             if(files == null){
                 files = new ArrayList<File>();               
                 request.getServletContext().setAttribute("uploadedfiles", files);
             }
             files.add(destFile);
        } catch (Exception e) {
            e.printStackTrace();
              return ERROR;
        }
        return ActionSupport.NONE;
    }

Your done  Thats it!

Resizing Images without loss quality. You can run as thread in background

We can resize Image using in using Graphics2D that in awt package and other required classes too.  See the following core snippet you can understand easily:

    File imageFile = new File(imagePathToBeResize);

     BufferedImage bimage = ImageIO.read(imageFile);

     ScalerThread scalerThread = new ScalerThread();
     scalerThread.setFile(bimage);
     scalerThread.setWidth(Integer.parseInt(sizes[0]));
     scalerThread.setHeight(Integer.parseInt(sizes[1]));
     scalerThread.setArgumentType(ScalerThread.TYPE_STATIC);
     
     ByteArrayOutputStream baos = new ByteArrayOutputStream();
     baos = scalerThread.doInBackground();
     byte[] bytesOut = baos.toByteArray();

   FileOutputStream fos = new FileOutputStream(imagepath to be save);
         fos.write(bytesOut);
         fos.close();

Save the following file as ScalarThread  then your done with scaling Image and also you can Scale Image with rounded corners


required imports;

public class ScalerThread {
    public final static char HIGH_QUALITY = 0;
    public final static char LOW_QUALITY = 1;
    public final static char TYPE_STATIC = 0;
    public final static char TYPE_PROPORTIONAL = 1;
    public final static char LARGEST_SIDE = 0;
    public final static char SMALLEST_SIDE = 1;
   
    private int width,height;
    private boolean blur;
    private BufferedImage file;
    private String outputDir,prefix,format;
    private static RenderingHints renderingHints;
    private char quality,side;
    private char argumentType;

    public ScalerThread() {
        width = 500;
        height = 375;
       
        prefix = "scaled-";
        format = "jpg";
        blur = false;
       
        final HashMap<Key,Object> map = new HashMap<Key,Object>();
        map.put(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
        map.put(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BILINEAR);
        map.put(RenderingHints.KEY_RENDERING, RenderingHints.VALUE_RENDER_QUALITY);
        map.put(RenderingHints.KEY_ALPHA_INTERPOLATION, RenderingHints.VALUE_ALPHA_INTERPOLATION_QUALITY);
       
        renderingHints = new RenderingHints(map);
       
        quality = 0;
        argumentType = 0;
    }   
   
   
    private void qualityScale(BufferedImage buffer, BufferedImage source) {
        final Image a = source.getScaledInstance(width, height, BufferedImage.SCALE_SMOOTH);
        buffer.getGraphics().drawImage(a,0,0,buffer.getWidth(),buffer.getHeight(),null);
    }
   
    private void fastScale(BufferedImage buffer, BufferedImage source) {
        Graphics2D g = (Graphics2D)buffer.getGraphics();
        g.setRenderingHints(renderingHints);
        g.drawImage(source, 0, 0, buffer.getWidth(), buffer.getHeight(), null);
    }
    public static BufferedImage blurImage(BufferedImage image, float amount) {
       
        float amount2 = (1-amount)/8.0f;
       
        float[] blurKernel = {
        amount2, amount2, amount2,
        amount2, amount, amount2,
        amount2, amount2, amount2
        };

        BufferedImageOp op = new ConvolveOp(new Kernel(3, 3, blurKernel), ConvolveOp.EDGE_NO_OP, renderingHints);
        return op.filter(image, null);
    }
   

    public ByteArrayOutputStream doInBackground() throws Exception {
        if(file==null) return null;
       
        int type = BufferedImage.TYPE_INT_RGB;
        if(format.equals("png")) type = BufferedImage.TYPE_INT_ARGB;
        BufferedImage buffer;
        int tries;
        tries = 0;
        while(tries<2) {
            try{
                if(argumentType==TYPE_STATIC) {
                    buffer = staticScale(file,type);
                }else {
                    buffer = proportionalScale(file,type);
                }
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                ImageIO.write(buffer, format, baos);
                return baos;
            } catch(IOException ex) {
                System.out.println("Couldn't convert:");
            } catch(OutOfMemoryError mem) {
                System.out.println("Error! Ran out of memory while converting "+file.toString());
                System.gc();
                Thread.sleep(3000);
                tries++;
            }
        }
        System.gc();
        return null;
    }
   
    public ByteArrayOutputStream doRoundInBackground() throws Exception {
        if(file==null) return null;
        Image image = Toolkit.getDefaultToolkit().createImage(file.getSource());
        int width = getWidth();
        int height = getHeight();
        image = getScaledRoundedImage(image, width, height);
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        BufferedImage bufferedImageOut = new BufferedImage(image.getWidth(null), image.getHeight(null), BufferedImage.TYPE_INT_RGB);
            Graphics g = bufferedImageOut.getGraphics();
           g.fillRect(0,0, width, height);
            g.drawImage(image, 0, 0, null);
        ImageIO.write(bufferedImageOut, format, baos);
        return baos;
    }
   
    public static Image getScaledRoundedImage(Image image, int width, int height)
    {
        ImageIcon scaledImage;
        int scaledImageWidth;
        int scaledImageHeight;
        BufferedImage destImage;
        Graphics2D g;
        scaledImage = scale(new ImageIcon(image), width, height);//scaleIconWithinBounds(image, width, height);
        scaledImageWidth = scaledImage.getIconWidth();
        scaledImageHeight = scaledImage.getIconHeight();
        if(scaledImageHeight <= 0 || scaledImageWidth <= 0)
        {
            return null;
        }
        destImage = new BufferedImage(scaledImageWidth, scaledImageHeight, 2);
        g = destImage.createGraphics();
       
        g.fillRoundRect(0, 0, scaledImageWidth, scaledImageHeight, 10, 10);
        g.setColor(new Color(161,197, 225 ));
        g.setComposite(AlphaComposite.SrcIn);
       // g.setColor(new Color(161,197, 225 ));
        g.drawImage(scaledImage.getImage(), 0, 0, null);
        g.drawRoundRect(0, 0, width - 1, height - 1, 10, 10);
        g.dispose();
        g.dispose();
 
        return destImage;
    }
   
    public static ImageIcon scale(ImageIcon icon, int newHeight, int newWidth) {
        Image img = icon.getImage();
        int height = icon.getIconHeight();
        int width = icon.getIconWidth();
        height = newHeight;
        width = newWidth;
        img = img.getScaledInstance(width, height, Image.SCALE_SMOOTH);
        return new ImageIcon(img);
    }
   
    private BufferedImage staticScale(BufferedImage image,int type) {
        BufferedImage buffer;
        if(image.getWidth() > width) {
            buffer = new BufferedImage(width,height,type);
        } else {
            width = image.getWidth();
            height = image.getHeight();
            buffer = new BufferedImage(width, height, type);
        }
        if(quality==HIGH_QUALITY) {
            qualityScale(buffer,image);
        } else {
            fastScale(buffer,image);
        }
       
        return buffer;
    }
   
    private BufferedImage proportionalScale(BufferedImage image, int type) {
        BufferedImage buffer;
        double ls,prop;
        if(side==LARGEST_SIDE) {
            ls = Math.max(image.getWidth(), image.getHeight());
        }else{
            ls = Math.min(image.getWidth(), image.getHeight());
        }
        prop = ((double)width)/ls;
        buffer = new BufferedImage((int)(image.getWidth()*prop),(int)(image.getHeight()*prop),type);
        if(quality==HIGH_QUALITY) {
            qualityScale(buffer,image);
        } else {
            fastScale(buffer,image);
        }
        return buffer;
    }

    ...setter & getter methods

}
thats it.

How to Change the JCaptcha configurations?

Set up JCAPTCHA to your application

In this example, a simple Java servlet is used to generate a jcaptcha image, that user must enter letters appearing in the jcaptcha image

• Add the jcaptcha jar to your project - you can download here
• If application do not already have commons-logging and commons-collection project add them too.

• Add the following servlet path and URL pattern to your web.xml in your application:

<servlet>

        <servlet-name>jcaptcha</servlet-name>

        <servlet-class>com.struts2.security.ImageCaptchaServlet</servlet-class>

</servlet>

<servlet-mapping>

        <servlet-name>jcaptcha</servlet-name>

        <url-pattern>/jcaptcha.jpg</url-pattern>

</servlet-mapping>

• Add the follwing HTML script in your jsp

  <img src="jcaptcha.jpg" /> <input type="text" name="jcaptcha" value="" />

•  Servlet to generate Jcaptcha image.

       byte[] captchaChallengeAsJpeg = null;
       // the output stream to render the captcha image as jpeg into
        ByteArrayOutputStream jpegOutputStream = new ByteArrayOutputStream();
        try {
        // get the session id that will identify the generated captcha.
        //the same id must be used to validate the response, the session id is a good candidate!
        String captchaId = httpServletRequest.getSession().getId();
        // call the ImageCaptchaService getChallenge method
            BufferedImage challenge =null;
            if(httpServletRequest.getParameter("sm")!=null){
               /* challenge =CaptchaSmallServiceSingleton.getInstance().getImageChallengeForID(captchaId,
                        httpServletRequest.getLocale());*/
            }else{
                challenge =CaptchaServiceSingleton.getInstance().getImageChallengeForID(captchaId,
                            httpServletRequest.getLocale());
            }
            //System.out.println(challenge);
            // a jpeg encoder
            JPEGImageEncoder jpegEncoder =
                    JPEGCodec.createJPEGEncoder(jpegOutputStream);
            jpegEncoder.encode(challenge);
        } catch (IllegalArgumentException e) {
            httpServletResponse.sendError(HttpServletResponse.SC_NOT_FOUND);
            return;
        } catch (CaptchaServiceException e) {
            httpServletResponse.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
            return;
        }
                   
        captchaChallengeAsJpeg = jpegOutputStream.toByteArray();

        // flush it in the response
        httpServletResponse.setHeader("Cache-Control", "no-store");
        httpServletResponse.setHeader("Pragma", "no-cache");
        httpServletResponse.setDateHeader("Expires", 0);
        httpServletResponse.setContentType("image/jpeg");
        ServletOutputStream responseOutputStream =
                httpServletResponse.getOutputStream();
        responseOutputStream.write(captchaChallengeAsJpeg);
        responseOutputStream.flush();
        responseOutputStream.close();




• To generate default Jcaptcha image you the following code




public class CaptchaServiceSingleton {
   
    private static ImageCaptchaService instance = new DefaultManageableImageCaptchaService();
   
    public static ImageCaptchaService getInstance(){
        return instance;
    }
}

Verify your jcapthca string with sessionId,

• verifyString is your String submitted throw the registration form or something else.


//String verifyString = request.getParameter("yourparam");
 String captchaId = request.getSession().getId();
   boolean isResponseCorrect =               CaptchaServiceSingleton.getInstance().validateResponseForID(captchaId, verifyString);




• You can change your Image by configuration as follows:

1. Implement ListImageCaptchaEngine where you can change all your settings There are many generator classes for every class(for below of all generator)you can use any one of it.
1. Word Generator
2. Color Generator 
3. Background Generator
4. Font Generator 

 public class MyImageCaptchaEngine extends ListImageCaptchaEngine {
    @Override
    protected void buildInitialFactories() {
        WordGenerator wgen = new RandomWordGenerator("ABCDEFGHIJKLMNOPQRSTUVWXYZ123456789");
        RandomRangeColorGenerator cgen = new RandomRangeColorGenerator(new int[] { 0, 255 }, new int[] { 20, 100 }, new int[] { 20, 100 });

        TextPaster textPaster = new RandomTextPaster(new Integer(4), new Integer(5), cgen, Boolean.TRUE);

        BackgroundGenerator backgroundGenerator = new UniColorBackgroundGenerator(new Integer(240), new Integer(50), new Color(252,252,253));
        Font[] fontsList = new Font[] { new Font("Helvetica", Font.TYPE1_FONT, 10), new Font("Arial", 0, 14), new Font("Vardana", 0, 17), };

        FontGenerator fontGenerator = new RandomFontGenerator(new Integer(18), new Integer(30), fontsList);
        WordToImage wordToImage = new ComposedWordToImage(fontGenerator, backgroundGenerator, textPaster);
        this.addFactory(new GimpyFactory(wgen, wordToImage));
    }
}

•  change code as follows:




private static ImageCaptchaService instance = new DefaultManageableImageCaptchaService(
            new FastHashMapCaptchaStore(),
            new MyImageCaptchaEngine(),
            180,
            100000,
            75000);

IF you are using open JDK replacing following two lines

  JPEGImageEncoder jpegEncoder =
                    JPEGCodec.createJPEGEncoder(jpegOutputStream);
  jpegEncoder.encode(challenge);           









replace with: 

ImageIO.write(challenge, "jpg", jpegOutputStream);
 

And that's it!