顯示具有 Hadoop 標籤的文章。 顯示所有文章
顯示具有 Hadoop 標籤的文章。 顯示所有文章

2014年10月10日 星期五

[JAVA] Big Data(4) Hadoop Multiple Input

Multiple Input

在map-reduce時，若不同的資料來源要塞給不同的mapper，最後再一起塞進reducer運算，就需要使用Multiple Input 的功能。如下圖，有三個不同的資料來源，先分別進入不同的mapper，然後最後要進到同一個reducer。



首先要先import需要的的類別
org.apache.hadoop.mapreduce.lib.input.MultipleInputs

 然後在主程式(main)中寫入下面這一行，告訴電腦你要把哪一筆資料送進哪一個Mapper class的map函數

 MultipleInputs.addInputPath(Job名稱, 輸入資料的位址, 格式, Mapper class的名字);

 接下來看一個簡單到近乎無腦的例子
假設手上有三筆資料，都包含學號、科目和成績，但是長相就是不太一樣，現在我們要計算各科的平均分數

 第一筆
$HADOOP/hadoop fs -cat input_1 | head -3
WUJ-360100;math;56
WPY-802007;math;98
FKT-670008;science;67
第二筆 
$HADOOP/hadoop fs -cat input_2 | head -3
{Number:FJB-004150, Subject:math, Score:96}
{Number:QDG-300700, Subject:chinese, Score:90}
{Number:JVY-030140, Subject:chinese, Score:71}
第三筆 
$HADOOP/hadoop fs -cat input_3 | head -3
[Number=>ITM-501806; Subject=>science; Score=>82]
[Number=>QBE-003981; Subject=>math; Score=>85]
[Number=>EUJ-017009; Subject=>chinese; Score=>63]
以上三種長相的資料要分別送給三種不同的Mapper中處理，產生(subject, score)的pair然後統一送進一個Reducer做平均數的計算，所以要準備三種Mapper 
public static class Map1 extends Mapper
{
      public void map(LongWritable key, Text value, Context con) 
        throws IOException, InterruptedException
      {
              // get the student number
              String stNum = value.toString().split(";")[1];

              // get score
              int score = Integer.parseInt(value.toString().split(";")[2]);
              con.write(new Text(stNum), new IntWritable(score));
      }
}
public static class Map2 extends Mapper
{
      public void map(LongWritable key, Text value, Context con) 
        throws IOException, InterruptedException
      {       
              // "shave" the input value
              String line = value.toString().replaceAll("}", "");

              if(line.contains(",")){
                      // get the student number
                      String stNum = line.split(",")[1].split(":")[1];

                      // get score
                      int score = Integer.parseInt(line.split(",")[2].split(":")[1]);
                      con.write(new Text(stNum), new IntWritable(score));
              }
      }
}
public static class Map3 extends Mapper
{
      public void map(LongWritable key, Text value, Context con) 
        throws IOException, InterruptedException
      {
              // "shave" the input value
              String line=value.toString().replaceAll("[]\\[]", "");


              if(line.contains(";")){
                // get the student number
                String stNum = line.split(";")[1].split("=>")[1];

                // get score
                int score = Integer.parseInt(line.split(";")[2].split("=>")[1]);
                con.write(new Text(stNum), new IntWritable(score));
              }
      }
}
Reducer其實就只需要一個就可以了 
public static class Red extends Reducer
{
     public void reduce(Text stNum, Iterable scores, Context con)
      throws IOException , InterruptedException
      {
              int numerator = 0;
              int denominator = 0;
              for (IntWritable v : scores){
                  numerator += v.get();
                  denominator ++;
              }
              int avg = numerator/denominator;
              con.write(stNum, new IntWritable(avg));
      }
}
然後是比較麻煩的主程式
public static void main(String[] args) throws Exception
{
      Configuration conf=new Configuration();
      String[] files=new GenericOptionsParser(conf,args).getRemainingArgs();
      Path inPath1=new Path(files[0]);
      Path inPath2=new Path(files[1]);
      Path inPath3=new Path(files[2]);
      Path outPath=new Path(files[3]);
      FileSystem hdfs = outPath.getFileSystem(conf);
      if (hdfs.exists(outPath)){
        hdfs.delete(outPath, true);
      };

      Job exampleJob = new Job(conf,"example");
      exampleJob.setJarByClass(MpInputExp.class);
      exampleJob.setMapperClass(Map1.class);
      exampleJob.setMapperClass(Map2.class);
      exampleJob.setMapperClass(Map3.class);
      exampleJob.setReducerClass(Red.class);
      exampleJob.setOutputKeyClass(Text.class);
      exampleJob.setOutputValueClass(IntWritable.class);

      MultipleInputs.addInputPath(exampleJob, inPath1, TextInputFormat.class, Map1.class);
      MultipleInputs.addInputPath(exampleJob, inPath2, TextInputFormat.class, Map2.class);
      MultipleInputs.addInputPath(exampleJob, inPath3, TextInputFormat.class, Map3.class);
      
      FileOutputFormat.setOutputPath(exampleJob, outPath);
      System.exit(exampleJob.waitForCompletion(true) ? 0:1);
}
要注意MultipleInputs.addInputPath有沒有把Input和Mapper配對好

最後來看結果(打包部分省略，可以參考這裡)
$HADOOP/hadoop fs -getmerge output_exp output_exp
cat output_exp

science 68
chinese 70
math    68
送上所有JAVA code結束這惱人的一切 
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.MultipleInputs;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

public class MpInputExp
{
  public static class Map1 extends Mapper
  {
        public void map(LongWritable key, Text value, Context con) 
          throws IOException, InterruptedException
        {
                // get the student number
                String stNum = value.toString().split(";")[1];

                // get score
                int score = Integer.parseInt(value.toString().split(";")[2]);
                con.write(new Text(stNum), new IntWritable(score));
        }
  }
  public static class Map2 extends Mapper
  {
        public void map(LongWritable key, Text value, Context con) 
          throws IOException, InterruptedException
        {       
                // "shave" the input value
                String line = value.toString().replaceAll("}", "");

                if(line.contains(",")){
                        // get the student number
                        String stNum = line.split(",")[1].split(":")[1];

                        // get score
                        int score = Integer.parseInt(line.split(",")[2].split(":")[1]);
                        con.write(new Text(stNum), new IntWritable(score));
                }
        }
  }
  public static class Map3 extends Mapper
  {
        public void map(LongWritable key, Text value, Context con) 
          throws IOException, InterruptedException
        {
                // "shave" the input value
                String line=value.toString().replaceAll("[]\\[]", "");


                if(line.contains(";")){
                  // get the student number
                  String stNum = line.split(";")[1].split("=>")[1];

                  // get score
                  int score = Integer.parseInt(line.split(";")[2].split("=>")[1]);
                  con.write(new Text(stNum), new IntWritable(score));
                }
        }
  }
  public static class Red extends Reducer
  {
       public void reduce(Text stNum, Iterable scores, Context con)
        throws IOException , InterruptedException
        {
                int numerator = 0;
                int denominator = 0;
                for (IntWritable v : scores){
                    numerator += v.get();
                    denominator ++;
                }
                int avg = numerator/denominator;
                con.write(stNum, new IntWritable(avg));
        }
   }
  public static void main(String[] args) throws Exception
  {
        Configuration conf=new Configuration();
        String[] files=new GenericOptionsParser(conf,args).getRemainingArgs();
        Path inPath1=new Path(files[0]);
        Path inPath2=new Path(files[1]);
        Path inPath3=new Path(files[2]);
        Path outPath=new Path(files[3]);
        FileSystem hdfs = outPath.getFileSystem(conf);
        if (hdfs.exists(outPath)){
          hdfs.delete(outPath, true);
        };

        Job exampleJob = new Job(conf,"example");
        exampleJob.setJarByClass(MpInputExp.class);
        exampleJob.setMapperClass(Map1.class);
        exampleJob.setMapperClass(Map2.class);
        exampleJob.setMapperClass(Map3.class);
        exampleJob.setReducerClass(Red.class);
        exampleJob.setOutputKeyClass(Text.class);
        exampleJob.setOutputValueClass(IntWritable.class);

        MultipleInputs.addInputPath(exampleJob, inPath1, TextInputFormat.class, Map1.class);
        MultipleInputs.addInputPath(exampleJob, inPath2, TextInputFormat.class, Map2.class);
        MultipleInputs.addInputPath(exampleJob, inPath3, TextInputFormat.class, Map2.class);
        
        FileOutputFormat.setOutputPath(exampleJob, outPath);
        System.exit(exampleJob.waitForCompletion(true) ? 0:1);
  }
}
張貼者： 於 沒有留言:
標籤： , , , ,

2013年10月1日 星期二

[JAVA] Big Data(2) 終於寫出小兒科等級的mapreduce!!!

先來個最近當紅的半澤直樹



在Big Data的時代，很多工具都可以做mapreduce的計算，甚至不用知道mapreduce的原理
像是Hive, Pig
又或者是更有彈性的R的rmr2套件

說穿了以上的解決方案底層的邏輯都還是把較簡單的程式碼轉換成java code再執行
因此，為了炫耀或是其他更重要目的，用java寫mapreduce還是有其必要性的(肯定貌)

用java寫mapreduce的架構主要有三個部分：
1.Map class
2.Reduce class
3.主程式

在建立好Map和Reduce兩個class以後
再於主程式內用job方法呼叫
另外job方法也用來設定運行時的指令和資料輸入

說了一大堆，實際開始寫code才發現太複雜的mapreduce我也hadle不了啦(逃~~)
只好從小怪開始打起
wordcount的範例在網路上實在太多了有點蘚
所以就來做個平均數計意思意思一下

首先data長這樣，已經丟到hadoop上了是用逗點分隔的
我希望計算Tom和Mary等人三科的平均分數

$hadoop fs -cat chinese.txt
Tom,90
Mary,95
Lisa,94
Luke,87
Marx,76
Teresa,89

$hadoop fs -cat math.txt:
Tom,67
Mary,68
Lisa,75
Luke,82
Marx,66
Teresa,74

$hadoop fs -cat science.txt:
Tom,56
Mary,75
Lisa,85
Luke,96
Marx,74
Teresa,69


java code在這：

import java.io.IOException;
import java.util.StringTokenizer;
import java.util.Iterator;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.filecache.DistributedCache;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.Mapper.Context;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.hadoop.util.StringUtils;

public class Avg{

 //map步驟
 public static class Map extends Mapper {
 
  //撰寫map方法
  public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
      String line = value.toString();//數據先轉成字串(String)
      StringTokenizer script = new StringTokenizer(line, "\n");//分割數據
      while (script.hasMoreElements()) {
       StringTokenizer scriptLine = new StringTokenizer(script.nextToken());
           //將Key和Value用逗號(",")分開
              Text Name = new Text(scriptLine.nextToken(","));
              int Score = Integer.parseInt(scriptLine.nextToken(","));
              context.write(Name, new IntWritable(Score));
      }
  }
 }
 
 //reduce 步驟
 public static class Reduce extends Reducer {
         //撰寫reduce方法
   public void reduce(Text key, Iterable value, Context context) throws IOException, InterruptedException{
          int numerator = 0;//初始化分子
          int denominator = 0;//初始化分母
          for (IntWritable v : value) {
                  numerator += v.get();//分子累加
                  denominator ++;//分母每次+1
          }
          int avg = numerator/denominator;//相除
             context.write(key,new  IntWritable(avg));
         }
 }
 
 //主程式
 public static void main(String[] args) throws Exception {
 
   Configuration conf = new Configuration();
   String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
         Path dst_path = new Path (otherArgs[1]);
         FileSystem hdfs = dst_path.getFileSystem(conf);
 
         //檢查OUTPUT的路徑是否存在，有的話就宰了他!
         if (hdfs.exists(dst_path)){
          hdfs.delete(dst_path, true);
         };
 
         Job job = new Job(conf, "Avg");
            job.setJarByClass(Avg.class);
            job.setMapperClass(Map.class);
            job.setCombinerClass(Reduce.class);
            job.setReducerClass(Reduce.class);
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(IntWritable.class);
            FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
            FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));

            System.exit(job.waitForCompletion(true) ? 0 : 1);
         }
 }

寫完之後還沒結束
得先用commend line把code做成一個job

mkdir test_classes javac -classpath /{hadoop_home}/hadoop-core-0.20.2-cdh3u1.jar
:/{hadoop_home}/commons-cli-1.2.jar : -d test_classes Avg.java
jar -cvf ~/Avg.jar -C test_classes/ .

用javac編譯的時候記得要用-classpath指令把參考的library指給他(很重要)
之後再把它打包成.jar的檔案
之後就可以用hadoop jar指令執行了!
$hadoop jar ~/Avg.jar org.test.Avg {input} {output}

執行完來看看結果
$hadoop fs -cat ~/part-m-0000

Luke    88
Marx    72
Mary    79
Teresa  77
Tom     71



------ 後記：

其實這樣的東西不是很夠用 Ｔ＿Ｔ 稍微進階版的Multiple input可以參考另一篇

張貼者： 於 6 則留言:
標籤： , ,

2013年8月29日 星期四

[R] Big Data(1) rmr2套件 多重input

rmr2是一個可以使用R語言在Hadoop上做map-reduce的套件。

尚不知Hadoop和mapreduce為何物的人請參考這篇文章

以往做過的東西和網路上看得到的demo幾乎都是單一dataset利用鍵值(key)打散然後做平行運算。然而最近在處理的問題必須要把不同來源和結構(schema)的資料送到同一個node裡做運算，想說網路上找demo code卻遍尋不著，整個讓我非常頭大。

就在我試了老半天之後，竟然就成功了!!!於是就來寫一篇網誌告訴大家遇到好多個dataset的時候該怎麼辦~~~

用來demo的三組資料A、B、C
我們要做這件事：

資料長這樣(我隨便亂生的)。
> head(A)
  key         A1         A2         A3
1   1 -1.0348640  1.8177439  0.1352576
2   3 -1.3735558 -0.2437948 -0.4509937
3   2 -0.2034888  1.0297576  0.6305115
4   1  0.2270242  0.9087429 -0.4122123
5   3 -0.3290382 -0.4840644 -0.3688641
6   3 -0.1822808 -1.1303439 -0.4175791
> head(B)
  key           B1         B2          B3          B4
1   3  0.363861211  1.5158812 -1.20591630  0.08659873
2   3 -0.001122564  0.1037150  0.25809288  1.60135858
3   2  1.465006010 -0.4572000 -1.89767865 -0.60817508
4   2  1.964748118  0.8320135  0.50937176  0.77755846
5   1 -0.390667063  0.8493213  0.09075889  0.36958850
6   1 -0.768992997  0.7308232 -0.82277576  0.33674132
> head(C)
  key         C1         C2
1   1  0.4192436  0.1554905
2   2 -0.2125050 -1.4696090
3   1  0.3410178  0.3290381
4   2 -2.2716488  1.3529220
5   2 -0.2565037 -0.1575783
6   2 -0.4259541  0.1968482
接下來用函數split，把資料打散成一份一份的存成list格式。 
> A_wk<- split(A,A$key)
> head(A_wk)
$`1`
   key         A1         A2          A3
1    1 -1.0348640  1.8177439  0.13525756
4    1  0.2270242  0.9087429 -0.41221232
13   1 -0.3573558 -1.2751938 -0.03458945
14   1 -0.7453312 -1.1503369  1.08577621
16   1  0.8465141  0.2366092  0.91902192
19   1  1.4896751  0.3213586 -0.98961302
23   1 -0.3143099  2.0754432 -1.29391057

$`2`
   key          A1          A2          A3
3    2 -0.20348880  1.02975761  0.63051151
7    2  1.53614888  1.96886642  0.04558348
9    2 -0.07431669 -0.03700706 -1.47200277
10   2  0.66355253  0.07667024 -1.22673427
17   2  1.13416422 -1.69200417 -0.13861365
18   2 -0.99696590  0.24465904  0.54821302
24   2 -0.44196754 -0.28170710 -0.73912548
27   2 -1.34823336 -0.06120274  2.12261003
30   2  1.83861168 -0.02982669  0.14312250

$`3`
   key         A1         A2         A3
2    3 -1.3735558 -0.2437948 -0.4509937
5    3 -0.3290382 -0.4840644 -0.3688641
6    3 -0.1822808 -1.1303439 -0.4175791
8    3  1.2317304 -0.7572487 -0.4401060
11   3 -0.4940248 -0.1259619 -1.1145702
12   3 -1.4488153 -0.9855823 -0.7537385
15   3 -0.6147528  0.6804414 -0.7799006
20   3  0.5340705 -0.2427455 -1.5272875
21   3 -1.2019567 -0.1434495 -0.3046498
22   3  0.1311908 -0.4900816  0.8861471
25   3 -1.1544569 -0.1732862 -2.2312314
26   3 -1.3264688 -0.6784207  1.5171326
28   3  1.1866616 -1.9195358  0.3591871
29   3  0.7476575 -0.3390230 -1.6448516

> B_wk<- split(B,B$key)
> head(B_wk)
$`1`
   key          B1         B2          B3            B4
5    1 -0.39066706  0.8493213  0.09075889  0.3695884990
6    1 -0.76899300  0.7308232 -0.82277576  0.3367413230
8    1 -1.05567857 -0.5663445  0.32075285  1.0807403069
9    1 -0.07174419 -0.6553943 -0.30134811 -1.2155568454
13   1 -0.39760179  0.5973388 -0.43153826 -0.0003626449
14   1  0.38234556 -0.4762401  0.90686094 -3.8579677970
15   1 -0.73177601 -0.6438049 -1.52620752  0.0814186088
20   1  0.07927141 -1.7918052 -1.27799659  0.1533002628

$`2`
   key          B1         B2          B3         B4
3    2  1.46500601 -0.4572000 -1.89767865 -0.6081751
4    2  1.96474812  0.8320135  0.50937176  0.7775585
7    2  0.35856113  0.1048170 -0.76270331 -0.5114040
10   2  1.11808088 -0.5707235 -0.16225111 -1.0749321
12   2 -0.32032713 -0.7167343 -0.03320639  0.2495948
18   2  1.36768551  1.1874533  2.13816520  1.0105115
19   2  0.02825361  0.8781400 -1.44303311  1.2391620

$`3`
   key           B1         B2         B3          B4
1    3  0.363861211  1.5158812 -1.2059163  0.08659873
2    3 -0.001122564  0.1037150  0.2580929  1.60135858
11   3  0.180619368 -0.8830636  0.7562675  1.09992035
16   3  2.644110325 -1.8546195  2.4887309 -0.03694847
17   3  1.131906794  0.9559589 -1.9111856  1.16240718

> C_wk<- split(C,C$key)
> head(C_wk)
$`1`
   key          C1         C2
1    1  0.41924361  0.1554905
3    1  0.34101785  0.3290381
14   1 -1.27793718 -0.2728525
18   1 -0.72084207  0.3307406
20   1  0.05193866 -0.4465938
27   1  1.03553670  1.7562845
29   1  2.35281628 -1.6114928
37   1  0.02295450 -0.9392724

$`2`
   key         C1         C2
2    2 -0.2125050 -1.4696090
4    2 -2.2716488  1.3529220
5    2 -0.2565037 -0.1575783
6    2 -0.4259541  0.1968482
7    2 -1.1866331 -0.3687882
13   2  0.7102612 -1.1221971
15   2  0.1592025  0.2775758
17   2  0.3842816 -0.6379072
22   2  0.1516206 -1.0723437
24   2 -0.4070279 -0.2998299
25   2  0.7779358 -0.2862851
28   2 -0.6860207 -0.9731296
30   2 -0.1319815  1.4057571
31   2  0.2845947 -0.3443439
33   2  0.7927496  0.9126125
34   2  0.4733910 -1.6850074

$`3`
   key          C1           C2
8    3 -0.13961645  0.204808027
9    3  1.49999162 -0.435241747
10   3 -0.94473626  1.768523536
11   3 -1.68380914 -0.172574070
12   3  1.12455947  0.611700128
16   3  0.19875147 -1.356228028
19   3  0.78590745 -0.796733981
21   3 -1.02839096  0.254459297
23   3 -1.00747198  0.593401435
26   3  0.81168025 -1.447740656
32   3  0.74883355 -0.170628912
35   3 -0.82805688 -1.792942718
36   3 -0.68229982  0.001420327
38   3  1.11185787  1.452839232
39   3  0.01296254 -0.676003236
40   3 -0.70681824 -0.027688693

打散之後我們用keyval函數把list的物件名稱(用names函數提取)當作key值(其實就是把打散的資料和key值兜在一起)
> Total<- keyval(c(names(A_wk),names(B_wk),names(C_wk)),c(A_wk,B_wk,C_wk))

之後放進mapreduce裡面! 因為已經用keyval把資料賦予key值，所以這裡我就不需要map了，直接在reduce裡面把資料還原成我要的東西。 先把資料用欄位名稱打散(在這裡不同dataset欄位名稱是不同的，如果有一樣的話請加上其他條件)，打散之後再把它們合併成data.frame 

> Demo<- mapreduce(
+     input = to.dfs(Total),
+     reduce = function(k,v){
+         test<- split(v,sapply(v,function(x) paste(colnames(x),collapse = "")))
+         test<- lapply(test,function(x) Reduce(rbind,x))
+         keyval(k,list(test[[1]],test[[2]],test[[3]]))
+     }
+ )
然後來看結果 
> from.dfs(Demo)
$key
[1] "1" "1" "1" "2" "2" "2" "3" "3" "3"

$val
$val[[1]]
   key         A1         A2          A3
1    1 -1.0348640  1.8177439  0.13525756
4    1  0.2270242  0.9087429 -0.41221232
13   1 -0.3573558 -1.2751938 -0.03458945
14   1 -0.7453312 -1.1503369  1.08577621
16   1  0.8465141  0.2366092  0.91902192
19   1  1.4896751  0.3213586 -0.98961302
23   1 -0.3143099  2.0754432 -1.29391057

$val[[2]]
   key          B1         B2          B3            B4
5    1 -0.39066706  0.8493213  0.09075889  0.3695884990
6    1 -0.76899300  0.7308232 -0.82277576  0.3367413230
8    1 -1.05567857 -0.5663445  0.32075285  1.0807403069
9    1 -0.07174419 -0.6553943 -0.30134811 -1.2155568454
13   1 -0.39760179  0.5973388 -0.43153826 -0.0003626449
14   1  0.38234556 -0.4762401  0.90686094 -3.8579677970
15   1 -0.73177601 -0.6438049 -1.52620752  0.0814186088
20   1  0.07927141 -1.7918052 -1.27799659  0.1533002628

$val[[3]]
   key          C1         C2
1    1  0.41924361  0.1554905
3    1  0.34101785  0.3290381
14   1 -1.27793718 -0.2728525
18   1 -0.72084207  0.3307406
20   1  0.05193866 -0.4465938
27   1  1.03553670  1.7562845
29   1  2.35281628 -1.6114928
37   1  0.02295450 -0.9392724

$val[[4]]
   key          A1          A2          A3
3    2 -0.20348880  1.02975761  0.63051151
7    2  1.53614888  1.96886642  0.04558348
9    2 -0.07431669 -0.03700706 -1.47200277
10   2  0.66355253  0.07667024 -1.22673427
17   2  1.13416422 -1.69200417 -0.13861365
18   2 -0.99696590  0.24465904  0.54821302
24   2 -0.44196754 -0.28170710 -0.73912548
27   2 -1.34823336 -0.06120274  2.12261003
30   2  1.83861168 -0.02982669  0.14312250

$val[[5]]
   key          B1         B2          B3         B4
3    2  1.46500601 -0.4572000 -1.89767865 -0.6081751
4    2  1.96474812  0.8320135  0.50937176  0.7775585
7    2  0.35856113  0.1048170 -0.76270331 -0.5114040
10   2  1.11808088 -0.5707235 -0.16225111 -1.0749321
12   2 -0.32032713 -0.7167343 -0.03320639  0.2495948
18   2  1.36768551  1.1874533  2.13816520  1.0105115
19   2  0.02825361  0.8781400 -1.44303311  1.2391620

$val[[6]]
   key         C1         C2
2    2 -0.2125050 -1.4696090
4    2 -2.2716488  1.3529220
5    2 -0.2565037 -0.1575783
6    2 -0.4259541  0.1968482
7    2 -1.1866331 -0.3687882
13   2  0.7102612 -1.1221971
15   2  0.1592025  0.2775758
17   2  0.3842816 -0.6379072
22   2  0.1516206 -1.0723437
24   2 -0.4070279 -0.2998299
25   2  0.7779358 -0.2862851
28   2 -0.6860207 -0.9731296
30   2 -0.1319815  1.4057571
31   2  0.2845947 -0.3443439
33   2  0.7927496  0.9126125
34   2  0.4733910 -1.6850074

$val[[7]]
   key         A1         A2         A3
2    3 -1.3735558 -0.2437948 -0.4509937
5    3 -0.3290382 -0.4840644 -0.3688641
6    3 -0.1822808 -1.1303439 -0.4175791
8    3  1.2317304 -0.7572487 -0.4401060
11   3 -0.4940248 -0.1259619 -1.1145702
12   3 -1.4488153 -0.9855823 -0.7537385
15   3 -0.6147528  0.6804414 -0.7799006
20   3  0.5340705 -0.2427455 -1.5272875
21   3 -1.2019567 -0.1434495 -0.3046498
22   3  0.1311908 -0.4900816  0.8861471
25   3 -1.1544569 -0.1732862 -2.2312314
26   3 -1.3264688 -0.6784207  1.5171326
28   3  1.1866616 -1.9195358  0.3591871
29   3  0.7476575 -0.3390230 -1.6448516

$val[[8]]
   key           B1         B2         B3          B4
1    3  0.363861211  1.5158812 -1.2059163  0.08659873
2    3 -0.001122564  0.1037150  0.2580929  1.60135858
11   3  0.180619368 -0.8830636  0.7562675  1.09992035
16   3  2.644110325 -1.8546195  2.4887309 -0.03694847
17   3  1.131906794  0.9559589 -1.9111856  1.16240718

$val[[9]]
   key          C1           C2
8    3 -0.13961645  0.204808027
9    3  1.49999162 -0.435241747
10   3 -0.94473626  1.768523536
11   3 -1.68380914 -0.172574070
12   3  1.12455947  0.611700128
16   3  0.19875147 -1.356228028
19   3  0.78590745 -0.796733981
21   3 -1.02839096  0.254459297
23   3 -1.00747198  0.593401435
26   3  0.81168025 -1.447740656
32   3  0.74883355 -0.170628912
35   3 -0.82805688 -1.792942718
36   3 -0.68229982  0.001420327
38   3  1.11185787  1.452839232
39   3  0.01296254 -0.676003236
40   3 -0.70681824 -0.027688693
OK，和我想要的一樣!
前面三組dataset屬於key "1"
中間三組dataset屬於key "2"
後面三組dataset屬於key "3"
大功告成!!!
在reduce裡面可以加上更多的東西，像是資料表之間彼此join之類的運算。
張貼者： 於 沒有留言:
標籤： , , , , , ,
位置： 亞洲
訂閱： 文章 (Atom)