This example will use C gigas larvae data.
#Setting Variables
R1="filtered_Y38_control_GTGAAA_L005_R1"
R2="filtered_Y38_control_GTGAAA_L005_R2"
!gunzip /Volumes/web/trilobite/Crassostrea_gigas_HTSdata/{R1}.fastq.gz
!gunzip /Volumes/web/trilobite/Crassostrea_gigas_HTSdata/{R2}.fastq.gz
!/Volumes/Bay3/Software/FastQC/fastqc \
/Volumes/web/trilobite/Crassostrea_gigas_HTSdata/{R1}.fastq \
-o /Volumes/web/cnidarian/
Started analysis of filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 5% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 10% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 15% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 20% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 25% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 30% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 35% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 40% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 45% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 50% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 55% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 60% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 65% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 70% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 75% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 80% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 85% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 90% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 95% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Approx 100% complete for filtered_Y38_control_GTGAAA_L005_R1.fastq Analysis complete for filtered_Y38_control_GTGAAA_L005_R1.fastq
!/Volumes/Bay3/Software/FastQC/fastqc \
/Volumes/web/trilobite/Crassostrea_gigas_HTSdata/{R2}.fastq \
-q \
-o /Volumes/web/cnidarian/
!grep -A 5 "Overrepresented" /Volumes/web/cnidarian/{R1}_fastqc/fastqc_data.txt
>>Overrepresented sequences fail #Sequence Count Percentage Possible Source GATCGGAAGAGCACACGTCTGAACTCCAGTCACGTG 77030 2.233517550930463 TruSeq Adapter, Index 1 (97% over 35bp) >>END_MODULE >>Kmer Content fail #Sequence Count Obs/Exp Overall Obs/Exp Max Max Obs/Exp Position
!grep -A 6 "Overrepresented" /Volumes/web/cnidarian/{R2}_fastqc/fastqc_data.txt
>>Overrepresented sequences warn #Sequence Count Percentage Possible Source GATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTAGAT 21905 0.6351447741546384 Illumina Single End PCR Primer 1 (100% over 36bp) NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN 9464 0.27441269767630666 No Hit GATCGGAAGAGCGTCGGGTAGGGAAAGAGGGTAGAT 7464 0.2164218486322858 Illumina Single End PCR Primer 1 (96% over 29bp) GATCGGAAGAGCGTCGGGTAGGGAAAGAGTGTAGAT 6997 0.20288098538050695 Illumina Single End PCR Primer 1 (97% over 36bp) >>END_MODULE
!sed -n "/>>Overrepresented/,/>>END_MODULE/p" /Volumes/web/cnidarian/{R2}_fastqc/fastqc_data.txt
>>Overrepresented sequences warn #Sequence Count Percentage Possible Source GATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTAGAT 21905 0.6351447741546384 Illumina Single End PCR Primer 1 (100% over 36bp) NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN 9464 0.27441269767630666 No Hit GATCGGAAGAGCGTCGGGTAGGGAAAGAGGGTAGAT 7464 0.2164218486322858 Illumina Single End PCR Primer 1 (96% over 29bp) GATCGGAAGAGCGTCGGGTAGGGAAAGAGTGTAGAT 6997 0.20288098538050695 Illumina Single End PCR Primer 1 (97% over 36bp) >>END_MODULE
!cat /Volumes/web/cnidarian/YE_overrepresent.txt
>TruSeq Adapter, Index 1 GATCGGAAGAGCACACGTCTGAACTCCAGTCACGTG >Illumina Single End PCR Primer 1 GATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTAGAT
!/Volumes/Bay3/Software/ea-utils.1.1.2-537/fastq-mcf \
-o /Volumes/web/cnidarian/{R1}_tr.fastq \
-o /Volumes/web/cnidarian/{R2}_tr.fastq \
/Volumes/web/cnidarian/YE_overrepresent.txt \
/Volumes/web/trilobite/Crassostrea_gigas_HTSdata/{R1}.fastq \
/Volumes/web/trilobite/Crassostrea_gigas_HTSdata/{R2}.fastq
Scale used: 2.2 Phred: 33 Within-read Skew: Position 4 from the start of reads is skewed! Within-read Skew: Position 5 from the start of reads is skewed! Within-read Skew: Position 6 from the start of reads is skewed! Within-read Skew: Position 7 from the start of reads is skewed! Within-read Skew: Position 8 from the start of reads is skewed! Within-read Skew: Position 9 from the start of reads is skewed! Within-read Skew: Position 10 from the start of reads is skewed! Within-read Skew: Position 12 from the start of reads is skewed! Within-read Skew: Position 14 from the start of reads is skewed! Within-read Skew: Position 16 from the start of reads is skewed! Within-read Skew: Position 17 from the start of reads is skewed! Within-read Skew: Position 4 from the end of reads is skewed! Within-read Skew: Position 6 from the end of reads is skewed! Within-read Skew: Position 7 from the end of reads is skewed! Within-read Skew: Position 8 from the end of reads is skewed! Within-read Skew: Position 11 from the end of reads is skewed! Within-read Skew: Position 13 from the end of reads is skewed! Within-read Skew: Position 14 from the end of reads is skewed! Within-read Skew: Position 15 from the end of reads is skewed! Within-read Skew: Position 16 from the end of reads is skewed! Warning: Too much skewing found (36), disabling skew clipping Threshold used: 751 out of 300000 Adapter TruSeq Adapter, Index 1 (GATCGGAAGAGCACACGTCTGAACTCCAGTCACGTG): counted 8753 at the 'start' of '/Volumes/web/trilobite/Crassostrea_gigas_HTSdata/filtered_Y38_control_GTGAAA_L005_R1.fastq', clip set to 4 Files: 2 Total reads: 3448820 Too short after clip: 186681 Clipped 'start' reads (/Volumes/web/trilobite/Crassostrea_gigas_HTSdata/filtered_Y38_control_GTGAAA_L005_R1.fastq): Count 993, Mean: 4.51, Sd: 1.78 Trimmed 68594 reads (/Volumes/web/trilobite/Crassostrea_gigas_HTSdata/filtered_Y38_control_GTGAAA_L005_R1.fastq) by an average of 7.77 bases on quality < 7 Trimmed 140063 reads (/Volumes/web/trilobite/Crassostrea_gigas_HTSdata/filtered_Y38_control_GTGAAA_L005_R2.fastq) by an average of 21.59 bases on quality < 7
#confirming
!/Volumes/Bay3/Software/FastQC/fastqc \
/Volumes/web/cnidarian/{R1}_tr.fastq \
-q \
-o /Volumes/web/cnidarian/
!grep -A 5 "Overrepresented" /Volumes/web/cnidarian/{R1}_tr_fastqc/fastqc_data.txt
>>Overrepresented sequences pass >>END_MODULE >>Kmer Content fail #Sequence Count Obs/Exp Overall Obs/Exp Max Max Obs/Exp Position CCCCC 50 509094.34 1651751.6 32 CTCCC 130 10524.399 324060.84 24
#confirming
!/Volumes/Bay3/Software/FastQC/fastqc \
/Volumes/web/cnidarian/{R2}_tr.fastq \
-q \
-o /Volumes/web/cnidarian/
!grep -A 5 "Overrepresented" /Volumes/web/cnidarian/{R2}_tr_fastqc/fastqc_data.txt
>>Overrepresented sequences pass >>END_MODULE >>Kmer Content fail #Sequence Count Obs/Exp Overall Obs/Exp Max Max Obs/Exp Position GGGGG 60 331473.72 4440570.5 27 GAGGG 235 11672.065 199613.89 27
! /Volumes/Bay3/Software/BSMAP/bsmap-2.74/bsmap \
-a /Volumes/web/cnidarian/{R1}_tr.fastq \
-b /Volumes/web/cnidarian/{R2}_tr.fastq \
-d /Volumes/web/trilobite/Crassostrea_gigas_ensembl_tracks/Crassostrea_gigas.GCA_000297895.1.22.dna_sm.genome.fa.gz \
-o /Volumes/web/cnidarian/BiLar_YE_control_t.sam \
-p 4
BSMAP v2.74 Start at: Tue Jun 3 15:33:16 2014 Input reference file: /Volumes/web/cnidarian/oyster.v9.fa (format: FASTA) Load in 11969 db seqs, total size 558601156 bp. 30 secs passed total_kmers: 43046721 Create seed table. 119 secs passed max number of mismatches: read_length * 8% max gap size: 0 kmer cut-off ratio: 5e-07 max multi-hits: 100 max Ns: 5 seed size: 16 index interval: 4 quality cutoff: 0 base quality char: '!' min fragment size:28 max fragemt size:500 start from read #1 end at read #4294967295 additional alignment: T in reads => C in reference mapping strand (read_1): ++,-+ mapping strand (read_2): +-,-- Pair-end alignment(4 threads) Input read file #1: /Volumes/web/cnidarian/filtered_Y54_Mix_GTGGCC_L005_R1_tr.fastq (format: FASTQ) Input read file #2: /Volumes/web/cnidarian/filtered_Y54_Mix_GTGGCC_L005_R2_tr.fastq (format: FASTQ) Output file: /Volumes/web/cnidarian/BiLar_YE_mix_t.sam (format: SAM) Thread #1: 50000 read pairs finished. 290 secs passed Thread #0: 150000 read pairs finished. 294 secs passed Thread #2: 100000 read pairs finished. 294 secs passed Thread #3: 200000 read pairs finished. 297 secs passed Thread #1: 250000 read pairs finished. 446 secs passed Thread #0: 300000 read pairs finished. 447 secs passed Thread #2: 350000 read pairs finished. 449 secs passed Thread #3: 400000 read pairs finished. 450 secs passed Thread #1: 450000 read pairs finished. 589 secs passed Thread #0: 500000 read pairs finished. 591 secs passed Thread #2: 550000 read pairs finished. 593 secs passed Thread #3: 600000 read pairs finished. 597 secs passed Thread #1: 650000 read pairs finished. 724 secs passed Thread #0: 700000 read pairs finished. 724 secs passed Thread #2: 750000 read pairs finished. 728 secs passed Thread #3: 800000 read pairs finished. 732 secs passed Thread #0: 900000 read pairs finished. 848 secs passed Thread #1: 850000 read pairs finished. 848 secs passed Thread #2: 950000 read pairs finished. 850 secs passed Thread #3: 1000000 read pairs finished. 856 secs passed Thread #0: 1050000 read pairs finished. 979 secs passed Thread #1: 1100000 read pairs finished. 981 secs passed Thread #2: 1150000 read pairs finished. 982 secs passed Thread #3: 1200000 read pairs finished. 989 secs passed Thread #0: 1250000 read pairs finished. 1104 secs passed Thread #1: 1300000 read pairs finished. 1105 secs passed Thread #2: 1350000 read pairs finished. 1107 secs passed Thread #3: 1400000 read pairs finished. 1114 secs passed Thread #0: 1450000 read pairs finished. 1227 secs passed Thread #1: 1500000 read pairs finished. 1230 secs passed Thread #2: 1550000 read pairs finished. 1231 secs passed Thread #3: 1600000 read pairs finished. 1239 secs passed Thread #0: 1650000 read pairs finished. 1377 secs passed Thread #1: 1700000 read pairs finished. 1382 secs passed Thread #2: 1750000 read pairs finished. 1411 secs passed Thread #3: 1800000 read pairs finished. 1411 secs passed Thread #0: 1850000 read pairs finished. 1460 secs passed Thread #1: 1900000 read pairs finished. 1602 secs passed Thread #2: 1950000 read pairs finished. 1604 secs passed Thread #3: 2000000 read pairs finished. 1608 secs passed Thread #0: 2050000 read pairs finished. 1651 secs passed Thread #1: 2100000 read pairs finished. 1797 secs passed Thread #2: 2150000 read pairs finished. 1798 secs passed Thread #3: 2200000 read pairs finished. 1802 secs passed Thread #0: 2250000 read pairs finished. 1848 secs passed Thread #1: 2300000 read pairs finished. 1992 secs passed Thread #2: 2350000 read pairs finished. 1993 secs passed Thread #3: 2400000 read pairs finished. 1996 secs passed Thread #0: 2450000 read pairs finished. 2040 secs passed Thread #1: 2500000 read pairs finished. 2143 secs passed Thread #2: 2550000 read pairs finished. 2144 secs passed Thread #3: 2600000 read pairs finished. 2145 secs passed Thread #0: 2650000 read pairs finished. 2173 secs passed Thread #1: 2700000 read pairs finished. 2268 secs passed Thread #2: 2750000 read pairs finished. 2270 secs passed Thread #3: 2800000 read pairs finished. 2272 secs passed Thread #0: 2850000 read pairs finished. 2300 secs passed Thread #1: 2900000 read pairs finished. 2396 secs passed Thread #2: 2950000 read pairs finished. 2400 secs passed Thread #3: 3000000 read pairs finished. 2402 secs passed Thread #0: 3050000 read pairs finished. 2430 secs passed Thread #1: 3100000 read pairs finished. 2526 secs passed Thread #2: 3150000 read pairs finished. 2529 secs passed Thread #3: 3200000 read pairs finished. 2532 secs passed Thread #0: 3250000 read pairs finished. 2558 secs passed Thread #1: 3300000 read pairs finished. 2651 secs passed Thread #2: 3350000 read pairs finished. 2655 secs passed Thread #3: 3400000 read pairs finished. 2658 secs passed Thread #0: 3450000 read pairs finished. 2683 secs passed Thread #1: 3500000 read pairs finished. 2784 secs passed Thread #2: 3550000 read pairs finished. 2788 secs passed Thread #3: 3600000 read pairs finished. 2793 secs passed Thread #0: 3650000 read pairs finished. 2822 secs passed Thread #1: 3700000 read pairs finished. 2932 secs passed Thread #2: 3750000 read pairs finished. 2938 secs passed Thread #3: 3800000 read pairs finished. 2945 secs passed Thread #0: 3850000 read pairs finished. 2969 secs passed Thread #1: 3900000 read pairs finished. 3070 secs passed Thread #2: 3950000 read pairs finished. 3073 secs passed Thread #3: 4000000 read pairs finished. 3082 secs passed Thread #0: 4050000 read pairs finished. 3105 secs passed Thread #1: 4100000 read pairs finished. 3204 secs passed Thread #2: 4150000 read pairs finished. 3207 secs passed Thread #3: 4200000 read pairs finished. 3215 secs passed Thread #0: 4250000 read pairs finished. 3236 secs passed Thread #1: 4300000 read pairs finished. 3333 secs passed Thread #2: 4350000 read pairs finished. 3336 secs passed Thread #3: 4400000 read pairs finished. 3344 secs passed Thread #0: 4450000 read pairs finished. 3364 secs passed Thread #1: 4500000 read pairs finished. 3462 secs passed Thread #2: 4550000 read pairs finished. 3464 secs passed Thread #3: 4600000 read pairs finished. 3474 secs passed Thread #0: 4650000 read pairs finished. 3494 secs passed Thread #1: 4700000 read pairs finished. 3600 secs passed Thread #2: 4750000 read pairs finished. 3603 secs passed Thread #3: 4800000 read pairs finished. 3615 secs passed Thread #0: 4850000 read pairs finished. 3637 secs passed Thread #1: 4900000 read pairs finished. 3734 secs passed Thread #2: 4950000 read pairs finished. 3736 secs passed Thread #3: 5000000 read pairs finished. 3748 secs passed Thread #0: 5050000 read pairs finished. 3766 secs passed Thread #1: 5100000 read pairs finished. 3864 secs passed Thread #2: 5150000 read pairs finished. 3866 secs passed Thread #3: 5200000 read pairs finished. 3880 secs passed Thread #0: 5250000 read pairs finished. 3897 secs passed Thread #1: 5300000 read pairs finished. 3995 secs passed Thread #2: 5350000 read pairs finished. 3996 secs passed Thread #3: 5400000 read pairs finished. 4010 secs passed Thread #0: 5450000 read pairs finished. 4029 secs passed Thread #1: 5500000 read pairs finished. 4132 secs passed Thread #2: 5550000 read pairs finished. 4132 secs passed Thread #3: 5600000 read pairs finished. 4148 secs passed Thread #0: 5650000 read pairs finished. 4164 secs passed Thread #1: 5700000 read pairs finished. 4261 secs passed Thread #2: 5750000 read pairs finished. 4262 secs passed Thread #3: 5800000 read pairs finished. 4277 secs passed Thread #0: 5850000 read pairs finished. 4293 secs passed Thread #1: 5900000 read pairs finished. 4389 secs passed Thread #2: 5950000 read pairs finished. 4390 secs passed Thread #3: 6000000 read pairs finished. 4405 secs passed Thread #0: 6050000 read pairs finished. 4421 secs passed Thread #1: 6100000 read pairs finished. 4517 secs passed Thread #2: 6150000 read pairs finished. 4519 secs passed Thread #3: 6200000 read pairs finished. 4534 secs passed Thread #0: 6250000 read pairs finished. 4549 secs passed Thread #1: 6300000 read pairs finished. 4647 secs passed Thread #2: 6350000 read pairs finished. 4650 secs passed Thread #3: 6400000 read pairs finished. 4668 secs passed Thread #0: 6450000 read pairs finished. 4681 secs passed Thread #1: 6500000 read pairs finished. 4774 secs passed Thread #2: 6550000 read pairs finished. 4778 secs passed Thread #3: 6600000 read pairs finished. 4797 secs passed Thread #0: 6650000 read pairs finished. 4815 secs passed Thread #1: 6700000 read pairs finished. 4909 secs passed Thread #2: 6750000 read pairs finished. 4913 secs passed Thread #3: 6800000 read pairs finished. 4930 secs passed Thread #0: 6850000 read pairs finished. 4942 secs passed Thread #1: 6900000 read pairs finished. 5035 secs passed Thread #2: 6950000 read pairs finished. 5038 secs passed Thread #3: 7000000 read pairs finished. 5057 secs passed Thread #0: 7050000 read pairs finished. 5071 secs passed Thread #1: 7100000 read pairs finished. 5168 secs passed Thread #2: 7150000 read pairs finished. 5172 secs passed Thread #3: 7200000 read pairs finished. 5191 secs passed Thread #0: 7250000 read pairs finished. 5202 secs passed Thread #1: 7300000 read pairs finished. 5296 secs passed Thread #2: 7350000 read pairs finished. 5300 secs passed Thread #3: 7400000 read pairs finished. 5319 secs passed Thread #0: 7450000 read pairs finished. 5331 secs passed Thread #1: 7500000 read pairs finished. 5424 secs passed Thread #2: 7550000 read pairs finished. 5428 secs passed Thread #3: 7600000 read pairs finished. 5447 secs passed Thread #0: 7650000 read pairs finished. 5457 secs passed Thread #1: 7700000 read pairs finished. 5550 secs passed Thread #2: 7750000 read pairs finished. 5556 secs passed Thread #3: 7800000 read pairs finished. 5575 secs passed Thread #0: 7850000 read pairs finished. 5584 secs passed Thread #1: 7900000 read pairs finished. 5676 secs passed Thread #2: 7950000 read pairs finished. 5682 secs passed Thread #3: 8000000 read pairs finished. 5699 secs passed Thread #0: 8050000 read pairs finished. 5710 secs passed Thread #1: 8100000 read pairs finished. 5802 secs passed Thread #2: 8150000 read pairs finished. 5808 secs passed Thread #3: 8200000 read pairs finished. 5827 secs passed Thread #0: 8250000 read pairs finished. 5837 secs passed Thread #1: 8300000 read pairs finished. 5928 secs passed Thread #2: 8350000 read pairs finished. 5935 secs passed Thread #3: 8400000 read pairs finished. 5954 secs passed Thread #0: 8450000 read pairs finished. 5963 secs passed Thread #1: 8500000 read pairs finished. 6054 secs passed Thread #2: 8550000 read pairs finished. 6061 secs passed Thread #3: 8600000 read pairs finished. 6080 secs passed Thread #0: 8650000 read pairs finished. 6089 secs passed Thread #1: 8700000 read pairs finished. 6185 secs passed Thread #2: 8750000 read pairs finished. 6192 secs passed Thread #3: 8800000 read pairs finished. 6213 secs passed Thread #0: 8850000 read pairs finished. 6222 secs passed Thread #1: 8900000 read pairs finished. 6316 secs passed Thread #2: 8950000 read pairs finished. 6323 secs passed Thread #3: 9000000 read pairs finished. 6342 secs passed Thread #0: 9050000 read pairs finished. 6350 secs passed Thread #1: 9100000 read pairs finished. 6454 secs passed Thread #2: 9150000 read pairs finished. 6461 secs passed Thread #3: 9200000 read pairs finished. 6485 secs passed Thread #0: 9250000 read pairs finished. 6493 secs passed Thread #1: 9300000 read pairs finished. 6605 secs passed Thread #2: 9350000 read pairs finished. 6611 secs passed Thread #3: 9400000 read pairs finished. 6633 secs passed Thread #0: 9450000 read pairs finished. 6639 secs passed Thread #1: 9500000 read pairs finished. 6735 secs passed Thread #2: 9550000 read pairs finished. 6741 secs passed Thread #3: 9600000 read pairs finished. 6762 secs passed Thread #0: 9650000 read pairs finished. 6768 secs passed Thread #1: 9700000 read pairs finished. 6861 secs passed Thread #2: 9750000 read pairs finished. 6866 secs passed Thread #3: 9800000 read pairs finished. 6889 secs passed Thread #0: 9850000 read pairs finished. 6895 secs passed Thread #1: 9900000 read pairs finished. 6990 secs passed Thread #2: 9950000 read pairs finished. 6995 secs passed Thread #3: 10000000 read pairs finished. 7018 secs passed Thread #0: 10050000 read pairs finished. 7023 secs passed Thread #1: 10100000 read pairs finished. 7116 secs passed Thread #2: 10150000 read pairs finished. 7122 secs passed Thread #3: 10200000 read pairs finished. 7147 secs passed Thread #0: 10250000 read pairs finished. 7150 secs passed Thread #1: 10300000 read pairs finished. 7249 secs passed Thread #2: 10350000 read pairs finished. 7257 secs passed Thread #3: 10400000 read pairs finished. 7284 secs passed Thread #0: 10450000 read pairs finished. 7286 secs passed Thread #1: 10500000 read pairs finished. 7378 secs passed Thread #2: 10550000 read pairs finished. 7384 secs passed Thread #0: 10650000 read pairs finished. 7411 secs passed Thread #3: 10600000 read pairs finished. 7423 secs passed Thread #1: 10700000 read pairs finished. 7491 secs passed Thread #2: 10750000 read pairs finished. 7495 secs passed Thread #0: 10800000 read pairs finished. 7549 secs passed Thread #3: 10850000 read pairs finished. 7552 secs passed Thread #1: 10900000 read pairs finished. 7617 secs passed Thread #2: 10950000 read pairs finished. 7622 secs passed Thread #0: 11000000 read pairs finished. 7675 secs passed Thread #3: 11050000 read pairs finished. 7678 secs passed Thread #1: 11100000 read pairs finished. 7741 secs passed Thread #2: 11150000 read pairs finished. 7746 secs passed Thread #0: 11200000 read pairs finished. 7799 secs passed Thread #3: 11250000 read pairs finished. 7804 secs passed Thread #1: 11300000 read pairs finished. 7865 secs passed Thread #2: 11350000 read pairs finished. 7870 secs passed Thread #0: 11400000 read pairs finished. 7923 secs passed Thread #3: 11450000 read pairs finished. 7929 secs passed Thread #1: 11500000 read pairs finished. 7989 secs passed Thread #2: 11550000 read pairs finished. 7996 secs passed Thread #0: 11600000 read pairs finished. 8047 secs passed Thread #3: 11650000 read pairs finished. 8054 secs passed Thread #1: 11700000 read pairs finished. 8114 secs passed Thread #2: 11750000 read pairs finished. 8121 secs passed Thread #0: 11800000 read pairs finished. 8173 secs passed Thread #3: 11850000 read pairs finished. 8182 secs passed Thread #1: 11900000 read pairs finished. 8243 secs passed Thread #2: 11950000 read pairs finished. 8249 secs passed Thread #0: 12000000 read pairs finished. 8300 secs passed Thread #3: 12050000 read pairs finished. 8311 secs passed Thread #1: 12100000 read pairs finished. 8370 secs passed Thread #2: 12150000 read pairs finished. 8375 secs passed Thread #0: 12200000 read pairs finished. 8425 secs passed Thread #3: 12250000 read pairs finished. 8437 secs passed Thread #1: 12300000 read pairs finished. 8495 secs passed Thread #2: 12350000 read pairs finished. 8500 secs passed Thread #0: 12400000 read pairs finished. 8550 secs passed Thread #3: 12450000 read pairs finished. 8563 secs passed Thread #1: 12500000 read pairs finished. 8619 secs passed Thread #2: 12550000 read pairs finished. 8625 secs passed Thread #0: 12600000 read pairs finished. 8674 secs passed Thread #3: 12650000 read pairs finished. 8688 secs passed Thread #1: 12700000 read pairs finished. 8744 secs passed Thread #2: 12750000 read pairs finished. 8750 secs passed Thread #0: 12800000 read pairs finished. 8799 secs passed Thread #3: 12850000 read pairs finished. 8813 secs passed Thread #1: 12900000 read pairs finished. 8868 secs passed Thread #2: 12950000 read pairs finished. 8875 secs passed Thread #0: 13000000 read pairs finished. 8924 secs passed Thread #3: 13050000 read pairs finished. 8938 secs passed Thread #1: 13100000 read pairs finished. 8994 secs passed Thread #2: 13150000 read pairs finished. 9002 secs passed Thread #0: 13200000 read pairs finished. 9050 secs passed Thread #3: 13250000 read pairs finished. 9067 secs passed Thread #3: 13412322 read pairs finished. 9098 secs passed Thread #1: 13300000 read pairs finished. 9115 secs passed Thread #2: 13350000 read pairs finished. 9121 secs passed Thread #0: 13400000 read pairs finished. 9136 secs passed Total number of aligned reads: pairs: 7934451 (59%) single a: 2762270 (21%) single b: 2440630 (18%) Done. Finished at Tue Jun 3 18:05:32 2014 Total time consumed: 9136 secs
! /Users/Shared/Apps/bsmap-2.74/bsmap \
-a /Volumes/web/cnidarian/{R1}_tr.fastq \
-b /Volumes/web/cnidarian/{R2}_tr.fastq \
-d /Volumes/web/trilobite/Crassostrea_gigas_ensembl_tracks/Crassostrea_gigas.GCA_000297895.1.22.dna_sm.genome.fa.gz \
-o /Volumes/web/cnidarian/BiLar_YE_control_t.sam \
-p 8
BSMAP v2.74 Start at: Fri Jun 6 08:18:53 2014 Input reference file: /Volumes/web/trilobite/Crassostrea_gigas_ensembl_tracks/Crassostrea_gigas.GCA_000297895.1.22.dna_sm.genome.fa.gz (format: gzipped FASTA) Load in 7658 db seqs, total size 557717710 bp. 15 secs passed total_kmers: 43046721 Create seed table. 33 secs passed max number of mismatches: read_length * 8% max gap size: 0 kmer cut-off ratio: 5e-07 max multi-hits: 100 max Ns: 5 seed size: 16 index interval: 4 quality cutoff: 0 base quality char: '!' min fragment size:28 max fragemt size:500 start from read #1 end at read #4294967295 additional alignment: T in reads => C in reference mapping strand (read_1): ++,-+ mapping strand (read_2): +-,-- Pair-end alignment(8 threads) Input read file #1: /Volumes/web/cnidarian/filtered_Y38_control_GTGAAA_L005_R1_tr.fastq (format: FASTQ) Input read file #2: /Volumes/web/cnidarian/filtered_Y38_control_GTGAAA_L005_R2_tr.fastq (format: FASTQ) Output file: /Volumes/web/cnidarian/BiLar_YE_control_t.sam (format: SAM) Thread #1: 50000 read pairs finished. 80 secs passed Thread #0: 100000 read pairs finished. 83 secs passed Thread #3: 150000 read pairs finished. 84 secs passed Thread #4: 200000 read pairs finished. 85 secs passed Thread #5: 250000 read pairs finished. 87 secs passed Thread #7: 350000 read pairs finished. 88 secs passed Thread #2: 300000 read pairs finished. 89 secs passed Thread #6: 400000 read pairs finished. 90 secs passed Thread #1: 450000 read pairs finished. 131 secs passed Thread #0: 500000 read pairs finished. 134 secs passed Thread #3: 550000 read pairs finished. 136 secs passed Thread #4: 600000 read pairs finished. 137 secs passed Thread #5: 650000 read pairs finished. 138 secs passed Thread #7: 700000 read pairs finished. 141 secs passed Thread #6: 800000 read pairs finished. 147 secs passed Thread #2: 750000 read pairs finished. 148 secs passed Thread #1: 850000 read pairs finished. 181 secs passed Thread #0: 900000 read pairs finished. 183 secs passed Thread #3: 950000 read pairs finished. 185 secs passed Thread #4: 1000000 read pairs finished. 187 secs passed Thread #5: 1050000 read pairs finished. 188 secs passed Thread #7: 1100000 read pairs finished. 190 secs passed Thread #6: 1150000 read pairs finished. 199 secs passed Thread #2: 1200000 read pairs finished. 202 secs passed Thread #1: 1250000 read pairs finished. 233 secs passed Thread #0: 1300000 read pairs finished. 236 secs passed Thread #3: 1350000 read pairs finished. 238 secs passed Thread #4: 1400000 read pairs finished. 239 secs passed Thread #5: 1450000 read pairs finished. 240 secs passed Thread #7: 1500000 read pairs finished. 243 secs passed Thread #6: 1550000 read pairs finished. 252 secs passed Thread #2: 1600000 read pairs finished. 256 secs passed Thread #1: 1650000 read pairs finished. 285 secs passed Thread #0: 1700000 read pairs finished. 288 secs passed Thread #3: 1750000 read pairs finished. 291 secs passed Thread #4: 1800000 read pairs finished. 292 secs passed Thread #5: 1850000 read pairs finished. 293 secs passed Thread #7: 1900000 read pairs finished. 296 secs passed Thread #6: 1950000 read pairs finished. 304 secs passed Thread #2: 2000000 read pairs finished. 310 secs passed Thread #1: 2050000 read pairs finished. 339 secs passed Thread #0: 2100000 read pairs finished. 342 secs passed Thread #3: 2150000 read pairs finished. 345 secs passed Thread #4: 2200000 read pairs finished. 346 secs passed Thread #5: 2250000 read pairs finished. 347 secs passed Thread #7: 2300000 read pairs finished. 351 secs passed Thread #6: 2350000 read pairs finished. 361 secs passed Thread #2: 2400000 read pairs finished. 365 secs passed Thread #1: 2450000 read pairs finished. 395 secs passed Thread #0: 2500000 read pairs finished. 397 secs passed Thread #3: 2550000 read pairs finished. 401 secs passed Thread #4: 2600000 read pairs finished. 402 secs passed Thread #5: 2650000 read pairs finished. 403 secs passed Thread #7: 2700000 read pairs finished. 407 secs passed Thread #6: 2750000 read pairs finished. 417 secs passed Thread #2: 2800000 read pairs finished. 422 secs passed Thread #1: 2850000 read pairs finished. 451 secs passed Thread #0: 2900000 read pairs finished. 453 secs passed Thread #3: 2950000 read pairs finished. 457 secs passed Thread #4: 3000000 read pairs finished. 458 secs passed Thread #5: 3050000 read pairs finished. 459 secs passed Thread #7: 3100000 read pairs finished. 461 secs passed Thread #0: 3262139 read pairs finished. 462 secs passed Thread #6: 3150000 read pairs finished. 464 secs passed Thread #2: 3200000 read pairs finished. 465 secs passed Thread #1: 3250000 read pairs finished. 471 secs passed Total number of aligned reads: pairs: 1817795 (56%) single a: 771957 (24%) single b: 635567 (19%) Done. Finished at Fri Jun 6 08:26:44 2014 Total time consumed: 471 secs
!head /Volumes/web/cnidarian/BiLar_YE_control_t.sam
@HD VN:1.0 @SQ SN:C12728 LN:200 @SQ SN:C12722 LN:200 @SQ SN:C12706 LN:200 @SQ SN:C12718 LN:200 @SQ SN:C12732 LN:200 @SQ SN:C12726 LN:200 @SQ SN:C12708 LN:200 @SQ SN:C12730 LN:200 @SQ SN:C12724 LN:200
#where is bsmap
#bsmap="/Users/Shared/Apps/bsmap-2.74/"
bsmap="/Volumes/Bay3/Software/BSMAP/bsmap-2.74/"
#genome
#genome="/Users/Steven/Downloads/Crassostrea_gigas.GCA_000297895.1.22.dna_sm.genome.fa"
genome="/Volumes/web/trilobite/Crassostrea_gigas_ensembl_tracks/Crassostrea_gigas.GCA_000297895.1.22.dna_sm.genome.fa"
!python {bsmap}methratio.py -d {genome} -u -z -g -o /Volumes/web/cnidarian/YE_control_22sm_methratio_out.txt -s {bsmap}samtools /Volumes/web/cnidarian/BiLar_YE_control_t.sam
@ Fri Jun 6 08:34:48 2014: reading reference /Volumes/web/trilobite/Crassostrea_gigas_ensembl_tracks/Crassostrea_gigas.GCA_000297895.1.22.dna_sm.genome.fa ... @ Fri Jun 6 08:36:31 2014: reading /Volumes/web/cnidarian/BiLar_YE_control_t.sam ... [samopen] SAM header is present: 7658 sequences. @ Fri Jun 6 08:44:24 2014: combining CpG methylation from both strands ... @ Fri Jun 6 08:45:40 2014: writing /Volumes/web/cnidarian/YE_control_22sm_methratio_out.txt ... @ Fri Jun 6 08:52:48 2014: done. total 3509466 valid mappings, 16975201 covered cytosines, average coverage: 1.15 fold.
!head /Volumes/web/cnidarian/YE_control_22sm_methratio_out.txt
chr pos strand context ratio eff_CT_count C_count CT_count rev_G_count rev_GA_count CI_lower CI_upper C12798 41 + AACAA 0.000 1.00 0 1 0 0 0.000 0.793 C12798 45 + AACTT 0.000 1.00 0 1 0 0 0.000 0.793 C12798 52 + ATCCC 0.000 1.00 0 1 0 0 0.000 0.793 C12798 53 + TCCCC 0.000 1.00 0 1 0 0 0.000 0.793 C12798 54 + CCCCT 0.000 1.00 0 1 0 0 0.000 0.793 C12798 55 + CCCTT 0.000 1.00 0 1 0 0 0.000 0.793 C12798 58 + TTCAC 0.000 1.00 0 1 0 0 0.000 0.793 C12798 60 + CACCT 0.000 1.00 0 1 0 0 0.000 0.793 C12798 61 + ACCTA 0.000 1.00 0 1 0 0 0.000 0.793
#command for only obtaining the context '__CG_'
#!grep "[A-Z][A-Z]CG[A-Z]" </Volumes/web/cnidarian/YE_mix_22sm_methratio_out.txt> /Volumes/web/cnidarian/YE_mix_22sm_methratio_outCG.txt
#5x coverage
!awk '{if ($8 >= 3) print $1,$2-1,$2+1,"CpG",$5}' </Volumes/web/cnidarian/YE_control_22sm_methratio_outCG.txt> /Volumes/web/cnidarian/YE_control_22sm_outCG3x.igv
!tr ' ' "\t" </Volumes/web/cnidarian/YE_control_22sm_outCG3x.igv> /Volumes/web/cnidarian/filt_YE_control_22smCG3x.igv
!head /Volumes/web/cnidarian/YE_mix_22sm_methratio_out.txt
chr pos strand context ratio eff_CT_count C_count CT_count rev_G_count rev_GA_count CI_lower CI_upper C12706 112 + TTCTA 0.000 1.00 0 1 0 0 0.000 0.793 C12706 135 + GACTC 0.000 1.00 0 1 0 0 0.000 0.793 C12706 137 + CTCTT 0.000 1.00 0 1 0 0 0.000 0.793 C12706 148 + ATCAA 0.000 1.00 0 1 0 0 0.000 0.793 C12706 151 + AACTT 0.000 1.00 0 1 0 0 0.000 0.793 C12706 159 + AGCCT 0.000 1.00 0 1 0 0 0.000 0.793 C12706 160 + GCCTA 0.000 1.00 0 1 0 0 0.000 0.793 C12706 163 + TACAA 0.000 1.00 0 1 0 0 0.000 0.793 C12706 170 - AAGGT 0.000 1.00 0 1 2 2 0.000 0.793
#command for only obtaining the context '__CG_'
#!grep "[A-Z][A-Z]CG[A-Z]" </Volumes/web/cnidarian/YE_mix_22sm_methratio_out.txt> /Volumes/web/cnidarian/YE_mix_22sm_methratio_outCG.txt
#5x coverage
!awk '{if ($8 >= 3) print $1"_"$2-1,$1,$2-1,$2+1,"CpG",$5}' </Volumes/web/cnidarian/YE_mix_22sm_methratio_outCG.txt> /Volumes/web/cnidarian/YE_mix_22sm_outCG3x.igv
!tr ' ' "\t" </Volumes/web/cnidarian/YE_mix_22sm_outCG3x.igv> /Volumes/web/cnidarian/filt_YE_mix_22smCG3xj.txt
!head /Volumes/web/cnidarian/filt_YE_mix_22smCG3xj.txt
C12764_118 C12764 118 120 CpG 0.000 C12764_128 C12764 128 130 CpG 0.000 C12764_131 C12764 131 133 CpG 0.000 C12778_113 C12778 113 115 CpG 0.000 C12802_118 C12802 118 120 CpG 0.000 C12860_104 C12860 104 106 CpG 0.000 C12860_112 C12860 112 114 CpG 0.000 C12860_115 C12860 115 117 CpG 0.000 C12860_120 C12860 120 122 CpG 0.000 C12860_158 C12860 158 160 CpG 0.000
#command for only obtaining the context '__CG_'
#!grep "[A-Z][A-Z]CG[A-Z]" </Volumes/web/cnidarian/YE_control_22sm_methratio_out.txt> /Volumes/web/cnidarian/YE_control_22sm_methratio_outCG.txt
#5x coverage
!awk '{if ($8 >= 3) print $1"_"$2-1,$1,$2-1,$2+1,"CpG",$5}' </Volumes/web/cnidarian/YE_control_22sm_methratio_outCG.txt> /Volumes/web/cnidarian/YE_control_22sm_outCG3x.igv
!tr ' ' "\t" </Volumes/web/cnidarian/YE_control_22sm_outCG3x.igv> /Volumes/web/cnidarian/filt_YE_control_22smCG3xj.txt
!head /Volumes/web/cnidarian/filt_YE_control_22smCG3xj.txt
C13288_100 C13288 100 102 CpG 0.000 C13288_107 C13288 107 109 CpG 0.000 C13448_36 C13448 36 38 CpG 0.000 C13800_97 C13800 97 99 CpG 0.000 C14308_113 C14308 113 115 CpG 0.000 C14308_120 C14308 120 122 CpG 0.000 C14966_93 C14966 93 95 CpG 0.000 C15066_81 C15066 81 83 CpG 0.333 C15066_88 C15066 88 90 CpG 0.000 C15066_260 C15066 260 262 CpG 0.000
spd="/Users/sr320/sqlshare-pythonclient/tools/"
!python {spd}singleupload.py -d filt_YE_control_22smCG3x /Volumes/web/cnidarian/filt_YE_control_22smCG3xj.txt
processing chunk line 0 to 77742 (0.0385191440582 s elapsed) pushing /Volumes/web/cnidarian/filt_YE_control_22smCG3xj.txt... parsing 41B4A049... finished filt_YE_control_22smCG3x
!python {spd}singleupload.py -d filt_YE_mix_22smCG3x /Volumes/web/cnidarian/filt_YE_mix_22smCG3xj.txt
processing chunk line 0 to 1434021 (8.93834710121 s elapsed) pushing /Volumes/web/cnidarian/filt_YE_mix_22smCG3xj.txt... parsing ED260C3F... finished filt_YE_mix_22smCG3x
SELECT * FROM [sr320@washington.edu].[filt_YE_control_22smCG3x]c left join [sr320@washington.edu].[filt_YE_mix_22smCG3x]m on c.Column1=m.Column1 where m.Column1 <> ' '
!head /Volumes/web/cnidarian/YE_mvc_3xjoin.csv
!wc -l /Volumes/web/cnidarian/YE_mvc_3xjoin.csv
35430 /Volumes/web/cnidarian/YE_mvc_3xjoin.csv
!head /Volumes/web/cnidarian/YE_mixHYPO.txt
ratio_control Column1 Column2 Column3 Column4 Column5 ratio_mix 1 C26536_1511 C26536 1511 1513 CpG 0 1 scaffold1179_1120799 scaffold1179 1120799 1120801 CpG 0 1 scaffold1189_665949 scaffold1189 665949 665951 CpG 0 1 scaffold1532_1178565 scaffold1532 1178565 1178567 CpG 0 1 scaffold1532_888871 scaffold1532 888871 888873 CpG 0 1 scaffold1656_129879 scaffold1656 129879 129881 CpG 0 1 scaffold1682_233873 scaffold1682 233873 233875 CpG 0 1 scaffold1733_67696 scaffold1733 67696 67698 CpG 0 1 scaffold1860_178219 scaffold1860 178219 178221 CpG 0
!wc -l /Volumes/web/cnidarian/YE_mixHYPO.txt
279 /Volumes/web/cnidarian/YE_mixHYPO.txt
!head /Volumes/web/cnidarian/YE_mixHYPER.txt
ratio_control Column1 Column2 Column3 Column4 Column5 ratio_mix 0 C27742_4663 C27742 4663 4665 CpG 1 0 scaffold1021_12097 scaffold1021 12097 12099 CpG 1 0 scaffold1032_211526 scaffold1032 211526 211528 CpG 1 0 scaffold105_434416 scaffold105 434416 434418 CpG 1 0 scaffold1086_656185 scaffold1086 656185 656187 CpG 1 0 scaffold1086_656188 scaffold1086 656188 656190 CpG 1 0 scaffold1087_9508 scaffold1087 9508 9510 CpG 1 0 scaffold109_358907 scaffold109 358907 358909 CpG 1 0 scaffold1132_613494 scaffold1132 613494 613496 CpG 1
!wc -l /Volumes/web/cnidarian/YE_mixHYPER.txt
544 /Volumes/web/cnidarian/YE_mixHYPER.txt
#make a mix_hyper igv track..
!awk '{print $3,$4,$5,$6,$7}' </Volumes/web/cnidarian/YE_mixHYPER.txt> /Volumes/web/cnidarian/YE_mixHYPER.igv
!head /Volumes/web/cnidarian/YE_mixHYPER.igv
Column2 Column3 Column4 Column5 ratio_mix C27742 4663 4665 CpG 1 scaffold1021 12097 12099 CpG 1 scaffold1032 211526 211528 CpG 1 scaffold105 434416 434418 CpG 1 scaffold1086 656185 656187 CpG 1 scaffold1086 656188 656190 CpG 1 scaffold1087 9508 9510 CpG 1 scaffold109 358907 358909 CpG 1 scaffold1132 613494 613496 CpG 1
!tail -n +2 /Volumes/web/cnidarian/YE_mixHYPER.igv > /Volumes/web/cnidarian/YE_mixHYPERc.igv
!head /Volumes/web/cnidarian/YE_mixHYPERc.igv
C27742 4663 4665 CpG 1 scaffold1021 12097 12099 CpG 1 scaffold1032 211526 211528 CpG 1 scaffold105 434416 434418 CpG 1 scaffold1086 656185 656187 CpG 1 scaffold1086 656188 656190 CpG 1 scaffold1087 9508 9510 CpG 1 scaffold109 358907 358909 CpG 1 scaffold1132 613494 613496 CpG 1 scaffold1142 97241 97243 CpG 1
!awk '{print $1,$2,$3,"HYPER"}' </Volumes/web/cnidarian/YE_mixHYPERc.igv > /Volumes/web/cnidarian/YE_mixHYPER.bed
!head /Volumes/web/cnidarian/YE_mixHYPER.bed
C27742 4663 4665 HYPER scaffold1021 12097 12099 HYPER scaffold1032 211526 211528 HYPER scaffold105 434416 434418 HYPER scaffold1086 656185 656187 HYPER scaffold1086 656188 656190 HYPER scaffold1087 9508 9510 HYPER scaffold109 358907 358909 HYPER scaffold1132 613494 613496 HYPER scaffold1142 97241 97243 HYPER
#make a mix_hypo igv track..
!awk '{print $3,$4,$5,$6,-$1}' </Volumes/web/cnidarian/YE_mixHYPO.txt> /Volumes/web/cnidarian/YE_mixHYPO.igv
!head /Volumes/web/cnidarian/YE_mixHYPO.igv
Column2 Column3 Column4 Column5 -0 C26536 1511 1513 CpG -1 scaffold1179 1120799 1120801 CpG -1 scaffold1189 665949 665951 CpG -1 scaffold1532 1178565 1178567 CpG -1 scaffold1532 888871 888873 CpG -1 scaffold1656 129879 129881 CpG -1 scaffold1682 233873 233875 CpG -1 scaffold1733 67696 67698 CpG -1 scaffold1860 178219 178221 CpG -1
!tail -n +2 /Volumes/web/cnidarian/YE_mixHYPO.igv > /Volumes/web/cnidarian/YE_mixHYPOc.igv
!head /Volumes/web/cnidarian/YE_mixHYPOc.igv
C26536 1511 1513 CpG -1 scaffold1179 1120799 1120801 CpG -1 scaffold1189 665949 665951 CpG -1 scaffold1532 1178565 1178567 CpG -1 scaffold1532 888871 888873 CpG -1 scaffold1656 129879 129881 CpG -1 scaffold1682 233873 233875 CpG -1 scaffold1733 67696 67698 CpG -1 scaffold1860 178219 178221 CpG -1 scaffold1891 93285 93287 CpG -1
!awk '{print $1,$2,$3,"HYPO"}' </Volumes/web/cnidarian/YE_mixHYPOc.igv > /Volumes/web/cnidarian/YE_mixHYPO.bed
!intersectbed
/bin/sh: intersectbed: command not found