Notebook

Helpful Tips for using the mosaicking pipeline¶

Most of the program's run time is spent on downloading and uploading imaging data. So the program could be significantly sped up if the investigator already has the survey data avaliable locally.

If a mosacing is conducted on the whole-catalog, the job will probably take a long time (~1 week for SDSS on RC3). You can use nohup to run the job on a remote machine to log off without terminating the process: nohup python bulk_run.py

Other options such as qsub, at, batch may also be used.

It is okay to run different procedures on different surveys and different parts of the sky simultaneously as long as the imaging bands do not have overlapping bands to prevent file-already-exist errors

Place @ inside rc3.txt (RC3 Data file) to denote where rc3_Catalog methods should start. This often comes in handy when debugging.

To run several (or single image), a quick and easy way is to place @ in rc3.txt and just run the on the survey that you want (SDSS: bulk_run.py ; DSS: bulk_run_dss.py; 2MASS: bulk_run_twomass.py), then terminate the process when finish running on the objects.

Example:¶

Inside rc3.txt,

99.9008333333 20.7444444444 0.0303283477042 19385 99.925 65.4611111111 0.0235422924575 19390 99.9579166667 -38.2452777778 0.0296379902266 19391 @ 100.02875 60.0827777778 0.0491868205428 19397 100.0625 -72.4558333333 0.0246518065188 19400 100.134166667 50.105 0.0152001806863 19409 100.180833333 -58.5255555556 0.0491868205428 19413

In [4]:

cd ../pipeline/

In [5]:

from rc3Catalog import RC3Catalog
from sdss import SDSS

Mosaicking starts from PGC19397:

In [ ]:

rc3_cat = RC3Catalog()
rc3_cat.mosaicAll(SDSS())

Working on PGC19397, at(100.02875 , 60.0827777778)
------------------mosaic_band----------------------
Now mosaic_band on 19397
(100.02875, 60.0827777778, 0.0737802308142)
Working on PGC19400, at(100.0625 , -72.4558333333)
------------------mosaic_band----------------------
Now mosaic_band on 19400
(100.0625, -72.4558333333, 0.0369777097782)
Working on PGC19409, at(100.134166667 , 50.105)

Sometimes the program may crash because of Server-side errors or bad Network Connection:

for SDSS:

[['PGC 71234', '350.619333', '29.180944'], [' <font face="Arial', ' Helvetica', ' Genev', ' SunSans-Regular', ' sans-serif ">\r\n'], [' Description: An application error occurred on the server. The current cust om error settings for this application prevent the details o f the application error from being viewed remotely (for security reasons). It could', ' however', ' be viewed by browser s running on the local server machine.\r’]]

for DSS:

File "/usr/local/lib/python2.7/dist-packages/astroquery/vizier/core.py", line 367, in query_region_async timeout=self.TIMEOUT) File "/usr/local/lib/python2.7/dist-packages/astroquery/query.py", line 172, in request response = query.request(self.__session, self.cache_location) File "/usr/local/lib/python2.7/dist-packages/astroquery/query.py", line 72, in request timeout=self.timeout)) File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 387, in send raise Timeout(e) requests.exceptions.Timeout: HTTPConnectionPool(host='vizier.u-strasbg.fr', port=80): Read timed out. (read timeout=60)

In these cases, restart the job starting from where you left off,using the trick described above.

When conducting the post-processing procedures,it is usually safer to save a backup copy of the whole data product file, so that you can always start from this step instead of taking a week or more to regenerate the data product.

Network and downloading speed will be much slower if you are running multiple mosaicing jobs (ex. bulk_run_dss.py and bulk_run.py) on the same machine.