Episode on data access and parallelization #86

fnattino · 2022-01-24T18:05:11Z

This is work-in-progress to address #82 .

@rbavery: I have added a notebook with a first sketch of how the episode on data access/parallelization could look like, any feedback is more than welcome!

rbavery · 2022-01-25T03:02:48Z

This is looking awesome!

Accessing Data Episode

I like the Objectives of this lesson. I think we can potentially split out the Process satellite images in "chunk" to take advantage of parallelization. into it's own lesson. This would mean we'd have a lesson focused on Data Access, which ends around this cell

import rioxarray

# ... or we can open them directly (and stream content only when necessary)
blue_band_href = assets["B02"].href
blue_band = rioxarray.open_rasterio(blue_band_href)
blue_band

and a separate Parallalizing Raster computation with Dask lesson.

I think the final cell for the Access Data Episode could be saving out the raster with rioxarry. this would involve reassigning the CRS to the mosaicked xarray DataArray we produced with stackstac and then using the .rio.toraster method. we can borrow from this example my colleague @alexmandel worked on https://github.com/PacificCommunity/DigitalEarthPacific/blob/demo/writeraster/notebooks/demo/cloudless-mosaic-sentinel2.ipynb

Parallalizing Raster computation with Dask

I love that you already cover guidelines on how to set the chunk size! An additional topic to cover here could be how to tell if your code is running faster with dask or without dask. For this we could cover using time, the dask profiler, or some other easily accessible profiling tool in jupyter notebooks. I think we should also have a section describing dask's lazy computation mode and how to take advantage of that to inspect metadata prior to downloading the actual scene data.

For the Raster calculations portion, instead of Raster calculations using stackstac I suggest we show how to mosaic a collection of scenes. there's stackstac's internal method which just flattens: https://stackstac.readthedocs.io/en/latest/api/main/stackstac.mosaic.html#stackstac.mosaic

I think it would be valuable to show that solution and for a median composite.

Setup instructions will also need to be updated with new dependencies. I've seen the most success with not pinning specific versions to allow a more flexible solve for different machines: https://carpentries-incubator.github.io/geospatial-python/setup.html

A third episode focused on working with a cool looking mosaic could focus on xarray-spatial's raster calc funcs. One idea: computing spectral indices, thresholding them, and polygonizing the result (maybe areas with especially high NDVI): https://github.com/makepath/xarray-spatial

rbavery · 2022-01-25T03:06:13Z

I also like the inclusion of the Dask task graph image. including other images of intermediate results, such as plots of the blue band, could be good to include prior to the final challenge. Also when this gets formatted to the lesson markdown, I think we can create a set of tooltips that refer to other sources for folks to read up on COG, STAC, and Dask, while also briefly summarizing their utility for geospatial.

fnattino · 2022-01-26T22:21:15Z

Hi @rbavery , I have created a first version of a full data access episode. Basically, I have converted the Jupyter notebook that you already had a look at into a .md file and I have added some explanatory text in between the code blocks. Whenever you have time to review it, I would be happy to have any kind of feedback - thanks in advance!

I have also added a first exercise following up on your idea to have participants exploring a STAC catalog even before having the search tool introduced - what do you think about having it formulated in this way?

Still working on the second episode (on parallel raster computation with Dask).

rbavery · 2022-01-26T22:22:09Z

@fnattino thanks I'll give this a review this evening

rbavery

thanks for all your work on this @fnattino. I think it's close to being able to merge. I'm meeting with NASA DEVELOP folks next week and I think this is already in great shape to teach if there's time.

_episodes/XX-access-data.md

rbavery · 2022-01-27T02:15:21Z

_episodes/XX-access-data.md

+> ## Exercise: Discover a STAC catalog
+> Open the following STAC API link using your web browser: https://earth-search.aws.element84.com/v0.
+> Navigate through the links to find out which collections are available and how many scenes are indexed. Where may one
+> find information on how to query the API for the desired scenes? Can you find out which parameters can be provided
+> in the queries?
+{: .challenge}
+


I think we might want to show learners a graphical tool to browse STAC Catalogs. This one shows the spatial extent and summarizes the information about any STAC catalog url you paste into it.

https://radiantearth.github.io/stac-browser

for example: https://radiantearth.github.io/stac-browser/#/external/earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a

Sounds great, and thanks for the tips - I hadn't seen yet the new STAC browser! Unfortunately the filtering tools do not seem to work with the Earth Search STAC API (maybe because this is an older STAC API version, 0.9), but I have read this is still a "demo" version, so things might be fixed soon. Anyway, for the purpose of the exercise, i.e. browsing through the items, works very well!

rbavery · 2022-01-27T02:31:04Z

_episodes/XX-access-data.md

+# save processed image to disk
+visual_clip.rio.to_raster("amsterdam_tci.tif", driver="COG")
+~~~
+{: .language-python}


this is awesome. I think we should end with a challenge so they can reproduce these steps and build some muscle memory for how interacting with a STAC API via pystac and then working with the result in rioxarray feels.

I think a good option would be to direct them to this STAC catalog and have them download data that intersects a specific lat, lon and date (specified in the challenge text): https://radiantearth.github.io/stac-browser/#/external/earth-search.aws.element84.com/v0/collections/landsat-8-l1-c1

the solution to the challenge could be to save a single band at that location and date to disk with rioxarray

Good point! I have added a challenge using the Landsat 8 dataset. This collection unfortunately seems not to be continuously updated here (and at a certain point might be dropped?) so we might have to find new sources in future!

rbavery · 2022-01-27T02:38:40Z

_episodes/XX-access-data.md

+~~~
+{: .language-python}
+
+> ## Exercise: Discover a STAC catalog


before this exercise I think we can show an image of the radiant earth stac browser to give people a visual of what information a STAC catalog contains. Looking at the lesson webpage, it's a dense in the amount of text before the first image so I think this will make the first part of the lesson more engaging for someone who is browsing the lesson material or seeking out guidance on STAC.

I have added a figure - the best "composition" I could come up with.. If you have suggestions for improvements, let me know!

_episodes/XX-access-data.md

rbavery · 2022-01-27T21:34:49Z

@fnattino thanks for addressing these reviews! once this data access episode is finished, can we merge that PR and finish the parallelization episode in a separate PR? Feel free to merge this as is now, I or somebody could add a challenge later unless you are already working on it.

fnattino · 2022-01-27T21:42:37Z

Hi @rbavery - thanks a lot for having already a look. I am finishing up the last challenge, I'll ping you as soon as I have pushed it!

fnattino · 2022-01-27T22:55:57Z

Hi @rbavery, this is it - I have added the final challenge.

I have also updated the setup instructions and the environment.yaml file, adding pystac_client to the dependencies.

Merging this first and opening a second PR for the parallelisation episode sounds good - I have removed the corresponding notebook from this branch.

One last thing: should this become episode 19? I could set the number and merge if this is alright with you. Really thanks a lot for all the feedback and suggestions!

rbavery · 2022-01-27T23:07:09Z

Fantastic!!! Yes let's make this episode 19 for now. Really looking forward to teaching this! Lgtm feel free to merge.

…bench-intro Update episodes 1-4

add first version of episode on data access

dbb94eb

fnattino added 2 commits January 25, 2022 18:16

split data-access and data-parallelization

232c5e8

add first draft of data access episode

69d38de

fnattino marked this pull request as ready for review January 26, 2022 22:21

fnattino requested a review from rbavery January 26, 2022 22:21

rbavery requested changes Jan 27, 2022

View reviewed changes

fnattino added 3 commits January 27, 2022 16:17

include some Ryan's suggestions

e63b6e9

fix exercise 1, make use of the STAC browser

a71f211

add STAC browser screenshot

a3d5efb

rbavery approved these changes Jan 27, 2022

View reviewed changes

fnattino added 4 commits January 27, 2022 23:04

add final challenge

77536b2

update data access notebook according to episode

19e19a7

remove notebook on parallelization

af58bc7

add pystac_client to dependencies

f1097ec

number the episode as 19

8e23faf

fnattino merged commit c57024e into carpentries-incubator:gh-pages Jan 28, 2022

fnattino deleted the data-access branch January 28, 2022 07:50

fnattino mentioned this pull request Jan 28, 2022

Episode on parallel raster computations #90

Merged

fnattino mentioned this pull request Apr 1, 2022

Stackstac episode #82

Open

rogerkuou pushed a commit that referenced this pull request Aug 8, 2023

Merge pull request #86 from esciencecenter-digital-skills/update-work…

bf51de3

…bench-intro Update episodes 1-4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Episode on data access and parallelization #86

Episode on data access and parallelization #86

Episode on data access and parallelization #86

Episode on data access and parallelization #86

Conversation

Accessing Data Episode

Parallalizing Raster computation with Dask

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment