Hi all, I was invited to repost this (originally posted last year) to be included in the second chance pool!
The USGS created topographic maps by hand from 1884 to 2006 (when they started making their topographic map series digitally). I stumbled upon the fact that all 183,000 of their hand-drawn maps were digitized and uploaded to a public S3 bucket [0] in Cloud-Optimized GeoTIFF format [1].
Because they're in this format, it's possible to quickly mosaic images _on demand_, in a process called "dynamic tiling" [2]. The dynamic nature of it gives you a lot more flexibility, because you aren't creating static tiles once.
In contrast, the hardest part about this project is the time variation. The USGS created many different scales of maps at many different time periods, so figuring out how to combine them into a few mosaics is quite tricky. With more time invested in this, it would be ideal to have a search capability to show user-specified time horizons. (Edit: I talk about this process a bit more in the Github README [3]).
More recently I've been spending time doing similar dynamic tiling projects, but focusing on satellite imagery, to enable analytic processing of satellite imagery in the browser [4].
[0]: `aws s3 ls s3://prd-tnm/StagedProducts/Maps/HistoricalTopo/GeoTIFF/`
This looks amazing! I wouldn't be surprised if this can be useful for e.g. urban economics, to compute measures of how cities and urban areas grew. Like LA eating into the San Fernando Valley, for instance.
Oh hi Sergio! Hope you're doing well. I'm not in the economics field anymore, but it would be cool to see if an economist could use this data. It's probably a little harder to use because you'd have to apply some vectorization techniques on the maps to get more usable road network data.
Definitely true! But if there's a good question then that's definitely doable.
Also I noted your change of field; your github repos and starred repos that pop up on my github main page are much more interesting that the usual r/stata repos
Thanks agains for the contributions, GIS and carto are definitely a super fun and useful field with a greater bang-for-the-buck than most economic papers!
The S3 bucket is hosted/managed by the U.S. Geological Survey. The only costs borne by them are the hosting, S3 requests, and S3 egress. In this case there's no egress since I fetch from the same region, so the marginal cost of this experiment to them is roughly $0.000001 per tile a user loads.
They've chosen to leave this particular bucket fully public for the time being. If usage got too high, they could convert the bucket to a requester pays bucket, where the requester (me) would pay for the S3 requests and egress. They've done this for their Landsat bucket (usgs-landsat), which is set as requester pays.
If the developer does not need to deal with a long-running process abstraction, and is billed only by actual compute time and not idle time, I think the consensus is to call that "serverless"
S3 isn't a server, it's an object store. This uses Lambda (the serverless part) to fetch source data on demand, combine, reproject to web mercator, and send to the client.
There are varied definitions of serverless; it isn't "as serverless" as the browser loading data directly from S3, but an AWS marketing person would call this usage of Lambda serverless.
The USGS created topographic maps by hand from 1884 to 2006 (when they started making their topographic map series digitally). I stumbled upon the fact that all 183,000 of their hand-drawn maps were digitized and uploaded to a public S3 bucket [0] in Cloud-Optimized GeoTIFF format [1].
Because they're in this format, it's possible to quickly mosaic images _on demand_, in a process called "dynamic tiling" [2]. The dynamic nature of it gives you a lot more flexibility, because you aren't creating static tiles once.
In contrast, the hardest part about this project is the time variation. The USGS created many different scales of maps at many different time periods, so figuring out how to combine them into a few mosaics is quite tricky. With more time invested in this, it would be ideal to have a search capability to show user-specified time horizons. (Edit: I talk about this process a bit more in the Github README [3]).
More recently I've been spending time doing similar dynamic tiling projects, but focusing on satellite imagery, to enable analytic processing of satellite imagery in the browser [4].
[0]: `aws s3 ls s3://prd-tnm/StagedProducts/Maps/HistoricalTopo/GeoTIFF/`
[1]: https://www.cogeo.org/
[2]: https://kylebarron.dev/blog/cog-mosaic/overview
[3]: https://github.com/kylebarron/usgs-topo-tiler
[4]: https://www.unfolded.ai/blog/2021-04-28-raster-layer/