On serverless data scraping, cloud object storage, MinIO and rclone

by | Nov 18, 2021 | administration, big data, data in action, fun, IBM, ibmcloud, IT, notebook, Python, tools

Building a data lake the serverless wayThis fall, I started another side project. It involves mobility data and its analysis and visualization. As first step and still ongoing, I am building up a data lake (and maybe integrating it into a data fabric). For that, I regularly download data from various sources, then upload it to a Cloud Object Storage/S3 bucket. Once in cloud storage, I process it